blog/atom.xml (view raw)
1<?xml version="1.0" encoding="UTF-8"?>
2<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
3 <title>Lonami's Site - My Blog</title>
4 <link href="https://lonami.dev/blog/atom.xml" rel="self" type="application/atom+xml"/>
5 <link href="https://lonami.dev/blog/"/>
6 <generator uri="https://www.getzola.org/">Zola</generator>
7 <updated>2021-03-13T00:00:00+00:00</updated>
8 <id>https://lonami.dev/blog/atom.xml</id>
9 <entry xml:lang="en">
10 <title>Writing our own Cheat Engine: Pointers</title>
11 <published>2021-03-13T00:00:00+00:00</published>
12 <updated>2021-03-13T00:00:00+00:00</updated>
13 <link href="https://lonami.dev/blog/woce-6/" type="text/html"/>
14 <id>https://lonami.dev/blog/woce-6/</id>
15 <content type="html"><p>This is part 6 on the <em>Writing our own Cheat Engine</em> series:</p>
16<ul>
17<li><a href="/blog/woce-1">Part 1: Introduction</a> (start here if you're new to the series!)</li>
18<li><a href="/blog/woce-2">Part 2: Exact Value scanning</a></li>
19<li><a href="/blog/woce-3">Part 3: Unknown initial value</a></li>
20<li><a href="/blog/woce-4">Part 4: Floating points</a></li>
21<li><a href="/blog/woce-5">Part 5: Code finder</a></li>
22<li>Part 6: Pointers</li>
23</ul>
24<p>In part 5 we wrote our very own debugger. We learnt that Cheat Engine is using hardware breakpoints to watch memory change, and how to do the same ourselves. We also learnt that hardware points are not the only way to achieve the effect of watchpoints, although they certainly are the fastest and cleanest approach.</p>
25<p>In this post, we will be reusing some of that knowledge to find out a closely related value, the <em>pointer</em> that points to the real value<sup class="footnote-reference"><a href="#1">1</a></sup>. As a quick reminder, a pointer is nothing but an <code>usize</code><sup class="footnote-reference"><a href="#2">2</a></sup> representing the address of another portion of memory, in this case, the actual value we will be scanning for. A pointer is a value that, well, points elsewhere. In Rust we normally use reference instead, which are safer (typed and their lifetime is tracked) than pointers, but in the end we can achieve the same with both.</p>
26<p>Why care about pointers? It turns out that things, such as your current health in-game, are very unlikely to end up in the same memory position when you restart the game (or even change to another level, or even during gameplay). So, if you perform a scan and find that the address where your health is stored is <code>0x73AABABE</code>, you might be tempted to save it and reuse it next time you launch the game. Now you don't need to scan for it again! Alas, as soon as you restart the game, the health is now stored at <code>0x5AADBEEF</code>.</p>
27<p>Not all hope is lost! The game must <em>somehow</em> have a way to reliably find this value, and the way it's done is with pointers. There will always be some base address that holds a pointer, and the game code knows where to find this pointer. If we are also able to find the pointer at said base address, and follow it ourselves (&quot;dereferencing&quot; it), we can perform the same steps the game is doing, and reliably find the health no matter how much we restart the game<sup class="footnote-reference"><a href="#3">3</a></sup>.</p>
28<h2 id="code-finder">Code finder</h2>
29<details open><summary>Cheat Engine Tutorial: Step 6</summary>
30<blockquote>
31<p>In the previous step I explained how to use the Code finder to handle changing locations. But that method alone makes it difficult to find the address to set the values you want. That's why there are pointers:</p>
32<p>At the bottom you'll find 2 buttons. One will change the value, and the other changes the value AND the location of the value. For this step you don't really need to know assembler, but it helps a lot if you do.</p>
33<p>First find the address of the value. When you've found it use the function to find out what accesses this address.</p>
34<p>Change the value again, and a item will show in the list. Double click that item. (or select and click on more info) and a new window will open with detailed information on what happened when the instruction ran.</p>
35<p>If the assembler instruction doesn't have anything between a '[' and ']' then use another item in the list. If it does it will say what it think will be the value of the pointer you need.</p>
36<p>Go back to the main cheat engine window (you can keep this extra info window open if you want, but if you close it, remember what is between the [ and ]) and do a 4 byte scan in hexadecimal for the value the extra info told you. When done scanning it may return 1 or a few hundred addresses. Most of the time the address you need will be the smallest one. Now click on manually add and select the pointer checkbox.</p>
37<p>The window will change and allow you to type in the address of a pointer and a offset. Fill in as address the address you just found. If the assembler instruction has a calculation (e.g: [esi+12]) at the end then type the value in that's at the end. else leave it 0. If it was a more complicated instruction look at the calculation.</p>
38<p>Example of a more complicated instruction:</p>
39<p>[EAX*2+EDX+00000310] eax=4C and edx=00801234.</p>
40<p>In this case EDX would be the value the pointer has, and EAX*2+00000310 the offset, so the offset you'd fill in would be 2*4C+00000310=3A8. (this is all in hex, use calc.exe from windows in scientific mode to calculate).</p>
41<p>Back to the tutorial, click OK and the address will be added, If all went right the address will show P-&gt;xxxxxxx, with xxxxxxx being the address of the value you found. If thats not right, you've done something wrong. Now, change the value using the pointer you added in 5000 and freeze it. Then click Change pointer, and if all went right the next button will become visible.</p>
42<p><em>extra</em>: And you could also use the pointer scanner to find the pointer to this address.</p>
43</blockquote>
44</details>
45<h2 id="on-access-watchpoints">On-access watchpoints</h2>
46<p>Last time we managed to learn how hardware breakpoints were being set by observing Cheat Engine's behaviour. I think it's now time to handle this properly instead. We'll check out the <a href="https://wiki.osdev.org/CPU_Registers_x86#Debug_Registers">CPU Registers x86 page on OSDev</a> to learn about it:</p>
47<ul>
48<li>DR0, DR1, DR2 and DR3 can hold a memory address each. This address will be used by the breakpoint.</li>
49<li>DR4 is actually an <a href="https://en.wikipedia.org/wiki/X86_debug_register">obsolete synonym</a> for DR6.</li>
50<li>DR5 is another obsolete synonym, this time for DR7.</li>
51<li>DR6 is debug status. The four lowest bits indicate which breakpoint was hit, and the four highest bits contain additional information. We should make sure to clear this ourselves when a breakpoint is hit.</li>
52<li>DR7 is debug control, which we need to study more carefully.</li>
53</ul>
54<p>Each debug register DR0 through DR3 has two corresponding bits in DR7, starting from the lowest-order bit, to indicate whether the corresponding register is a <strong>L</strong>ocal or <strong>G</strong>lobal breakpoint. So it looks like this:</p>
55<pre><code> Meaning: [ .. .. | G3 | L3 | G2 | L2 | G1 | L1 | G0 | L0 ]
56Bit-index: 31-08 | 07 | 06 | 05 | 04 | 03 | 02 | 01 | 00
57</code></pre>
58<p>Cheat Engine was using local breakpoints, because the zeroth bit was set. Probably because we don't want these breakpoints to infect other programs! Because we were using only one breakpoint, only the lowermost bit was being set. The local 1st, 2nd and 3rd bits were unset.</p>
59<p>Now, each debug register DR0 through DR4 has four additional bits in DR7, two for the <strong>C</strong>ondition and another two for the <strong>S</strong>ize:</p>
60<pre><code> Meaning: [ S3 | C3 | S2 | C2 | S1 | C1 | S0 | C0 | .. .. ]
61Bit-index: 31 30 | 29 28 | 27 26 | 25 24 | 23 22 | 21 20 | 19 18 | 17 16 | 15-00
62</code></pre>
63<p>The two bits of the condition mean the following:</p>
64<ul>
65<li><code>00</code> execution breakpoint.</li>
66<li><code>01</code> write watchpoint.</li>
67<li><code>11</code> read/write watchpoint.</li>
68<li><code>10</code> unsupported I/O read/write.</li>
69</ul>
70<p>When we were using Cheat Engine to add write watchpoints, the bits 17 and 16 were indeed set to <code>01</code>, and the bits 19 and 18 were set to <code>11</code>. Hm, but <em>11<sub>2</sub> = 3<sub>10</sub></em> , and yet, we were watching writes to 4 bytes. So what's up with this? Is there a different mapping for the size which isn't documented at the time of writing? Seems we need to learn from Cheat Engine's behaviour one more time.</p>
71<p>For reference, this is what DR7 looked like when we added a single write watchpoint:</p>
72<pre><code>hex: 000d_0001
73bin: 00000000_00001101_00000000_00000001
74</code></pre>
75<p>And this is the code I will be using to check the breakpoints of different sizes:</p>
76<pre><code>thread::enum_threads(pid)
77 .unwrap()
78 .into_iter()
79 .for_each(|tid| {
80 let thread = thread::Thread::open(tid).unwrap();
81 let ctx = thread.get_context().unwrap();
82 eprintln!(&quot;hex: {:08x}&quot;, ctx.Dr7);
83 eprintln!(&quot;bin: {:032b}&quot;, ctx.Dr7);
84 });
85</code></pre>
86<p>Let's compare this to watchpoints for sizes 1, 2, 4 and 8 bytes:</p>
87<pre><code>1 byte
88hex: 0001_0401
89bin: 00000000_00000001_00000100_00000001
90
912 bytes
92hex: 0005_0401
93bin: 00000000_00000101_00000100_00000001
94
954 bytes
96hex: 000d_0401
97bin: 00000000_00001101_00000100_00000001
98
998 bytes
100hex: 0009_0401
101bin: 00000000_00001001_00000100_00000001
102 ^ wut?
103</code></pre>
104<p>I have no idea what's up with that stray tenth bit. Its use does not seem documented, and things worked fine without it, so we'll ignore it. The lowest bit is set to indicate we're using DR0, bits 17 and 16 represent the write watchpoint, and the size seems to be as follows:</p>
105<ul>
106<li><code>00</code> for a single byte.</li>
107<li><code>01</code> for two bytes (a &quot;word&quot;).</li>
108<li><code>11</code> for four bytes (a &quot;double word&quot;).</li>
109<li><code>10</code> for eight bytes (a &quot;quadruple word&quot;).</li>
110</ul>
111<p>Doesn't make much sense if you ask me, but we'll roll with it. Just to confirm, this is what the &quot;on-access&quot; breakpoint looks like according to Cheat Engine:</p>
112<pre><code>hex: 000f_0401
113bin: 00000000_00001111_00000100_00000001
114</code></pre>
115<p>So it all checks out! The bit pattern is <code>11</code> for read/write (technically, a write is also an access). Let's implement this!</p>
116<h2 id="proper-breakpoint-handling">Proper breakpoint handling</h2>
117<p>The first thing we need to do is represent the possible breakpoint conditions:</p>
118<pre><code class="language-rust" data-lang="rust">#[repr(u8)]
119pub enum Condition {
120 Execute = 0b00,
121 Write = 0b01,
122 Access = 0b11,
123}
124</code></pre>
125<p>And also the legal breakpoint sizes:</p>
126<pre><code class="language-rust" data-lang="rust">#[repr(u8)]
127pub enum Size {
128 Byte = 0b00,
129 Word = 0b01,
130 DoubleWord = 0b11,
131 QuadWord = 0b10,
132}
133</code></pre>
134<p>We are using <code>#[repr(u8)]</code> so that we can convert a given variant into the corresponding bit pattern. With the right types defined in order to set a breakpoint, we can start implementing the method that will set them (inside <code>impl Thread</code>):</p>
135<pre><code class="language-rust" data-lang="rust">pub fn add_breakpoint(&amp;self, addr: usize, cond: Condition, size: Size) -&gt; io::Result&lt;Breakpoint&gt; {
136 let mut context = self.get_context()?;
137 todo!()
138}
139</code></pre>
140<p>First, let's try finding an &quot;open spot&quot; where we could set our breakpoint. We will &quot;slide&quot; a the <code>0b11</code> bitmask over the lowest eight bits, and if and only if both the local and global bits are unset, then we're free to set a breakpoint at this index<sup class="footnote-reference"><a href="#4">4</a></sup>:</p>
141<pre><code class="language-rust" data-lang="rust">let index = (0..4)
142 .find_map(|i| ((context.Dr7 &amp; (0b11 &lt;&lt; (i * 2))) == 0).then(|| i))
143 .ok_or_else(|| io::Error::new(io::ErrorKind::Other, &quot;no debug register available&quot;))?;
144</code></pre>
145<p>Once an <code>index</code> is found, we can set the address we want to watch in the corresponding register and update the debug control bits:</p>
146<pre><code class="language-rust" data-lang="rust">let addr = addr as u64;
147match index {
148 0 =&gt; context.Dr0 = addr,
149 1 =&gt; context.Dr1 = addr,
150 2 =&gt; context.Dr2 = addr,
151 3 =&gt; context.Dr3 = addr,
152 _ =&gt; unreachable!(),
153}
154
155let clear_mask = !((0b1111 &lt;&lt; (16 + index * 4)) | (0b11 &lt;&lt; (index * 2)));
156context.Dr7 &amp;= clear_mask;
157
158context.Dr7 |= 1 &lt;&lt; (index * 2);
159
160let sc = (((size as u8) &lt;&lt; 2) | (cond as u8)) as u64;
161context.Dr7 |= sc &lt;&lt; (16 + index * 4);
162
163self.set_context(&amp;context)?;
164Ok(Breakpoint {
165 thread: self,
166 clear_mask,
167})
168</code></pre>
169<p>Note that we're first creating a &quot;clear mask&quot;. We switch on all the bits that we may use for this breakpoint, and then negate. Effectively, <code>Dr7 &amp; clear_mask</code> will make sure we don't leave any bit high on accident. We apply the mask before OR-ing the rest of bits to also clear any potential garbage on the size and condition bits. Next, we set the bit to enable the new local breakpoint, and also store the size and condition bits at the right location.</p>
170<p>With the context updated, we can set it back and return the <code>Breakpoint</code>. It stores the <code>thread</code> and the <code>clear_mask</code> so that it can clean up on <code>Drop</code>. We are technically relying on <code>Drop</code> to run behaviour here, but the cleanup is done on a best-effort basis. If the user intentionally forgets the <code>Breakpoint</code>, maybe they want the <code>Breakpoint</code> to forever be set.</p>
171<p>This logic is begging for a testcase though; I'll split it into a new <code>Breakpoint::update_dbg_control</code> method and test that out:</p>
172<pre><code class="language-rust" data-lang="rust">
173#[cfg(test)]
174mod tests {
175 use super::*;
176
177 #[test]
178 fn brk_add_one() {
179 // DR7 starts with garbage which should be respected.
180 let (clear_mask, dr, dr7) =
181 Breakpoint::update_dbg_control(0x1700, Condition::Write, Size::DoubleWord).unwrap();
182
183 assert_eq!(clear_mask, 0xffff_ffff_fff0_fffc);
184 assert_eq!(dr, DebugRegister::Dr0);
185 assert_eq!(dr7, 0x0000_0000_000d_1701);
186 }
187
188 #[test]
189 fn brk_add_two() {
190 let (clear_mask, dr, dr7) = Breakpoint::update_dbg_control(
191 0x0000_0000_000d_0001,
192 Condition::Write,
193 Size::DoubleWord,
194 )
195 .unwrap();
196
197 assert_eq!(clear_mask, 0xffff_ffff_ff0f_fff3);
198 assert_eq!(dr, DebugRegister::Dr1);
199 assert_eq!(dr7, 0x0000_0000_00dd_0005);
200 }
201
202 #[test]
203 fn brk_try_add_when_max() {
204 assert!(Breakpoint::update_dbg_control(
205 0x0000_0000_dddd_0055,
206 Condition::Write,
207 Size::DoubleWord
208 )
209 .is_none());
210 }
211}
212</code></pre>
213<pre><code>running 3 tests
214test thread::tests::brk_add_one ... ok
215test thread::tests::brk_add_two ... ok
216test thread::tests::brk_try_add_when_max ... ok
217</code></pre>
218<p>Very good! With proper breakpoint handling usable, we can continue.</p>
219<h2 id="inferring-the-pointer-value">Inferring the pointer value</h2>
220<p>After scanning memory for the location we're looking for (say, our current health), we then add an access watchpoint, and wait for an exception to occur. As a reminder, here's the page with the <a href="https://docs.microsoft.com/en-us/windows/win32/debug/debugging-events">Debugging Events</a>:</p>
221<pre><code class="language-rust" data-lang="rust">let addr = ...;
222let mut threads = ...;
223
224let _watchpoints = threads
225 .iter_mut()
226 .map(|thread| {
227 thread
228 .add_breakpoint(addr, thread::Condition::Access, thread::Size::DoubleWord)
229 .unwrap()
230 })
231 .collect::&lt;Vec&lt;_&gt;&gt;();
232
233loop {
234 let event = debugger.wait_event(None).unwrap();
235 if event.dwDebugEventCode == winapi::um::minwinbase::EXCEPTION_DEBUG_EVENT {
236 let exc = unsafe { event.u.Exception() };
237 if exc.ExceptionRecord.ExceptionCode == winapi::um::minwinbase::EXCEPTION_SINGLE_STEP {
238 todo!();
239 }
240 }
241 debugger.cont(event, true).unwrap();
242}
243</code></pre>
244<p>Now, inside the <code>todo!()</code> we will want to do a few things, namely printing out the instructions &quot;around this location&quot; and dumping the entire thread context on screen. To print the instructions, we need to import <code>iced_x86</code> again, iterate over all memory regions to find the region where the exception happened, read the corresponding bytes, decode the instructions, and when we find the one with a corresponding instruction pointer, print &quot;around it&quot;:</p>
245<pre><code class="language-rust" data-lang="rust">use iced_x86::{Decoder, DecoderOptions, Formatter, Instruction, NasmFormatter};
246
247let addr = exc.ExceptionRecord.ExceptionAddress as usize;
248let region = process
249 .memory_regions()
250 .into_iter()
251 .find(|region| {
252 let base = region.BaseAddress as usize;
253 base &lt;= addr &amp;&amp; addr &lt; base + region.RegionSize
254 })
255 .unwrap();
256
257let bytes = process
258 .read_memory(region.BaseAddress as usize, region.RegionSize)
259 .unwrap();
260
261let mut decoder = Decoder::new(64, &amp;bytes, DecoderOptions::NONE);
262decoder.set_ip(region.BaseAddress as _);
263
264let mut formatter = NasmFormatter::new();
265let mut output = String::new();
266
267let instructions = decoder.into_iter().collect::&lt;Vec&lt;_&gt;&gt;();
268for (i, ins) in instructions.iter().enumerate() {
269 if ins.next_ip() as usize == addr {
270 let low = i.saturating_sub(5);
271 let high = (i + 5).min(instructions.len());
272 for j in low..high {
273 let ins = &amp;instructions[j];
274 print!(&quot;{} {:016X} &quot;, if j == i { &quot;&gt;&gt;&gt;&quot; } else { &quot; &quot; }, ins.ip());
275 let k = (ins.ip() - region.BaseAddress as usize as u64) as usize;
276 let instr_bytes = &amp;bytes[k..k + ins.len()];
277 for b in instr_bytes.iter() {
278 print!(&quot;{:02X}&quot;, b);
279 }
280 if instr_bytes.len() &lt; 10 {
281 for _ in 0..10usize.saturating_sub(instr_bytes.len()) {
282 print!(&quot; &quot;);
283 }
284 }
285
286 output.clear();
287 formatter.format(ins, &amp;mut output);
288 println!(&quot; {}&quot;, output);
289 }
290 break;
291 }
292}
293debugger.cont(event, true).unwrap();
294break;
295</code></pre>
296<p>The result is pretty fancy:</p>
297<pre><code> 000000010002CAAC 48894DF0 mov [rbp-10h],rcx
298 000000010002CAB0 488955F8 mov [rbp-8],rdx
299 000000010002CAB4 48C745D800000000 mov qword [rbp-28h],0
300 000000010002CABC 90 nop
301 000000010002CABD 488B050CA02D00 mov rax,[rel 100306AD0h]
302&gt;&gt;&gt; 000000010002CAC4 8B00 mov eax,[rax]
303 000000010002CAC6 8945EC mov [rbp-14h],eax
304 000000010002CAC9 B9E8030000 mov ecx,3E8h
305 000000010002CACE E88D2FFEFF call 000000010000FA60h
306 000000010002CAD3 8945E8 mov [rbp-18h],eax
307</code></pre>
308<p>Cool! So <code>rax</code> is holding an address, meaning it's a pointer, and the value it reads (dereferences) is stored back into <code>eax</code> (because it does not need <code>rax</code> anymore). Alas, the current thread context has the register state <em>after</em> the instruction was executed, and <code>rax</code> no longer contains the address at this point. However, notice how the previous instruction writes a fixed value to <code>rax</code>, and then that value is used to access memory, like so:</p>
309<pre><code class="language-rust" data-lang="rust">let eax = memory[memory[0x100306AD0]];
310</code></pre>
311<p>The value at <code>memory[0x100306AD0]</code> <em>is</em> the pointer! No offsets are used, because nothing is added to the pointer after it's read. This means that, if we simply scan for the address we were looking for, we should find out where the pointer is stored:</p>
312<pre><code class="language-rust" data-lang="rust">let addr = ...;
313let scan = process.scan_regions(&amp;regions, Scan::Exact(addr as u64));
314
315scan.into_iter().for_each(|region| {
316 region.locations.iter().for_each(|ptr_addr| {
317 println!(&quot;[{:x}] = {:x}&quot;, ptr_addr, addr);
318 });
319});
320</code></pre>
321<p>And just like that:</p>
322<pre><code>[100306ad0] = 15de9f0
323</code></pre>
324<p>Notice how the pointer address found matches with the offset used by the instructions:</p>
325<pre><code> 000000010002CABD 488B050CA02D00 mov rax,[rel 100306AD0h]
326 this is the same as the value we just found ^^^^^^^^^^
327</code></pre>
328<p>Very interesting indeed. We were actually very lucky to have only found a single memory location containing the pointer value, <code>0x15de9f0</code>. Cheat Engine somehow knows that this value is always stored at <code>0x100306ad0</code> (or rather, at <code>Tutorial-x86_64.exe+306AD0</code>), because the address shows green. How does it do this?</p>
329<h2 id="base-addresses">Base addresses</h2>
330<p>Remember back in <a href="/blog/woce-2">part 2</a> when we introduced the memory regions? They're making a comeback! A memory region contains both the current memory protection option <em>and</em> the protection level when the region was created. If we try printing out the protection levels for both the memory region containing the value, and the memory region containing the pointer, this is what we get (the addresses differ from the ones previously because I restarted the tutorial):</p>
331<pre><code>Region holding the value:
332 BaseAddress: 0xb0000
333 AllocationBase: 0xb0000
334 AllocationProtect: 0x4
335 RegionSize: 1007616
336 State: 4096
337 Protect: 4
338 Type: 0x20000
339
340Region holding the pointer:
341 BaseAddress: 0x100304000
342 AllocationBase: 0x100000000
343 AllocationProtect: 0x80
344 RegionSize: 28672
345 State: 4096
346 Protect: 4
347 Type: 0x1000000
348</code></pre>
349<p>Interesting! According to the <a href="https://docs.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-memory_basic_information"><code>MEMORY_BASIC_INFORMATION</code> page</a>, the type for the first region is <code>MEM_PRIVATE</code>, and the type for the second region is <code>MEM_IMAGE</code> which:</p>
350<blockquote>
351<p>Indicates that the memory pages within the region are mapped into the view of an image section.</p>
352</blockquote>
353<p>The protection also changes from <code>PAGE_EXECUTE_WRITECOPY</code> to simply <code>PAGE_READWRITE</code>, but I don't think it's relevant. Neither the type seems to be much more relevant. In <a href="/blog/woce-2">part 2</a> we also mentioned the concept of &quot;base address&quot;, but decided against using it, because starting to look for regions at address zero seemed to work fine. However, it would make sense that fixed &quot;addresses&quot; start at some known &quot;base&quot;. Let's try getting the <a href="https://stackoverflow.com/a/26573045/4759433">base address for all loaded modules</a>. Currently, we only get the address for the base module, in order to retrieve its name, but now we need them all:</p>
354<pre><code class="language-rust" data-lang="rust">pub fn enum_modules(&amp;self) -&gt; io::Result&lt;Vec&lt;winapi::shared::minwindef::HMODULE&gt;&gt; {
355 let mut size = 0;
356 if unsafe {
357 winapi::um::psapi::EnumProcessModules(
358 self.handle.as_ptr(),
359 ptr::null_mut(),
360 0,
361 &amp;mut size,
362 )
363 } == FALSE
364 {
365 return Err(io::Error::last_os_error());
366 }
367
368 let mut modules = Vec::with_capacity(size as usize / mem::size_of::&lt;HMODULE&gt;());
369 if unsafe {
370 winapi::um::psapi::EnumProcessModules(
371 self.handle.as_ptr(),
372 modules.as_mut_ptr(),
373 (modules.capacity() * mem::size_of::&lt;HMODULE&gt;()) as u32,
374 &amp;mut size,
375 )
376 } == FALSE
377 {
378 return Err(io::Error::last_os_error());
379 }
380
381 unsafe {
382 modules.set_len(size as usize / mem::size_of::&lt;HMODULE&gt;());
383 }
384
385 Ok(modules)
386}
387</code></pre>
388<p>The first call is used to retrieve the correct <code>size</code>, then we allocate just enough, and make the second call. The returned type are pretty much memory addresses, so let's see if we can find regions that contain them:</p>
389<pre><code class="language-rust" data-lang="rust">let mut bases = 0;
390let modules = process.enum_modules().unwrap();
391let regions = process.memory_regions();
392regions.iter().for_each(|region| {
393 if modules.iter().any(|module| {
394 let base = region.AllocationBase as usize;
395 let addr = *module as usize;
396 base &lt;= addr &amp;&amp; addr &lt; base + region.RegionSize
397 }) {
398 bases += 1;
399 }
400});
401
402println!(
403 &quot;{}/{} regions have a module address within them&quot;,
404 bases,
405 regions.len()
406);
407</code></pre>
408<pre><code>41/353 regions have a module address within them
409</code></pre>
410<p>Exciting stuff! It appears <code>base == addr</code> also does the trick<sup class="footnote-reference"><a href="#5">5</a></sup>, so now we could build a <code>bases: HashSet&lt;usize&gt;</code> and simply check if <code>bases.contains(&amp;region.AllocationBase as usize)</code> to determine whether <code>region</code> is a base address or not<sup class="footnote-reference"><a href="#6">6</a></sup>. So there we have it! The address holding the pointer value does fall within one of these &quot;base regions&quot;. You can also get the name from one of these module addresses, and print it in the same way as Cheat Engine does it (such as <code>Tutorial-x86_64.exe+306AD0</code>).</p>
411<h2 id="finale">Finale</h2>
412<p>So, there's no &quot;automated&quot; solution to all of this? That's the end? Well, yes, once you have a pointer you can dereference it once and then write to the given address to complete the tutorial step! I can understand how this would feel a bit underwhelming, but in all fairness, we were required to pretty-print assembly to guess what pointer address we could potentially need to look for. There is an <a href="https://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-software-developer-vol-2a-manual.html">stupidly large amount of instructions</a>, and I'm sure a lot of them can access memory, so automating that would be rough. We were lucky that the instructions right before the one that hit the breakpoint were changing the memory address, but you could imagine this value coming from somewhere completely different. It could also be using a myriad of different techniques to apply the offset. I would argue manual intervention is a must here<sup class="footnote-reference"><a href="#7">7</a></sup>.</p>
413<p>We have learnt how to pretty-print instructions, and had a very gentle introduction to figuring out what we may need to look for. The code to retrieve the loaded modules, and their corresponding regions, will come in handy later on. Having access to this information lets us know when to stop looking for additional pointers. As soon as a pointer is found within a memory region corresponding to a base module, we're done! Also, I know the title doesn't really much the contents of this entry (sorry about that), but I'm just following the convention of calling it whatever the Cheat Engine tutorial calls them.</p>
414<p>The <a href="https://github.com/lonami/memo">code for this post</a> is available over at my GitHub. You can run <code>git checkout step6</code> after cloning the repository to get the right version of the code, although you will have to <code>checkout</code> to individual commits if you want to review, for example, how the instructions were printed out. Only the code necessary to complete the step is included at the <code>step6</code> tag.</p>
415<p>In the next post, we'll tackle the sixth step of the tutorial: Code Injection. This will be pretty similar to part 5, but instead of writing out a simple NOP instruction, we will have to get a bit more creative.</p>
416<h3 id="footnotes">Footnotes</h3>
417<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
418<p>This will only be a gentle introduction to pointers. Part 8 of this series will have to rely on more advanced techniques.</p>
419</div>
420<div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup>
421<p>Kind of. The size of a pointer isn't necessarily the size as <code>usize</code>, although <code>usize</code> is guaranteed to be able of representing every possible address. For our purposes, we can assume a pointer is as big as <code>usize</code>.</p>
422</div>
423<div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup>
424<p>Game updates are likely to pull more code and shuffle stuff around. This is unfortunately a difficult problem to solve. But storing a pointer which is usable across restarts for as long as the game doesn't update is still a pretty darn big improvement over having to constantly scan for the locations we care about. Although if you're smart enough to look for certain unique patterns, even if the code is changed, finding those patterns will give you the new updated address, so it's not <em>impossible</em>.</p>
425</div>
426<div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup>
427<p><code>bool::then</code> is a pretty recent addition at the time of writing (1.50.0), so make sure you <code>rustup update</code> if it's erroring out!</p>
428</div>
429<div class="footnote-definition" id="5"><sup class="footnote-definition-label">5</sup>
430<p>I wasn't sure if there would be some metadata before the module base address but within the region, so I went with the range check. What <em>is</em> important however is using <code>AllocationBase</code>, not <code>BaseAddress</code>. They're different, and this did bite me.</p>
431</div>
432<div class="footnote-definition" id="6"><sup class="footnote-definition-label">6</sup>
433<p>As usual, I have no idea if this is how Cheat Engine is doing it, but it seems reasonable.</p>
434</div>
435<div class="footnote-definition" id="6"><sup class="footnote-definition-label">6</sup>
436<p>But nothing's stopping you from implementing some heuristics to get the job done for you. If you run some algorithm in your head to find what the pointer value could be, you can program it in Rust as well, although I don't think it's worth the effort.</p>
437</div>
438</content>
439 </entry>
440 <entry xml:lang="en">
441 <title>Writing our own Cheat Engine: Code finder</title>
442 <published>2021-03-06T00:00:00+00:00</published>
443 <updated>2021-03-06T00:00:00+00:00</updated>
444 <link href="https://lonami.dev/blog/woce-5/" type="text/html"/>
445 <id>https://lonami.dev/blog/woce-5/</id>
446 <content type="html"><p>This is part 5 on the <em>Writing our own Cheat Engine</em> series:</p>
447<ul>
448<li><a href="/blog/woce-1">Part 1: Introduction</a> (start here if you're new to the series!)</li>
449<li><a href="/blog/woce-2">Part 2: Exact Value scanning</a></li>
450<li><a href="/blog/woce-3">Part 3: Unknown initial value</a></li>
451<li><a href="/blog/woce-4">Part 4: Floating points</a></li>
452<li>Part 5: Code finder</li>
453<li><a href="/blog/woce-6">Part 6: Pointers</a></li>
454</ul>
455<p>In part 4 we spent a good deal of time trying to make our scans generic, and now we have something that works<sup class="footnote-reference"><a href="#1">1</a></sup>! Now that the scanning is fairly powerful and all covered, the Cheat Engine tutorial shifts focus into slightly more advanced techniques that you will most certainly need in anything bigger than a toy program.</p>
456<p>It's time to write our very own <strong>debugger</strong> in Rust!</p>
457<h2 id="code-finder">Code finder</h2>
458<details open><summary>Cheat Engine Tutorial: Step 5</summary>
459<blockquote>
460<p>Sometimes the location something is stored at changes when you restart the game, or even while you're playing… In that case you can use 2 things to still make a table that works. In this step I'll try to describe how to use the Code Finder function.</p>
461<p>The value down here will be at a different location each time you start the tutorial, so a normal entry in the address list wouldn't work. First try to find the address. (You've got to this point so I assume you know how to.)</p>
462<p>When you've found the address, right-click the address in Cheat Engine and choose &quot;Find out what writes to this address&quot;. A window will pop up with an empty list.</p>
463<p>Then click on the Change value button in this tutorial, and go back to Cheat Engine. If everything went right there should be an address with assembler code there now.</p>
464<p>Click it and choose the replace option to replace it with code that does nothing. That will also add the code address to the code list in the advanced options window. (Which gets saved if you save your table.)</p>
465<p>Click on stop, so the game will start running normal again, and close to close the window. Now, click on Change value, and if everything went right the Next button should become enabled.</p>
466<p>Note: When you're freezing the address with a high enough speed it may happen that next becomes visible anyhow</p>
467</blockquote>
468</details>
469<h2 id="baby-steps-to-debugging">Baby steps to debugging</h2>
470<p>Although I have used debuggers before, I have never had a need to write one myself so it's time for some research.</p>
471<p>Searching on DuckDuckGo, I can find entire series to <a href="http://system.joekain.com/debugger/">Writing a Debugger</a>. We would be done by now if only that series wasn't written for Linux. The Windows documentation contains a section called <a href="https://docs.microsoft.com/en-us/windows/win32/debug/creating-a-basic-debugger">Creating a Basic Debugger</a>, but as far as I can tell, it only teaches you the <a href="https://docs.microsoft.com/en-us/windows/win32/debug/debugging-functions">functions</a> needed to configure the debugging loop. Which mind you, we will need, but in due time.</p>
472<p>According to <a href="https://www.gironsec.com/blog/2013/12/writing-your-own-debugger-windows-in-c/">Writing your own windows debugger in C</a>, the steps needed to write a debugger are:</p>
473<ul>
474<li><a href="https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-suspendthread"><code>SuspendThread(proc)</code></a>. It makes sense that we need to pause all the threads<sup class="footnote-reference"><a href="#2">2</a></sup> before messing around with the code the program is executing, or things are very prone to go wrong.</li>
475<li><a href="https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-getthreadcontext"><code>GetThreadContext(proc)</code></a>. This function retrieves the appropriate context of the specified thread and is highly processor specific. It basically takes a snapshot of all the registers. Think of registers like extremely fast, but also extremely limited, memory the processor uses.</li>
476<li><a href="https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-debugbreakprocess"><code>DebugBreakProcess</code></a>. Essentially <a href="https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/x86-instructions#miscellaneous">writes out the 0xCC opcode</a>, <code>int 3</code> in assembly, also known as software breakpoint. It's written wherever the Register Instruction Pointer (RIP<sup class="footnote-reference"><a href="#3">3</a></sup>) currently points to, so in essence, when the thread resumes, it will immediately <a href="https://stackoverflow.com/q/3915511/">trigger the breakpoint</a>.</li>
477<li><a href="https://docs.microsoft.com/en-us/windows/win32/api/debugapi/nf-debugapi-continuedebugevent"><code>ContinueDebugEvent</code></a>. Presumably continues debugging.</li>
478</ul>
479<p>There are pages documenting <a href="https://docs.microsoft.com/en-us/windows/win32/debug/debugging-events">all of the debug events</a> that our debugger will be able to handle.</p>
480<p>Okay, nice! Software breakpoints seem to be done by writing out memory to the region where the program is reading instructions from. We know how to write memory, as that's what all the previous posts have been doing to complete the corresponding tutorial steps. After the breakpoint is executed, all we need to do is <a href="https://stackoverflow.com/q/3747852/">restore the original memory back</a> so that the next time the program executes the code it sees no difference.</p>
481<p>But a software breakpoint will halt execution when the code executes the interrupt instruction. This step of the tutorial wants us to find <em>what writes to a memory location</em>. Where should we place the breakpoint to detect such location? Writing out the instruction to the memory we want to break in won't do; it's not an instruction, it's just data.</p>
482<p>The name may have given it away. If we're talking about software breakpoints, it makes sense that there would exist such a thing as <a href="https://en.wikipedia.org/wiki/Breakpoint#Hardware"><em>hardware</em> breakpoints</a>. Because they're tied to the hardware, they're highly processor-specific, but luckily for us, the processor on your usual desktop computer probably has them! Even the <a href="https://interrupt.memfault.com/blog/cortex-m-breakpoints">cortex-m</a> does. The wikipedia page also tells us the name of the thing we're looking for, watchpoints:</p>
483<blockquote>
484<p>Other kinds of conditions can also be used, such as the reading, writing, or modification of a specific location in an area of memory. This is often referred to as a conditional breakpoint, a data breakpoint, or a watchpoint.</p>
485</blockquote>
486<p>A breakpoint that triggers when a specific memory location is written to is exactly what we need, and <a href="https://stackoverflow.com/a/19109153/">x86 has debug registers D0 to D3 to track memory addresses</a>. As far as I can tell, there is no API in specific to mess with the registers. But we don't need any of that! We can just go ahead and <a href="https://doc.rust-lang.org/stable/unstable-book/library-features/asm.html">write some assembly by hand</a> to access these registers. At the time of writing, inline assembly is unstable, so we need a nightly compiler. Run <code>rustup toolchain install nightly</code> if you haven't yet, and execute the following code with <code>cargo +nightly run</code>:</p>
487<pre><code class="language-rust" data-lang="rust">#![feature(asm)] // top of the file
488
489fn main() {
490 let x: u64 = 123;
491 unsafe {
492 asm!(&quot;mov dr7, {}&quot;, in(reg) x);
493 }
494}
495
496</code></pre>
497<p><code>dr7</code> stands is the <a href="https://en.wikipedia.org/wiki/X86_debug_register">debug control register</a>, and running this we get…</p>
498<pre><code>&gt;cargo +nightly run
499 Compiling memo v0.1.0
500 Finished dev [unoptimized + debuginfo] target(s) in 0.74s
501 Running `target\debug\memo.exe`
502error: process didn't exit successfully: `target\debug\memo.exe` (exit code: 0xc0000096, STATUS_PRIVILEGED_INSTRUCTION)
503</code></pre>
504<p>…an exception! In all fairness, I have no idea what that code would have done. So maybe the <code>STATUS_PRIVILEGED_INSTRUCTION</code> is just trying to protect us. Can we read from the register instead, and see it's default value?</p>
505<pre><code class="language-rust" data-lang="rust">let x: u64;
506unsafe {
507 asm!(&quot;mov {}, dr7&quot;, out(reg) x);
508}
509assert_eq!(x, 5);
510</code></pre>
511<pre><code>&gt;cargo +nightly run
512...
513error: process didn't exit successfully: `target\debug\memo.exe` (exit code: 0xc0000096, STATUS_PRIVILEGED_INSTRUCTION)
514</code></pre>
515<p>Nope. Okay, it seems directly reading from or writing to the debug register is a ring-0 thing. Surely there's a way around this. But first we should figure out how to enumerate and pause all the threads.</p>
516<h2 id="pausing-all-the-threads">Pausing all the threads</h2>
517<p>It seems there is no straightforward way to enumerate the threads. One has to <a href="https://stackoverflow.com/a/1206915/">create a &quot;toolhelp&quot;</a> and poll the entries. I won't bore you with the details. Let's add <code>tlhelp32</code> to the crate features of <code>winapi</code> and try it out:</p>
518<pre><code class="language-rust" data-lang="rust">
519#[derive(Debug)]
520pub struct Toolhelp {
521 handle: winapi::um::winnt::HANDLE,
522}
523
524impl Drop for Toolhelp {
525 fn drop(&amp;mut self) {
526 unsafe { winapi::um::handleapi::CloseHandle(self.handle) };
527 }
528}
529
530pub fn enum_threads(pid: u32) -&gt; io::Result&lt;Vec&lt;u32&gt;&gt; {
531 const ENTRY_SIZE: u32 = mem::size_of::&lt;winapi::um::tlhelp32::THREADENTRY32&gt;() as u32;
532
533 // size_of(dwSize + cntUsage + th32ThreadID + th32OwnerProcessID)
534 const NEEDED_ENTRY_SIZE: u32 = 4 * mem::size_of::&lt;DWORD&gt;() as u32;
535
536 // SAFETY: it is always safe to attempt to call this function.
537 let handle = unsafe {
538 winapi::um::tlhelp32::CreateToolhelp32Snapshot(winapi::um::tlhelp32::TH32CS_SNAPTHREAD, 0)
539 };
540 if handle == winapi::um::handleapi::INVALID_HANDLE_VALUE {
541 return Err(io::Error::last_os_error());
542 }
543 let toolhelp = Toolhelp { handle };
544
545 let mut result = Vec::new();
546 let mut entry = winapi::um::tlhelp32::THREADENTRY32 {
547 dwSize: ENTRY_SIZE,
548 cntUsage: 0,
549 th32ThreadID: 0,
550 th32OwnerProcessID: 0,
551 tpBasePri: 0,
552 tpDeltaPri: 0,
553 dwFlags: 0,
554 };
555
556 // SAFETY: we have a valid handle, and point to memory we own with the right size.
557 if unsafe { winapi::um::tlhelp32::Thread32First(toolhelp.handle, &amp;mut entry) } != FALSE {
558 loop {
559 if entry.dwSize &gt;= NEEDED_ENTRY_SIZE &amp;&amp; entry.th32OwnerProcessID == pid {
560 result.push(entry.th32ThreadID);
561 }
562
563 entry.dwSize = ENTRY_SIZE;
564 // SAFETY: we have a valid handle, and point to memory we own with the right size.
565 if unsafe { winapi::um::tlhelp32::Thread32Next(toolhelp.handle, &amp;mut entry) } == FALSE {
566 break;
567 }
568 }
569 }
570
571 Ok(result)
572}
573</code></pre>
574<p>Annoyingly, invalid handles returned by <a href="https://docs.microsoft.com/en-us/windows/win32/api/tlhelp32/nf-tlhelp32-createtoolhelp32snapshot"><code>CreateToolhelp32Snapshot</code></a>, are <code>INVALID_HANDLE_VALUE</code> (which is -1), not null. But that's not a big deal, we simply can't use <code>NonNull</code> here. The function ignores the process identifier when using <code>TH32CS_SNAPTHREAD</code>, used to include all threads, and we need to compare the process identifier ourselves.</p>
575<p>In summary, we create a &quot;toolhelp&quot; (wrapped in a helper <code>struct</code> so that whatever happens, <code>Drop</code> will clean it up), initialize a thread enntry (with everything but the structure size to zero) and call <code>Thread32First</code> the first time, <code>Thread32Next</code> subsequent times. It seems to work all fine!</p>
576<pre><code class="language-rust" data-lang="rust">dbg!(process::enum_threads(pid));
577</code></pre>
578<pre><code>[src\main.rs:46] process::enum_threads(pid) = Ok(
579 [
580 10560,
581 ],
582)
583</code></pre>
584<p>According to this, the Cheat Engine tutorial is only using one thread. Good to know. Much like processes, threads need to be opened before we can use them, with <a href="https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-openthread"><code>OpenThread</code></a>:</p>
585<pre><code class="language-rust" data-lang="rust">pub struct Thread {
586 tid: u32,
587 handle: NonNull&lt;c_void&gt;,
588}
589
590impl Thread {
591 pub fn open(tid: u32) -&gt; io::Result&lt;Self&gt; {
592 // SAFETY: the call doesn't have dangerous side-effects
593 NonNull::new(unsafe {
594 winapi::um::processthreadsapi::OpenThread(
595 winapi::um::winnt::THREAD_SUSPEND_RESUME,
596 FALSE,
597 tid,
598 )
599 })
600 .map(|handle| Self { tid, handle })
601 .ok_or_else(io::Error::last_os_error)
602 }
603
604 pub fn tid(&amp;self) -&gt; u32 {
605 self.tid
606 }
607}
608
609impl Drop for Thread {
610 fn drop(&amp;mut self) {
611 unsafe { winapi::um::handleapi::CloseHandle(self.handle.as_mut()) };
612 }
613}
614</code></pre>
615<p>Just your usual RAII pattern. The thread is opened with permission to suspend and resume it. Let's try to pause the handles with <a href="https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-suspendthread"><code>SuspendThread</code></a> to make sure that this thread is actually the one we're looking for:</p>
616<pre><code class="language-rust" data-lang="rust">pub fn suspend(&amp;mut self) -&gt; io::Result&lt;usize&gt; {
617 // SAFETY: the handle is valid.
618 let ret = unsafe {
619 winapi::um::processthreadsapi::SuspendThread(self.handle.as_ptr())
620 };
621 if ret == -1i32 as u32 {
622 Err(io::Error::last_os_error())
623 } else {
624 Ok(ret as usize)
625 }
626}
627
628pub fn resume(&amp;mut self) -&gt; io::Result&lt;usize&gt; {
629 // SAFETY: the handle is valid.
630 let ret = unsafe {
631 winapi::um::processthreadsapi::ResumeThread(self.handle.as_ptr())
632 };
633 if ret == -1i32 as u32 {
634 Err(io::Error::last_os_error())
635 } else {
636 Ok(ret as usize)
637 }
638}
639</code></pre>
640<p>Both suspend and resume return the previous &quot;suspend count&quot;. It's kind of like a barrier or semaphore where the thread only runs if the suspend count is zero. Trying it out:</p>
641<pre><code class="language-rust" data-lang="rust">let mut threads = thread::enum_threads(pid)
642 .unwrap()
643 .into_iter()
644 .map(Thread::open)
645 .collect::&lt;Result&lt;Vec&lt;_&gt;, _&gt;&gt;()
646 .unwrap();
647
648threads
649 .iter_mut()
650 .for_each(|thread| {
651 println!(&quot;Pausing thread {} for 10 seconds…&quot;, thread.tid());
652 thread.suspend().unwrap();
653
654 std::thread::sleep(std::time::Duration::from_secs(10));
655
656 println!(&quot;Wake up, {}!&quot;, thread.tid());
657 thread.resume().unwrap();
658 });
659</code></pre>
660<p>If you run this code with the process ID of the Cheat Engine tutorial, you will see that the tutorial window freezes for ten seconds! Because the main and only thread is paused, it cannot process any window events, so it becomes unresponsive. It is now &quot;safe&quot; to mess around with the thread context.</p>
661<h2 id="setting-hardware-breakpoints">Setting hardware breakpoints</h2>
662<p>I'm definitely not the first person to wonder <a href="https://social.msdn.microsoft.com/Forums/en-US/0cb3360d-3747-42a7-bc0e-668c5d9ee1ee/how-to-set-a-hardware-breakpoint">How to set a hardware breakpoint?</a>. This is great, because it means I don't need to ask that question myself. It appears we need to change the debug register <em>via the thread context</em>.</p>
663<p>One has to be careful to use the right context structure. Confusingly enough, <a href="https://stackoverflow.com/q/17504174/"><code>WOW64_CONTEXT</code></a> is 32 bits, not 64. <code>CONTEXT</code> alone seems to be the right one:</p>
664<pre><code class="language-rust" data-lang="rust">pub fn get_context(&amp;self) -&gt; io::Result&lt;winapi::um::winnt::CONTEXT&gt; {
665 let context = MaybeUninit::&lt;winapi::um::winnt::CONTEXT&gt;::zeroed();
666 // SAFETY: it's a C struct, and all-zero is a valid bit-pattern for the type.
667 let mut context = unsafe { context.assume_init() };
668 context.ContextFlags = winapi::um::winnt::CONTEXT_ALL;
669
670 // SAFETY: the handle is valid and structure points to valid memory.
671 if unsafe {
672 winapi::um::processthreadsapi::GetThreadContext(self.handle.as_ptr(), &amp;mut context)
673 } == FALSE
674 {
675 Err(io::Error::last_os_error())
676 } else {
677 Ok(context)
678 }
679}
680</code></pre>
681<p>Trying it out:</p>
682<pre><code class="language-rust" data-lang="rust">thread.suspend().unwrap();
683
684let context = thread.get_context().unwrap();
685println!(&quot;Dr0: {:016x}&quot;, context.Dr0);
686println!(&quot;Dr7: {:016x}&quot;, context.Dr7);
687println!(&quot;Dr6: {:016x}&quot;, context.Dr6);
688println!(&quot;Rax: {:016x}&quot;, context.Rax);
689println!(&quot;Rbx: {:016x}&quot;, context.Rbx);
690println!(&quot;Rcx: {:016x}&quot;, context.Rcx);
691println!(&quot;Rip: {:016x}&quot;, context.Rip);
692</code></pre>
693<pre><code>Dr0: 0000000000000000
694Dr7: 0000000000000000
695Dr6: 0000000000000000
696Rax: 0000000000001446
697Rbx: 0000000000000000
698Rcx: 0000000000000000
699Rip: 00007ffda4259904
700</code></pre>
701<p>Looks about right! Hm, I wonder what happens if I use Cheat Engine to add the watchpoint on the memory location we care about?</p>
702<pre><code>Dr0: 000000000157e650
703Dr7: 00000000000d0001
704</code></pre>
705<p>Look at that! The debug registers changed! DR0 contains the location we want to watch for writes, and the debug control register DR7 changed. Cheat Engine sets the same values on all threads (for some reason I now see more than one thread printed for the tutorial, not sure what's up with that; maybe the single-thread is the weird one out).</p>
706<p>Hmm, what happens if I watch for access instead of write?</p>
707<pre><code>Dr0: 000000000157e650
708Dr7: 00000000000f0001
709</code></pre>
710<p>What if I set both?</p>
711<pre><code>Dr0: 000000000157e650
712Dr7: 0000000000fd0005
713</code></pre>
714<p>Most intriguing! This was done by telling Cheat Engine to find &quot;what writes&quot; to the address, then &quot;what accesses&quot; the address. I wonder if the order matters?</p>
715<pre><code>Dr0: 000000000157e650
716Dr7: 0000000000df0005
717</code></pre>
718<p>&quot;What accesses&quot; and then &quot;what writes&quot; does change it. Very well! We're only concerned in a single breakpoint, so we won't worry about this, but it's good to know that we can inspect what Cheat Engine is doing. It's also interesting to see how Cheat Engine is using hardware breakpoints and not software breakpoints.</p>
719<p>For simplicity, our code is going to assume that we're the only ones messing around with the debug registers, and that there will only be a single debug register in use. Make sure to add <code>THREAD_SET_CONTEXT</code> to the permissions when opening the thread handle:</p>
720<pre><code class="language-rust" data-lang="rust">pub fn set_context(&amp;self, context: &amp;winapi::um::winnt::CONTEXT) -&gt; io::Result&lt;()&gt; {
721 // SAFETY: the handle is valid and structure points to valid memory.
722 if unsafe {
723 winapi::um::processthreadsapi::SetThreadContext(self.handle.as_ptr(), context)
724 } == FALSE
725 {
726 Err(io::Error::last_os_error())
727 } else {
728 Ok(())
729 }
730}
731
732pub fn watch_memory_write(&amp;self, addr: usize) -&gt; io::Result&lt;()&gt; {
733 let mut context = self.get_context()?;
734 context.Dr0 = addr as u64;
735 context.Dr7 = 0x00000000000d0001;
736 self.set_context(&amp;context)?;
737 todo!()
738}
739</code></pre>
740<p>If we do this (and temporarily get rid of the <code>todo!()</code>), trying to change the value in the Cheat Engine tutorial will greet us with a warm message:</p>
741<blockquote>
742<p><strong>Tutorial-x86_64</strong></p>
743<p>External exception 80000004.</p>
744<p>Press OK to ignore and risk data corruption.<br />
745Press Abort to kill the program.</p>
746<p><kbd>OK</kbd> <kbd>Abort</kbd></p>
747</blockquote>
748<p>There is no debugger attached yet that could possibly handle this exception, so the exception just propagates. Let's fix that.</p>
749<h2 id="handling-debug-events">Handling debug events</h2>
750<p>Now that we've succeeded on setting breakpoints, we can actually follow the steps described in <a href="https://docs.microsoft.com/en-us/windows/win32/debug/creating-a-basic-debugger">Creating a Basic Debugger</a>. It starts by saying that we should use <a href="https://docs.microsoft.com/en-us/windows/win32/api/debugapi/nf-debugapi-debugactiveprocess"><code>DebugActiveProcess</code></a> to attach our processor, the debugger, to the process we want to debug, the debuggee. This function lives under the <code>debugapi</code> header, so add it to <code>winapi</code> features:</p>
751<pre><code class="language-rust" data-lang="rust">pub struct DebugToken {
752 pid: u32,
753}
754
755pub fn debug(pid: u32) -&gt; io::Result&lt;DebugToken&gt; {
756 if unsafe { winapi::um::debugapi::DebugActiveProcess(pid) } == FALSE {
757 return Err(io::Error::last_os_error());
758 };
759 let token = DebugToken { pid };
760 if unsafe { winapi::um::winbase::DebugSetProcessKillOnExit(FALSE) } == FALSE {
761 return Err(io::Error::last_os_error());
762 };
763 Ok(token)
764}
765
766impl Drop for DebugToken {
767 fn drop(&amp;mut self) {
768 unsafe { winapi::um::debugapi::DebugActiveProcessStop(self.pid) };
769 }
770}
771</code></pre>
772<p>Once again, we create a wrapper <code>struct</code> with <code>Drop</code> to stop debugging the process once the token is dropped. The call to <code>DebugSetProcessKillOnExit</code> in our <code>debug</code> method ensures that, if our process (the debugger) dies, the process we're debugging (the debuggee) stays alive. We don't want to be restarting the entire Cheat Engine tutorial every time our Rust code crashes!</p>
773<p>With the debugger attached, we can wait for debug events. We will put this method inside of <code>impl DebugToken</code>, so that the only way you can call it is if you successfully attached to another process:</p>
774<pre><code class="language-rust" data-lang="rust">impl DebugToken {
775 pub fn wait_event(
776 &amp;self,
777 timeout: Option&lt;Duration&gt;,
778 ) -&gt; io::Result&lt;winapi::um::minwinbase::DEBUG_EVENT&gt; {
779 let mut result = MaybeUninit::uninit();
780 let timeout = timeout
781 .map(|d| d.as_millis().try_into().ok())
782 .flatten()
783 .unwrap_or(winapi::um::winbase::INFINITE);
784
785 // SAFETY: can only wait for events with a token, so the debugger is active.
786 if unsafe { winapi::um::debugapi::WaitForDebugEvent(result.as_mut_ptr(), timeout) } == FALSE
787 {
788 Err(io::Error::last_os_error())
789 } else {
790 // SAFETY: the call returned non-zero, so the structure is initialized.
791 Ok(unsafe { result.assume_init() })
792 }
793 }
794}
795</code></pre>
796<p><code>WaitForDebugEvent</code> wants a timeout in milliseconds, so our function lets the user pass the more Rusty <code>Duration</code> type. <code>None</code> will indicate &quot;there is no timeout&quot;, i.e., it's infinite. If the duration is too large to fit in the <code>u32</code> (<code>try_into</code> fails), it will also be infinite.</p>
797<p>If we attach the debugger, set the hardware watchpoint, and modify the memory location from the tutorial, an event with <code>dwDebugEventCode = 3</code> will be returned! Now, back to the page with the <a href="https://docs.microsoft.com/en-us/windows/win32/debug/debugging-events">Debugging Events</a>… Gah! It only has the name of the constants, not the values. Well, good thing <a href="https://docs.rs/">docs.rs</a> has a source view! We can just check the values in the <a href="https://docs.rs/winapi/0.3.9/src/winapi/um/minwinbase.rs.html#203-211">source code for <code>winapi</code></a>:</p>
798<pre><code class="language-rust" data-lang="rust">pub const EXCEPTION_DEBUG_EVENT: DWORD = 1;
799pub const CREATE_THREAD_DEBUG_EVENT: DWORD = 2;
800pub const CREATE_PROCESS_DEBUG_EVENT: DWORD = 3;
801pub const EXIT_THREAD_DEBUG_EVENT: DWORD = 4;
802pub const EXIT_PROCESS_DEBUG_EVENT: DWORD = 5;
803pub const LOAD_DLL_DEBUG_EVENT: DWORD = 6;
804pub const UNLOAD_DLL_DEBUG_EVENT: DWORD = 7;
805pub const OUTPUT_DEBUG_STRING_EVENT: DWORD = 8;
806pub const RIP_EVENT: DWORD = 9;
807</code></pre>
808<p>So, we've got a <code>CREATE_PROCESS_DEBUG_EVENT</code>:</p>
809<blockquote>
810<p>Generated whenever a new process is created in a process being debugged or whenever the debugger begins debugging an already active process. The system generates this debugging event before the process begins to execute in user mode and before the system generates any other debugging events for the new process.</p>
811</blockquote>
812<p>It makes sense that this is our first event. By the way, if you were trying this out with a <code>sleep</code> lying around in your code, you may have noticed that the window froze until the debugger terminated. That's because:</p>
813<blockquote>
814<p>When the system notifies the debugger of a debugging event, it also suspends all threads in the affected process. The threads do not resume execution until the debugger continues the debugging event by using <a href="https://docs.microsoft.com/en-us/windows/win32/api/debugapi/nf-debugapi-continuedebugevent"><code>ContinueDebugEvent</code></a>.</p>
815</blockquote>
816<p>Let's call <code>ContinueDebugMethod</code> but also wait on more than one event and see what happens:</p>
817<pre><code class="language-rust" data-lang="rust">for _ in 0..10 {
818 let event = debugger.wait_event(None).unwrap();
819 println!(&quot;Got {}&quot;, event.dwDebugEventCode);
820 debugger.cont(event, true).unwrap();
821}
822</code></pre>
823<pre><code>Got 3
824Got 6
825Got 6
826Got 6
827Got 6
828Got 6
829Got 6
830Got 6
831Got 6
832Got 6
833</code></pre>
834<p>That's a lot of <code>LOAD_DLL_DEBUG_EVENT</code>. Pumping it up to one hundred and also showing the index we get the following:</p>
835<pre><code>0. Got 3
8361. Got 6
837...
83840. Got 6
83941. Got 2
84042. Got 1
84143. Got 4
842</code></pre>
843<p>In order, we got:</p>
844<ul>
845<li>One <code>CREATE_PROCESS_DEBUG_EVENT</code>.</li>
846<li>Forty <code>LOAD_DLL_DEBUG_EVENT</code>.</li>
847<li>One <code>CREATE_THREAD_DEBUG_EVENT</code>.</li>
848<li>One <code>EXCEPTION_DEBUG_EVENT</code>.</li>
849<li>One <code>EXIT_THREAD_DEBUG_EVENT</code>.</li>
850</ul>
851<p>And, if after all this, you change the value in the Cheat Engine tutorial (thus triggering our watch point), we get <code>EXCEPTION_DEBUG_EVENT</code>!</p>
852<blockquote>
853<p>Generated whenever an exception occurs in the process being debugged. Possible exceptions include attempting to access inaccessible memory, executing breakpoint instructions, attempting to divide by zero, or any other exception noted in Structured Exception Handling.</p>
854</blockquote>
855<p>If we print out all the fields in the <a href="https://docs.microsoft.com/en-us/windows/win32/api/minwinbase/ns-minwinbase-exception_debug_info"><code>EXCEPTION_DEBUG_INFO</code></a> structure:</p>
856<pre><code>Watching writes to 10e3a0 for 10s
857First chance: 1
858ExceptionCode: 2147483652
859ExceptionFlags: 0
860ExceptionRecord: 0x0
861ExceptionAddress: 0x10002c5ba
862NumberParameters: 0
863ExceptionInformation: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
864</code></pre>
865<p>The <code>ExceptionCode</code>, which is <code>0x80000004</code>, corresponds with <code>EXCEPTION_SINGLE_STEP</code>:</p>
866<blockquote>
867<p>A trace trap or other single-instruction mechanism signaled that one instruction has been executed.</p>
868</blockquote>
869<p>The <code>ExceptionAddress</code> is supposed to be &quot;the address where the exception occurred&quot;. Very well! I have already completed this step of the tutorial, and I know the instruction is <code>mov [rax],edx</code> (or, as Cheat Engine shows, the bytes <code>89 10</code> in hexadecimal). The opcode for the <code>nop</code> instruction is <code>90</code> in hexadecimal, so if we replace two bytes at this address, we should be able to complete the tutorial.</p>
870<p>Note that we also need to flush the instruction cache, as noted in the Windows documentation:</p>
871<blockquote>
872<p>Debuggers frequently read the memory of the process being debugged and write the memory that contains instructions to the instruction cache. After the instructions are written, the debugger calls the <a href="https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-flushinstructioncache"><code>FlushInstructionCache</code></a> function to execute the cached instructions.</p>
873</blockquote>
874<p>So we add a new method to <code>impl Process</code>:</p>
875<pre><code class="language-rust" data-lang="rust">/// Flushes the instruction cache.
876///
877/// Should be called when writing to memory regions that contain code.
878pub fn flush_instruction_cache(&amp;self) -&gt; io::Result&lt;()&gt; {
879 // SAFETY: the call doesn't have dangerous side-effects.
880 if unsafe {
881 winapi::um::processthreadsapi::FlushInstructionCache(
882 self.handle.as_ptr(),
883 ptr::null(),
884 0,
885 )
886 } == FALSE
887 {
888 Err(io::Error::last_os_error())
889 } else {
890 Ok(())
891 }
892}
893</code></pre>
894<p>And write some quick and dirty code to get this done:</p>
895<pre><code class="language-rust" data-lang="rust">let addr = ...;
896println!(&quot;Watching writes to {:x} for 10s&quot;, addr);
897threads.iter_mut().for_each(|thread| {
898 thread.watch_memory_write(addr).unwrap();
899});
900loop {
901 let event = debugger.wait_event(None).unwrap();
902 if event.dwDebugEventCode == 1 {
903 let exc = unsafe { event.u.Exception() };
904 if exc.ExceptionRecord.ExceptionCode == 2147483652 {
905 let addr = exc.ExceptionRecord.ExceptionAddress as usize;
906 match process.write_memory(addr, &amp;[0x90, 0x90]) {
907 Ok(_) =&gt; eprintln!(&quot;Patched [{:x}] with NOP&quot;, addr),
908 Err(e) =&gt; eprintln!(&quot;Failed to patch [{:x}] with NOP: {}&quot;, addr, e),
909 };
910 process.flush_instruction_cache().unwrap();
911 debugger.cont(event, true).unwrap();
912 break;
913 }
914 }
915 debugger.cont(event, true).unwrap();
916}
917</code></pre>
918<p>Although it seems to work:</p>
919<pre><code>Watching writes to 15103f0 for 10s
920Patched [10002c5ba] with NOP
921</code></pre>
922<p>It really doesn't:</p>
923<blockquote>
924<p><strong>Tutorial-x86_64</strong></p>
925<p>Access violation.</p>
926<p>Press OK to ignore and risk data corruption.<br />
927Press Abort to kill the program.</p>
928<p><kbd>OK</kbd> <kbd>Abort</kbd></p>
929</blockquote>
930<p>Did we write memory somewhere we shouldn't? The documentation does mention &quot;segment-relative&quot; and &quot;linear virtual addresses&quot;:</p>
931<blockquote>
932<p><code>GetThreadSelectorEntry</code> returns the descriptor table entry for a specified selector and thread. Debuggers use the descriptor table entry to convert a segment-relative address to a linear virtual address. The <code>ReadProcessMemory</code> and <code>WriteProcessMemory</code> functions require linear virtual addresses.</p>
933</blockquote>
934<p>But nope! This isn't the problem. The problem is that the <code>ExceptionRecord.ExceptionAddress</code> is <em>after</em> the execution happened, so it's already 2 bytes beyond where it should be. We were accidentally writing out the first half of the next instruction, which, yeah, could not end good.</p>
935<p>So does it work if I do this instead?:</p>
936<pre><code class="language-rust" data-lang="rust">process.write_memory(addr - 2, &amp;[0x90, 0x90])
937// ^^^ new
938</code></pre>
939<p>This totally does work. Step 5: complete 🎉</p>
940<h2 id="properly-patching-instructions">Properly patching instructions</h2>
941<p>You may not be satisfied at all with our solution. Not only are we hardcoding some magic constants to set hardware watchpoints, we're also relying on knowledge specific to the Cheat Engine tutorial (insofar that we're replacing two bytes worth of instruction with NOPs).</p>
942<p>Properly supporting more than one hardware breakpoint, along with supporting different types of breakpoints, is definitely doable. The meaning of the bits for the debug registers is well defined, and you can definitely study that to come up with <a href="https://github.com/mmorearty/hardware-breakpoints">something more sophisticated</a> and support multiple different breakpoints. But for now, that's out of the scope of this series. The tutorial only wants us to use an on-write watchpoint, and our solution is fine and portable for that use case.</p>
943<p>However, relying on the size of the instructions is pretty bad. The instructions x86 executes are of variable length, so we can't possibly just look back until we find the previous instruction, or even naively determine its length. A lot of unrelated sequences of bytes are very likely instructions themselves. We need a disassembler. No, we're not writing our own<sup class="footnote-reference"><a href="#4">4</a></sup>.</p>
944<p>Searching on <a href="https://crates.io">crates.io</a> for &quot;disassembler&quot; yields a few results, and the first one I've found is <a href="https://crates.io/crates/iced-x86">iced-x86</a>. I like the name, it has a decent amount of GitHub stars, and it was last updated less than a month ago. I don't know about you, but I think we've just hit a jackpot!</p>
945<p>It's quite heavy though, so I will add it behind a feature gate, and users that want it may opt into it:</p>
946<pre><code class="language-toml" data-lang="toml">[features]
947patch-nops = [&quot;iced-x86&quot;]
948
949[dependencies]
950iced-x86 = { version = &quot;1.10.3&quot;, optional = true }
951</code></pre>
952<p>You can make use of it with <code>cargo run --features=patch-nops</code>. I don't want to turn this blog post into a tutorial for <code>iced-x86</code>, but in essence, we need to make use of its <code>Decoder</code>. Here's the plan:</p>
953<ol>
954<li>Find the memory region corresponding to the address we want to patch.</li>
955<li>Read the entire region.</li>
956<li>Decode the read bytes until the instruction pointer reaches our address.</li>
957<li>Because we just parsed the previous instruction, we know its length, and can be replaced with NOPs.</li>
958</ol>
959<pre><code class="language-rust" data-lang="rust">#[cfg(feature = &quot;patch-nops&quot;)]
960pub fn nop_last_instruction(&amp;self, addr: usize) -&gt; io::Result&lt;()&gt; {
961 use iced_x86::{Decoder, DecoderOptions, Formatter, Instruction, NasmFormatter};
962
963 let region = self
964 .memory_regions()
965 .into_iter()
966 .find(|region| {
967 let base = region.BaseAddress as usize;
968 base &lt;= addr &amp;&amp; addr &lt; base + region.RegionSize
969 })
970 .ok_or_else(|| io::Error::new(io::ErrorKind::Other, &quot;no matching region found&quot;))?;
971
972 let bytes = self.read_memory(region.BaseAddress as usize, region.RegionSize)?;
973
974 let mut decoder = Decoder::new(64, &amp;bytes, DecoderOptions::NONE);
975 decoder.set_ip(region.BaseAddress as _);
976
977 let mut instruction = Instruction::default();
978 while decoder.can_decode() {
979 decoder.decode_out(&amp;mut instruction);
980 if instruction.next_ip() as usize == addr {
981 return self
982 .write_memory(instruction.ip() as usize, &amp;vec![0x90; instruction.len()])
983 .map(drop);
984 }
985 }
986
987 Err(io::Error::new(
988 io::ErrorKind::Other,
989 &quot;no matching instruction found&quot;,
990 ))
991}
992</code></pre>
993<p>Pretty straightforward! We can set the &quot;instruction pointer&quot; of the decoder so that it matches with the address we're reading from. The <code>next_ip</code> method comes in really handy. Overall, it's a bit inefficient, because we could reuse the regions retrieved previously, but other than that, there is not much room for improvement.</p>
994<p>With this, we are no longer hardcoding the instruction size or guessing which instruction is doing what. You may wonder, what if the region does not start with valid executable code? It could be possible that the instructions are in some memory region with garbage except for a very specific location with real code. I don't know how Cheat Engine handles this, but I think it's reasonable to assume that the region starts with valid code.</p>
995<p>As far as I can tell (after having asked a bit around), the encoding is usually self synchronizing (similar to UTF-8), so eventually we should end up with correct instructions. But someone can still intentionally write real code between garbage data which we would then disassemble incorrectly. This is a problem on all variable-length ISAs. Half a solution is to <a href="https://stackoverflow.com/q/3983735/">start at the entry point</a>, decode all instructions, and follow the jumps. The other half would be correctly identifying jumps created just to trip a disassembler up, and jumps pointing to dynamically-calculated addresses!</p>
996<h2 id="finale">Finale</h2>
997<p>That was quite a deep dive! We have learnt about the existence of the various breakpoint types (software, hardware, and even behaviour, such as watchpoints), how to debug a separate process, and how to correctly update the code other process is running on-the-fly. The <a href="https://github.com/lonami/memo">code for this post</a> is available over at my GitHub. You can run <code>git checkout step5</code> after cloning the repository to get the right version of the code.</p>
998<p>Although we've only talked about <em>setting</em> breakpoints, there are of course <a href="https://reverseengineering.stackexchange.com/a/16547">ways of detecting them</a>. There's <a href="https://www.codeproject.com/Articles/30815/An-Anti-Reverse-Engineering-Guide">entire guides about it</a>. Again, we currently hardcode the fact we want to add a single watchpoint using the first debug register. A proper solution here would be to actually calculate the needs that need to be set, as well as keeping track of how many breakpoints have been added so far.</p>
999<p>Hardware breakpoints are also limited, since they're simply a bunch of registers, and our machine does not have infinite registers. How are other debuggers like <code>gdb</code> able to create a seemingly unlimited amount of breakpoints? Well, the GDB wiki actually has a page on <a href="https://sourceware.org/gdb/wiki/Internals%20Watchpoints">Internals Watchpoints</a>, and it's really interesting! <code>gdb</code> essentially single-steps through the entire program and tests the expressions after every instruction:</p>
1000<blockquote>
1001<p>Software watchpoints are very slow, since GDB needs to single-step the program being debugged and test the value of the watched expression(s) after each instruction.</p>
1002</blockquote>
1003<p>However, that's not the only way. One could <a href="https://stackoverflow.com/a/7805842/">change the protection level</a> of the region of interest (for example, remove the write permission), and when the program tries to write there, it will fail! In any case, the GDB wiki is actually a pretty nice resource. It also has a section on <a href="https://sourceware.org/gdb/wiki/Internals/Breakpoint%20Handling">Breakpoint Handling</a>, which contains some additional insight.</p>
1004<p>With regards to code improvements, <code>DebugToken::wait_event</code> could definitely be both nicer and safer to use, with a custom <code>enum</code>, so the user does not need to rely on magic constants or having to resort to <code>unsafe</code> access to get the right <code>union</code> variant.</p>
1005<p>In the <a href="/blog/woce-6">next post</a>, we'll tackle the sixth step of the tutorial: Pointers. It reuses the debugging techniques presented here to backtrack where the pointer for our desired value is coming from, so here we will need to actually <em>understand</em> what the instructions are doing, not just patching them out!</p>
1006<h3 id="footnotes">Footnotes</h3>
1007<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
1008<p>I'm not super happy about the design of it all, but we won't actually need anything beyond scanning for integers for the rest of the steps so it doesn't really matter.</p>
1009</div>
1010<div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup>
1011<p>There seems to be a way to pause the entire process in one go, with the <a href="https://stackoverflow.com/a/4062698/">undocumented <code>NtSuspendProcess</code></a> function!</p>
1012</div>
1013<div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup>
1014<p>It really is called that. The naming went from &quot;IP&quot; (instruction pointer, 16 bits), to &quot;EIP&quot; (extended instruction pointer, 32 bits) and currently &quot;RIP&quot; (64 bits). The naming convention for upgraded registers is the same (RAX, RBX, RCX, and so on). The <a href="https://wiki.osdev.org/CPU_Registers_x86_64">OS Dev wiki</a> is a great resource for this kind of stuff.</p>
1015</div>
1016<div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup>
1017<p>Well, we don't need an entire disassembler. Knowing the length of each instruction is enough, but that on its own is also a lot of work.</p>
1018</div>
1019</content>
1020 </entry>
1021 <entry xml:lang="en">
1022 <title>Writing our own Cheat Engine: Floating points</title>
1023 <published>2021-02-28T00:00:00+00:00</published>
1024 <updated>2021-02-28T00:00:00+00:00</updated>
1025 <link href="https://lonami.dev/blog/woce-4/" type="text/html"/>
1026 <id>https://lonami.dev/blog/woce-4/</id>
1027 <content type="html"><p>This is part 4 on the <em>Writing our own Cheat Engine</em> series:</p>
1028<ul>
1029<li><a href="/blog/woce-1">Part 1: Introduction</a> (start here if you're new to the series!)</li>
1030<li><a href="/blog/woce-2">Part 2: Exact Value scanning</a></li>
1031<li><a href="/blog/woce-3">Part 3: Unknown initial value</a></li>
1032<li>Part 4: Floating points</li>
1033<li><a href="/blog/woce-5">Part 5: Code finder</a></li>
1034<li><a href="/blog/woce-6">Part 6: Pointers</a></li>
1035</ul>
1036<p>In part 3 we did a fair amount of plumbing in order to support scan modes beyond the trivial &quot;exact value scan&quot;. As a result, we have abstracted away the <code>Scan</code>, <code>CandidateLocations</code> and <code>Value</code> types as a separate <code>enum</code> each. Scanning for changed memory regions in an opened process can now be achieved with three lines of code:</p>
1037<pre><code class="language-rust" data-lang="rust">let regions = process.memory_regions();
1038let first_scan = process.scan_regions(&amp;regions, Scan::InRange(0, 500));
1039let second_scan = process.rescan_regions(&amp;first_scan, Scan::DecreasedBy(7));
1040</code></pre>
1041<p>How's that for programmability? No need to fire up Cheat Engine's GUI anymore!</p>
1042<p>The <code>first_scan</code> in the example above remembers all the found <code>Value</code> within the range specified by <code>Scan</code>. Up until now, we have only worked with <code>i32</code>, so that's the type the scans expect and what they work with.</p>
1043<p>Now it's time to introduce support for different types, like <code>f32</code>, <code>i64</code>, or even more atypical ones, like arbitrary sequences of bytes (think of strings) or even numbers in big-endian.</p>
1044<p>Tighten your belt, because this post is quite the ride. Let's get right into it!</p>
1045<h2 id="floating-points">Floating points</h2>
1046<details open><summary>Cheat Engine Tutorial: Step 4</summary>
1047<blockquote>
1048<p>In the previous tutorial we used bytes to scan, but some games store information in so called 'floating point' notations.
1049(probably to prevent simple memory scanners from finding it the easy way). A floating point is a value with some digits behind the point. (like 5.12 or 11321.1)</p>
1050<p>Below you see your health and ammo. Both are stored as Floating point notations, but health is stored as a float and ammo is stored as a double.
1051Click on hit me to lose some health, and on shoot to decrease your ammo with 0.5</p>
1052<p>You have to set BOTH values to 5000 or higher to proceed.</p>
1053<p>Exact value scan will work fine here, but you may want to experiment with other types too.</p>
1054<p>Hint: It is recommended to disable &quot;Fast Scan&quot; for type double</p>
1055</blockquote>
1056</details>
1057<h2 id="generic-values">Generic values</h2>
1058<p>The <code>Value</code> enumeration holds scanned values, and is currently hardcoded to store <code>i32</code>. The <code>Scan</code> type also holds a value, the value we want to scan for. Changing it to support other types is trivial:</p>
1059<pre><code class="language-rust" data-lang="rust">pub enum Scan&lt;T&gt; {
1060 Exact(T),
1061 Unknown,
1062 Decreased,
1063 // ...other variants...
1064}
1065
1066pub enum Value&lt;T&gt; {
1067 Exact(T),
1068 AnyWithin(Vec&lt;u8&gt;),
1069}
1070</code></pre>
1071<p><code>AnyWithin</code> is the raw memory, and <code>T</code> can be interpreted from any sequence of bytes thanks to our friend <a href="https://doc.rust-lang.org/stable/std/mem/fn.transmute.html"><code>mem::transmute</code></a>. This change alone is enough to store an arbitrary <code>T</code>! So we're done now? Not really, no.</p>
1072<p>First of all, we need to update all the places where <code>Scan</code> or <code>Value</code> are used. Our first stop is the scanned <code>Region</code>, which holds the found <code>Value</code>:</p>
1073<pre><code class="language-rust" data-lang="rust">pub struct Region&lt;T&gt; {
1074 pub info: MEMORY_BASIC_INFORMATION,
1075 pub locations: CandidateLocations,
1076 pub value: Value&lt;T&gt;,
1077}
1078</code></pre>
1079<p>Then, we need to update everywhere <code>Region</code> is used, and on and on… All in all this process is just repeating <code>cargo check</code>, letting the compiler vent on you, and taking good care of it by fixing the errors. It's quite reassuring to know you will not miss a single place. Thank you, compiler!</p>
1080<p>But wait, how could scanning for a decreased value work for any <code>T</code>? The type is not <code>Ord</code>, we should add some trait bounds. And also, what happens if the type is not <code>Copy</code>? It could implement <code>Drop</code><sup class="footnote-reference"><a href="#1">1</a></sup>, and we will be transmuting from raw bytes, which would trigger the <code>Drop</code> implementation when we're done with the value! Not memory safe at all! And how could we possibly cast raw memory to the type without knowing its siz– oh nevermind, <a href="https://doc.rust-lang.org/stable/std/marker/trait.Sized.html"><code>T</code> is already <code>Sized</code> by default</a>. But anyway, we need the other bounds.</p>
1081<p>In order to not repeat ourselves, we will implement a new <code>trait</code>, let's say <code>Scannable</code>, which requires all other bounds:</p>
1082<pre><code class="language-rust" data-lang="rust">pub trait Scannable: Copy + PartialEq + PartialOrd {}
1083
1084impl&lt;T: Copy + PartialEq + PartialOrd&gt; Scannable for T {}
1085</code></pre>
1086<p>And fix our definitions:</p>
1087<pre><code class="language-rust" data-lang="rust">pub enum Scan&lt;T: Scannable&gt; { ... }
1088pub enum Value&lt;T: Scannable&gt; { ... }
1089pub struct Region&lt;T: Scannable&gt; { ... }
1090
1091// ...and the many other places referring to T
1092</code></pre>
1093<p>Every type which is <code>Copy</code>, <code>PartialEq</code> and <code>PartialOrd</code> can be scanned over<sup class="footnote-reference"><a href="#2">2</a></sup>, because we <code>impl Scan for T</code> where the bounds are met. Unfortunately, we cannot require <code>Eq</code> or <code>Ord</code> because the floating point types do not implement it.</p>
1094<h2 id="transmuting-memory">Transmuting memory</h2>
1095<p>Also known as reinterpreting a bunch of bytes as something else, or perhaps it stands for &quot;summoning the demon&quot;:</p>
1096<blockquote>
1097<p><code>transmute</code> is <strong>incredibly</strong> unsafe. There are a vast number of ways to cause <a href="https://doc.rust-lang.org/stable/reference/behavior-considered-undefined.html">undefined behavior</a> with this function. <code>transmute</code> should be the absolute last resort.</p>
1098</blockquote>
1099<p>Types like <code>i32</code> define methods such as <a href="https://doc.rust-lang.org/stable/std/primitive.i32.html#method.from_ne_bytes"><code>from_ne_bytes</code></a> and <a href="https://doc.rust-lang.org/stable/std/primitive.i32.html#method.to_ne_bytes"><code>to_ne_bytes</code></a> which convert raw bytes from and into its native representation. This is all really nice, but unfortunately, there's no standard trait in the Rust's standard library to &quot;interpret a type <code>T</code> as the byte sequence of its native representation&quot;. <code>transmute</code>, however, does exist, and similar to any other <code>unsafe</code> function, it's safe to call <strong>as long as we respect its invariants</strong>. What are these invariants<sup class="footnote-reference"><a href="#3">3</a></sup>?</p>
1100<blockquote>
1101<p>Both types must have the same size</p>
1102</blockquote>
1103<p>Okay, we can just assert that the window length matches the type's length. What else?</p>
1104<blockquote>
1105<p>Neither the original, nor the result, may be an <a href="https://doc.rust-lang.org/nomicon/what-unsafe-does.html">invalid value</a>.</p>
1106</blockquote>
1107<p>What's an invalid value?</p>
1108<blockquote>
1109<ul>
1110<li>a <code>bool</code> that isn't 0 or 1</li>
1111<li>an <code>enum</code> with an invalid discriminant</li>
1112<li>a null <code>fn</code> pointer</li>
1113<li>a <code>char</code> outside the ranges [0x0, 0xD7FF] and [0xE000, 0x10FFFF]</li>
1114<li>a <code>!</code> (all values are invalid for this type)</li>
1115<li>an integer (<code>i*</code>/<code>u*</code>), floating point value (<code>f*</code>), or raw pointer read from uninitialized memory, or uninitialized memory in a <code>str</code>.</li>
1116<li>a reference/<code>Box</code> that is dangling, unaligned, or points to an invalid value.</li>
1117<li>a wide reference, <code>Box</code>, or raw pointer that has invalid metadata:
1118<ul>
1119<li><code>dyn Trait</code> metadata is invalid if it is not a pointer to a vtable for <code>Trait</code> that matches the actual dynamic trait the pointer or reference points to</li>
1120<li>slice metadata is invalid if the length is not a valid <code>usize</code> (i.e., it must not be read from uninitialized memory)</li>
1121</ul>
1122</li>
1123<li>a type with custom invalid values that is one of those values, such as a <code>NonNull</code> that is null. (Requesting custom invalid values is an unstable feature, but some stable libstd types, like <code>NonNull</code>, make use of it.)</li>
1124</ul>
1125</blockquote>
1126<p>Okay, that's actually an awful lot. Types like <code>bool</code> implement all the trait bounds we defined, and it would be insta-UB to ever try to cast them from arbitrary bytes. The same goes for <code>char</code>, and all <code>enum</code> are out of our control, too. At least we're safe on the &quot;memory is initialized&quot; front.</p>
1127<p>Dang it, I really wanted to use <code>transmute</code>! But if we were to use it for arbitrary types, it would trigger undefined behaviour sooner than later.</p>
1128<p>We have several options here:</p>
1129<ul>
1130<li>Make it an <code>unsafe trait</code>. Implementors will be responsible for ensuring that the type they're implementing it for can be safely transmuted from and into.</li>
1131<li><a href="https://rust-lang.github.io/api-guidelines/future-proofing.html">Seal the <code>trait</code></a> and implement it only for types we know are safe<sup class="footnote-reference"><a href="#4">4</a></sup>, like <code>i32</code>.</li>
1132<li>Add methods to the <code>trait</code> definition that do the conversion of the type into its native representation.</li>
1133</ul>
1134<p>We will go with the first option<sup class="footnote-reference"><a href="#5">5</a></sup>, because I really want to use <code>transmute</code>, and I want users to be able to implement the trait on their own types.</p>
1135<p>In any case, we need to change our <code>impl</code> to something more specific, in order to prevent it from automatically implementing the trait for types for which their memory representation has invalid values. So we get rid of this:</p>
1136<pre><code class="language-rust" data-lang="rust">pub trait Scannable: Copy + PartialEq + PartialOrd {}
1137
1138impl&lt;T: Copy + PartialEq + PartialOrd&gt; Scannable for T {}
1139</code></pre>
1140<p>And replace it with this:</p>
1141<pre><code class="language-rust" data-lang="rust">pub unsafe trait Scannable: Copy + PartialEq + PartialOrd {}
1142
1143macro_rules! impl_many {
1144 ( unsafe impl $trait:tt for $( $ty:ty ),* ) =&gt; {
1145 $( unsafe impl $trait for $ty {} )*
1146 };
1147}
1148
1149// SAFETY: all these types respect `Scannable` invariants.
1150impl_many!(unsafe impl Scannable for i8, u8, i16, u16, i32, u32, i64, u64, f32, f64);
1151</code></pre>
1152<p>Making a small macro for things like these is super useful. You could of course write <code>unsafe impl Scannable for T</code> for all ten <code>T</code> as well, but that introduces even more <code>unsafe</code> to read. Last but not least, let's replace the hardcoded <code>i32::from_ne_bytes</code> and <code>i32::to_ne_bytes</code> with <code>mem::transmute</code>.</p>
1153<p>All the <code>windows(4)</code> need to be replaced with <code>windows(mem::size_of::&lt;T&gt;())</code> because the size may no longer be <code>4</code>. All the <code>i32::from_ne_bytes(...)</code> need to be replaced with <code>mem::transmute::&lt;_, T&gt;(...)</code>. We explicitly write out <code>T</code> to make sure the compiler doesn't accidentally infer something we didn't intend.</p>
1154<p>And… it doesn't work at all. We're working with byte slices of arbitrary length. We cannot transmute a <code>&amp;[]</code> type, which is 16 bytes (8 for the pointer and 8 for the length), to our <code>T</code>. My plan to use transmute can't possibly work here. Sigh.</p>
1155<h2 id="not-quite-transmuting-memory">Not quite transmuting memory</h2>
1156<p>Okay, we can't transmute, because we don't have a sized value, we only have a slice of bytes pointing somewhere else. What we <em>could</em> do is reinterpret the pointer to those bytes as a different type, and then dereference it! This is still a form of &quot;transmutation&quot;, just without using <code>transmute</code>.</p>
1157<pre><code class="language-rust" data-lang="rust">let value = unsafe { *(window.as_ptr() as *const T) };
1158</code></pre>
1159<p>Woop! You can compile this and test it out on the step 2 and 3 of the tutorial, using <code>i32</code>, and it will still work! Something troubles me, though. Can you see what it is?</p>
1160<p>When we talked about invalid values, it had a note about unaligned references:</p>
1161<blockquote>
1162<p>a reference/<code>Box</code> that is dangling, unaligned, or points to an invalid value.</p>
1163</blockquote>
1164<p>Our <code>window</code> is essentially a reference to <code>T</code>. The only difference is we're working at the pointer level, but they're pretty much references. Let's see what the documentation for <a href="https://doc.rust-lang.org/std/primitive.pointer.html"><code>pointer</code></a> has to say as well, since we're dereferencing pointers:</p>
1165<blockquote>
1166<p>when a raw pointer is dereferenced (using the <code>*</code> operator), it must be non-null and aligned.</p>
1167</blockquote>
1168<p>It must be aligned. The only reason why our data is aligned is because we are also performing a &quot;fast scan&quot;, so we only look at aligned locations. This is a time bomb waiting to blow up. Is there any other way to <a href="https://doc.rust-lang.org/std/ptr/fn.read.html"><code>read</code></a> from a pointer which is safer?</p>
1169<blockquote>
1170<p><code>src</code> must be properly aligned. Use <a href="https://doc.rust-lang.org/std/ptr/fn.read_unaligned.html"><code>read_unaligned</code></a> if this is not the case.</p>
1171</blockquote>
1172<p>Bingo! Both <code>read</code> and <code>read_unaligned</code>, unlike dereferencing the pointer, will perform a copy, but if it can make the code less prone to blowing up, I'll take it<sup class="footnote-reference"><a href="#6">6</a></sup>. Let's change the code one more time:</p>
1173<pre><code class="language-rust" data-lang="rust">let current = unsafe { window.as_ptr().cast::&lt;T&gt;().read_unaligned() };
1174</code></pre>
1175<p>I prefer to avoid type annotations in variables where possible, which is why I use the <a href="https://www.reddit.com/r/rust/comments/3fimgp/why_double_colon_rather_that_dot/ctozkd0/">turbofish</a> so often. You can get rid of the cast and use a type annotation instead, but make sure the type is known, otherwise it will think it's <code>u8</code> because <code>window</code> is a <code>&amp;[u8]</code>.</p>
1176<p>Now, this is all cool and good. You can replace <code>i32</code> with <code>f32</code> for <code>T</code> and you'll be able to get halfway done with the step 4 of Cheat Engine's tutorial. Unfortunately, as it is, this code is not enough to complete step 4 with exact scans<sup class="footnote-reference"><a href="#7">7</a></sup>. You see, comparing floating point values is not as simple as checking for bitwise equality. We were actually really lucky that the <code>f32</code> part works! But the values in the <code>f64</code> part are not as precise as our inputs, so our exact scan fails.</p>
1177<p>Using a fixed type parameter is pretty limiting as well. On the one hand, it is nice that, if you scan for <code>i32</code>, the compiler statically guarantees that subsequent scans will also happen on <code>i32</code> and thus be compatible. On the other, this requires us to know the type at compile time, which for an interactive program, is not possible. While we <em>could</em> create different methods for each supported type and, at runtime, decide to which we should jump, I am not satisfied with that solution. It also means we can't switch from scanning an <code>u32</code> to an <code>i32</code>, for whatever reason.</p>
1178<p>So we need to work around this once more.</p>
1179<h2 id="rethinking-the-scans">Rethinking the scans</h2>
1180<p>What does our scanning function need, really? It needs a way to compare two chunks of memory as being equal or not (as we have seen, this isn't trivial with types such as floating point numbers) and, for other types of scans, it needs to be able to produce an ordering, or calculate a difference.</p>
1181<p>Instead of having a our trait require the bounds <code>PartialEq</code> and <code>PartialOrd</code>, we can define our own methods to compare <code>Self</code> with <code>&amp;[u8]</code>. It still should be <code>Clone</code>, so we can pass it around without worrying about lifetimes:</p>
1182<pre><code class="language-rust" data-lang="rust">// Callers must `assert_eq!(memory.len(), mem::size_of::&lt;Self&gt;())`.
1183unsafe fn eq(&amp;self, memory: &amp;[u8]) -&gt; bool;
1184unsafe fn cmp(&amp;self, memory: &amp;[u8]) -&gt; Ordering;
1185</code></pre>
1186<p>This can be trivially implemented for all integer types:</p>
1187<pre><code class="language-rust" data-lang="rust">macro_rules! impl_scannable_for_int {
1188 ( $( $ty:ty ),* ) =&gt; {
1189 $(
1190 // SAFETY: caller is responsible to `assert_eq!(memory.len(), mem::size_of::&lt;T&gt;())`
1191 impl Scannable for $ty {
1192 unsafe fn eq(&amp;self, memory: &amp;[u8]) -&gt; bool {
1193 let other = unsafe { memory.as_ptr().cast::&lt;$ty&gt;().read_unaligned() };
1194 *self == other
1195 }
1196
1197 unsafe fn cmp(&amp;self, memory: &amp;[u8]) -&gt; Ordering {
1198 let other = unsafe { memory.as_ptr().cast::&lt;$ty&gt;().read_unaligned() };
1199 &lt;$ty as Ord&gt;::cmp(self, &amp;other)
1200 }
1201 }
1202 )*
1203 };
1204}
1205
1206impl_scannable_for_int!(i8, u8, i16, u16, i32, u32, i64, u64);
1207</code></pre>
1208<p>The funny <code>&lt;$ty as Ord&gt;</code> is because I decided to call the method <code>Scannable::cmp</code>, so I have to disambiguate between it and <code>Ord::cmp</code>. We can go ahead and update the code using <code>Scannable</code> to use these new functions instead.</p>
1209<p>Now, you may have noticed I only implemented it for the integer types. That's because floats need some extra care. Unfortunately, floating point types do not have any form of &quot;precision&quot; embedded in them, so we can't accurately say &quot;compare these floats to the precision level the user specified&quot;. What we can do, however, is drop a few bits from the mantissa, so &quot;relatively close&quot; quantities are considered equal. It's definitely not as good as comparing floats to the user's precision, but it will get the job done.</p>
1210<p>I'm going to arbitrarily say that we are okay comparing with &quot;half&quot; the precision. We can achieve that by masking half of the bits from the mantissa to zero:</p>
1211<pre><code class="language-rust" data-lang="rust">
1212macro_rules! impl_scannable_for_float {
1213 ( $( $ty:ty : $int_ty:ty ),* ) =&gt; {
1214 $(
1215 #[allow(unused_unsafe)] // mind you, it is necessary
1216 impl Scannable for $ty {
1217 unsafe fn eq(&amp;self, memory: &amp;[u8]) -&gt; bool {
1218 const MASK: $int_ty = !((1 &lt;&lt; (&lt;$ty&gt;::MANTISSA_DIGITS / 2)) - 1);
1219
1220 // SAFETY: caller is responsible to `assert_eq!(memory.len(), mem::size_of::&lt;T&gt;())`
1221 let other = unsafe { memory.as_ptr().cast::&lt;$ty&gt;().read_unaligned() };
1222 let left = &lt;$ty&gt;::from_bits(self.to_bits() &amp; MASK);
1223 let right = &lt;$ty&gt;::from_bits(other.to_bits() &amp; MASK);
1224 left == right
1225 }
1226
1227 ...
1228 }
1229 )*
1230 };
1231}
1232
1233impl_scannable_for_float!(f32: u32, f64: u64);
1234</code></pre>
1235<p>You may be wondering what's up with that weird <code>MASK</code>. Let's visualize it with a <a href="https://en.wikipedia.org/wiki/Bfloat16_floating-point_format"><code>f16</code></a>. This type has 16 bits, 1 for sign, 5 for exponent, and 10 for the mantissa:</p>
1236<pre><code>S EEEEE MMMMMMMMMM
1237</code></pre>
1238<p>If we substitute the constant with the numeric value and operate:</p>
1239<pre><code class="language-rust" data-lang="rust">!((1 &lt;&lt; (10 / 2)) - 1)
1240!((1 &lt;&lt; 5) - 1)
1241!(0b00000000_00100000 - 1)
1242!(0b00000000_00011111)
12430b11111111_11100000
1244</code></pre>
1245<p>So effectively, half of the mantisssa bit will be masked to 0. For the <code>f16</code> example, this makes us lose 5 bits of precision. Comparing two floating point values with their last five bits truncated is equivalent to checking if they are &quot;roughly equal&quot;!</p>
1246<p>When Cheat Engine scans for floating point values, several additional settings show, and one such option is &quot;truncated&quot;. I do not know if it behaves like this, but it might.</p>
1247<p>Let's try this out:</p>
1248<pre><code class="language-rust" data-lang="rust">#[test]
1249fn f32_roughly_eq() {
1250 let left = 0.25f32;
1251 let right = 0.25000123f32;
1252 let memory = unsafe { mem::transmute::&lt;_, [u8; 4]&gt;(right) };
1253 assert_ne!(left, right);
1254 assert!(unsafe { Scannable::eq(&amp;left, &amp;memory) });
1255}
1256</code></pre>
1257<pre><code>&gt;cargo test f32_roughly_eq
1258
1259running 1 test
1260test scan::candidate_location_tests::f32_roughly_eq ... ok
1261</code></pre>
1262<p>Huzzah! The <code>assert_ne!</code> makes sure that a normal comparision would fail, and then we <code>assert!</code> that our custom one passes the test. When the user performs an exact scan, the code will be more tolerant to the user's less precise inputs, which overall should result in a nicer experience.</p>
1263<h2 id="dynamically-sized-scans">Dynamically sized scans</h2>
1264<p>The second problem we need to solve is the possibility of the size not being known at compile time<sup class="footnote-reference"><a href="#8">8</a></sup>. While we can go as far as scanning over strings of a known length, this is rather limiting, because we need to know the length at compile time<sup class="footnote-reference"><a href="#9">9</a></sup>. Heap allocated objects are another problem, because we don't want to compare the memory representation of the stack object, but likely the memory where they point to (such as <code>String</code>).</p>
1265<p>Instead of using <code>mem::size_of</code>, we can add a new method to our <code>Scannable</code>, <code>size</code>, which will tell us the size required of the memory view we're comparing against:</p>
1266<pre><code class="language-rust" data-lang="rust">unsafe impl Scannable {
1267 ...
1268
1269 fn size(&amp;self) -&gt; usize;
1270}
1271</code></pre>
1272<p>It is <code>unsafe</code> to implement, because we are relying on the returned value to be truthful and unchanging. It should be safe to call, because it cannot have any invariants. Unfortunately, signaling &quot;unsafe to implement&quot; is done by marking the entire trait as <code>unsafe</code>, since &quot;unsafe to call&quot; is reserved for <code>unsafe fn</code>, and even though the rest of methods are not necessarily unsafe to implement, they're treated as such.</p>
1273<p>At the moment, <code>Scannable</code> cannot be made into a trait object because it is <a href="https://doc.rust-lang.org/stable/error-index.html#E0038">not object safe</a>. This is caused by the <code>Clone</code> requirement on all <code>Scannable</code> object, which in turn needs the types to be <code>Sized</code> because <code>clone</code> returns <code>Self</code>. Because of this, the size must be known.</p>
1274<p>However, we <em>can</em> move the <code>Clone</code> requirement to the methods that need it! This way, <code>Scannable</code> can remain object safe, enabling us to do the following:</p>
1275<pre><code class="language-rust" data-lang="rust">unsafe impl&lt;T: AsRef&lt;dyn Scannable&gt; + AsMut&lt;dyn Scannable&gt;&gt; Scannable for T {
1276 unsafe fn eq(&amp;self, memory: &amp;[u8]) -&gt; bool {
1277 self.as_ref().eq(memory)
1278 }
1279
1280 unsafe fn cmp(&amp;self, memory: &amp;[u8]) -&gt; Ordering {
1281 self.as_ref().cmp(memory)
1282 }
1283
1284 fn mem_view(&amp;self) -&gt; &amp;[u8] {
1285 self.as_ref().mem_view()
1286 }
1287
1288 fn size(&amp;self) -&gt; usize {
1289 self.as_ref().size()
1290 }
1291}
1292</code></pre>
1293<p>Any type which can be interpreted as a reference to <code>Scannable</code> is also a scannable! This enables us to perform scans over <code>Box&lt;dyn i32&gt;</code>, where the type is known at runtime! Or rather, it would, if <code>Box&lt;dyn T&gt;</code> implemented <code>Clone</code>, which it can't<sup class="footnote-reference"><a href="#10">10</a></sup> because that's what prompted this entire issue. Dang it! I can't catch a breath today!</p>
1294<p>Okay, let's step back. Why did we need our scannables to be clone in the first place? When we perform exact scans, we store the original value in the region, which we don't own, so we clone it. But what if we <em>did</em> own the value? Instead of taking the <code>Scan</code> by reference, which holds <code>T: Scannable</code>, we could take it by value. If we get rid of all the <code>Clone</code> bounds and update <code>Scan::run</code> to take <code>self</code>, along with updating all the things that take a <code>Region</code> to take them by value as well, it should all work out.</p>
1295<p>But it does not. If we take <code>Scan</code> by value, with it not being <code>Clone</code>, we simply can't use it to scan over multiple regions. After the first region, we have lost the <code>Scan</code>.</p>
1296<p>Let's take a second step back. We are scanning memory, and we want to compare memory, but we want to treat the memory with different semantics (for example, if we treat it as <code>f32</code>, we want to check for rough equality). Instead of storing the <em>value</em> itself, we could store its <em>memory representation</em>, and when we compare memory representations, we can do so under certain semantics.</p>
1297<p>First off, let's revert getting rid of all <code>Clone</code>. Wherever we stored a <code>T</code>, we will now store a <code>Vec&lt;u8&gt;</code>. We will still use a type parameter to represent the &quot;implementations of <code>Scannable</code>&quot;. For this to work, our definitions need to use <code>T</code> somewhere, or else the compiler refuses to compile the code with error <a href="https://doc.rust-lang.org/stable/error-index.html#E0392">E0392</a>. For this, I will stick a <a href="https://doc.rust-lang.org/stable/std/marker/struct.PhantomData.html"><code>PhantomData</code></a> in the <code>Exact</code> variant. It's a bit pointless to include it in all variants, and <code>Exact</code> seems the most appropriated:</p>
1298<pre><code class="language-rust" data-lang="rust">pub enum Scan&lt;T: Scannable&gt; {
1299 Exact(Vec&lt;u8&gt;, PhantomData&lt;T&gt;),
1300 Unknown,
1301 ...
1302}
1303</code></pre>
1304<p>This keeps in line with <code>Value</code>:</p>
1305<pre><code class="language-rust" data-lang="rust">pub enum Value&lt;T: Scannable&gt; {
1306 Exact(Vec&lt;u8&gt;, PhantomData&lt;T&gt;),
1307 ...
1308}
1309</code></pre>
1310<p>Our <code>Scannable</code> will no longer work on <code>T</code> and <code>&amp;[u8]</code>. Instead, it will work on two <code>&amp;[u8]</code>. We will also need a way to interpret a <code>T</code> as <code>&amp;[u8]</code>, which we can achieve with a new method, <code>mem_view</code>. This method interprets the raw memory representation of <code>self</code> as its raw bytes. It also lets us get rid of <code>size</code>, because we can simply do <code>mem_view().len()</code>. It's still <code>unsafe</code> to implement, because it should return the same length every time:</p>
1311<pre><code class="language-rust" data-lang="rust">pub unsafe trait Scannable {
1312 // Callers must `assert_eq!(left.len(), right.len(), self.mem_view().len())`.
1313 unsafe fn eq(left: &amp;[u8], right: &amp;[u8]) -&gt; bool;
1314 unsafe fn cmp(left: &amp;[u8], right: &amp;[u8]) -&gt; Ordering;
1315 fn mem_view(&amp;self) -&gt; &amp;[u8];
1316}
1317</code></pre>
1318<p>But now we can't use it in trait object, so the following no longer works:</p>
1319<pre><code class="language-rust" data-lang="rust">unsafe impl&lt;T: AsRef&lt;dyn Scannable&gt; + AsMut&lt;dyn Scannable&gt;&gt; Scannable for T {
1320 ...
1321}
1322</code></pre>
1323<p>Ugh! Well, to be fair, we no longer have a &quot;scannable&quot; at this point. It's more like a scan mode that tells us how memory should be compared according to a certain type. Let's split the trait into two: one for the scan mode, and other for &quot;things which are scannable&quot;:</p>
1324<pre><code class="language-rust" data-lang="rust">pub trait ScanMode {
1325 unsafe fn eq(left: &amp;[u8], right: &amp;[u8]) -&gt; bool;
1326 unsafe fn cmp(left: &amp;[u8], right: &amp;[u8]) -&gt; Ordering;
1327}
1328
1329pub unsafe trait Scannable {
1330 type Mode: ScanMode;
1331
1332 fn mem_view(&amp;self) -&gt; &amp;[u8];
1333}
1334</code></pre>
1335<p>Note that we have an associated <code>type Mode</code> which contains the corresponding <code>ScanMode</code>. If we used a trait bound such as <code>Scannable: ScanMode</code>, we'd be back to square one: it would inherit the method definitions that don't use <code>&amp;self</code> and thus cannot be used as trait objects.</p>
1336<p>With these changes, it is possible to implement <code>Scannable</code> for any <code>dyn Scannable</code>:</p>
1337<pre><code class="language-rust" data-lang="rust">unsafe impl&lt;T: ScanMode + AsRef&lt;dyn Scannable&lt;Mode = Self&gt;&gt;&gt; Scannable for T {
1338 type Mode = Self;
1339
1340 fn mem_view(&amp;self) -&gt; &amp;[u8] {
1341 self.as_ref().mem_view()
1342 }
1343}
1344</code></pre>
1345<p>We do have to adjust a few places of the code to account for both <code>Scannable</code> and <code>ScanMode</code>, but all in all, it's pretty straightforward. Things like <code>Value</code> don't need to store the <code>Scannable</code> anymore, just a <code>Vec&lt;u8&gt;</code>. It also doesn't need the <code>ScanMode</code>, because it's not going to be scanning anything on its own. This applies transitively to <code>Region</code> which was holding a <code>Value</code>.</p>
1346<p><code>Value</code> <em>does</em> need to be updated to store the size of the region we are scanning for, however, because we need that information when running a subsequent scan. For all <code>Scan</code> that don't have a explicit thing to scan for (like <code>Decreased</code>), the <code>size</code> also needs to be stored in them.</p>
1347<p>Despite all our efforts, we're still unable to return an <code>Scannable</code> chosen at runtime.</p>
1348<pre><code class="language-rust" data-lang="rust">fn prompt_user_for_scan() -&gt; Scan&lt;Box&lt;dyn Scannable&lt;Mode = ???&gt;&gt;&gt; {
1349 todo!()
1350}
1351</code></pre>
1352<p>As far as I can tell, there's simply no way to specify that type. We want to return a type which is scannable, which has itself (which is also a <code>ScanMode</code>) as the corresponding mode. Even if we just tried to return the mode, we simply can't, because it's not object-safe. Is this the end of the road?</p>
1353<h2 id="specifying-the-scan-mode">Specifying the scan mode</h2>
1354<p>We need a way to pass an arbitrary scan mode to our <code>Scan</code>. This scan mode should go in tandem with <code>Scannable</code> types, because it would be unsafe otherwise. We've seen that using a type just doesn't cut it. What else can we do?</p>
1355<p>Using an enumeration is a no-go, because I want users to be able to extend it further. I also would like to avoid having to update the <code>enum</code> and all the matches every time I come up with a different type combination. And it could get pretty complicated if I ever built something dynamically, such as letting the user combine different scans in one pass.</p>
1356<p>So what if we make <code>Scannable</code> return a value that implements the functions we need?</p>
1357<pre><code class="language-rust" data-lang="rust">pub struct ScanMode {
1358 eq: unsafe fn(left: &amp;[u8], right: &amp;[u8]) -&gt; bool,
1359 cmp: unsafe fn(left: &amp;[u8], right: &amp;[u8]) -&gt; Ordering,
1360}
1361</code></pre>
1362<p>It's definitely… non-conventional. But hey, now we're left with the <code>Scannable</code> trait, which is object-safe, and does not have any type parameters!</p>
1363<pre><code class="language-rust" data-lang="rust">pub unsafe trait Scannable {
1364 fn mem_view(&amp;self) -&gt; &amp;[u8];
1365 fn scan_mode(&amp;self) -&gt; ScanMode;
1366}
1367</code></pre>
1368<p>It is a bit weird, but defining local functions and using those in the returned value is a nice way to keep things properly scoped:</p>
1369<pre><code class="language-rust" data-lang="rust">macro_rules! impl_scannable_for_int {
1370 ( $( $ty:ty ),* ) =&gt; {
1371 $(
1372 unsafe impl Scannable for $ty {
1373 fn mem_view(&amp;self) -&gt; &amp;[u8] {
1374 unsafe { std::slice::from_raw_parts(self as *const _ as *const u8, mem::size_of::&lt;$ty&gt;()) }
1375 }
1376
1377 fn scan_mode(&amp;self) -&gt; ScanMode {
1378 unsafe fn eq(left: &amp;[u8], right: &amp;[u8]) -&gt; bool {
1379 ...
1380 }
1381
1382 unsafe fn cmp(left: &amp;[u8], right: &amp;[u8]) -&gt; Ordering {
1383 ...
1384 }
1385
1386 ScanMode { eq, cmp }
1387 }
1388 }
1389 )*
1390 };
1391}
1392</code></pre>
1393<p>Our <code>Scan</code> needs to store the <code>Scannable</code> type, and not just the memory, once again. For variants that don't need any value, they can store the <code>ScanMode</code> and size instead.</p>
1394<p>Does this solution work? Yes! It's possible to return a <code>Box&lt;dyn Scannable&gt;</code> from a function, and underneath, it may be using any type which is <code>Scannable</code>. Is this the best solution? Well, that's hard to say. This is <em>one</em> of the possible solutions.</p>
1395<p>We have been going around in circles for quite some time now, so I'll leave it there. It's a solution, which may not be pretty, but it works. With these changes, the code is capable of completing all of the steps in the Cheat Engine tutorial up until point!</p>
1396<h2 id="finale">Finale</h2>
1397<p>If there's one lesson to learn from this post, it's that there is often no single correct solution to a problem. We could have approached the scan types in many, many ways (and we tried quite a few!), but in the end, choosing one option or the other comes down to your (sometimes self-imposed) requirements.</p>
1398<p>You may <a href="https://github.com/lonami/memo">obtain the code for this post</a> over at my GitHub. You can run <code>git checkout step4</code> after cloning the repository to get the right version of the code. The code has gone through a lot of iterations, and I'd still like to polish it a bit more, so it might slightly differ from the code presented in this entry.</p>
1399<p>If you feel adventurous, Cheat Engine has different options for scanning floating point types: &quot;rounded (default)&quot;, &quot;rounded (extreme)&quot;, and truncated. Optionally, it can scan for &quot;simple values only&quot;. You could go ahead and toy around with these!</p>
1400<p>We didn't touch on types with different lengths, such as strings. You could support UTF-8, UTF-16, or arbitrary byte sequences. This post also didn't cover scanning for multiple things at once, known as &quot;groupscan commands&quot;, although from what I can tell, these are just a nice way to scan for arbitrary byte sequences.</p>
1401<p>We also didn't look into supporting different the same scan with different alignments. All these things may be worth exploring depending on your requirements. You could even get rid of such genericity and go with something way simpler. Supporting <code>i32</code>, <code>f32</code> and <code>f64</code> is enough to complete the Cheat Engine tutorial. But I wanted something more powerful, although my solution currently can't scan for a sequence such as &quot;exact type, unknown, exact matching the unknown&quot;. So yeah.</p>
1402<p>In the <a href="/blog/woce-5">next post</a>, we'll tackle the fifth step of the tutorial: Code finder. Cheat Engine attaches its debugger to the process for this one, and then replaces the instruction that performs the write with a different no-op so that nothing is written anymore. This will be quite the challenge!</p>
1403<h3 id="footnotes">Footnotes</h3>
1404<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
1405<p><a href="https://doc.rust-lang.org/stable/std/ops/trait.Drop.html#copy-and-drop-are-exclusive"><code>Copy</code> and <code>Drop</code> are exclusive</a>. See also <a href="https://doc.rust-lang.org/stable/error-index.html#E0184">E0184</a>.</p>
1406</div>
1407<div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup>
1408<p>If you added more scan types that require additional bounds, make sure to add them too. For example, the &quot;decreased by&quot; scan requires the type to <code>impl Sub</code>.</p>
1409</div>
1410<div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup>
1411<p>This is a good time to remind you to read the documentation. It is of special importance when dealing with <code>unsafe</code> methods; I recommend reading it a couple times.</p>
1412</div>
1413<div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup>
1414<p>Even with this option, it would not be a bad idea to make the trait <code>unsafe</code>.</p>
1415</div>
1416<div class="footnote-definition" id="5"><sup class="footnote-definition-label">5</sup>
1417<p>Not for long. As we will find out later, this approach has its limitations.</p>
1418</div>
1419<div class="footnote-definition" id="6"><sup class="footnote-definition-label">6</sup>
1420<p>We can still perform the pointer dereference when we know it's aligned. This would likely be an optimization, although it would definitely complicate the code more.</p>
1421</div>
1422<div class="footnote-definition" id="7"><sup class="footnote-definition-label">7</sup>
1423<p>It <em>would</em> work if you scanned for unknown values and then checked for decreased values repeatedly. But we can't just leave exact scan broken!</p>
1424</div>
1425<div class="footnote-definition" id="8"><sup class="footnote-definition-label">8</sup>
1426<p>Unfortunately, this makes some optimizations harder or even impossible to perform. Providing specialized functions for types where the size is known at compile time could be worth doing. Programming is all tradeoffs.</p>
1427</div>
1428<div class="footnote-definition" id="9"><sup class="footnote-definition-label">9</sup>
1429<p><a href="https://blog.rust-lang.org/2021/02/26/const-generics-mvp-beta.html">Rust 1.51</a>, which was not out at the time of writing, would make it a lot easier to allow scanning for fixed-length sequences of bytes, thanks to const generics.</p>
1430</div>
1431<div class="footnote-definition" id="10"><sup class="footnote-definition-label">10</sup>
1432<p>Workarounds do exist, such as <a href="https://crates.io/crates/dyn-clone">dtolnay's <code>dyn-clone</code></a>. But I would rather not go that route.</p>
1433</div>
1434</content>
1435 </entry>
1436 <entry xml:lang="en">
1437 <title>Writing our own Cheat Engine: Unknown initial value</title>
1438 <published>2021-02-19T00:00:00+00:00</published>
1439 <updated>2021-02-19T00:00:00+00:00</updated>
1440 <link href="https://lonami.dev/blog/woce-3/" type="text/html"/>
1441 <id>https://lonami.dev/blog/woce-3/</id>
1442 <content type="html"><p>This is part 3 on the <em>Writing our own Cheat Engine</em> series:</p>
1443<ul>
1444<li><a href="/blog/woce-1">Part 1: Introduction</a> (start here if you're new to the series!)</li>
1445<li><a href="/blog/woce-2">Part 2: Exact Value scanning</a></li>
1446<li>Part 3: Unknown initial value</li>
1447<li><a href="/blog/woce-4">Part 4: Floating points</a></li>
1448<li><a href="/blog/woce-5">Part 5: Code finder</a></li>
1449<li><a href="/blog/woce-6">Part 6: Pointers</a></li>
1450</ul>
1451<p>In part 2 we left off with a bit of a cliff-hanger. Our little program is now able to scan for an exact value, remember the couple hundred addresses pointing to said value, and perform subsequent scans to narrow the list of addresses down until we're left with a handful of them.</p>
1452<p>However, it is not always the case that you have an exact value to work with. The best you can do in these cases is guess what the software might be storing. For example, it could be a floating point for your current movement speed in a game, or an integer for your current health.</p>
1453<p>The problem with this is that there are far too many possible locations storing our desired value. If you count misaligned locations, this means there is a different location to address every single byte in memory. A program with one megabyte of memory already has a <em>million</em> of addresses. Clearly, we need to do better than performing one million memory reads<sup class="footnote-reference"><a href="#1">1</a></sup>.</p>
1454<p>This post will shift focus a bit from using <code>winapi</code> to possible techniques to perform the various scans.</p>
1455<h2 id="unknown-initial-value">Unknown initial value</h2>
1456<details open><summary>Cheat Engine Tutorial: Step 3</summary>
1457<blockquote>
1458<p>Ok, seeing that you've figured out how to find a value using exact value let's move on to the next step.</p>
1459<p>First things first though. Since you are doing a new scan, you have to click on New Scan first, to start a new scan. (You may think this is straighforward, but you'd be surprised how many people get stuck on that step) I won't be explaining this step again, so keep this in mind
1460Now that you've started a new scan, let's continue</p>
1461<p>In the previous test we knew the initial value so we could do a exact value, but now we have a status bar where we don't know the starting value.
1462We only know that the value is between 0 and 500. And each time you click 'hit me' you lose some health. The amount you lose each time is shown above the status bar.</p>
1463<p>Again there are several different ways to find the value. (like doing a decreased value by... scan), but I'll only explain the easiest. &quot;Unknown initial value&quot;, and decreased value.
1464Because you don't know the value it is right now, a exact value wont do any good, so choose as scantype 'Unknown initial value', again, the value type is 4-bytes. (most windows apps use 4-bytes)click first scan and wait till it's done.</p>
1465<p>When it is done click 'hit me'. You'll lose some of your health. (the amount you lost shows for a few seconds and then disappears, but you don't need that)
1466Now go to Cheat Engine, and choose 'Decreased Value' and click 'Next Scan'
1467When that scan is done, click hit me again, and repeat the above till you only find a few.</p>
1468<p>We know the value is between 0 and 500, so pick the one that is most likely the address we need, and add it to the list.
1469Now change the health to 5000, to proceed to the next step.</p>
1470</blockquote>
1471</details>
1472<h2 id="dense-memory-locations">Dense memory locations</h2>
1473<p>The key thing to notice here is that, when we read memory from another process, we do so over <em>entire regions</em>. A memory region is represented by a starting offset, a size, and a bunch of other things like protection level.</p>
1474<p>When running the first scan for an unknown value, all we need to remember is the starting offset and size for every single region. All the candidate locations that could point to our value fall within this range, so it is enough for us to store the range definition, and not every location within it.</p>
1475<p>To gain a better understanding of what this means, let's come up with a more specific scenario. With our current approach of doing things, we store an address (<code>usize</code>) for every location pointing to our desired value. In the case of unknown values, all locations are equally valid, since we don't know what value they should point to yet, and any value they point to is good. With this representation, we would end up with a very large vector:</p>
1476<pre><code class="language-rust" data-lang="rust">let locations = vec![0x2000, 0x2001, ..., 0x20ff, 0x2100];
1477</code></pre>
1478<p>This representation is dense. Every single number in the range <code>0x2000..=0x2100</code> is present. So why bother storing the values individually when the range is enough?:</p>
1479<pre><code class="language-rust" data-lang="rust">let locations = EntireRegion { range: 0x2000..=0x2100 };
1480</code></pre>
1481<p>Much better! With two <code>usize</code>, one for the starting location and another for the end, we can indicate that we care about all the locations falling in that range.</p>
1482<p>In fact, some accessible memory regions immediately follow eachother, so we could even compact this further and merge regions which are together. But due to their potential differences with regards to protection levels, we will not attempt to merge regions.</p>
1483<p>We don't want to get rid of the old way of storing locations, because once we start narrowing them down, we will want to go back to storing just a few candidates. To keep things tidy, let's introduce a new <code>enum</code> representing either possibility:</p>
1484<pre><code class="language-rust" data-lang="rust">use std::ops::Range;
1485
1486pub enum CandidateLocations {
1487 Discrete {
1488 locations: Vec&lt;usize&gt;,
1489 },
1490 Dense {
1491 range: Range&lt;usize&gt;,
1492 }
1493}
1494</code></pre>
1495<p>Let's also introduce another <code>enum</code> to perform the different scan types. For the time being, we will only worry about looking for <code>i32</code> in memory:</p>
1496<pre><code class="language-rust" data-lang="rust">pub enum Scan {
1497 Exact(i32),
1498 Unknown,
1499}
1500</code></pre>
1501<h2 id="storing-scanned-values">Storing scanned values</h2>
1502<p>When scanning for exact values, it's not necessary to store the value found. We already know they're all the same, for example, value <code>42</code>. However, if the value is unknown, we do need to store it so that we can compare it in a subsequent scan to see if the value is the same or it changed. This means the value can be &quot;any within&quot; the read memory chunk:</p>
1503<pre><code class="language-rust" data-lang="rust">pub enum Value {
1504 Exact(i32),
1505 AnyWithin(Vec&lt;u8&gt;),
1506}
1507</code></pre>
1508<p>For every region in memory, there will be some candidate locations and a value (or value range) we need to compare against in subsequent scans:</p>
1509<pre><code class="language-rust" data-lang="rust">pub struct Region {
1510 pub info: winapi::um::winnt::MEMORY_BASIC_INFORMATION,
1511 pub locations: CandidateLocations,
1512 pub value: Value,
1513}
1514</code></pre>
1515<p>With all the data structures needed setup, we can finally refactor our old scanning code into a new method capable of dealing with all these cases. For brevity, I will omit the exact scan, as it remains mostly unchanged:</p>
1516<pre><code class="language-rust" data-lang="rust">use winapi::um::winnt::MEMORY_BASIC_INFORMATION;
1517
1518...
1519
1520// inside `impl Process`
1521pub fn scan_regions(&amp;self, regions: &amp;[MEMORY_BASIC_INFORMATION], scan: Scan) -&gt; Vec&lt;Region&gt; {
1522 regions
1523 .iter()
1524 .flat_map(|region| match scan {
1525 Scan::Exact(n) =&gt; todo!(&quot;old scan implementation&quot;),
1526 Scan::Unknown =&gt; {
1527 let base = region.BaseAddress as usize;
1528 match self.read_memory(region.BaseAddress as _, region.RegionSize) {
1529 Ok(memory) =&gt; Some(Region {
1530 info: region.clone(),
1531 locations: CandidateLocations::Dense {
1532 range: base..base + region.RegionSize,
1533 },
1534 value: Value::AnyWithin(memory),
1535 }),
1536 Err(_) =&gt; None,
1537 }
1538 }
1539 })
1540 .collect()
1541}
1542</code></pre>
1543<p>Time to try it out!</p>
1544<pre><code class="language-rust" data-lang="rust">impl CandidateLocations {
1545 pub fn len(&amp;self) -&gt; usize {
1546 match self {
1547 CandidateLocations::Discrete { locations } =&gt; locations.len(),
1548 CandidateLocations::Dense { range } =&gt; range.len(),
1549 }
1550 }
1551}
1552
1553...
1554
1555fn main() {
1556 // -snip-
1557
1558 println!(&quot;Scanning {} memory regions&quot;, regions.len());
1559 let last_scan = process.scan_regions(&amp;regions, Scan::Unknown);
1560 println!(
1561 &quot;Found {} locations&quot;,
1562 last_scan.iter().map(|r| r.locations.len()).sum::&lt;usize&gt;()
1563 );
1564}
1565</code></pre>
1566<pre><code>Scanning 88 memory regions
1567Found 3014656 locations
1568</code></pre>
1569<p>If we consider misaligned locations, there is a lot of potential addresses where we could look for. Running the same scan on Cheat Engine yields <code>2,449,408</code> addresses, which is pretty close. It's probably skipping some additional regions that we are considering. Emulating Cheat Engine to perfection is not a concern for us at the moment, so I'm not going to investigate what regions it actually uses.</p>
1570<h2 id="comparing-scanned-values">Comparing scanned values</h2>
1571<p>Now that we have performed the initial scan and have stored all the <code>CandidateLocations</code> and <code>Value</code>, we can re-implement the &quot;next scan&quot; step to handle any variant of our <code>Scan</code> enum. This enables us to mix-and-match any <code>Scan</code> mode in any order. For example, one could perform an exact scan, then one for decreased values, or start with unknown scan and scan for unchanged values.</p>
1572<p>The tutorial suggests using &quot;decreased value&quot; scan, so let's start with that:</p>
1573<pre><code class="language-rust" data-lang="rust">pub enum Scan {
1574 Exact(i32),
1575 Unknown,
1576 Decreased, // new!
1577}
1578</code></pre>
1579<p>Other scanning modes, such as decreased by a known amount rather than any decrease, increased, unchanged, changed and so on, are not very different from the &quot;decreased&quot; scan, so I won't bore you with the details.</p>
1580<p>I will use a different method to perform a &quot;rescan&quot;, since the first one is a bit more special in that it doesn't start with any previous values:</p>
1581<pre><code class="language-rust" data-lang="rust">pub fn rescan_regions(&amp;self, regions: &amp;[Region], scan: Scan) -&gt; Vec&lt;Region&gt; {
1582 regions
1583 .iter()
1584 .flat_map(|region| match scan {
1585 Scan::Decreased =&gt; {
1586 let mut locations = Vec::new();
1587 match region.locations {
1588 CandidateLocations::Dense { range } =&gt; {
1589 match self.read_memory(range.start, range.end - range.start) {
1590 Ok(memory) =&gt; match region.value {
1591 Value::AnyWithin(previous) =&gt; {
1592 memory
1593 .windows(4)
1594 .zip(previous.windows(4))
1595 .enumerate()
1596 .step_by(4)
1597 .for_each(|(offset, (new, old))| {
1598 let new = i32::from_ne_bytes([
1599 new[0], new[1], new[2], new[3],
1600 ]);
1601 let old = i32::from_ne_bytes([
1602 old[0], old[1], old[2], old[3],
1603 ]);
1604 if new &lt; old {
1605 locations.push(range.start + offset);
1606 }
1607 });
1608
1609 Some(Region {
1610 info: region.info.clone(),
1611 locations: CandidateLocations::Discrete { locations },
1612 value: Value::AnyWithin(memory),
1613 })
1614 }
1615 _ =&gt; todo!(),
1616 },
1617 _ =&gt; todo!(),
1618 }
1619 }
1620 _ =&gt; todo!(),
1621 }
1622 }
1623 _ =&gt; todo!(),
1624 })
1625 .collect()
1626}
1627</code></pre>
1628<p>If you've skimmed over that, I do not blame you. Here's the summary: for every existing region, when executing the scan mode &quot;decreased&quot;, if the previous locations were dense, read the entire memory region. On success, if the previous values were a chunk of memory, iterate over the current and old memory at the same time, and for every aligned <code>i32</code>, if the new value is less, store it.</p>
1629<p>It's also making me ill. Before I leave a mess on the floor, does it work?</p>
1630<pre><code class="language-rust" data-lang="rust">std::thread::sleep(std::time::Duration::from_secs(10));
1631let last_scan = process.rescan_regions(&amp;last_scan, Scan::Decreased);
1632println!(
1633 &quot;Found {} locations&quot;,
1634 last_scan.iter().map(|r| r.locations.len()).sum::&lt;usize&gt;()
1635);
1636</code></pre>
1637<pre><code class="language-rust" data-lang="rust">Found 3014656 locations
1638Found 177 locations
1639</code></pre>
1640<p>Okay, great, let's clean up this mess…</p>
1641<h2 id="refactoring">Refactoring</h2>
1642<p>Does it also make you uncomfortable to be writing something that you know will end up <em>huge</em> unless you begin refactoring other parts right now? I definitely feel that way. But I think it's good discipline to push through with something that works first, even if it's nasty, before going on a tangent. Now that we have the basic implementation working, let's take on this monster before it eats us alive.</p>
1643<p>First things first, that method is inside an <code>impl</code> block. The deepest nesting level is 13. I almost have to turn around my chair to read the entire thing out!</p>
1644<p>Second, we're nesting four matches. Three of them we care about: scan, candidate location, and value. If each of these <code>enum</code> has <code>S</code>, <code>C</code> and <code>V</code> variants respectively, writing each of these by hand will require <code>S * C * V</code> different implementations! Cheat Engine offers 10 different scans, I can think of at least 3 different ways to store candidate locations, and another 3 ways to store the values found. That's <code>10 * 3 * 3 = 90</code> different combinations. I am not willing to write out all these<sup class="footnote-reference"><a href="#2">2</a></sup>, so we need to start introducing some abstractions. Just imagine what a monster function you would end with! The horror!</p>
1645<p>Third, why is the scan being executed in the process? This is something that should be done in the <code>impl Scan</code> instead!</p>
1646<p>Let's begin the cleanup:</p>
1647<pre><code class="language-rust" data-lang="rust">pub fn rescan_regions(&amp;self, regions: &amp;[Region], scan: Scan) -&gt; Vec&lt;Region&gt; {
1648 todo!()
1649}
1650</code></pre>
1651<p>I already feel ten times better.</p>
1652<p>Now, this method will unconditionally read the entire memory region, even if the scan or the previous candidate locations don't need it<sup class="footnote-reference"><a href="#3">3</a></sup>. In the worst case with a single discrete candidate location, we will be reading a very large chunk of memory when we could have read just the 4 bytes needed for the <code>i32</code>. On the bright side, if there <em>are</em> more locations in this memory region, we will get read of them at the same time<sup class="footnote-reference"><a href="#4">4</a></sup>. So even if we're moving more memory around all the time, it isn't <em>too</em> bad.</p>
1653<pre><code class="language-rust" data-lang="rust">regions
1654 .iter()
1655 .flat_map(
1656 |region| match self.read_memory(region.info.BaseAddress as _, region.info.RegionSize) {
1657 Ok(memory) =&gt; todo!(),
1658 Err(err) =&gt; {
1659 eprintln!(
1660 &quot;Failed to read {} bytes at {:?}: {}&quot;,
1661 region.info.RegionSize, region.info.BaseAddress, err,
1662 );
1663 None
1664 }
1665 },
1666 )
1667 .collect()
1668</code></pre>
1669<p>Great! If reading memory succeeds, we want to rerun the scan:</p>
1670<pre><code class="language-rust" data-lang="rust">Ok(memory) =&gt; Some(scan.rerun(region, memory)),
1671</code></pre>
1672<p>The rerun will live inside <code>impl Scan</code>:</p>
1673<pre><code class="language-rust" data-lang="rust">pub fn rerun(&amp;self, region: &amp;Region, memory: Vec&lt;u8&gt;) -&gt; Region {
1674 match self {
1675 Scan::Exact(_) =&gt; self.run(region.info.clone(), memory),
1676 Scan::Unknown =&gt; region.clone(),
1677 Scan::Decreased =&gt; todo!(),
1678 }
1679}
1680</code></pre>
1681<p>An exact scan doesn't care about any previous values, so it behaves like a first scan. The first scan is done by the <code>run</code> function (it contains the implementation factored out of the <code>Process::scan_regions</code> method), which only needs the region information and the current memory chunk we just read.</p>
1682<p>The unknown scan leaves the region unchanged: any value stored is still valid, because it is unknown what we're looking for.</p>
1683<p>The decreased scan will have to iterate over all the candidate locations, and compare them with the current memory chunk. But this time, we'll abstract this iteration too:</p>
1684<pre><code class="language-rust" data-lang="rust">impl Region {
1685 fn iter_locations&lt;'a&gt;(
1686 &amp;'a self,
1687 new_memory: &amp;'a [u8],
1688 ) -&gt; impl Iterator&lt;Item = (usize, i32, i32)&gt; + 'a {
1689 match &amp;self.locations {
1690 CandidateLocations::Dense { range } =&gt; range.clone().step_by(4).map(move |addr| {
1691 let old = self.value_at(addr);
1692 let new = i32::from_ne_bytes([
1693 new_memory[0],
1694 new_memory[1],
1695 new_memory[2],
1696 new_memory[3],
1697 ]);
1698 (addr, old, new)
1699 }),
1700 _ =&gt; todo!(),
1701 }
1702 }
1703}
1704</code></pre>
1705<p>For a dense candidate location, we iterate over all the 4-aligned addresses (fast scan for <code>i32</code> values), and yield <code>(current address, old value, new value)</code>. This way, the <code>Scan</code> can do anything it wants with the old and new values, and if it finds a match, it can use the address.</p>
1706<p>The <code>value_at</code> method will deal with all the <code>Value</code> variants:</p>
1707<pre><code class="language-rust" data-lang="rust">fn value_at(&amp;self, addr: usize) -&gt; i32 {
1708 match &amp;self.value {
1709 Value::AnyWithin(chunk) =&gt; {
1710 let base = addr - self.info.BaseAddress as usize;
1711 let bytes = &amp;chunk[base..base + 4];
1712 i32::from_ne_bytes([bytes[0], bytes[1], bytes[2], bytes[3]])
1713 }
1714 _ =&gt; todo!(),
1715 }
1716}
1717</code></pre>
1718<p>This way, <code>iter_locations</code> can easily use any value type. With this, we have all <code>enum</code> covered: <code>Scan</code> in <code>rerun</code>, <code>CandidateLocation</code> in <code>iter_locations</code>, and <code>Value</code> in <code>value_at</code>. Now we can add as many variants as we want, and we will only need to update a single <code>match</code> arm for each of them. Let's implement <code>Scan::Decreased</code> and try it out:</p>
1719<pre><code class="language-rust" data-lang="rust">pub fn rerun(&amp;self, region: &amp;Region, memory: Vec&lt;u8&gt;) -&gt; Region {
1720 match self {
1721 Scan::Decreased =&gt; Region {
1722 info: region.info.clone(),
1723 locations: CandidateLocations::Discrete {
1724 locations: region
1725 .iter_locations(&amp;memory)
1726 .flat_map(|(addr, old, new)| if new &lt; old { Some(addr) } else { None })
1727 .collect(),
1728 },
1729 value: Value::AnyWithin(memory),
1730 },,
1731 }
1732}
1733</code></pre>
1734<pre><code>Found 3014656 locations
1735Found 223791 locations
1736</code></pre>
1737<p>Hmm… before we went down from <code>3014656</code> to <code>177</code> locations, and now we went down to <code>223791</code>. Where did we go wrong?</p>
1738<p>After spending several hours on this, I can tell you where we went wrong. <code>iter_locations</code> is always accessing the memory range <code>0..4</code>, and not the right address. Here's the fix:</p>
1739<pre><code class="language-rust" data-lang="rust">CandidateLocations::Dense { range } =&gt; range.clone().step_by(4).map(move |addr| {
1740 let old = self.value_at(addr);
1741 let base = addr - self.info.BaseAddress as usize;
1742 let bytes = &amp;new_memory[base..base + 4];
1743 let new = i32::from_ne_bytes([bytes[0], bytes[1], bytes[2], bytes[3]]);
1744 (addr, old, new)
1745}),
1746</code></pre>
1747<h2 id="going-beyond">Going beyond</h2>
1748<p>Let's take a look at other possible <code>Scan</code> types. Cheat Engine supports the following initial scan types:</p>
1749<ul>
1750<li>Exact Value</li>
1751<li>Bigger than…</li>
1752<li>Smaller than…</li>
1753<li>Value between…</li>
1754<li>Unknown initial value</li>
1755</ul>
1756<p>&quot;Bigger than&quot; and &quot;Smaller than&quot; can both be represented by &quot;Value between&quot;, so it's pretty much just three.</p>
1757<p>For subsequent scans, in addition to the scan types described above, we find:</p>
1758<ul>
1759<li>Increased value</li>
1760<li>Increased value by…</li>
1761<li>Decreased value</li>
1762<li>Decreased value by…</li>
1763<li>Changed value</li>
1764<li>Unchanged value</li>
1765</ul>
1766<p>Not only does Cheat Engine provide all of these scans, but all of them can also be negated. For example, &quot;find values that were not increased by 7&quot;. One could imagine to also support things like &quot;increased value by range&quot;. For the increased and decreased scans, Cheat Engine also supports &quot;at least xx%&quot;, so that if the value changed within the specified percentage interval, it will be considered.</p>
1767<p>What about <code>CandidateLocations</code>? I can't tell you how Cheat Engine stores these, but I can tell you that <code>CandidateLocations::Discrete</code> can still be quite inefficient. Imagine you've started with a scan for unknown values and then ran a scan for unchanged valueus. Most values in memory will have been unchanged, but with our current implementation, we are now storing an entire <code>usize</code> address for each of these. One option would be to introduce <code>CandidateLocations::Sparse</code>, which would be a middle ground. You could implement it like <code>Dense</code> and include a vector of booleans telling you which values to consider, or go smaller and use a bitstring or bit vector. You could use a sparse vector data structure.</p>
1768<p><code>Value</code> is very much like <code>CandidateLocations</code>, except that it stores a value to compare against and not an address. Here we can either have an exact value, or an older copy of the memory. Again, keeping a copy of the entire memory chunk when all we need is a handful of values is inefficient. You could keep a mapping from addresses to values if you don't have too many. Or you could shrink and fragment the copied memory in a more optimal way. There's a lot of room for improvement!</p>
1769<p>What if, despite all of the efforts above, we still don't have enough RAM to store all this information? The Cheat Engine Tutorial doesn't use a lot of memory, but as soon as you try scanning bigger programs, like games, you may find yourself needing several gigabytes worth of memory to remember all the found values in order to compare them in subsequent scans. You may even need to consider dumping all the regions to a file and read from it to run the comparisons. For example, running a scan for &quot;unknown value&quot; in Cheat Engine brings its memory up by the same amount of memory used by the process scanned (which makes sense), but as soon as I ran a scan for &quot;unchanged value&quot; over the misaligned values, Cheat Engine's disk usage skyrocketed to 1GB/s (!) for several seconds on my SSD. After it finished, memory usage went down to normal. It was very likely writing out all candidate locations to disk.</p>
1770<h2 id="finale">Finale</h2>
1771<p>There is a lot of things to learn from Cheat Engine just by observing its behaviour, and we're only scratching its surface.</p>
1772<p>In the <a href="/blog/woce-4">next post</a>, we'll tackle the fourth step of the tutorial: Floating points. So far, we have only been working with <code>i32</code> for simplicity. We will need to update our code to be able to account for different data types, which will make it easy to support other types like <code>i16</code>, <code>i64</code>, or even strings, represented as an arbitrary sequence of bytes.</p>
1773<p>As usual, you can <a href="https://github.com/lonami/memo">obtain the code for this post</a> over at my GitHub. You can run <code>git checkout step3</code> after cloning the repository to get the right version of the code. This version is a bit cleaner than the one presented in the blog, and contains some of the things described in the <a href="https://lonami.dev/blog/woce-3/#going-beyond">Going beyond</a> section. Until next time!</p>
1774<h3 id="footnotes">Footnotes</h3>
1775<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
1776<p>Well, technically, we will perform a million memory reads<sup class="footnote-reference"><a href="#5">5</a></sup>. The issue here is the million calls to <code>ReadProcessMemory</code>, not reading memory per se.</p>
1777</div>
1778<div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup>
1779<p>Not currently. After a basic implementation works, writing each implementation by hand and fine-tuning them by treating each of them as a special case could yield significant speed improvements. So although it would be a lot of work, this option shouldn't be ruled out completely.</p>
1780</div>
1781<div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup>
1782<p>You could ask the candidate locations where one should read, which would still keep the code reasonably simple.</p>
1783</div>
1784<div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup>
1785<p>You could also optimize for this case by determining both the smallest and largest address, and reading enough to cover them both. Or apply additional heuristics to only do so if the ratio of the size you're reading compared to the size you need isn't too large and abort the joint read otherwise. There is a lot of room for optimization here.</p>
1786</div>
1787<div class="footnote-definition" id="5"><sup class="footnote-definition-label">5</sup>
1788<p>(A footnote in a footnote?) The machine registers, memory cache and compiler will all help lower this cost, so the generated executable might not actually need that many reads from RAM. But that's getting way too deep into the details now.</p>
1789</div>
1790</content>
1791 </entry>
1792 <entry xml:lang="en">
1793 <title>Writing our own Cheat Engine: Exact Value scanning</title>
1794 <published>2021-02-12T00:00:00+00:00</published>
1795 <updated>2021-02-19T00:00:00+00:00</updated>
1796 <link href="https://lonami.dev/blog/woce-2/" type="text/html"/>
1797 <id>https://lonami.dev/blog/woce-2/</id>
1798 <content type="html"><p>This is part 2 on the <em>Writing our own Cheat Engine</em> series:</p>
1799<ul>
1800<li><a href="/blog/woce-1">Part 1: Introduction</a> (start here if you're new to the series!)</li>
1801<li>Part 2: Exact Value scanning</li>
1802<li><a href="/blog/woce-3">Part 3: Unknown initial value</a></li>
1803<li><a href="/blog/woce-4">Part 4: Floating points</a></li>
1804<li><a href="/blog/woce-5">Part 5: Code finder</a></li>
1805<li><a href="/blog/woce-6">Part 6: Pointers</a></li>
1806</ul>
1807<p>In the introduction, we spent a good deal of time enumerating all running processes just so we could find out the pid we cared about. With the pid now in our hands, we can do pretty much anything to its corresponding process.</p>
1808<p>It's now time to read the process' memory and write to it. If our process was a single-player game, this would enable us to do things like setting a very high value on the player's current health pool, making us invincible. This technique will often not work for multi-player games, because the server likely knows your true current health (the most you could probably do is make the client render an incorrect value). However, if the server is crappy and it trusts the client, then you're still free to mess around with your current health.</p>
1809<p>Even if we don't want to write to the process' memory, reading is still very useful. Maybe you could enhance your experience by making a custom overlay that displays useful information, or something that makes noise if it detects the life is too low, or even simulating a keyboard event to automatically recover some mana when you're running low.</p>
1810<p>Be warned about anti-cheat systems. Anything beyond a basic game is likely to have some protection measures in place, making the analysis more difficult (perhaps the values are scrambled in memory), or even pinging the server if it detects something fishy.</p>
1811<p><strong>I am not responsible for any bans!</strong> Use your brain before messing with online games, and don't ruin the fun for everyone else. If you get caught for cheating, I don't want to know about it.</p>
1812<p>Now that all <a href="https://www.urbandictionary.com/define.php?term=script%20kiddie">script kiddies</a> have left the room, let's proceed with the post.</p>
1813<h2 id="exact-value-scanning">Exact Value scanning</h2>
1814<details open><summary>Cheat Engine Tutorial: Step 2</summary>
1815<blockquote>
1816<p>Now that you have opened the tutorial with Cheat Engine let's get on with the next step.</p>
1817<p>You can see at the bottom of this window is the text Health: xxx. Each time you click 'Hit me' your health gets decreased.</p>
1818<p>To get to the next step you have to find this value and change it to 1000</p>
1819<p>To find the value there are different ways, but I'll tell you about the easiest, 'Exact Value': First make sure value type is set to at least 2-bytes or 4-bytes. 1-byte will also work, but you'll run into an easy to fix problem when you've found the address and want to change it. The 8-byte may perhaps works if the bytes after the address are 0, but I wouldn't take the bet. Single, double, and the other scans just don't work, because they store the value in a different way.</p>
1820<p>When the value type is set correctly, make sure the scantype is set to 'Exact Value'. Then fill in the number your health is in the value box. And click 'First Scan'. After a while (if you have a extremely slow pc) the scan is done and the results are shown in the list on the left</p>
1821<p>If you find more than 1 address and you don't know for sure which address it is, click 'Hit me', fill in the new health value into the value box, and click 'Next Scan'. Repeat this until you're sure you've found it. (that includes that there's only 1 address in the list.....)</p>
1822<p>Now double click the address in the list on the left. This makes the address pop-up in the list at the bottom, showing you the current value. Double click the value, (or select it and press enter), and change the value to 1000.</p>
1823<p>If everything went ok the next button should become enabled, and you're ready for the next step.</p>
1824<p>Note: If you did anything wrong while scanning, click &quot;New Scan&quot; and repeat the scanning again. Also, try playing around with the value and click 'hit me'</p>
1825</blockquote>
1826</details>
1827<h2 id="our-first-scan">Our First Scan</h2>
1828<p>The Cheat Engine tutorial talks about &quot;value types&quot; and &quot;scan types&quot; like &quot;exact value&quot;.</p>
1829<p>The <strong>value types</strong> will help us narrow down <em>what</em> we're looking for. For example, the integer type <code>i32</code> is represented in memory as 32 bits, or 4 bytes. However, <code>f32</code> is <em>also</em> represented by 4 bytes, and so is <code>u32</code>. Or perhaps the 4 bytes represent RGBA values of a color! So any 4 bytes in memory can be interpreted in many ways, and it's up to us to decide which way we interpret the bytes in.</p>
1830<p>When programming, numbers which are 32-bit wide are common, as they're a good (and fast) size to work with. Scanning for this type is often a good bet. For positive numbers, <code>i32</code> is represented the same as <code>u32</code> in memory, so even if the value turns out to not be signed, the scan is likely to work. Focusing on <code>i32</code> will save us from scanning for <code>f32</code> or even other types, like interpreting 8 bytes for <code>i64</code>, <code>f64</code>, or less bytes like <code>i16</code>.</p>
1831<p>The <strong>scan types</strong> will help us narrow down <em>how</em> we're looking for a value. Scanning for an exact value means what you think it does: interpret all 4 bytes in the process' memory as our value type, and check if they exactly match our value. This will often yield a lot of candidates, but it will be enough to get us started. Variations of the exact scan include checking for all values below a threshold, above, in between, or even just… unknown.</p>
1832<p>What's the point of scanning for unknown values if <em>everything</em> in memory is unknown? Sometimes you don't have a concrete value. Maybe your health pool is a bar and it nevers tell you how much health you actually have, just a visual indicator of your percentage left, even if the health is not stored as a percentage. As we will find later on, scanning for unknown values is more useful than it might appear at first.</p>
1833<p>We can access the memory of our own program by guessing random pointers and trying to read from them. But Windows isolates the memory of each program, so no pointer we could ever guess will let us read from the memory of another process. Luckily for us, searching for &quot;read process memory winapi&quot; leads us to the <a href="https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-readprocessmemory"><code>ReadProcessMemory</code></a> function. Spot on.</p>
1834<pre><code class="language-rust" data-lang="rust">pub fn read_memory(&amp;self, addr: usize, n: usize) -&gt; io::Result&lt;Vec&lt;u8&gt;&gt; {
1835 todo!()
1836}
1837</code></pre>
1838<p>Much like trying to dereference a pointer pointing to released memory or even null, reading from an arbitrary address can fail for the same reasons (and more). We will want to signal this with <code>io::Result</code>. It's funny to note that, even though we're doing something that seems wildly unsafe (reading arbitrary memory, even if the other process is mutating it at the same time), the function is perfectly safe. If we cannot read something, it will return <code>Err</code>, but if it succeeds, it has taken a snapshot of the memory of the process, and the returned value will be correctly initialized.</p>
1839<p>The function will be defined inside our <code>impl Process</code>, since it conveniently holds an open handle to the process in question. It takes <code>&amp;self</code>, because we do not need to mutate anything in the <code>Process</code> instance. After adding the <code>memoryapi</code> feature to <code>Cargo.toml</code>, we can perform the call:</p>
1840<pre><code class="language-rust" data-lang="rust">let mut buffer = Vec::&lt;u8&gt;::with_capacity(n);
1841let mut read = 0;
1842
1843// SAFETY: the buffer points to valid memory, and the buffer size is correctly set.
1844if unsafe {
1845 winapi::um::memoryapi::ReadProcessMemory(
1846 self.handle.as_ptr(),
1847 addr as *const _,
1848 buffer.as_mut_ptr().cast(),
1849 buffer.capacity(),
1850 &amp;mut read,
1851 )
1852} == FALSE
1853{
1854 Err(io::Error::last_os_error())
1855} else {
1856 // SAFETY: the call succeeded and `read` contains the amount of bytes written.
1857 unsafe { buffer.set_len(read as usize) };
1858 Ok(buffer)
1859}
1860</code></pre>
1861<p>Great! But the address space is somewhat large. 64 bits large. Eighteen quintillion, four hundred and forty-six quadrillion, seven hundred and forty-four trillion, seventy-three billion, seven hundred and nine million, five hundred and fifty-one thousand, six hundred and sixteen<sup class="footnote-reference"><a href="#1">1</a></sup> large. You gave up reading that, didn't you? Anyway, 18'446'744'073'709'551'616 is a <em>big</em> number.</p>
1862<p>I am not willing to wait for the program to scan over so many values. I don't even have 16 <a href="https://en.wikipedia.org/wiki/Orders_of_magnitude_(data)">exbibytes</a> of RAM installed on my laptop yet<sup class="footnote-reference"><a href="#2">2</a></sup>! What's up with that?</p>
1863<h2 id="memory-regions">Memory regions</h2>
1864<p>The program does not actually have all that memory allocated (surprise!). Random-guessing an address is extremely likely to point out to invalid memory. Reading from the start of the address space all the way to the end would not be any better. And we <strong>need</strong> to do better.</p>
1865<p>We need to query for the memory regions allocated to the program. For this purpose we can use <a href="https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualqueryex"><code>VirtualQueryEx</code></a>.</p>
1866<blockquote>
1867<p>Retrieves information about a range of pages within the virtual address space of a specified process.</p>
1868</blockquote>
1869<p>We have enumerated things before, and this function is not all that different.</p>
1870<pre><code class="language-rust" data-lang="rust">fn memory_regions(&amp;self) -&gt; io::Result&lt;winapi::um::winnt::MEMORY_BASIC_INFORMATION&gt; {
1871 let mut info = MaybeUninit::uninit();
1872
1873 // SAFETY: the info structure points to valid memory.
1874 let written = unsafe {
1875 winapi::um::memoryapi::VirtualQueryEx(
1876 self.handle.as_ptr(),
1877 std::ptr::null(),
1878 info.as_mut_ptr(),
1879 mem::size_of::&lt;winapi::um::winnt::MEMORY_BASIC_INFORMATION&gt;(),
1880 )
1881 };
1882 if written == 0 {
1883 Err(io::Error::last_os_error())
1884 } else {
1885 // SAFETY: a non-zero amount was written to the structure
1886 Ok(unsafe { info.assume_init() })
1887 }
1888}
1889</code></pre>
1890<p>We start with a base address of zero<sup class="footnote-reference"><a href="#3">3</a></sup> (<code>std::ptr::null()</code>), and ask the function to tell us what's in there. Let's try it out, with the <code>impl-debug</code> crate feature in <code>Cargo.toml</code>:</p>
1891<pre><code class="language-rust" data-lang="rust">dbg!(process.memory_regions());
1892</code></pre>
1893<pre><code>&gt;cargo run
1894Compiling memo v0.1.0
1895
1896error[E0277]: `winapi::um::winnt::MEMORY_BASIC_INFORMATION` doesn't implement `std::fmt::Debug`
1897 --&gt; src\main.rs:185:5
1898 |
1899185 | dbg!(process.memory_regions());
1900 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `winapi::um::winnt::MEMORY_BASIC_INFORMATION` cannot be formatted using `{:?}` because it doesn't implement `std::fmt::Debug`
1901</code></pre>
1902<p>That's annoying. It seems not everything has an <code>impl std::fmt::Debug</code>, and <a href="https://github.com/retep998/winapi-rs/issues/548#issuecomment-355278090">you're supposed to send a PR</a> if you want it to have debug, even if the <code>impl-debug</code> feature is set. I'm surprised they don't auto-generate all of this and have to rely on manually adding <code>Debug</code> as needed? Oh well, let's get rid of the feature and print it out ourselves:</p>
1903<pre><code>eprintln!(
1904 &quot;Region:
1905 BaseAddress: {:?}
1906 AllocationBase: {:?}
1907 AllocationProtect: {:?}
1908 RegionSize: {:?}
1909 State: {:?}
1910 Protect: {:?}
1911 Type: {:?}&quot;,
1912 region.BaseAddress,
1913 region.AllocationBase,
1914 region.AllocationProtect,
1915 region.RegionSize,
1916 region.State,
1917 region.Protect,
1918 region.Type,
1919);
1920</code></pre>
1921<p>Hopefully we don't need to do this often:</p>
1922<pre><code>&gt;cargo run
1923 Compiling memo v0.1.0
1924 Finished dev [unoptimized + debuginfo] target(s) in 0.60s
1925 Running `target\debug\memo.exe`
1926
1927Region:
1928 BaseAddress: 0x0
1929 AllocationBase: 0x0
1930 AllocationProtect: 0
1931 RegionSize: 65536
1932 State: 65536
1933 Protect: 1
1934 Type: 0
1935</code></pre>
1936<p>Awesome! There is a region at <code>null</code>, and the <code>AllocationProtect</code> of zero indicates that &quot;the caller does not have access&quot; when the region was created. However, <code>Protect</code> is <code>1</code>, and that is the <em>current</em> protection level. A value of one indicates <a href="https://docs.microsoft.com/en-us/windows/win32/memory/memory-protection-constants"><code>PAGE_NOACCESS</code></a>:</p>
1937<blockquote>
1938<p>Disables all access to the committed region of pages. An attempt to read from, write to, or execute the committed region results in an access violation.</p>
1939</blockquote>
1940<p>Now that we know that the first region starts at 0 and has a size of 64 KiB, we can simply query for the page at <code>(current base + current size)</code> to fetch the next region. Essentially, we want to loop until it fails, after which we'll know there are no more pages<sup class="footnote-reference"><a href="#4">4</a></sup>:</p>
1941<pre><code class="language-rust" data-lang="rust">pub fn memory_regions(&amp;self) -&gt; Vec&lt;winapi::um::winnt::MEMORY_BASIC_INFORMATION&gt; {
1942 let mut base = 0;
1943 let mut regions = Vec::new();
1944 let mut info = MaybeUninit::uninit();
1945
1946 loop {
1947 // SAFETY: the info structure points to valid memory.
1948 let written = unsafe {
1949 winapi::um::memoryapi::VirtualQueryEx(
1950 self.handle.as_ptr(),
1951 base as *const _,
1952 info.as_mut_ptr(),
1953 mem::size_of::&lt;winapi::um::winnt::MEMORY_BASIC_INFORMATION&gt;(),
1954 )
1955 };
1956 if written == 0 {
1957 break regions;
1958 }
1959 // SAFETY: a non-zero amount was written to the structure
1960 let info = unsafe { info.assume_init() };
1961 base = info.BaseAddress as usize + info.RegionSize;
1962 regions.push(info);
1963 }
1964}
1965</code></pre>
1966<p><code>RegionSize</code> is:</p>
1967<blockquote>
1968<p>The size of the region beginning at the base address in which all pages have identical attributes, in bytes.</p>
1969</blockquote>
1970<p>…which also hints that the value we want is &quot;base address&quot;, not the &quot;allocation base&quot;. With these two values, we can essentially iterate over all the page ranges:</p>
1971<pre><code class="language-rust" data-lang="rust">dbg!(process.memory_regions().len());
1972</code></pre>
1973<pre><code>&gt;cargo run
1974 Compiling memo v0.1.0
1975 Finished dev [unoptimized + debuginfo] target(s) in 0.63s
1976 Running `target\debug\memo.exe`
1977
1978[src\main.rs:189] process.memory_regions().len() = 367
1979</code></pre>
1980<p>That's a lot of pages!</p>
1981<h2 id="protection-levels">Protection levels</h2>
1982<p>Let's try to narrow the amount of pages down. How many pages aren't <code>PAGE_NOACCESS</code>?</p>
1983<pre><code class="language-rust" data-lang="rust">dbg!(process
1984 .memory_regions()
1985 .into_iter()
1986 .filter(|p| p.Protect != winapi::um::winnt::PAGE_NOACCESS)
1987 .count());
1988</code></pre>
1989<pre><code>295
1990</code></pre>
1991<p>Still a fair bit! Most likely, there are just a few interleaved <code>NOACCESS</code> pages, and the rest are allocated each with different protection levels. How much memory do we need to scan through?</p>
1992<pre><code class="language-rust" data-lang="rust">dbg!(process
1993 .memory_regions()
1994 .into_iter()
1995 .filter(|p| p.Protect != winapi::um::winnt::PAGE_NOACCESS)
1996 .map(|p| p.RegionSize)
1997 .sum::&lt;usize&gt;());
1998</code></pre>
1999<pre><code>4480434176
2000</code></pre>
2001<p>Wait, what? What do you mean over 4 GiB? The Task Manager claims that the Cheat Engine Tutorial is only using 2.1 MB worth of RAM! Perhaps we can narrow down the <a href="https://docs.microsoft.com/en-us/windows/win32/memory/memory-protection-constants">protection levels</a> a bit more. If you look at the scan options in Cheat Engine, you will notice the &quot;Memory Scan Options&quot; groupbox. By default, it only scans for memory that is writable, and doesn't care if it's executable or not:</p>
2002<pre><code class="language-rust" data-lang="rust">let mask = winnt::PAGE_EXECUTE_READWRITE
2003 | winnt::PAGE_EXECUTE_WRITECOPY
2004 | winnt::PAGE_READWRITE
2005 | winnt::PAGE_WRITECOPY;
2006
2007dbg!(process
2008 .memory_regions()
2009 .into_iter()
2010 .filter(|p| (p.Protect &amp; mask) != 0)
2011 .map(|p| p.RegionSize)
2012 .sum::&lt;usize&gt;());
2013</code></pre>
2014<p>Each memory protection level has its own bit, so we can OR them all together to have a single mask. When ANDing this mask with the protection level, if any bit is set, it will be non-zero, meaning we want to keep this region.</p>
2015<p>Don't ask me why there isn't a specific bit for &quot;write&quot;, &quot;read&quot;, &quot;execute&quot;, and there are only bits for combinations. I guess this way Windows forbids certain combinations.</p>
2016<pre><code>2580480
2017</code></pre>
2018<p>Hey, that's close to the value shown by the Task Manager! A handfull of megabytes is a lot more manageable than 4 entire gigabytes.</p>
2019<h2 id="actually-running-our-first-scan">Actually running our First Scan</h2>
2020<p>Okay, we have all the memory regions from which the program can read, write, or execute. Now we also can read the memory in these regions:</p>
2021<pre><code class="language-rust" data-lang="rust">let regions = process
2022 .memory_regions()
2023 .into_iter()
2024 .filter(|p| (p.Protect &amp; mask) != 0)
2025 .collect::&lt;Vec&lt;_&gt;&gt;();
2026
2027println!(&quot;Scanning {} memory regions&quot;, regions.len());
2028
2029regions.into_iter().for_each(|region| {
2030 match process.read_memory(region.BaseAddress as _, region.RegionSize) {
2031 Ok(memory) =&gt; todo!(),
2032 Err(err) =&gt; eprintln!(
2033 &quot;Failed to read {} bytes at {:?}: {}&quot;,
2034 region.RegionSize, region.BaseAddress, err,
2035 ),
2036 }
2037})
2038</code></pre>
2039<p>All that's left is for us to scan for a target value. To do this, we want to iterate over all the <a href="https://doc.rust-lang.org/stable/std/primitive.slice.html#method.windows"><code>slice::windows</code></a> of size equal to the size of our scan type.</p>
2040<pre><code class="language-rust" data-lang="rust">let target: i32 = ...;
2041let target = target.to_ne_bytes();
2042
2043// -snip-
2044
2045// inside the Ok match, replacing the todo!() -- this is where the first scan happens
2046Ok(memory) =&gt; memory
2047 .windows(target.len())
2048 .enumerate()
2049 .for_each(|(offset, window)| {
2050 if window == target {
2051 println!(
2052 &quot;Found exact value at [{:?}+{:x}]&quot;,
2053 region.BaseAddress, offset
2054 );
2055 }
2056 })
2057</code></pre>
2058<p>We convert the 32-bit exact target value to its memory representation as a byte array in <a href="https://doc.rust-lang.org/stable/std/primitive.i32.html#method.to_ne_bytes">native byte order</a>. This way we can compare the target bytes with the window bytes. Another option is to interpret the window bytes as an <code>i32</code> with <code>from_be_bytes</code>, but <code>slice::windows</code> gives us slices of type <code>&amp;[u8]</code>, and <code>from_be_bytes</code> wants an <code>[u8; 4]</code> array, so it's a bit more annoying to convert.</p>
2059<p>This is enough to find the value in the process' memory!</p>
2060<pre><code>Found exact value at [0x10000+aec]
2061Failed to read 12288 bytes at 0x13f8000: Only part of a ReadProcessMemory or WriteProcessMemory request was completed. (os error 299)
2062Found exact value at [0x14f0000+3188]
2063Found exact value at [0x14f0000+ac74]
2064...
2065Found exact value at [0x10030e000+1816]
2066Found exact value at [0x7ff8f7b93000+441a]
2067...
2068Found exact value at [0x7ff8fb381000+4023]
2069</code></pre>
2070<p>The tutorial starts out with health &quot;100&quot;, which is what I scanned. Apparently, there are nearly a hundred of <code>100</code>-valued integers stored in the memory of the tutorial.</p>
2071<p>Attentive readers will notice that some values are located at an offset modulo 4. In Cheat Engine, this is known as &quot;Fast Scan&quot;, which is enabled by default with an alignment of 4. Most of the time, values are aligned in memory, and this alignment often corresponds with the size of the type itself. For 4-byte integers, it's common that they're 4-byte aligned.</p>
2072<p>We can perform a fast scan ourselves with <a href="https://doc.rust-lang.org/stable/std/iter/trait.Iterator.html#method.step_by"><code>step_by</code></a><sup class="footnote-reference"><a href="#5">5</a></sup>:</p>
2073<pre><code class="language-rust" data-lang="rust">memory
2074 .windows(target.len())
2075 .enumerate()
2076 .step_by(4)
2077 .for_each(...)
2078</code></pre>
2079<p>As a bonus, over half the addresses are gone, so we have less results to worry about<sup class="footnote-reference"><a href="#6">6</a></sup>.</p>
2080<h2 id="next-scan">Next Scan</h2>
2081<p>The first scan gave us way too many results. We have no way to tell which is the correct one, as they all have the same value. What we need to do is a <em>second</em> scan at the <em>locations we just found</em>. This way, we can get a second reading, and compare it against a new value. If it's the same, we're on good track, and if not, we can discard a location. Repeating this process lets us cut the hundreds of potential addresses to just a handful of them.</p>
2082<p>For example, let's say we're scanning our current health of <code>100</code> in a game. This gives us over a hundred addresses that point to the value of <code>100</code>. If we go in-game and get hit<sup class="footnote-reference"><a href="#7">7</a></sup> by some enemy and get our health down to, say, <code>99</code> (we have a lot of defense), we can then read the memory at the hundred memory locations we found before. If this second reading is not <code>99</code>, we know the address does not actually point to our health pool and it just happened to also contain a <code>100</code> on the first scan. This address can be removed from the list of potential addresses pointing to our health.</p>
2083<p>Let's do that:</p>
2084<pre><code class="language-rust" data-lang="rust">// new vector to hold the locations, before getting into `memory.windows`' for-each
2085let mut locations = Vec::with_capacity(regions.len());
2086
2087// -snip-
2088
2089// updating the `println!(&quot;Found exact value...&quot;)` to store the location instead.
2090if window == target {
2091 locations.push(region.BaseAddress as usize + offset);
2092}
2093
2094// -snip-
2095
2096// performing a second scan on the locations the first scan found.
2097let target: i32 = ...;
2098let target = target.to_ne_bytes();
2099locations.retain(|addr| match process.read_memory(*addr, target.len()) {
2100 Ok(memory) =&gt; memory == target,
2101 Err(_) =&gt; false,
2102});
2103
2104println!(&quot;Now have {} locations&quot;, locations.len());
2105</code></pre>
2106<p>We create a vector to store all the locations the first scan finds, and then retain those that match a second target value. You may have noticed that we perform a memory read, and thus a call to the Windows API, for every single address. With a hundred locations to read from, this is not a big deal, but oftentimes you will have tens of thousands of addresses. For the time being, we will not worry about this inefficiency, but we will get back to it once it matters:</p>
2107<pre><code>Scanning 98 memory regions
2108Which exact value to scan for?: 100
2109Failed to read 12288 bytes at 0x13f8000: Only part of a ReadProcessMemory or WriteProcessMemory request was completed. (os error 299)
2110...
2111Found 49 locations
2112Which exact value to scan for next?: 99
2113Now have 1 locations
2114</code></pre>
2115<p>Sweet! In a real-world scenario, you will likely need to perform these additional scans a couple of times, and even then, there may be more than one value left no matter what.</p>
2116<p>For good measure, we'll wrap our <code>retain</code> in a <code>while</code> loop<sup class="footnote-reference"><a href="#8">8</a></sup>:</p>
2117<pre><code class="language-rust" data-lang="rust">while locations.len() != 1 {
2118 let target: i32 = ...;
2119 let target = target.to_ne_bytes();
2120 locations.retain(...);
2121}
2122</code></pre>
2123<h2 id="modifying-memory">Modifying memory</h2>
2124<p>Now that we have very likely locations pointing to our current health in memory, all that's left is writing our new desired value to gain infinite health<sup class="footnote-reference"><a href="#9">9</a></sup>. Much like how we're able to read memory with <code>ReadProcessMemory</code>, we can write to it with <a href="https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-writeprocessmemory"><code>WriteProcessMemory</code></a>. Its usage is straightforward:</p>
2125<pre><code class="language-rust" data-lang="rust">pub fn write_memory(&amp;self, addr: usize, value: &amp;[u8]) -&gt; io::Result&lt;usize&gt; {
2126 let mut written = 0;
2127
2128 // SAFETY: the input value buffer points to valid memory.
2129 if unsafe {
2130 winapi::um::memoryapi::WriteProcessMemory(
2131 self.handle.as_ptr(),
2132 addr as *mut _,
2133 value.as_ptr().cast(),
2134 value.len(),
2135 &amp;mut written,
2136 )
2137 } == FALSE
2138 {
2139 Err(io::Error::last_os_error())
2140 } else {
2141 Ok(written)
2142 }
2143}
2144</code></pre>
2145<p>Similar to how writing to a file can return short, writing to a memory location could also return short. Here we mimic the API for writing files and return the number of bytes written. The documentation indicates that we could actually ignore the amount written by passing <code>ptr::null_mut()</code> as the last parameter, but it does no harm to retrieve the written count as well.</p>
2146<pre><code class="language-rust" data-lang="rust">let new_value: i32 = ...;
2147locations
2148 .into_iter()
2149 .for_each(|addr| match process.write_memory(addr, &amp;new_value) {
2150 Ok(n) =&gt; eprintln!(&quot;Written {} bytes to [{:x}]&quot;, n, addr),
2151 Err(e) =&gt; eprintln!(&quot;Failed to write to [{:x}]: {}&quot;, addr, e),
2152 });
2153</code></pre>
2154<p>And just like that:</p>
2155<pre><code>Now have 1 location(s)
2156Enter new memory value: 1000
2157Failed to write to [15d8b90]: Access is denied. (os error 5)
2158</code></pre>
2159<p>…oh noes. Oh yeah. The documentation, which I totally didn't forget to read, mentions:</p>
2160<blockquote>
2161<p>The handle must have <code>PROCESS_VM_WRITE</code> and <code>PROCESS_VM_OPERATION</code> access to the process.</p>
2162</blockquote>
2163<p>We currently open our process with <code>PROCESS_QUERY_INFORMATION</code> and <code>PROCESS_VM_READ</code>, which is enough for reading, but not for writing. Let's adjust <code>OpenProcess</code> to accomodate for our new requirements:</p>
2164<pre><code class="language-rust" data-lang="rust">winapi::um::processthreadsapi::OpenProcess(
2165 winnt::PROCESS_QUERY_INFORMATION
2166 | winnt::PROCESS_VM_READ
2167 | winnt::PROCESS_VM_WRITE
2168 | winnt::PROCESS_VM_OPERATION,
2169 FALSE,
2170 pid,
2171)
2172</code></pre>
2173<p>Behold:</p>
2174<pre><code>Now have 1 location(s)
2175Enter new memory value: 1000
2176Written 4 bytes to [15d8b90]
2177</code></pre>
2178<p><img src="https://user-images.githubusercontent.com/6297805/107829541-3f4f2d00-6d8a-11eb-87c4-e2f2d505afbc.png" alt="Tutorial complete with memo" /></p>
2179<p>Isn't that active <em>Next</em> button just beautiful?</p>
2180<h2 id="finale">Finale</h2>
2181<p>This post somehow ended up being longer than part one, but look at what we've achieved! We completed a step of the Cheat Engine Tutorial <em>without using Cheat Engine</em>. Just pure Rust. Figuring out how a program works and reimplementing it yourself is a great way to learn what it's doing behind the scenes. And now that this code is yours, you can extend it as much as you like, without being constrained by Cheat Engine's UI. You can automate it as much as you want.</p>
2182<p>And we're not even done. The current tutorial has nine steps, and three additional graphical levels.</p>
2183<p>In the <a href="/blog/woce-3">next post</a>, we'll tackle the third step of the tutorial: Unknown initial value. This will pose a challenge, because with just 2 MiB of memory, storing all the 4-byte aligned locations would require 524288 addresses (<code>usize</code>, 8 bytes). This adds up to twice as much memory as the original program (4 MiB), but that's not our main concern, having to perform over five hundred thousand API calls is!</p>
2184<p>Remember that you can <a href="https://github.com/lonami/memo">obtain the code for this post</a> over at my GitHub. You can run <code>git checkout step2</code> after cloning the repository to get the right version of the code.</p>
2185<h3 id="footnotes">Footnotes</h3>
2186<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
2187<p>I did in fact use an online tool to spell it out for me.</p>
2188</div>
2189<div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup>
2190<p>16 GiB is good enough for my needs. I don't think I'll ever upgrade to 16 EiB.</p>
2191</div>
2192<div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup>
2193<p>Every address we query should have a corresponding region, even if it's not allocated or we do not have access. This is why we can query for the memory address zero to get its corresponding region.</p>
2194</div>
2195<div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup>
2196<p>Another option is to <a href="https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/nf-sysinfoapi-getsysteminfo"><code>GetSystemInfo</code></a> to determine the <code>lpMinimumApplicationAddress</code> and <code>lpMaximumApplicationAddress</code> and only work within bounds.</p>
2197</div>
2198<div class="footnote-definition" id="5"><sup class="footnote-definition-label">5</sup>
2199<p>Memory regions are page-aligned, which is a large power of two. Our alignment of 4 is much lower than this, so we're guaranteed to start off at an aligned address.</p>
2200</div>
2201<div class="footnote-definition" id="6"><sup class="footnote-definition-label">6</sup>
2202<p>If it turns out that the value was actually misaligned, we will miss it. You will notice this if, after going through the whole process, there are no results. It could mean that either the value type is wrong, or the value type is misaligned. In the worst case, the value is not stored directly but is rather computed with something like <code>maximum - stored</code>, or XORed with some magic value, or a myriad other things.</p>
2203</div>
2204<div class="footnote-definition" id="7"><sup class="footnote-definition-label">7</sup>
2205<p>You could do this without getting hit, and just keep on repeating the scan for the same value over and over again. This does work, but the results are suboptimal, because there are also many other values that didn't change. Scanning for a changed value is a better option.</p>
2206</div>
2207<div class="footnote-definition" id="8"><sup class="footnote-definition-label">8</sup>
2208<p>You could actually just go ahead and try to modify the memory at the hundred addresses you just found, although don't be surprised if the program starts to misbehave!</p>
2209</div>
2210<div class="footnote-definition" id="9"><sup class="footnote-definition-label">9</sup>
2211<p>Okay, we cannot fit infinity in an <code>i32</code>. However, we can fit sufficiently large numbers. Like <code>1000</code>, which is enough to complete the tutorial.</p>
2212</div>
2213</content>
2214 </entry>
2215 <entry xml:lang="en">
2216 <title>Writing our own Cheat Engine: Introduction</title>
2217 <published>2021-02-07T00:00:00+00:00</published>
2218 <updated>2021-02-19T00:00:00+00:00</updated>
2219 <link href="https://lonami.dev/blog/woce-1/" type="text/html"/>
2220 <id>https://lonami.dev/blog/woce-1/</id>
2221 <content type="html"><p>This is part 1 on the <em>Writing our own Cheat Engine</em> series:</p>
2222<ul>
2223<li>Part 1: Introduction</li>
2224<li><a href="/blog/woce-2">Part 2: Exact Value scanning</a></li>
2225<li><a href="/blog/woce-3">Part 3: Unknown initial value</a></li>
2226<li><a href="/blog/woce-4">Part 4: Floating points</a></li>
2227<li><a href="/blog/woce-5">Part 5: Code finder</a></li>
2228<li><a href="/blog/woce-6">Part 6: Pointers</a></li>
2229</ul>
2230<p><a href="https://cheatengine.org/">Cheat Engine</a> is a tool designed to modify single player games and contains other useful tools within itself that enable its users to debug games or other applications. It comes with a memory scanner, (dis)assembler, inspection tools and a handful other things. In this series, we will be writing our own tiny Cheat Engine capable of solving all steps of the tutorial, and diving into how it all works underneath.</p>
2231<p>Needless to say, we're doing this for private and educational purposes only. One has to make sure to not violate the EULA or ToS of the specific application we're attaching to. This series, much like cheatengine.org, does not condone the illegal use of the code shared.</p>
2232<p>Cheat Engine is a tool for Windows, so we will be developing for Windows as well. However, you can also <a href="https://stackoverflow.com/q/12977179/4759433">read memory from Linux-like systems</a>. <a href="https://github.com/scanmem/scanmem">GameConqueror</a> is a popular alternative to Cheat Engine on Linux systems, so if you feel adventurous, you could definitely follow along too! The techniques shown in this series apply regardless of how we read memory from a process. You will learn a fair bit about doing FFI in Rust too.</p>
2233<p>We will be developing the application in Rust, because it enables us to interface with the Windows API easily, is memory safe (as long as we're careful with <code>unsafe</code>!), and is speedy (we will need this for later steps in the Cheat Engine tutorial). You could use any language of your choice though. For example, <a href="https://lonami.dev/blog/ctypes-and-windows/">Python also makes it relatively easy to use the Windows API</a>. You don't need to be a Rust expert to follow along, but this series assumes some familiarity with C-family languages. Slightly advanced concepts like the use of <code>unsafe</code> or the <code>MaybeUninit</code> type will be briefly explained. What a <code>fn</code> is or what <code>let</code> does will not be explained.</p>
2234<p><a href="https://github.com/cheat-engine/cheat-engine/">Cheat Engine's source code</a> is mostly written in Pascal and C. And it's <em>a lot</em> of code, with a very flat project structure, and files ranging in the thousand lines of code each. It's daunting<sup class="footnote-reference"><a href="#1">1</a></sup>. It's a mature project, with a lot of knowledge encoded in the code base, and a lot of features like distributed scanning or an entire disassembler. Unfortunately, there's not a lot of comments. For these reasons, I'll do some guesswork when possible as to how it's working underneath, rather than actually digging into what Cheat Engine is actually doing.</p>
2235<p>With that out of the way, let's get started!</p>
2236<h2 id="welcome-to-the-cheat-engine-tutorial">Welcome to the Cheat Engine Tutorial</h2>
2237<details open><summary>Cheat Engine Tutorial: Step 1</summary>
2238<blockquote>
2239<p>This tutorial will teach you the basics of cheating in video games. It will also show you foundational aspects of using Cheat Engine (or CE for short). Follow the steps below to get started.</p>
2240<ol>
2241<li>Open Cheat Engine if it currently isn't running.</li>
2242<li>Click on the &quot;Open Process&quot; icon (it's the top-left icon with the computer on it, below &quot;File&quot;.).</li>
2243<li>With the Process List window now open, look for this tutorial's process in the list. It will look something like &gt; &quot;00001F98-Tutorial-x86_64.exe&quot; or &quot;0000047C-Tutorial-i386.exe&quot;. (The first 8 numbers/letters will probably be different.)</li>
2244<li>Once you've found the process, click on it to select it, then click the &quot;Open&quot; button. (Don't worry about all the &gt; other buttons right now. You can learn about them later if you're interested.)</li>
2245</ol>
2246<p>Congratulations! If you did everything correctly, the process window should be gone with Cheat Engine now attached to the &gt; tutorial (you will see the process name towards the top-center of CE).</p>
2247<p>Click the &quot;Next&quot; button below to continue, or fill in the password and click the &quot;OK&quot; button to proceed to that step.)</p>
2248<p>If you're having problems, simply head over to forum.cheatengine.org, then click on &quot;Tutorials&quot; to view beginner-friendly &gt; guides!</p>
2249</blockquote>
2250</details>
2251<h2 id="enumerating-processes">Enumerating processes</h2>
2252<p>Our first step is attaching to the process we want to work with. But we need a way to find that process in the first place! Having to open the task manager, look for the process we care about, noting down the process ID (PID), and slapping it in the source code is not satisfying at all. Instead, let's enumerate all the processes from within the program, and let the user select one by typing its name.</p>
2253<p>From a quick <a href="https://ddg.gg/winapi%20enumerate%20all%20processes">DuckDuckGo search</a>, we find an official tutorial for <a href="https://docs.microsoft.com/en-us/windows/win32/psapi/enumerating-all-processes">Enumerating All Processes</a>, which leads to the <a href="https://docs.microsoft.com/en-us/windows/win32/api/psapi/nf-psapi-enumprocesses"><code>EnumProcesses</code></a> call. Cool! Let's slap in the <a href="https://crates.io/crates/winapi"><code>winapi</code></a> crate on <code>Cargo.toml</code>, because I don't want to write all the definitions by myself:</p>
2254<pre><code class="language-toml" data-lang="toml">[dependencies]
2255winapi = { version = &quot;0.3.9&quot;, features = [&quot;psapi&quot;] }
2256</code></pre>
2257<p>Because <a href="https://docs.microsoft.com/en-us/windows/win32/api/psapi/nf-psapi-enumprocesses"><code>EnumProcesses</code></a> is in <code>Psapi.h</code> (you can see this in the online page of its documentation), we know we'll need the <code>psapi</code> crate feature. Another option is to search for it in the <a href="https://docs.rs/winapi/"><code>winapi</code> documentation</a> and noting down the parent module where its stored.</p>
2258<p>The documentation for the method has the following remark:</p>
2259<blockquote>
2260<p>It is a good idea to use a large array, because it is hard to predict how many processes there will be at the time you call <strong>EnumProcesses</strong>.</p>
2261</blockquote>
2262<p><em>Sidenote: reading the documentation for the methods we'll use from the Windows API is extremely important. There's a lot of gotchas involved, so we need to make sure we're extra careful.</em></p>
2263<p>1024 is a pretty big number, so let's go with that:</p>
2264<pre><code class="language-rust" data-lang="rust">use std::io;
2265use std::mem;
2266use winapi::shared::minwindef::{DWORD, FALSE};
2267
2268pub fn enum_proc() -&gt; io::Result&lt;Vec&lt;u32&gt;&gt; {
2269 let mut pids = Vec::&lt;DWORD&gt;::with_capacity(1024);
2270 let mut size = 0;
2271 // SAFETY: the pointer is valid and the size matches the capacity.
2272 if unsafe {
2273 winapi::um::psapi::EnumProcesses(
2274 pids.as_mut_ptr(),
2275 (pids.capacity() * mem::size_of::&lt;DWORD&gt;()) as u32,
2276 &amp;mut size,
2277 )
2278 } == FALSE
2279 {
2280 return Err(io::Error::last_os_error());
2281 }
2282
2283 todo!()
2284}
2285</code></pre>
2286<p>We allocate enough space<sup class="footnote-reference"><a href="#2">2</a></sup> for 1024 <code>pids</code> in a vector<sup class="footnote-reference"><a href="#3">3</a></sup>, and pass a mutable pointer to the contents to <code>EnumProcesses</code>. Note that the size of the array is in <em>bytes</em>, not items, so we need to multiply the capacity by the size of <code>DWORD</code>. The API likes to use <code>u32</code> for sizes, unlike Rust which uses <code>usize</code>, so we need a cast.</p>
2287<p>Last, we need another mutable variable where the amount of bytes written is stored, <code>size</code>.</p>
2288<blockquote>
2289<p>If the function fails, the return value is zero. To get extended error information, call <a href="https://docs.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-getlasterror"><code>GetLastError</code></a>.</p>
2290</blockquote>
2291<p>That's precisely what we do. If it returns false (zero), we return the last OS error. Rust provides us with <a href="https://doc.rust-lang.org/stable/std/io/struct.Error.html#method.last_os_error"><code>std::io::Error::last_os_error</code></a>, which essentially makes that same call but returns a proper <code>io::Error</code> instance. Cool!</p>
2292<blockquote>
2293<p>To determine how many processes were enumerated, divide the <em>lpcbNeeded</em> value by <code>sizeof(DWORD)</code>.</p>
2294</blockquote>
2295<p>Easy enough:</p>
2296<pre><code class="language-rust" data-lang="rust">let count = size as usize / mem::size_of::&lt;DWORD&gt;();
2297// SAFETY: the call succeeded and count equals the right amount of items.
2298unsafe { pids.set_len(count) };
2299Ok(pids)
2300</code></pre>
2301<p>Rust doesn't know that the memory for <code>count</code> items were initialized by the call, but we do, so we make use of the <a href="https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.set_len"><code>Vec::set_len</code></a> call to indicate this. The Rust documentation even includes a FFI similar to our code!</p>
2302<p>Let's give it a ride:</p>
2303<pre><code class="language-rust" data-lang="rust">fn main() {
2304 dbg!(enum_proc().unwrap().len());
2305}
2306</code></pre>
2307<pre><code>&gt;cargo run
2308 Compiling memo v0.1.0
2309 Finished dev [unoptimized + debuginfo] target(s) in 0.20s
2310 Running `target\debug\memo.exe`
2311[src\main.rs:27] enum_proc().unwrap().len() = 178
2312</code></pre>
2313<p>It works! But currently we only have a bunch of process identifiers, with no way of knowing which process they refer to.</p>
2314<blockquote>
2315<p>To obtain process handles for the processes whose identifiers you have just obtained, call the <a href="https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-openprocess"><code>OpenProcess</code></a> function.</p>
2316</blockquote>
2317<p>Oh!</p>
2318<h2 id="opening-a-process">Opening a process</h2>
2319<p>The documentation for <code>OpenProcess</code> also contains the following:</p>
2320<blockquote>
2321<p>When you are finished with the handle, be sure to close it using the <a href="https://lonami.dev/blog/woce-1/closehandle"><code>CloseHandle</code></a> function.</p>
2322</blockquote>
2323<p>This sounds to me like the perfect time to introduce a custom <code>struct Process</code> with an <code>impl Drop</code>! We're using <code>Drop</code> to cleanup resources, not behaviour, so it's fine. <a href="https://internals.rust-lang.org/t/pre-rfc-leave-auto-trait-for-reliable-destruction/13825">Using <code>Drop</code> to cleanup behaviour is a bad idea</a>. But anyway, let's get back to the code:</p>
2324<pre><code class="language-rust" data-lang="rust">use std::ptr::NonNull;
2325use winapi::ctypes::c_void;
2326
2327pub struct Process {
2328 pid: u32,
2329 handle: NonNull&lt;c_void&gt;,
2330}
2331
2332impl Process {
2333 pub fn open(pid: u32) -&gt; io::Result&lt;Self&gt; {
2334 todo!()
2335 }
2336}
2337
2338impl Drop for Process {
2339 fn drop(&amp;mut self) {
2340 todo!()
2341 }
2342}
2343</code></pre>
2344<p>For <code>open</code>, we'll want to use <code>OpenProcess</code> (and we also need to add the <code>processthreadsapi</code> feature to the <code>winapi</code> dependency in <code>Cargo.toml</code>). It returns a <code>HANDLE</code>, which is a nullable mutable pointer to <code>c_void</code>. If it's null, the call failed, and if it's non-null, it succeeded and we have a valid handle. This is why we use Rust's <a href="https://doc.rust-lang.org/stable/std/ptr/struct.NonNull.html"><code>NonNull</code></a>:</p>
2345<pre><code class="language-rust" data-lang="rust">// SAFETY: the call doesn't have dangerous side-effects.
2346NonNull::new(unsafe { winapi::um::processthreadsapi::OpenProcess(0, FALSE, pid) })
2347 .map(|handle| Self { pid, handle })
2348 .ok_or_else(io::Error::last_os_error)
2349</code></pre>
2350<p><code>NonNull</code> will return <code>Some</code> if the pointer is non-null. We map the non-null pointer to a <code>Process</code> instance with <code>Self { .. }</code>. <code>ok_or_else</code> converts the <code>Option</code> to a <code>Result</code> with the error builder function we provide if it was <code>None</code>.</p>
2351<p>The first parameter is a bitflag of permissions we want to have. For now, we can leave it as zero (all bits unset, no specific permissions granted). The second one is whether we want to inherit the handle, which we don't, and the third one is the process identifier. Let's close the resource handle on <code>Drop</code> (after adding <code>handleapi</code> to the crate features):</p>
2352<pre><code class="language-rust" data-lang="rust">// SAFETY: the handle is valid and non-null.
2353unsafe { winapi::um::handleapi::CloseHandle(self.handle.as_mut()) };
2354</code></pre>
2355<p><code>CloseHandle</code> can actually fail (for example, on double-close), but given our invariants, it won't. You could add an <code>assert!</code> to panic if this is not the case.</p>
2356<p>We can now open processes, and they will be automatically closed on <code>Drop</code>. Does any of this work though?</p>
2357<pre><code class="language-rust" data-lang="rust">fn main() {
2358 let mut success = 0;
2359 let mut failed = 0;
2360 enum_proc().unwrap().into_iter().for_each(|pid| match Process::open(pid) {
2361 Ok(_) =&gt; success += 1,
2362 Err(_) =&gt; failed += 1,
2363 });
2364
2365 eprintln!(&quot;Successfully opened {}/{} processes&quot;, success, success + failed);
2366}
2367</code></pre>
2368<pre><code>&gt;cargo run
2369 Compiling memo v0.1.0
2370 Finished dev [unoptimized + debuginfo] target(s) in 0.36s
2371 Running `target\debug\memo.exe`
2372Successfully opened 0/191 processes
2373</code></pre>
2374<p>…nope. Maybe the documentation for <code>OpenProcess</code> says something?</p>
2375<blockquote>
2376<p><code>dwDesiredAccess</code></p>
2377<p>The access to the process object. This access right is checked against the security descriptor for the process. This parameter can be <strong>one or more</strong> of the process access rights.</p>
2378</blockquote>
2379<p>One or more, but we're setting zero permissions. I told you, reading the documentation is important<sup class="footnote-reference"><a href="#4">4</a></sup>! The <a href="https://docs.microsoft.com/en-us/windows/win32/procthread/process-security-and-access-rights">Process Security and Access Rights</a> page lists all possible values we could use. <code>PROCESS_QUERY_INFORMATION</code> seems to be appropriated:</p>
2380<blockquote>
2381<p>Required to retrieve certain information about a process, such as its token, exit code, and priority class</p>
2382</blockquote>
2383<pre><code class="language-rust" data-lang="rust">OpenProcess(winapi::um::winnt::PROCESS_QUERY_INFORMATION, ...)
2384</code></pre>
2385<p>Does this fix it?</p>
2386<pre><code class="language-rust" data-lang="rust">&gt;cargo run
2387 Compiling memo v0.1.0
2388 Finished dev [unoptimized + debuginfo] target(s) in 0.36s
2389 Running `target\debug\memo.exe`
2390Successfully opened 69/188 processes
2391</code></pre>
2392<p><em>Nice</em>. It does solve it. But why did we only open 69 processes out of 188? Does it help if we run our code as administrator? Let's search for <code>cmd</code> in the Windows menu and right click to Run as administrator, then <code>cd</code> into our project and try again:</p>
2393<pre><code>&gt;cargo run
2394 Finished dev [unoptimized + debuginfo] target(s) in 0.01s
2395 Running `target\debug\memo.exe`
2396Successfully opened 77/190 processes
2397</code></pre>
2398<p>We're able to open a few more, so it does help. In general, we'll want to run as administrator, so normal programs can't sniff on what we're doing, and so that we have permission to do more things.</p>
2399<h2 id="getting-the-name-of-a-process">Getting the name of a process</h2>
2400<p>We're not done enumerating things just yet. To get the &quot;name&quot; of a process, we need to enumerate the modules that it has loaded, and only then can we get the module base name. The first module is the program itself, so we don't need to enumerate <em>all</em> modules, just the one is enough.</p>
2401<p>For this we want <a href="https://docs.microsoft.com/en-us/windows/win32/api/psapi/nf-psapi-enumprocessmodules"><code>EnumProcessModules</code></a> and <a href="https://docs.microsoft.com/en-us/windows/win32/api/psapi/nf-psapi-getmodulebasenamea"><code>GetModuleBaseNameA</code></a>. I'm using the ASCII variant of <code>GetModuleBaseName</code> because I'm too lazy to deal with UTF-16 of the <code>W</code> (wide, unicode) variants.</p>
2402<pre><code class="language-rust" data-lang="rust">use std::mem::MaybeUninit;
2403use winapi::shared::minwindef::HMODULE;
2404
2405pub fn name(&amp;self) -&gt; io::Result&lt;String&gt; {
2406 let mut module = MaybeUninit::&lt;HMODULE&gt;::uninit();
2407 let mut size = 0;
2408 // SAFETY: the pointer is valid and the size is correct.
2409 if unsafe {
2410 winapi::um::psapi::EnumProcessModules(
2411 self.handle.as_ptr(),
2412 module.as_mut_ptr(),
2413 mem::size_of::&lt;HMODULE&gt;() as u32,
2414 &amp;mut size,
2415 )
2416 } == FALSE
2417 {
2418 return Err(io::Error::last_os_error());
2419 }
2420
2421 // SAFETY: the call succeeded, so module is initialized.
2422 let module = unsafe { module.assume_init() };
2423 todo!()
2424}
2425</code></pre>
2426<p><code>EnumProcessModules</code> takes a pointer to an array of <code>HMODULE</code>. We could use a <code>Vec</code> of capacity one to hold the single module, but in memory, a pointer a single item can be seen as a pointer to an array of items. <code>MaybeUninit</code> helps us reserve enough memory for the one item we need.</p>
2427<p>With the module handle, we can retrieve its base name:</p>
2428<pre><code class="language-rust" data-lang="rust">let mut buffer = Vec::&lt;u8&gt;::with_capacity(64);
2429// SAFETY: the handle, module and buffer are all valid.
2430let length = unsafe {
2431 winapi::um::psapi::GetModuleBaseNameA(
2432 self.handle.as_ptr(),
2433 module,
2434 buffer.as_mut_ptr().cast(),
2435 buffer.capacity() as u32,
2436 )
2437};
2438if length == 0 {
2439 return Err(io::Error::last_os_error());
2440}
2441
2442// SAFETY: the call succeeded and length represents bytes.
2443unsafe { buffer.set_len(length as usize) };
2444Ok(String::from_utf8(buffer).unwrap())
2445</code></pre>
2446<p>Similar to how we did with <code>EnumProcesses</code>, we create a buffer that will hold the ASCII string of the module's base name<sup class="footnote-reference"><a href="#5">5</a></sup>. The call wants us to pass a pointer to a mutable buffer of <code>i8</code>, but Rust's <code>String::from_utf8</code> wants a <code>Vec&lt;u8&gt;</code>, so instead we declare a buffer of <code>u8</code> and <code>.cast()</code> the pointer in the call. You could also do this with <code>as _</code>, and Rust would infer the right type, but <code>cast</code> is neat.</p>
2447<p>We <code>unwrap</code> the creation of the UTF-8 string because the buffer should contain only ASCII characters (which are also valid UTF-8). We could use the <code>unsafe</code> variant to create the string, but what if somehow it contains non-ASCII characters? The less <code>unsafe</code>, the better.</p>
2448<p>Let's see it in action:</p>
2449<pre><code class="language-rust" data-lang="rust">fn main() {
2450 enum_proc()
2451 .unwrap()
2452 .into_iter()
2453 .for_each(|pid| match Process::open(pid) {
2454 Ok(proc) =&gt; match proc.name() {
2455 Ok(name) =&gt; println!(&quot;{}: {}&quot;, pid, name),
2456 Err(e) =&gt; println!(&quot;{}: (failed to get name: {})&quot;, pid, e),
2457 },
2458 Err(e) =&gt; eprintln!(&quot;failed to open {}: {}&quot;, pid, e),
2459 });
2460}
2461</code></pre>
2462<pre><code>&gt;cargo run
2463 Compiling memo v0.1.0
2464 Finished dev [unoptimized + debuginfo] target(s) in 0.32s
2465 Running `target\debug\memo.exe`
2466failed to open 0: The parameter is incorrect. (os error 87)
2467failed to open 4: Access is denied. (os error 5)
2468...
2469failed to open 5940: Access is denied. (os error 5)
24705608: (failed to get name: Access is denied. (os error 5))
2471...
24721704: (failed to get name: Access is denied. (os error 5))
2473failed to open 868: Access is denied. (os error 5)
2474...
2475</code></pre>
2476<p>That's not good. What's up with that? Maybe…</p>
2477<blockquote>
2478<p>The handle must have the <code>PROCESS_QUERY_INFORMATION</code> and <code>PROCESS_VM_READ</code> access rights.</p>
2479</blockquote>
2480<p>…I should've read the documentation. Okay, fine:</p>
2481<pre><code class="language-rust" data-lang="rust">use winapi::um::winnt;
2482OpenProcess(winnt::PROCESS_QUERY_INFORMATION | winnt::PROCESS_VM_READ, ...)
2483</code></pre>
2484<pre><code>&gt;cargo run
2485 Compiling memo v0.1.0 (C:\Users\L\Desktop\memo)
2486 Finished dev [unoptimized + debuginfo] target(s) in 0.35s
2487 Running `target\debug\memo.exe`
2488failed to open 0: The parameter is incorrect. (os error 87)
2489failed to open 4: Access is denied. (os error 5)
2490...
24919348: cheatengine-x86_64.exe
24923288: Tutorial-x86_64.exe
24938396: cmd.exe
24944620: firefox.exe
24957964: cargo.exe
249610052: cargo.exe
24975756: memo.exe
2498</code></pre>
2499<p>Hooray 🎉! There's some processes we can't open, but that's because they're system processes. Security works!</p>
2500<h2 id="finale">Finale</h2>
2501<p>That was a fairly long post when all we did was print a bunch of pids and their corresponding name. But in all fairness, we also laid out a good foundation for what's coming next.</p>
2502<p>You can <a href="https://github.com/lonami/memo">obtain the code for this post</a> over at my GitHub. At the end of every post, the last commit will be tagged, so you can <code>git checkout step1</code> to see the final code for any blog post.</p>
2503<p>In the <a href="/blog/woce-2">next post</a>, we'll tackle the second step of the tutorial: Exact Value scanning.</p>
2504<h3 id="footnotes">Footnotes</h3>
2505<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
2506<p>You could say I simply love reinventing the wheel, which I do, but in this case, the codebase contains <em>far</em> more features than we're interested in. The (apparent) lack of structure and documentation regarding the code, along with the unfortunate <a href="https://github.com/cheat-engine/cheat-engine/issues/60">lack of license</a> for the source code, make it a no-go. There's a license, but I think that's for the distributed program itself.</p>
2507</div>
2508<div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup>
2509<p>If it turns out that there are more than 1024 processes, our code will be unaware of those extra processes. The documentation suggests to perform the call again with a larger buffer if <code>count == provided capacity</code>, but given I have under 200 processes on my system, it seems unlikely we'll reach this limit. If you're worried about hitting this limit, simply use a larger limit or retry with a larger vector.</p>
2510</div>
2511<div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup>
2512<p>C code would likely use <a href="https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-globalalloc"><code>GlobalAlloc</code></a> here, but Rust's <code>Vec</code> handles the allocation for us, making the code both simpler and more idiomatic. In general, if you see calls to <code>GlobalAlloc</code> when porting some code to Rust, you can probably replace it with a <code>Vec</code>.</p>
2513</div>
2514<div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup>
2515<p>This will be a recurring theme.</p>
2516</div>
2517<div class="footnote-definition" id="5"><sup class="footnote-definition-label">5</sup>
2518<p>…and similar to <code>EnumProcesses</code>, if the name doesn't fit in our buffer, the result will be truncated.</p>
2519</div>
2520</content>
2521 </entry>
2522 <entry xml:lang="en">
2523 <title>Data Mining, Warehousing and Information Retrieval</title>
2524 <published>2020-07-03T00:00:00+00:00</published>
2525 <updated>2020-07-03T00:00:00+00:00</updated>
2526 <link href="https://lonami.dev/blog/university/" type="text/html"/>
2527 <id>https://lonami.dev/blog/university/</id>
2528 <content type="html"><p>During university, there were a few subjects where I had to write blog posts for (either as evaluable tasks or just for fun). I thought it was really fun and I wanted to preserve that work here, with the hopes it's interesting to someone.</p>
2529<p>The posts series were auto-generated from the original HTML files and manually anonymized later.</p>
2530<ul>
2531<li><a href="/blog/mdad">Data Mining and Data Warehousing</a></li>
2532<li><a href="/blog/ribw">Information Retrieval and Web Search</a></li>
2533</ul>
2534</content>
2535 </entry>
2536 <entry xml:lang="en">
2537 <title>My new computer</title>
2538 <published>2020-06-19T00:00:00+00:00</published>
2539 <updated>2020-07-03T00:00:00+00:00</updated>
2540 <link href="https://lonami.dev/blog/new-computer/" type="text/html"/>
2541 <id>https://lonami.dev/blog/new-computer/</id>
2542 <content type="html"><p>This post will be mostly me ranting about setting up a new laptop, but I also just want to share my upgrade. If you're considering installing Arch Linux with dual-boot for Windows, maybe this post will help. Or perhaps you will learn something new to troubleshoot systems in the future. Let's begin!</p>
2543<p>Last Sunday, I ordered a Asus Rog Strix G531GT-BQ165 for 900€ (on a 20% discount) with the following specifications:</p>
2544<ul>
2545<li>Intel® Core i7-9750H (6 cores, 12MB cache, 2.6GHz up to 4.5GHz, 64-bit)</li>
2546<li>16GB RAM (8GB*2) DDR4 2666MHz</li>
2547<li>512GB SSD M.2 PCIe® NVMe</li>
2548<li>Display 15.6&quot; (1920x1080/16:9) 60Hz</li>
2549<li>Graphics NVIDIA® GeForce® GTX1650 4GB GDDR5 VRAM</li>
2550<li>LAN 10/100/1000</li>
2551<li>Wi-Fi 5 (802.11ac) 2x2 RangeBoost</li>
2552<li>Bluetooth 5.0</li>
2553<li>48Wh battery with 3 cells</li>
2554<li>3 x USB 3.1 (GEN1)</li>
2555</ul>
2556<p>I was mostly interested in a general upgrade (better processor, disk, more RAM), although the graphics card is a really nice addition which will allow me to take some time off on more games. After using it for a bit, I really love the feel of the keyboard, and I love the lack of numpad! (No sarcasm, I really don't like numpads.)</p>
2557<p>This is an upgrade from my previous laptop (Asus X554LA-XX822T), which I won in a competition before entering university in a programming challenge. It has served me really well for the past five years, and had the following specifications:</p>
2558<ul>
2559<li>Intel® Core™ i5-5200U</li>
2560<li>4GB RAM DDR3L 1600MHz (which I upgraded to have 8GB)</li>
2561<li>1TB HDD</li>
2562<li>Display 15.6&quot; (1366x768/16:9)</li>
2563<li>Intel® HD Graphics 4400</li>
2564<li>LAN 10/100/1000</li>
2565<li>Wifi 802.11 bgn</li>
2566<li>Bluetooth 4.0</li>
2567<li>Battery 2 cells</li>
2568<li>1 x USB 2.0</li>
2569<li>2 x USB 3.0</li>
2570</ul>
2571<p>Prior to this one, I had a Lenovo (also won in the same competition of the previous year), and prior to that (just for the sake of history), it was HP Pavilion, AMD A4-3300M processor, which unfortunately ended with heating problems. But that's very old now.</p>
2572<h2 id="laptop-arrival">Laptop arrival</h2>
2573<p>The laptop arrived 2 days ago at roughly 19:00, which I put charged for 3 hours as the book said. The day after, nightmares began!</p>
2574<p>Trying to boot it the first two times was fun, as it comes with a somewhat loud sound on boot. I don't know why they would do this, and I immediately turned it off in the BIOS.</p>
2575<h2 id="installation-journey">Installation journey</h2>
2576<p>I spent all of yesterday trying to setup Windows and Arch Linux (and didn't even finish, it took me this morning too and even now it's only half functional). I absolutely <em>hate</em> the amount of partitions the Windows installer creates on a clean disk. So instead, I first went with Arch Linux, and followed the <a href="https://wiki.archlinux.org/index.php/Installation_guide">installation guide on the Arch wiki</a>. Pre-installation, setting up the wireless network, creating the partitions and formatting them went all good. I decided to avoid GRUB at first and go with rEFInd, but alas I missed a big warning on the wiki and after reboot (I would later find out) it was not mounting root properly, so all I had was whatever was in the Initramfs. Reboot didn't work, so I had to hold the power button.</p>
2577<p>Anyway, once the partitions were created, I went to install Windows (there was a lot of back and forth burning different <code>.iso</code> images on the USB, which was a bit annoying because it wasn't the fastest thing in the world). This was pretty painless, and the process was standard: select advanced to let me choose the right partition, pick the one, say &quot;no&quot; to everything in the services setup, and done. But this was the first Windows <code>.iso</code> I tried. It was an old revision, and the drivers were causing issues when running (something weird about their <code>.dll</code>, manually installing the <code>.ini</code> driver files seemed to work?). The Nvidia drivers didn't want to be installed on such an old revision, after updating everything I could via Windows updates. So back I went to burning a newer Windows <code>.iso</code> and going through the same process again…</p>
2578<p>Once Windows was ready and I verified that I could boot to it correctly, it was time to have a second go at Arch Linux. And I went through the setup at least three times, getting it wrong every single time, formatting root every single time, redownloading the packages every single pain. If only had I known earlier what the issue was!</p>
2579<p>Why bother with Arch? I was pretty happy with Linux Mint, and I lowkey wanted to try NixOS, but I had used Arch before and it's a really nice distro overall (up-to-date, has AUR, quite minimal, imperative), except for trying to install rEFInd while chrooted…</p>
2580<p>In the end I managed to get something half-working, I still need to properly configure WiFi and pulseaudio in my system but hey it works.</p>
2581<p>I like to be able to dual-boot Windows and Linux because Linux is amazing for productivity, but unfortunately, some games only work fine on Windows. Might as well have both systems and use one for gaming, while the other is my daily driver.</p>
2582<h2 id="setting-up-arch-linux">Setting up Arch Linux</h2>
2583<p>This is the process I followed to install Arch Linux in the end, along with a brief explanation on what I think the things are doing and why we are doing them. I think the wiki could do a better job at this, but I also know it's hard to get it right for everyone. Something I do dislike is the link colour, after opening a link it becomes gray and it's a lot easier to miss the fact that it is a link in the first place, which was tough when re-reading it because some links actually matter a lot. Furthermore, important information may just be a single line, also easy to skim over. Anyway, on to the installation process…</p>
2584<p>The first thing we want to do is configure our keyboard layout or else the keys won't correspond to what we expect:</p>
2585<pre><code class="language-sh" data-lang="sh">loadkeys es
2586</code></pre>
2587<p>Because we're on a recent system, we want to verify that UEFI works correctly. If we see files listed, then it works fine:</p>
2588<pre><code class="language-sh" data-lang="sh">ls /sys/firmware/efi/efivars
2589</code></pre>
2590<p>The next thing we want to do is configure the WiFi, because I don't have any ethernet cable nearby. To do this, we check what network interfaces our laptop has (we're looking for the one prefixed with &quot;w&quot;, presumably for wireless, such as &quot;wlan0&quot; or &quot;wlo1&quot;), we set it up, scan for available wireless network, and finally connect. In my case, the network has WPA security so we rely on <code>wpa_supplicant</code> to connect, passing the SSID (network name) and password:</p>
2591<pre><code class="language-sh" data-lang="sh">ip link
2592ip link set &lt;IFACE&gt; up
2593iw dev &lt;IFACE&gt; scan | less
2594wpa_supplicant -B -i &lt;IFACE&gt; -c &lt;(wpa_passphrase &lt;SSID&gt; &lt;PASS&gt;)
2595</code></pre>
2596<p>After that's done, pinging an IP address like &quot;1.1.1.1&quot; should Just Work™, but to be able to resolve hostnames, we need to also setup a nameserver. I'm using Cloudflare's, but you could use any other:</p>
2597<pre><code class="language-sh" data-lang="sh">echo nameserver 1.1.1.1 &gt; /etc/resolv.conf
2598ping archlinux.org
2599^C
2600</code></pre>
2601<p>If the ping works, then network works! If you still have issues, you may need to <a href="https://wiki.archlinux.org/index.php/Network_configuration#Static_IP_address">manually configure a static IP address</a> and add a route with the address of your, well, router. This basically shows if we have any address, adds a static address (so people know who we are), shows what route we have, and adds a default one (so our packets know where to go):</p>
2602<pre><code class="language-sh" data-lang="sh">ip address show
2603ip address add &lt;YOUR ADDR&gt;/24 broadcast + dev &lt;IFACE&gt;
2604ip route show
2605ip route add default via &lt;ROUTER ADDR&gt; dev &lt;IFACE&gt;
2606</code></pre>
2607<p>Now that we have network available, we can enable NTP to synchronize our system time (this may be required for network operations where certificates have a validity period, not sure; in any case nobody wants a wrong system time):</p>
2608<pre><code class="language-sh" data-lang="sh">timedatectl set-ntp true
2609</code></pre>
2610<p>After that, we can manage our disk and partitions using <code>fdisk</code>. We want to define partitions to tell the system where it should live. To determine the disk name, we first list them, and then edit it. <code>fdisk</code> is really nice and reminds you at every step that help can be accessed with &quot;m&quot;, which you should constantly use to guide you through.</p>
2611<pre><code class="language-sh" data-lang="sh">fdisk -l
2612fdisk /dev/&lt;DISK&gt;
2613</code></pre>
2614<p>The partitions I made are the following:</p>
2615<ul>
2616<li>A 100MB one for the EFI system.</li>
2617<li>A 32GB one for Linux' root <code>/</code> partition.</li>
2618<li>A 200GB one for Linux' home <code>/home</code> partition.</li>
2619<li>The rest was unallocated for Windows because I did this first.</li>
2620</ul>
2621<p>I like to have <code>/home</code> and <code>/</code> separate because I can reinstall root without losing anything from home (projects, music, photos, screenshots, videos…).</p>
2622<p>After the partitions are made, we format them in FAT32 and EXT4 which are good defaults for EFI, root and home. They need to have a format, or else they won't be usable:</p>
2623<pre><code class="language-sh" data-lang="sh">mkfs.fat -F32 /dev/&lt;DISK&gt;&lt;PART1&gt;
2624mkfs.ext4 /dev/&lt;DISK&gt;&lt;PART2&gt;
2625mkfs.ext4 /dev/&lt;DISK&gt;&lt;PART3&gt;
2626</code></pre>
2627<p>Because the laptop was new, there was no risk to lose anything, but if you're doing a install on a previous system, be very careful with the partition names. Make sure they match with the ones in <code>fdisk -l</code>.</p>
2628<p>Now that we have usable partitions, we need to mount them or they won't be accessible. We can do this with <code>mount</code>:</p>
2629<pre><code class="language-sh" data-lang="sh">mount /dev/&lt;DISK&gt;&lt;PART2&gt; /mnt
2630mkdir /mnt/efi
2631mount /dev/&lt;DISK&gt;&lt;PART1&gt; /mnt/efi
2632mkdir /mnt/home
2633mount /dev/&lt;DISK&gt;&lt;PART3&gt; /mnt/home
2634</code></pre>
2635<p>Remember to use the correct partitions while mounting. We mount everything so that the system knows which partitions we care about, which we will let know about later on.</p>
2636<p>Next step is to setup the basic Arch Linux system on root, which can be done with <code>pacstrap</code>. What follows the directory is a list of packages, and you may choose any you wish (at least add <code>base</code>, <code>linux</code> and <code>linux-firmware</code>). These can be installed later, but I'd recommend having them from the beginning, just in case:</p>
2637<pre><code class="language-sh" data-lang="sh">pacstrap /mnt base linux linux-firmware sudo vim-minimal dhcpcd wpa_supplicant man-db man-pages intel-ucode grub efibootmgr os-prober ntfs-3g
2638</code></pre>
2639<p>Because my system has an intel CPU, I also installed <code>intel-ucode</code>.</p>
2640<p>Next up is generating the <code>fstab</code> file, which we tell to use UUIDs to be on the safe side through <code>-U</code>. This file is important, because without it the system won't know what partitions exist and will happily only boot with the initramfs, without anything of what we just installed at root. Not knowing this made me restart the entire installation process a few times.</p>
2641<pre><code class="language-sh" data-lang="sh">genfstab -U /mnt &gt;&gt; /mnt/etc/fstab
2642</code></pre>
2643<p>After that's done, we can change our root into our mount point and finish up configuration. We setup our timezone (so DST can be handled correctly if needed), synchronize the hardware clock (to persist the current time to the BIOS), uncomment our locales (exit <code>vim</code> by pressing ESC, then type <code>:wq</code> and press enter), generate locale files (which some applications need), configure language and keymap, update the hostname of our laptop and what indicate what <code>localhost</code> means…</p>
2644<pre><code class="language-sh" data-lang="sh">ln -sf /usr/share/zoneinfo/&lt;REGION&gt;/&lt;CITY&gt; /etc/localtime
2645hwclock --systohc
2646vim /etc/locale.gen
2647locale-gen
2648echo LANG=es_ES.UTF-8 &gt; /etc/locale.conf
2649echo KEYMAP=es &gt; /etc/vconsole.conf
2650echo &lt;HOST&gt; /etc/hostname
2651cat &lt;&lt;EOF &gt; /etc/hosts
2652127.0.0.1 localhost
2653::1 localhost
2654127.0.1.1 &lt;HOST&gt;.localdomain &lt;HOST&gt;
2655EOF
2656</code></pre>
2657<p>Really, we could've done all of this later, and the same goes for setting root's password with <code>passwd</code> or creating users (some of the groups you probably want are <code>power</code> and <code>wheel</code>).</p>
2658<p>The important part here is installing GRUB (which also needed the <code>efibootmgr</code> package):</p>
2659<pre><code class="language-sh" data-lang="sh">grub-install --target=x86_64-efi --efi-directory=/efi --bootloader-id=GRUB
2660</code></pre>
2661<p>If we want GRUB to find our Windows install, we also need the <code>os-prober</code> and <code>ntfs-3g</code> packages that we installed earlier with <code>pacstrap</code>, and with those we need to mount the Windows partition somewhere. It doesn't matter where. With that done, we can generate the GRUB configuration file which lists all the boot options:</p>
2662<pre><code class="language-sh" data-lang="sh">mkdir /windows
2663mount /dev/&lt;DISK&gt;&lt;PART5&gt; /windows
2664grub-mkconfig -o /boot/grub/grub.cfg
2665</code></pre>
2666<p>(In my case, I installed Windows before completing the Arch install, which created an additional partition in between).</p>
2667<p>With GRUB ready, we can exit the chroot and reboot the system, and if all went well, you should be greeted with a choice of operating system to use:</p>
2668<pre><code class="language-sh" data-lang="sh">exit
2669reboot
2670</code></pre>
2671<p>If for some reason you need to find what mountpoints were active prior to rebooting (to <code>unmount</code> them for example), you can use <code>findmnt</code>.</p>
2672<p>Before GRUB I tried rEFInd, which as I explained had issues with for missing a warning. Then I tried systemd-boot, which did not pick up Arch at first. That's where the several reinstalls come from, I didn't want to work with a half-worked system so I mostly redid the entire process quite a few times.</p>
2673<h2 id="migrating-to-the-new-laptop">Migrating to the new laptop</h2>
2674<p>I had a external disk formatted with NTFS. Of course, after moving every file I cared about from my previous Linux install caused all the permissions to reset. All my <code>.git</code> repositories, dirty with file permission changes! This is going to take a while to fix, or maybe I should just <code>git config core.fileMode false</code>. Here is a <a href="https://stackoverflow.com/a/2083563">lovely command</a> to sort them out on a per-repository basis:</p>
2675<pre><code class="language-sh" data-lang="sh">git diff --summary | grep --color 'mode change 100644 =&gt; 100755' | cut -d' ' -f7- | xargs -d'\n' chmod -x
2676</code></pre>
2677<p>I never realized how much I had stored over the years, but it really was a lot. While moving things to the external disk, I tried to do some cleanup, such as removing some build artifacts which needlessly occupy space, or completely skipping all the binary application files. If I need those I will install them anyway. The process was mostly focused on finding all the projects and program data that I did care about, or even some game saves. Nothing too difficult, but definitely time consuming.</p>
2678<h2 id="tuning-arch">Tuning Arch</h2>
2679<p>Now that our system is ready, install <code>pacman-contrib</code> to grab a copy of the <code>rankmirrors</code> speed. It should help speed up the download of whatever packages you want to install, since it will help us <a href="https://wiki.archlinux.org/index.php/Mirrors#List_by_speed">rank the mirrors by download speed</a>. Making a copy of the file is important, otherwise whenever you try to install something it will fail saying it can't find anything.</p>
2680<pre><code class="language-sh" data-lang="sh">cp /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.backup
2681sed -i 's/^#Server/Server/' /etc/pacman.d/mirrorlist.backup
2682rankmirrors -n 6 /etc/pacman.d/mirrorlist.backup | tee /etc/pacman.d/mirrorlist
2683</code></pre>
2684<p>This will take a while, but it should be well worth it. We're using <code>tee</code> to see the progress as it goes.</p>
2685<p>Some other packages I installed after I had a working system in no particular order:</p>
2686<ul>
2687<li><code>xfce4</code> and <code>xorg-server</code>. I just love the simplicity of XFCE.</li>
2688<li><code>xfce4-whiskermenu-plugin</code>, a really nice start menu.</li>
2689<li><code>xfce4-pulseaudio-plugin</code> and <code>pavucontrol</code>, to quickly adjust the audio with my mouse.</li>
2690<li><code>xfce4-taskmanager</code>, a GUI alternative I generally prefer to <code>htop</code>.</li>
2691<li><code>pulseaudio</code> and <code>pulseaudio-alsa</code> to get nice integration with XFCE4 and audio mixing.</li>
2692<li><code>firefox</code>, which comes with fonts too. A really good web browser.</li>
2693<li><code>git</code>, to commit <del>crimes</del> code.</li>
2694<li><code>code</code>, a wonderful editor which I used to write this blog entry.</li>
2695<li><code>nano</code>, so much nicer to write a simple commit message.</li>
2696<li><code>python</code> and <code>python-pip</code>, my favourite language to toy around ideas or use as a calculator.</li>
2697<li><code>telegram-desktop</code>, for my needs on sharing memes.</li>
2698<li><code>cmus</code> and <code>mpv</code>, a simple terminal music player and media player.</li>
2699<li><code>openssh</code>, to connect into any VPS I have access to.</li>
2700<li><code>base-devel</code>, necessary to build most projects I'll find myself working with (or even compiling some projects Rust which I installed via <code>rustup</code>).</li>
2701<li><code>flac</code>, <code>libmad</code>, <code>opus</code>, and <code>libvorbis</code>, to be able to play more audio files.</li>
2702<li><code>inkscape</code>, to make random drawings.</li>
2703<li><code>ffmpeg</code>, to convert media or record screen.</li>
2704<li><code>xclip</code>, to automatically copy screenshots to my clipboard.</li>
2705<li><code>gvfs</code>, needed by Thunar to handle mounting and having a trash (perma-deletion by default can be nasty sometimes).</li>
2706<li><code>noto-fonts</code>, <code>noto-fonts-cjk</code>, <code>noto-fonts-extra</code> and <code>noto-fonts-emoji</code>, if you don't want missing gliphs everywhere.</li>
2707<li><code>xfce4-notifyd</code> and <code>libnotify</code>, for notifications.</li>
2708<li><code>cronie</code>, to be able to <code>crontab -e</code>. Make sure to <code>system enable cronie</code>.</li>
2709<li><code>xarchiver</code> (with <code>p7zip</code>, <code>zip</code>, <code>unzip</code> and <code>unrar</code>) to uncompress stuff.</li>
2710<li><code>xreader</code> to read <code>.pdf</code> files.</li>
2711<li><code>sqlitebrowser</code> is always nice to tinker around with SQLite databases.</li>
2712<li><code>jre8-openjdk</code> if you want to run Java applications.</li>
2713<li><code>smartmontools</code> is nice with a SSD to view your disk statistics.</li>
2714</ul>
2715<p>After that, I configured my Super L key to launch <code>xfce4-popup-whiskermenu</code> so that it opens the application menu, pretty much the same as it would on Windows, moved the panels around and configured them to my needs, and it feels like home once more.</p>
2716<p>I made some mistakes while <a href="https://wiki.archlinux.org/index.php/Systemd-networkd">configuring systemd-networkd</a> and accidentally added a service that was incorrect, which caused boot to wait for it to timeout before completing. My boot time was taking 90 seconds longer because of this! <a href="https://www.reddit.com/r/archlinux/comments/4nv9yi/my_arch_greets_me_now_with_a_start_job/">The solution was to remove said service</a>, so this is something to look out for.</p>
2717<p>In order to find what was taking long, I had to edit the <a href="https://wiki.archlinux.org/index.php/kernel_parameters">kernel parameters</a> to remove the <code>quiet</code> option. I prefer seeing the output on what my computer is doing anyway, because it gives me a sense of progress and most importantly is of great value when things go wrong. Another interesting option is <code>noauto,x-systemd.automount</code>, which makes a disk lazily-mounted. If you have a slow disk, this could help speed things up.</p>
2718<p>If you see a service taking long, you can also use <code>systemd-analyze blame</code> to see what takes the longest, and <code>systemctl list-dependencies</code> is also helpful to find what services are active.</p>
2719<p>My <code>locale charmap</code> was spitting out a bunch of warnings:</p>
2720<pre><code class="language-sh" data-lang="sh">$ locale charmap
2721locale: Cannot set LC_CTYPE to default locale: No such file or directory
2722locale: Cannot set LC_MESSAGES to default locale: No such file or directory
2723locale: Cannot set LC_ALL to default locale: No such file or directory
2724ANSI_X3.4-1968
2725</code></pre>
2726<p>…ANSI encoding? Immediately I added the following to <code>~/.bashrc</code> and <code>~/.profile</code>:</p>
2727<pre><code class="language-sh" data-lang="sh">export LC_ALL=en_US.UTF-8
2728export LANG=en_US.UTF-8
2729export LANGUAGE=en_US.UTF-8
2730</code></pre>
2731<p>For some reason, I also had to edit <code>xfce4-terminal</code>'s preferences in advanced to change the default character encoding to UTF-8. This also solved my issues with pasting things into the terminal, and also proper rendering! I guess pastes were not working because it had some characters that could not be encoded.</p>
2732<p>To have working notifications, I added the following to <code>~/.bash_profile</code> after <code>exec startx</code>:</p>
2733<pre><code class="language-sh" data-lang="sh">systemctl --user start xfce4-notifyd.service
2734</code></pre>
2735<p>I'm pretty sure there's a better way to do this, or maybe it's not even necessary, but this works for me.</p>
2736<p>Some of the other things I had left to do was setting up <code>sccache</code> to speed up Rust builds:</p>
2737<pre><code class="language-sh" data-lang="sh">cargo install sccache
2738echo export RUSTC_WRAPPER=sccache &gt;&gt; ~/.bashrc
2739</code></pre>
2740<p>Once I had <code>cargo</code> ready, installed <code>hacksaw</code> and <code>shotgun</code> with it to perform screenshots.</p>
2741<p>I also disabled the security delay when downloading files in Firefox because it's just annoying, in <code>about:config</code> setting <code>security.dialog_enable_delay</code> to <code>0</code>, and added the <a href="https://alisdair.mcdiarmid.org/kill-sticky-headers/">Kill sticky headers</a> to my bookmarks (you may prefer <a href="https://github.com/t-mart/kill-sticky">the updated version</a>).</p>
2742<p>The <code>utils-linux</code> comes with a <code>fstrim</code> utility to <a href="https://wiki.archlinux.org/index.php/Solid_state_drive#Periodic_TRIM">trim the SSD weekly</a>, which I want enabled via <code>systemctl enable fstrim.timer</code> (you may also want to <code>start</code> it if you don't reboot often). For more SSD tips, check <a href="https://easylinuxtipsproject.blogspot.com/p/ssd.html">How to optimize your Solid State Drive</a>.</p>
2743<p>If the sound is funky prior to reboot, try <code>pulseaudio --kill</code> and <code>pulseaudio --start</code>, or delete <code>~/.config/pulse</code>.</p>
2744<p>I haven't been able to get the brightness keys to work yet, but it's not a big deal, because scrolling on the power manager plugin of Xfce does work (and also <code>xbacklight</code> works, or writing directly to <code>/sys/class/backlight/*</code>).</p>
2745<h2 id="tuning-windows">Tuning Windows</h2>
2746<p>On the Windows side, I disabled the annoying Windows defender by running (<kbd>Ctrl+R</kbd>) <code>gpedit.msc</code> and editing:</p>
2747<ul>
2748<li><em>Computer Configuration &gt; Administrative Templates &gt; Windows Components &gt; Windows Defender » Turn off Windows Defender » Enable</em></li>
2749<li><em>User Configuration &gt; Administrative Templates &gt; Start Menu and Taskbar » Remove Notifications and Action Center » Enable</em></li>
2750</ul>
2751<p>I also updated the <a href="https://github.com/WindowsLies/BlockWindows/raw/master/hosts"><code>hosts</code> file</a> (located at <code>%windir%\system32\Drivers\etc\hosts</code>) with the hope that it will stop some of the telemetry.</p>
2752<p>Last, to have consistent time on Windows and Linux, I changed the following registry key for a <code>qword</code> with value <code>1</code>:</p>
2753<pre><code>HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation\RealTimeIsUniversal
2754</code></pre>
2755<p>(The key might not exist, but you can create it if that's the case).</p>
2756<p>All this time, my laptop had the keyboard lights on, which have been quite annoying. Apparently, they also can cause <a href="https://www.reddit.com/r/ValveIndex/comments/cm6pos/psa_uninstalldisable_aura_sync_lighting_if_you/">massive FPS drops</a>. I headed over to <a href="https://rog.asus.com/downloads/">Asus Rog downloads</a>, selected Aura Sync…</p>
2757<pre><code class="language-md" data-lang="md"># Not Found
2758
2759The requested URL /campaign/aura/us/Sync.html was not found on this server.
2760
2761Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
2762</code></pre>
2763<p>…great! I'll just find the <a href="https://www.asus.com/campaign/aura/global/">Aura site</a> somewhere else…</p>
2764<pre><code class="language-md" data-lang="md"># ASUS
2765
2766# We'll be back.
2767
2768Hi, our website is temporarily closed for service enhancements.
2769
2770We'll be back shortly.Thank you for your patience!
2771</code></pre>
2772<p>Oh come on. After waiting for the next day, I headed over, downloaded their software, tried to install it and it was an awful experience. It felt like I was purposedly installing malware. It spammed and flashed a lot of <code>cmd</code>'s on screen as if it was a virus. It was stuck at 100% doing that and then, Windows blue-screened with <code>KERNEL_MODE_HEAP_CORRUPTION</code>. Amazing. How do you screw up this bad?</p>
2773<p>Well, at least rebooting worked. I tried to <a href="https://answers.microsoft.com/en-us/windows/forum/all/unable-to-uninstall-asus-aura-sync-utility/e9bec36c-e62f-4773-80be-88fb68dace16">uninstall Aura, but of course that failed</a>. Using the <a href="https://support.microsoft.com/en-us/help/17588/windows-fix-problems-that-block-programs-being-installed-or-removed">troubleshooter to uninstall programs</a> helped me remove most of the crap that was installed.</p>
2774<p>After searching around how to disable the lights (because <a href="https://rog.asus.com/forum/showthread.php?112786-Option-to-Disable-Aura-Lights-on-Strix-G-series-(G531GT)-irrespective-of-OSes">my BIOS did not have this setting</a>), I stumbled upon <a href="https://rog.asus.com/us/innovation/armoury_crate/">&quot;Armoury Crate&quot;</a>. Okay, fine, I will install that.</p>
2775<p>The experience wasn't much better. It did the same thing with a lot of consoles flashing on screen. And of course, it resulted in another blue-screen, this time <code>KERNEL_SECURITY_CHECK_FAILURE</code>. To finish up, the BSOD kept happening as I rebooted the system. <del>Time to reinstall Windows once more.</del> After booting and crashing a few more times I could get into secure mode and perform the reinstall from there, which saved me from burning the <code>.iso</code> again.</p>
2776<p>Asus software might be good, but the software is utter crap.</p>
2777<p>After trying out <a href="https://github.com/wroberts/rogauracore">rogauracore</a> (which didn't list my model), it worked! I could disable the stupid lights from Linux, and <a href="https://gitlab.com/CalcProgrammer1/OpenRGB/-/wikis/home">OpenRGB</a> also works on Windows which may be worth checking out too.</p>
2778<p>Because <code>rougauracore</code> helped me and they linked to <a href="https://github.com/linuxhw/hw-probe/blob/master/README.md#appimage">hw-probe</a>, I decided to <a href="https://linux-hardware.org/?probe=0e3e48c501">run it on my system</a>, with the hopes it is useful for other people.</p>
2779<h2 id="closing-words">Closing words</h2>
2780<p>I hope the installation journey is at least useful to someone, or that you enjoyed reading about it all. If not, sorry!</p>
2781</content>
2782 </entry>
2783 <entry xml:lang="en">
2784 <title>Tips for Outpost</title>
2785 <published>2020-05-10T00:00:00+00:00</published>
2786 <updated>2020-05-22T00:00:00+00:00</updated>
2787 <link href="https://lonami.dev/blog/tips-outpost/" type="text/html"/>
2788 <id>https://lonami.dev/blog/tips-outpost/</id>
2789 <content type="html"><p><a href="https://store.steampowered.com/app/1127110/Outpost/">Outpost</a> is a fun little game by Open Mid Interactive that has popped in recently in my recommended section of Steam, and I decided to give it a try.</p>
2790<p>It's a fun tower-defense game with progression, different graphics and random world generation which makes it quite fun for a few hours. In this post I want to talk about some tips I found useful to get past night 50.</p>
2791<h2 id="build-pattern">Build Pattern</h2>
2792<p>At first, you may be inclined to design a checkerboard pattern like the following, where &quot;C&quot; is the Crystal shrine, &quot;S&quot; is a stone launcher and &quot;B&quot; is a booster:</p>
2793<p><img src="https://lonami.dev/blog/tips-outpost/outpost-bad-pattern.svg" alt="Bad Outpost build pattern" /></p>
2794<p>Indeed, this pattern will apply <strong>4</strong> boosts to every turret, but unfortunately, the other 4 slots of the booster are wasted! This is because boosters are able to power 8 different towers, and you really want to maximize that. Here's a better design:</p>
2795<p><img src="https://lonami.dev/blog/tips-outpost/outpost-good-pattern.svg" alt="Good Outpost build pattern" /></p>
2796<p>The shrine's tower does get boosted, but it's still not really worth it to boost it. This pattern works good, and it's really easy to tile: just repeat the same 3x3 pattern.</p>
2797<p>Nonetheless, we can do better. What if we applied multiple boosters to the same tower while still applying all 8 boosts?</p>
2798<p><img src="https://lonami.dev/blog/tips-outpost/outpost-best-pattern.svg" alt="Best Outpost build pattern" /></p>
2799<p>That's what peak performance looks like. You can actually apply multiple boosters to the same tower, and it works great.</p>
2800<p>Now, is it really worth it building anywhere except around the shrine? Not really. You never know where a boss will come from, so all sides need a lot of defense if you want to stand a chance.</p>
2801<p>The addition of traps in 1.6 is amazing. You want to build these outside your strong &quot;core&quot;, mostly to slow the enemies down so your turrets have more time to finish them off. Don't waste boosters on the traps, and build them at a reasonable distance from the center (the sixth tile is a good spot):</p>
2802<p><img src="https://lonami.dev/blog/tips-outpost/outpost-trap-pattern.svg" alt="Trap Outpost build pattern" /></p>
2803<p>If you gather enough materials, you can build more trap and cannon layers outside, roughly at enough distance to slow them for enough duration until they reach the next layer of traps, and so on. Probably a single gap of &quot;cannon, booster, cannon&quot; is enough between trap layers, just not in the center where you need a lot of fire power.</p>
2804<h2 id="talents">Talents</h2>
2805<p>Talents are the way progression works in the game. Generally, after a run, you will have enough experience to upgrade nearly all talents of roughly the same tier. However, some are worth upgrading more than others (which provide basically no value).</p>
2806<p>The best ones to upgrade are:</p>
2807<ul>
2808<li>Starting supplies. Amazing to get good tools early.</li>
2809<li>Shrine shield. Very useful to hold against tough bosses.</li>
2810<li>Better buildings (cannon, boosters, bed and traps). They're a must to deal the most damage.</li>
2811<li>Better pickaxe. Stone is limited, so better make good use of it.</li>
2812<li>Better chests. They provide an insane amount of resources early.</li>
2813<li>Winter slow. Turrets will have more time to deal damage, it's perfect.</li>
2814<li>More time. Useful if you're running out, although generally you enter nights early after having a good core anyway.</li>
2815<li>More rocks. Similar to a better pickaxe, more stone is always better.</li>
2816</ul>
2817<p>Some decent ones:</p>
2818<ul>
2819<li>In-shrine turret. It's okay to get past the first night without building but not much beyond that.</li>
2820<li>Better axe and greaves. Great to save some energy and really nice quality of life to move around.</li>
2821<li>Tree growth. Normally there's enough trees for this not to be an issue but it can save some time gathering wood.</li>
2822<li>Wisps. They're half-decent since they can provide materials once you max out or max out expensive gear.</li>
2823</ul>
2824<p>Some okay ones:</p>
2825<ul>
2826<li>Extra XP while playing. Generally not needed due to the way XP scales per night, but can be a good boost.</li>
2827<li>Runestones. Not as reliable as chests but some can grant more energy per day.</li>
2828</ul>
2829<p>Some crap ones:</p>
2830<ul>
2831<li>Boosts for other seasons. I mean, winter is already the best, no use there.</li>
2832<li>Bow. The bow is very useless at the moment, it's not worth your experience.</li>
2833<li>More energy per bush. Not really worth hunting for bushes since you will have enough energy to do well.</li>
2834</ul>
2835<h2 id="turrets">Turrets</h2>
2836<p>Always build the highest tier, there's no point in anything lower than that. You will need to deal a lot of damage in a small area, which means space is a premium.</p>
2837<h2 id="boosters">Boosters</h2>
2838<p>If you're very early in the game, I recommend alternating both the flag and torch in a checkerboard pattern where the boosters should go in the pattern above. This way your towers will get extra speed and extra range, which works great.</p>
2839<p>When you're in mid-game (stone launchers, gears and campfires), I do not recommend using campfires. The issue is their range boost is way too long, and the turrets will miss quite a few shots. It's better to put all your power into fire speed for increased DPS, at least near the center. If you manage to build too far out and some of the turrets hardly ever shoot, you may put campfires there.</p>
2840<p>In end-game, of course alternate both of the highest tier upgrades. They are really good, and provide the best benefit / cost ratio.</p>
2841<h2 id="gathering-materials">Gathering Materials</h2>
2842<p>It is <strong>very</strong> important to use all your energy every day! Otherwise it will go to waste, and you will need a lot of materials.</p>
2843<p>As of 1.6, you can mine two things at once if they're close enough! I don't know if this is intended or a bug, but it sure is great.</p>
2844<p>Once you're in mid-game, your stone-based fort should stand pretty well against the nights on its own. After playing for a while you will notice, if your base can defend a boss, then it will have no issue carrying you through the nights until the next boss. You can (and should!) spend the nights gathering materials, but only when you're confident that the night won't run out.</p>
2845<p>Before the boss hits (every fifth night), come back to your base and use all of your materials. This is the next fort upgrade that will carry it the five next nights.</p>
2846<p>You may also speed up time during night, but make sure you use all your energy before hand. And also take care, in the current version of the game speeding up time only speeds up monster movement, not the fire rate or projectile speed of your turrets! This means they will miss more shots and can be pretty dangerous. If you're speeding up time, consider speeding it up for a little bit, then go back to normal until things are more calm, and repeat.</p>
2847<p>If you're in the end-game, try to rush for chests. They provide a huge amount of materials which is really helpful to upgrade all your tools early so you can make sure to get the most out of every rock left in the map.</p>
2848<p>In the end-game, after all stone has been collected, you don't really need to use all of your energy anymore. Just enough to have enough wood to build with the remaining stone. This will also be nice with the bow upgrades, which admitedly can get quite powerful, but it's best to have a strong fort first.</p>
2849<h2 id="season">Season</h2>
2850<p>In my opinion, winter is just the best of the seasons. You don't <em>really</em> need that much energy (it gets tiresome), or extra tree drops, or luck. Slower movement means your turrets will be able to shoot enemies for longer, dealing more damage over time, giving them more chance to take enemies out before they reach the shrine.</p>
2851<p>Feel free to re-roll the map a few times (play and exit, or even restart the game) until you get winter if you want to go for The Play.</p>
2852<h2 id="gear">Gear</h2>
2853<p>In my opinion, you really should rush for the best pickaxe you can afford. Stone is a limited resource that doesn't regrow like trees, so once you run out, it's over. Better to make the best use out of it with a good pickaxe!</p>
2854<p>You may also upgrade your greaves, we all known faster movement is a <em>really</em> nice quality of life improvement.</p>
2855<p>Of course, you will eventually upgrade your axe to chop wood (otherwise it's wasted energy, really), but it's not as much of a priority as the pickaxe.</p>
2856<p>Now, the bow is completely useless. Don't bother with it. Your energy is better spent gathering materials to build permanent turrets that deal constant damage while you're away, and the damage adds up with every extra turret you build.</p>
2857<p>With regards to items you carry (like sword, or helmet), look for these (from best to worst):</p>
2858<ul>
2859<li>Less minion life.</li>
2860<li>Chance to not consume energy.</li>
2861<li>+1 turret damage.</li>
2862<li>Extra energy.</li>
2863<li>+1 drop from trees or stones.</li>
2864<li>+1 free wood or stone per day.</li>
2865</ul>
2866<p>Less minion life, nothing to say. You will need it near end-game.</p>
2867<p>The chance to not consume energy is better the more energy you have. With a 25% chance not to consume energy, you can think of it as 1 extra energy for every 4 energy you have on average.</p>
2868<p>Turret damage is a tough one, it's <em>amazing</em> mid-game (it basically doubles your damage) but falls short once you unlock the cannon where you may prefer other items. Definitely recommended if you're getting started. You may even try to roll it on low tiers by dying on the second night, because it's that good.</p>
2869<p>Extra energy is really good, because it means you can get more materials before it gets too rough. Make sure you have built at least two beds in the first night! This extra energy will pay of for the many nights to come.</p>
2870<p>The problem with free wood or stone per day is that you have, often, five times as much energy per day. By this I mean you can get easily 5 stone every day, which means 5 extra stone, whereas the other would provide just 1 per night. On a good run, you will get around 50 free stone or 250 extra stone. It's a clear winner.</p>
2871<p>In end-game, more quality of life are revealing chests so that you can rush them early, if you like to hunt for them try to make better use of the slot.</p>
2872<h2 id="closing-words">Closing words</h2>
2873<p>I hope you enjoy the game as much as I do! Movement is sometimes janky and there's the occassional lag spikes, but despite this it should provide at least a few good hours of gameplay. Beware however a good run can take up to an hour!</p>
2874</content>
2875 </entry>
2876 <entry xml:lang="en">
2877 <title>Python ctypes and Windows</title>
2878 <published>2019-06-19T00:00:00+00:00</published>
2879 <updated>2019-06-19T00:00:00+00:00</updated>
2880 <link href="https://lonami.dev/blog/ctypes-and-windows/" type="text/html"/>
2881 <id>https://lonami.dev/blog/ctypes-and-windows/</id>
2882 <content type="html"><p><a href="https://www.python.org/">Python</a>'s <a href="https://docs.python.org/3/library/ctypes.html"><code>ctypes</code></a> is quite a nice library to easily load and invoke C methods available in already-compiled <a href="https://en.wikipedia.org/wiki/Dynamic-link_library"><code>.dll</code> files</a> without any additional dependencies. And I <em>love</em> depending on as little as possible.</p>
2883<p>In this blog post, we will walk through my endeavors to use <code>ctypes</code> with the <a href="https://docs.microsoft.com/en-us/windows/desktop/api/">Windows API</a>, and do some cool stuff with it.</p>
2884<p>We will assume some knowledge of C/++ and Python, since we will need to read and write a bit of both. Please note that this post is only an introduction to <code>ctypes</code>, and if you need more information you should consult the <a href="https://docs.python.org/3/library/ctypes.html">Python's documentation for <code>ctypes</code></a>.</p>
2885<p>While the post focuses on Windows' API, the code here probably applies to unix-based systems with little modifications.</p>
2886<h2 id="basics">Basics</h2>
2887<p>First of all, let's learn how to load a library. Let's say we want to load <code>User32.dll</code>:</p>
2888<pre><code class="language-python" data-lang="python">import ctypes
2889
2890ctypes.windll.user32
2891</code></pre>
2892<p>Yes, it's that simple. When you access an attribute of <code>windll</code>, said library will load. Since Windows is case-insensitive, we will use lowercase consistently.</p>
2893<p>Calling a function is just as simple. Let's say you want to call <a href="https://docs.microsoft.com/en-us/windows/desktop/api/winuser/nf-winuser-setcursorpos"><code>SetCursorPos</code></a>, which is defined as follows:</p>
2894<pre><code class="language-c" data-lang="c">BOOL SetCursorPos(
2895 int X,
2896 int Y
2897);
2898</code></pre>
2899<p>Okay, it returns a <code>bool</code> and takes two inputs, <code>x</code> and <code>y</code>. So we can call it like so:</p>
2900<pre><code class="language-python" data-lang="python">ctypes.windll.user32.SetCursorPos(100, 100)
2901</code></pre>
2902<p>Try it! Your cursor will move!</p>
2903<h2 id="funky-stuff">Funky Stuff</h2>
2904<p>We can go a bit more crazy and make it form a spiral:</p>
2905<pre><code class="language-python" data-lang="python">import math
2906import time
2907
2908for i in range(200):
2909 x = int(500 + math.cos(i / 5) * i)
2910 y = int(500 + math.sin(i / 5) * i)
2911 ctypes.windll.user32.SetCursorPos(x, y)
2912 time.sleep(0.05)
2913</code></pre>
2914<p>Ah, it's always so pleasant to do random stuff when programming. Sure makes it more fun.</p>
2915<h2 id="complex-structures">Complex Structures</h2>
2916<p><code>SetCursorPos</code> was really simple. It took two parameters and they both were integers. Let's go with something harder. Let's go with <a href="https://docs.microsoft.com/en-us/windows/desktop/api/winuser/nf-winuser-sendinput"><code>SendInput</code></a>! Emulating input will be a fun exercise:</p>
2917<pre><code class="language-c" data-lang="c">UINT SendInput(
2918 UINT cInputs,
2919 LPINPUT pInputs,
2920 int cbSize
2921);
2922</code></pre>
2923<p>Okay, <code>LPINPUT</code>, what are you? Microsoft likes to prefix types with what they are. In this case, <code>LP</code> stands for &quot;Long Pointer&quot; (I guess?), so <code>LPINPUT</code> is just a Long Pointer to <a href="https://docs.microsoft.com/en-us/windows/desktop/api/winuser/ns-winuser-taginput"><code>INPUT</code></a>:</p>
2924<pre><code class="language-c" data-lang="c">typedef struct tagINPUT {
2925 DWORD type;
2926 union {
2927 MOUSEINPUT mi;
2928 KEYBDINPUT ki;
2929 HARDWAREINPUT hi;
2930 } DUMMYUNIONNAME;
2931} INPUT, *PINPUT, *LPINPUT;
2932</code></pre>
2933<p>Alright, that's new. We have a <code>struct</code> and <code>union</code>, two different concepts. We can define both with <code>ctypes</code>:</p>
2934<pre><code class="language-python" data-lang="python">INPUT_MOUSE = 0
2935INPUT_KEYBOARD = 1
2936INPUT_HARDWARE = 2
2937
2938class INPUT(ctypes.Structure):
2939 _fields_ = [
2940 ('type', ctypes.c_long),
2941 ...
2942 ]
2943</code></pre>
2944<p>Structures are classes that subclass <code>ctypes.Structure</code>, and you define their fields in the <code>_fields_</code> class-level variable, which is a list of tuples <code>(field name, field type)</code>.</p>
2945<p>The C structure had a <code>DWORD type</code>. <code>DWORD</code> is a <code>c_long</code>, and <code>type</code> is a name like any other, which is why we did <code>('type', ctypes.c_long)</code>.</p>
2946<p>But what about the union? It's anonymous, and we can't make anonymous unions (<em>citation needed</em>) with <code>ctypes</code>. We will give it a concrete name and a type.</p>
2947<p>Before defining the union, we need to define its inner structures, <a href="https://docs.microsoft.com/en-us/windows/desktop/api/winuser/ns-winuser-tagmouseinput"><code>MOUSEINPUT</code></a>, <a href="https://docs.microsoft.com/en-us/windows/desktop/api/winuser/ns-winuser-tagkeybdinput"><code>KEYBDINPUT</code></a> and <a href="https://docs.microsoft.com/en-us/windows/desktop/api/winuser/ns-winuser-taghardwareinput"><code>HARDWAREINPUT</code></a>. We won't be using them all, but since they count towards the final struct size (C will choose the largest structure as the final size), we need them, or Windows' API will get confused and refuse to work (personal experience):</p>
2948<pre><code class="language-python" data-lang="python">class MOUSEINPUT(ctypes.Structure):
2949 _fields_ = [
2950 ('dx', ctypes.c_long),
2951 ('dy', ctypes.c_long),
2952 ('mouseData', ctypes.c_long),
2953 ('dwFlags', ctypes.c_long),
2954 ('time', ctypes.c_long),
2955 ('dwExtraInfo', ctypes.POINTER(ctypes.c_ulong))
2956 ]
2957
2958
2959class KEYBDINPUT(ctypes.Structure):
2960 _fields_ = [
2961 ('wVk', ctypes.c_short),
2962 ('wScan', ctypes.c_short),
2963 ('dwFlags', ctypes.c_long),
2964 ('time', ctypes.c_long),
2965 ('dwExtraInfo', ctypes.POINTER(ctypes.c_ulong))
2966 ]
2967
2968
2969class HARDWAREINPUT(ctypes.Structure):
2970 _fields_ = [
2971 ('uMsg', ctypes.c_long),
2972 ('wParamL', ctypes.c_short),
2973 ('wParamH', ctypes.c_short)
2974 ]
2975
2976
2977class INPUTUNION(ctypes.Union):
2978 _fields_ = [
2979 ('mi', MOUSEINPUT),
2980 ('ki', KEYBDINPUT),
2981 ('hi', HARDWAREINPUT)
2982 ]
2983
2984
2985class INPUT(ctypes.Structure):
2986 _fields_ = [
2987 ('type', ctypes.c_long),
2988 ('value', INPUTUNION)
2989 ]
2990</code></pre>
2991<p>Some things to note:</p>
2992<ul>
2993<li>Pointers are defined as <code>ctypes.POINTER(inner type)</code>.</li>
2994<li>The field names can be anything you want. You can make them more &quot;pythonic&quot; if you want (such as changing <code>dwExtraInfo</code> for just <code>extra_info</code>), but I chose to stick with the original naming.</li>
2995<li>The union is very similar, but it uses <code>ctypes.Union</code> instead of <code>ctypes.Structure</code>.</li>
2996<li>We gave a name to the anonymous union, <code>INPUTUNION</code>, and used it inside <code>INPUT</code> with also a made-up name, <code>('value', INPUTUNION)</code>.</li>
2997</ul>
2998<p>Now that we have all the types we need defined, we can use them:</p>
2999<pre><code class="language-python" data-lang="python">KEYEVENTF_KEYUP = 0x0002
3000
3001def press(vk, down):
3002 inputs = INPUT(type=INPUT_KEYBOARD, value=INPUTUNION(ki=KEYBDINPUT(
3003 wVk=vk,
3004 wScan=0,
3005 dwFlags=0 if down else KEYEVENTF_KEYUP,
3006 time=0,
3007 dwExtraInfo=None
3008 )))
3009 ctypes.windll.user32.SendInput(1, ctypes.byref(inputs), ctypes.sizeof(inputs))
3010
3011
3012for char in 'HELLO':
3013 press(ord(char), down=True)
3014 press(ord(char), down=False)
3015</code></pre>
3016<p>Run it! It will press and release the keys <code>hello</code> to type the word <code>&quot;hello&quot;</code>!</p>
3017<p><code>vk</code> stands for &quot;virtual key&quot;. Letters correspond with their upper-case ASCII value, which is what we did above. You can find all the available keys in the page with all the <a href="https://docs.microsoft.com/en-us/windows/desktop/inputdev/virtual-key-codes">Virtual Key Codes</a>.</p>
3018<h2 id="dynamic-inputs-and-pointers">Dynamic Inputs and Pointers</h2>
3019<p>What happens if a method wants something by reference? That is, a pointer to your thing? For example, <a href="https://docs.microsoft.com/en-us/windows/desktop/api/winuser/nf-winuser-getcursorpos"><code>GetCursorPos</code></a>:</p>
3020<pre><code class="language-c" data-lang="c">typedef struct tagPOINT {
3021 LONG x;
3022 LONG y;
3023} POINT, *PPOINT, *NPPOINT, *LPPOINT;
3024
3025BOOL GetCursorPos(
3026 LPPOINT lpPoint
3027);
3028</code></pre>
3029<p>It wants a Long Pointer to <a href="https://docs.microsoft.com/en-us/windows/desktop/api/windef/ns-windef-point"><code>POINT</code></a>. We can do just that with <code>ctypes.byref</code>:</p>
3030<pre><code class="language-python" data-lang="python">class POINT(ctypes.Structure):
3031 _fields_ = [
3032 ('x', ctypes.c_long),
3033 ('y', ctypes.c_long)
3034 ]
3035
3036
3037def get_mouse():
3038 point = POINT()
3039 ctypes.windll.user32.GetCursorPos(ctypes.byref(point))
3040 # pass our point by ref ^^^^^
3041 # this lets GetCursorPos fill its x and y fields
3042
3043 return point.x, point.y
3044
3045
3046while True:
3047 print(get_mouse())
3048 time.sleep(0.05)
3049</code></pre>
3050<p>Now you can track the mouse position! Make sure to <code>Ctrl+C</code> the program when you're tired of it.</p>
3051<p>What happens if a method wants a dynamically-sized input?</p>
3052<pre><code class="language-python" data-lang="python">buffer = ctypes.create_string_buffer(size)
3053</code></pre>
3054<p>In that case, you can create an in-memory <code>buffer</code> of <code>size</code> with <code>ctypes.create_string_buffer</code>. It will return a character array of that size, which you can pass as a pointer directly (without <code>ctypes.byref</code>).</p>
3055<p>To access the buffer's contents, you can use either <code>.raw</code> or <code>.value</code>:</p>
3056<pre><code class="language-python" data-lang="python">entire_buffer_as_bytes = buffer.raw
3057up_until_null = buffer.value
3058</code></pre>
3059<p>When the method fills in the data, you can <code>cast</code> your buffer back into a pointer of a concrete type:</p>
3060<pre><code class="language-python" data-lang="python">result_ptr = ctypes.cast(buffer, ctypes.POINTER(ctypes.c_long))
3061</code></pre>
3062<p>And you can de-reference pointers with <code>.contents</code>:</p>
3063<pre><code class="language-python" data-lang="python">first_result = result_ptr.contents
3064</code></pre>
3065<h2 id="arrays">Arrays</h2>
3066<p>Arrays are defined as <code>type * size</code>. Your linter may not like that, and if you don't know the size beforehand, consider creating a 0-sized array. For example:</p>
3067<pre><code class="language-python" data-lang="python"># 10 longs
3068ten_longs = (ctypes.c_long * 10)()
3069for i in range(10):
3070 ten_longs[i] = 2 ** i
3071
3072# Unknown size of longs, e.g. inside some Structure
3073longs = (ctypes.c_long * 0)
3074
3075# Now you know how many longs it actually was
3076known_longs = ctypes.cast(
3077 ctypes.byref(longs),
3078 ctypes.POINTER(ctypes.c_long * size)
3079).contents
3080</code></pre>
3081<p>If there's a better way to initialize arrays, please let me know.</p>
3082<h2 id="wintypes">wintypes</h2>
3083<p>Under Windows, the <code>ctypes</code> module has a <code>wintypes</code> submodule. This one contains definitions like <code>HWND</code> which may be useful and can be imported as:</p>
3084<pre><code class="language-python" data-lang="python">from ctypes.wintypes import HWND, LPCWSTR, UINT
3085</code></pre>
3086<h2 id="callbacks">Callbacks</h2>
3087<p>Some functions (I'm looking at you, <a href="https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-enumwindows"><code>EnumWindows</code></a>) ask us to pass a callback. In this case, it wants a <a href="https://docs.microsoft.com/en-us/previous-versions/windows/desktop/legacy/ms633498(v=vs.85)"><code>EnumWindowsProc</code></a>:</p>
3088<pre><code class="language-c" data-lang="c">BOOL EnumWindows(
3089 WNDENUMPROC lpEnumFunc,
3090 LPARAM lParam
3091);
3092
3093BOOL CALLBACK EnumWindowsProc(
3094 _In_ HWND hwnd,
3095 _In_ LPARAM lParam
3096);
3097</code></pre>
3098<p>The naive approach won't work:</p>
3099<pre><code class="language-python" data-lang="python">def callback(hwnd, lParam):
3100 print(hwnd)
3101 return True
3102
3103ctypes.windll.user32.EnumWindows(callback, 0)
3104# ctypes.ArgumentError: argument 1: &lt;class 'TypeError'&gt;: Don't know how to convert parameter 1
3105# Aww.
3106</code></pre>
3107<p>Instead, you must wrap your function as a C definition like so:</p>
3108<pre><code class="language-python" data-lang="python">from ctypes.wintypes import BOOL, HWND, LPARAM
3109
3110EnumWindowsProc = ctypes.WINFUNCTYPE(BOOL, HWND, LPARAM)
3111
3112def callback(hwnd, lParam):
3113 print(hwnd)
3114 return True
3115
3116# Wrap the function in the C definition
3117callback = EnumWindowsProc(callback)
3118
3119ctypes.windll.user32.EnumWindows(callback, 0)
3120# Yay, it works.
3121</code></pre>
3122<p>You may have noticed this is what decorators do, wrap the function. So…</p>
3123<pre><code class="language-python" data-lang="python">from ctypes.wintypes import BOOL, HWND, LPARAM
3124
3125@ctypes.WINFUNCTYPE(BOOL, HWND, LPARAM)
3126def callback(hwnd, lParam):
3127 print(hwnd)
3128 return True
3129
3130ctypes.windll.user32.EnumWindows(callback, 0)
3131</code></pre>
3132<p>…will also work. And it is a <em>lot</em> fancier.</p>
3133<h2 id="closing-words">Closing Words</h2>
3134<p>With the knowledge above and some experimentation, you should be able to call and do (almost) anything you want. That was pretty much all I needed on my project anyway :)</p>
3135<p>We have been letting Python convert Python values into C values, but you can do so explicitly too. For example, you can use <code>ctypes.c_short(17)</code> to make sure to pass that <code>17</code> as a <code>short</code>. And if you have a <code>c_short</code>, you can convert or cast it to its Python <code>.value</code> as <code>some_short.value</code>. The same applies for integers, longs, floats, doubles… pretty much anything, char pointers (strings) included.</p>
3136<p>If you can't find something in their online documentation, you can always <a href="https://github.com/BurntSushi/ripgrep"><code>rg</code></a> for it in the <code>C:\Program Files (x86)\Windows Kits\10\Include\*</code> directory.</p>
3137<p>Note that the <code>ctypes.Structure</code>'s that you define can have more methods of your own. For example, you can write them a <code>__str__</code> to easily view its fields, or define a <code>@property</code> to re-interpret some data in a meaningful way.</p>
3138<p>For enumerations, you can pass just the right integer number, make a constant for it, or if you prefer, use a <a href="https://docs.python.org/3/library/enum.html#enum.IntEnum"><code>enum.IntEnum</code></a>. For example, <a href="https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/dism/dismloglevel-enumeration"><code>DismLogLevel</code></a> would be:</p>
3139<pre><code class="language-python" data-lang="python">class DismLogLevel(enum.IntEnum):
3140 DismLogErrors = 0
3141 DismLogErrorsWarnings = 1
3142 DismLogErrorsWarningsInfo = 2
3143</code></pre>
3144<p>And you <em>should</em> be able to pass <code>DismLogLevel.DismLogErrors</code> as the parameter now.</p>
3145<p>If you see a function definition like <code>Function(void)</code>, that's C's way of saying it takes no parameters, so just call it as <code>Function()</code>.</p>
3146<p>Make sure to pass all parameters, even if they seem optional they probably still want a <code>NULL</code> at least, and of course, read the documentation well. Some methods have certain pre-conditions.</p>
3147<p>Have fun hacking!</p>
3148</content>
3149 </entry>
3150 <entry xml:lang="en">
3151 <title>Shattered Pixel Dungeon</title>
3152 <published>2019-06-03T00:00:00+00:00</published>
3153 <updated>2019-06-03T00:00:00+00:00</updated>
3154 <link href="https://lonami.dev/blog/pixel-dungeon/" type="text/html"/>
3155 <id>https://lonami.dev/blog/pixel-dungeon/</id>
3156 <content type="html"><p><a href="https://shatteredpixel.com/shatteredpd/">Shattered Pixel Dungeon</a> is the classic roguelike RPG game with randomly-generated dungeons. As a new player, it was a bit frustrating to be constantly killed on the first levels of the dungeon, but with some practice it's easy to reach high levels if you can kill the first boss.</p>
3157<h2 id="basic-tips">Basic Tips</h2>
3158<p>The game comes with its own tips, but here's a short and straight-forward summary:</p>
3159<ul>
3160<li><strong>Don't rush into enemies</strong>. Abuse doors and small corridors to kill them one by one. You can use the clock on the bottom left to wait a turn without moving.</li>
3161<li><strong>Explore each level at full</strong>. You will find goodies and gain XP while doing so.</li>
3162<li><strong>Upon finding a special room</strong> (e.g. has a chest but is protected by piranhas), drink all potions that you found in that level until there's one that helps you (e.g. be invisible so piranhas leave you alone). There is guaranteed to be a helpful one per level with special rooms.</li>
3163<li><strong>Drink potions as early as possible</strong>. Harmful potions do less damage on early levels (and if you die, you lose less). This will keep them identified early for the rest of the game.</li>
3164<li><strong>Read scrolls as early as possible</strong> as well. This will keep them identified. It may be worth to wait until you have an item which may be cursed and until the level is clear, because some scrolls clean curses and others alert enemies.</li>
3165<li><strong>Food and health are resources</strong> that you have to <em>manage</em>, not keep them always at full. Even if you are starving and taking damage, you may not need to eat <em>just yet</em>, since food is scarce. Eat when you are low on health or in possible danger.</li>
3166<li><strong>Piranhas</strong>. Seriously, just leave them alone if you are melee. They're free food if you're playing ranged, though.</li>
3167<li><strong>Prefer armor over weapons</strong>. And make sure to identify or clean it from curses before wearing anything!</li>
3168<li><strong>Find a dew vial early</strong>. It's often a better idea to store dew (health) for later than to use it as soon as possible.</li>
3169</ul>
3170<h2 id="bosses">Bosses</h2>
3171<p>There is a boss every 5 levels.</p>
3172<ul>
3173<li><strong>Level 5 boss</strong>. Try to stay on water, but don't let <em>it</em> stay on water since it will heal. Be careful when he starts enraging.</li>
3174<li><strong>Level 10 boss</strong>. Ranged weapons are good against it.</li>
3175<li><strong>Level 15 boss</strong>. I somehow managed to tank it with a health potion.</li>
3176<li><strong>Level 20 boss</strong>. I didn't get this far just yet. You are advised to use scrolls of magic mapping in the last levels to skip straight to the boss, since there's nothing else of value.</li>
3177<li><strong>Level 25 boss</strong>. The final boss. Good job if you made it this far!</li>
3178</ul>
3179<h2 id="mage">Mage</h2>
3180<p>If you followed the basic tips, you will sooner or later make use of two scrolls of upgrade in a single run. This will unlock the mage class, which is ridiculously powerful. He starts with a ranged-weapon, a magic missile wand, which is really helpful to keep enemies at a distance. Normally, you want to use this at first to surprise attack them soon, and if you are low on charges, you may go melee on normal enemies if you are confident.</p>
3181<h2 id="luck">Luck</h2>
3182<p>This game is all about luck and patience! Some runs will be better than others, and you should thank and pray the RNG gods for them. If you don't, they will only give you cursed items and not a single scroll to clean them. So, good luck and enjoy playing!</p>
3183</content>
3184 </entry>
3185 <entry xml:lang="en">
3186 <title>Installing NixOS, Take 2</title>
3187 <published>2019-02-15T00:00:00+00:00</published>
3188 <updated>2019-02-16T00:00:00+00:00</updated>
3189 <link href="https://lonami.dev/blog/installing-nixos-2/" type="text/html"/>
3190 <id>https://lonami.dev/blog/installing-nixos-2/</id>
3191 <content type="html"><p>This is my second take at installing NixOS, after a while being frustrated with Arch Linux and the fact that a few kernel upgrades ago, the system crashed randomly from time to time. <code>journalctl</code> did not have any helpful hints and I thought reinstalling could be worthwhile anyway.</p>
3192<p>This time, I started with more knowledge! The first step is heading to the <a href="https://nixos.org">NixOS website</a> and downloading their minimal installation CD for 64 bits. I didn't go with their graphical live CD, because their <a href="https://nixos.org/nixos/manual">installation manual</a> is a wonderful resource that guides you nicely.</p>
3193<p>Once you have downloaded their <code>.iso</code>, you should probably verify it's <code>sha256sum</code> and make sure that it matches. The easiest thing to do in my opinion is using an USB to burn the image in it. Plug it in and check its device name with <code>fdisk -l</code>. In my case, it was <code>/dev/sdb</code>, so I went ahead with it and ran <code>dd if=nixos.iso of=/dev/sdb status=progress</code>. Make sure to run <code>sync</code> once that's done.</p>
3194<p>If either <code>dd</code> or <code>sync</code> seem &quot;stuck&quot; in the end, they are just flushing the changes to disk to make sure all is good. This is normal, and depends on your drives.</p>
3195<p>Now, reboot your computer with the USB plugged in and make sure to boot into it. You should be welcome with a pretty screen. Just select the first option and wait until it logs you in as root. Once you're there you probably want to <code>loadkeys es</code> or whatever your keyboard layout is, or you will have a hard time with passwords, since the characters are all over the place.</p>
3196<p>In a clean disk, you would normally create the partitions now. In my case, I already had the partitions made (100MB for the EFI system, where <code>/boot</code> lives, 40GB for the root <code>/</code> partition with my old Linux installation, and 700G for <code>/home</code>), so I didn't need to do anything here. The manual showcases <code>parted</code>, but I personally use <code>fdisk</code>, which has very helpful help I check every time I use it.</p>
3197<p><strong>Important</strong>: The <code>XY</code> in <code>/dev/sdXY</code> is probably different in your system! Make sure you use <code>fdisk -l</code> to see the correct letters and numbers!</p>
3198<p>With the partitions ready in my UEFI system, I formatted both <code>/</code> and <code>/boot</code> just to be safe with <code>mkfs.ext4 -L nixos /dev/sda2</code> and <code>mkfs.fat -F 32 -n boot /dev/sda1</code> (remember that these are the letters and numbers used in my partition scheme). Don't worry about the warning in the second command regarding lowercase letters and Windows. It's not really an issue.</p>
3199<p>Now, since we gave each partition a label, we can easily mount them through <code>mount /dev/disk/by-label/nixos /mnt</code> and, in UEFI systems, be sure to <code>mkdir -p /mnt/boot</code> and <code>mount /dev/disk/by-label/boot /mnt/boot</code>. I didn't bother setting up swap, since I have 8GB of RAM in my laptop and that's really enough for my use case.</p>
3200<p>With that done, we will now ask the configuration wizard to do some work for us (in particular, generate a template) with <code>nixos-generate-config --root /mnt</code>. This generates a very well documented file that we should edit right now (and this is important!) with whatever editor you prefer. I used <code>vim</code>, but you can change it for <code>nano</code> if you prefer.</p>
3201<p>On to the configuration file, we need to enable a few things, so <code>vim /mnt/etc/nixos/configuration.nix</code> and start scrolling down. We want to make sure to uncomment:</p>
3202<pre><code># We really want network!
3203networking.wireless.enable = true;
3204
3205# This &quot;fixes&quot; the keyboard layout. Put the one you use.
3206i18n = {
3207consoleKeyMap = &quot;es&quot;;
3208}
3209
3210# Timezones are tricky so let's get this right.
3211time.timeZone = &quot;Europe/Madrid&quot;;
3212
3213# We *really* want some base packages installed, such as
3214# wpa_supplicant, or we won't have a way to connect to the
3215# network once we install...
3216environment.systemPackages = with pkgs; [
3217wpa_supplicant wget curl vim neovim cmus mpv firefox git tdesktop
3218];
3219
3220# Printing is useful, sure, enable CUPS
3221services.printing.enable = true;
3222
3223# We have speakers, let's make use of them.
3224sound.enable = true;
3225hardware.pulseaudio.enable = true;
3226
3227# We want the X11 windowing system enabled, in Spanish.
3228services.xserver.enable = true;
3229services.xserver.layout = &quot;es&quot;;
3230
3231# I want a desktop manager in my laptop.
3232# I personally prefer XFCE, but the manual shows plenty
3233# of other options, such as Plasma, i3 WM, or whatever.
3234services.xserver.desktopManager.xfce.enable = true;
3235services.xserver.desktopManager.default = &quot;xfce&quot;;
3236
3237# Touchpad is useful (although sometimes annoying) in a laptop
3238services.xserver.libinput.enable = true;
3239
3240# We don't want to do everything as root!
3241users.users.lonami = {
3242isNormalUser = true;
3243uid = 1000;
3244home = &quot;/home/lonami&quot;;
3245extraGroups = [ &quot;wheel&quot; &quot;networkmanager&quot; &quot;audio&quot; ];
3246};
3247</code></pre>
3248<p><em>(Fun fact, I overlooked the configuration file until I wrote this and hadn't noticed sound/pulseaudio was there. It wasn't hard to find online how to enable it though!)</em></p>
3249<p>Now, let's modify <code>hardware-configuration.nix</code>. But if you have <code>/home</code> in a separate partition like me, you should run <code>blkid</code> to figure out its UUID. To avoid typing it out myself, I just ran <code>blkid &gt;&gt; /mnt/etc/nixos/hardware-configuration.nix</code> so that I could easily move it around with <code>vim</code>:</p>
3250<pre><code># (stuff...)
3251
3252fileSystems.&quot;/home&quot; =
3253{ device = &quot;/dev/disk/by-uuid/d344c686-cae7-4dd3-840e-308eddf86608&quot;;
3254fsType = &quot;ext4&quot;;
3255};
3256
3257# (more stuff...)
3258</code></pre>
3259<p>Note that, obviously, you should put your own partition's UUID there. Modifying the configuration is where I think the current NixOS' manual should have made more emphasis, at this step of the installation. They do detail it below, but that was already too late in my first attempt. Anyway, you can boot from the USB and run <code>nixos-install</code> as many times as you need until you get it working!</p>
3260<p>But before installing, we need to configure the network since there are plenty of things to download. If you want to work from WiFi, you should first figure out the name of your network card with <code>ip link show</code>. In my case it's called <code>wlp3s0</code>. So with that knowledge we can run <code>wpa_supplicant -B -i wlp3s0 -c &lt;(wpa_passphrase SSID key)</code>. Be sure to replace both <code>SSID</code> and <code>key</code> with the name of your network and password key, respectively. If they have spaces, surround them in quotes.</p>
3261<p>Another funny pitfall was typing <code>wpa_supplicant</code> in the command above twice (instead of <code>wpa_passphrase</code>). That sure spit out a few funny errors! Once you have ran that, wait a few seconds and <code>ping 1.1.1.1</code> to make sure that you can reach the internet. If you do, <code>^C</code> and let's install NixOS!</p>
3262<pre><code>nixos-install
3263</code></pre>
3264<p>Well, that was pretty painless. You can now <code>reboot</code> and enjoy your new, functional system.</p>
3265<h2 id="afterword">Afterword</h2>
3266<p>The process of installing NixOS was really painless once you have made sense out of what things mean. I was far more pleased this time than in my previous attempt, despite the four attempts I needed to have it up and running.</p>
3267<p>However not all is so good. I'm not sure where I went wrong, but the first time I tried with <code>i3</code> instead of <code>xfce</code>, all I was welcome with was a white, small terminal in the top left corner. I even generated a configuration file with <code>i3-config-wizard</code> to make sure it could detect my Mod1/Mod4 keys (which, it did), but even after rebooting, my commands weren't responding. For example, I couldn't manage to open another terminal with <code>Mod1+Enter</code>. I'm not even sure that I was in <code>i3</code>…</p>
3268<p>In my very first attempt, I pressed <code>Alt+F8</code> as suggested in the welcome message. This took me an offline copy of the manual, which is really nicely done. Funny enough, though, I couldn't exit <code>w3m</code>. Both <code>Q</code> and <code>B</code> to quit and take me back wouldn't work. Somehow, it kept throwing me back into <code>w3m</code>, so I had to forcibly shutdown.</p>
3269<p>In my second attempt, I also forgot to configure network, so I had no way to download <code>wpa_supplicant</code> without having <code>wpa_supplicant</code> itself to connect my laptop to the network! So, it was important to do that through the USB before installing it (which comes with the program preinstalled), just by making sure to add it in the configuration file.</p>
3270<p>Some other notes, if you can't reach the internet, don't add any DNS in <code>/etc/resolv.conf</code>. This should be done declaratively in <code>configuration.nix</code>.</p>
3271<p>In the end, I spent the entire afternoon playing around with it, taking breaks and what-not. I still haven't figured out why <code>nvim</code> was printing the literal escape character when going from normal to insert mode in the <code>xfce4-terminal</code> (and other actions also made it print this &quot;garbage&quot; to the console), why sometimes the network can reach the internet (and only some sites!) and sometimes not, and how to setup dualboot.</p>
3272<p>But despite all of this, I think it was a worth installing it again. One sure sees things from a different perspective, and gets the chance to write another blog post!</p>
3273<p>If there's something I overlooked or that could be done better, or maybe you can explain it differently, please be sure to <a href="https://lonami.dev/contact">contact me</a> to let me know!</p>
3274<h2 id="update">Update</h2>
3275<p>Well, that was surprisingly fast feedback. Thank you very much <a href="https://bb010g.keybase.pub/">@bb010g</a> for it! As they rightfully pointed out, one can avoid adding <code>/home</code> manually to <code>hardware-configuration.nix</code> if you mount it before generating the configuration files. However, the installation process doesn't need <code>/home</code> mounted, so I didn't do it.</p>
3276<p>The second weird issue with <code>w3m</code> is actually a funny one. <code>Alt+F8</code> <em>switches to another TTY</em>! That's why quitting the program wouldn't do anything. You'd still be in a different TTY! Normally, this is <code>Ctrl+Alt+FX</code>, so I hadn't even thought that this is what could be happening. Anyway, the solution is not quitting the program, but rather going back to the main TTY with <code>Alt+F1</code>. You can switch back and forth all you need to consult the manual.</p>
3277<p>More suggestions are having <a href="https://github.com/rycee/home-manager"><code>home-manager</code></a> manage the graphical sessions, since it should be easier to deal with than the alternatives.</p>
3278<p>Despite having followed the guide and having read it over and over several times, it seems like my thoughts in this blog post may be a bit messy. So I recommend you also reading through the guide to have two versions of all this, just in case.</p>
3279<p>Regarding network issues, they use <code>connman</code> so that may be worth checking out.</p>
3280<p>Regarding terminal issues with <code>nvim</code> printing the literal escape character, I was told off for not having checked what my <code>$TERM</code> was. I hadn't really looked into it much myself, just complained about it here, so sorry for being annoying about that. A quick search in the <code>nixpkgs</code> repository lets us find <a href="https://github.com/NixOS/nixpkgs/blob/release-18.09/pkgs/applications/editors/neovim/default.nix">neovim/default.nix</a>, with version 0.3.1. Looking at <a href="https://github.com/neovim/neovim">Neovim's main repository</a> we can see that this is a bit outdated, but that is fine.</p>
3281<p>If only I had bothered to look at <a href="https://github.com/neovim/neovim/wiki/FAQ#nvim-shows-weird-symbols-2-q-when-changing-modes">Neovim's wiki</a>, (which they found through <a href="https://github.com/neovim/neovim/issues/7749">Neovim's GitHub issues</a>) I would've seen that some terminals just don't support the program properly. The solution is, of course, to use a different terminal emulator with better support or to disable the <code>guicursor</code> in Neovim's config.</p>
3282<p>This is a pretty good life lesson. 30 seconds of searching, maybe two minutes and a half for also checking XFCE issues, are often more than enough to troubleshoot your issues. The internet is a big place and more people have surely came across the problem before, so make sure to look online first. In my defense I'll say that it didn't bother me so much so I didn't bother looking for that soon either.</p>
3283</content>
3284 </entry>
3285 <entry xml:lang="en">
3286 <title>Breaking Risk of Rain</title>
3287 <published>2019-01-12T00:00:00+00:00</published>
3288 <updated>2019-01-12T00:00:00+00:00</updated>
3289 <link href="https://lonami.dev/blog/breaking-ror/" type="text/html"/>
3290 <id>https://lonami.dev/blog/breaking-ror/</id>
3291 <content type="html"><p><a href="https://riskofraingame.com/">Risk of Rain</a> is a fun little game you can spend a lot of hours on. It's incredibly challenging for new players, and fun once you have learnt the basics. This blog will go through what I've learnt and how to play the game correctly.</p>
3292<h2 id="getting-started">Getting Started</h2>
3293<p>If you're new to the game, you may find it frustrating. You must learn very well to dodge.</p>
3294<p>Your first <a href="http://riskofrain.wikia.com/wiki/Category:Characters">character</a> will be <a href="http://riskofrain.wikia.com/wiki/Commando">Commando</a>. He's actually a very nice character. Use your third skill (dodge) to move faster, pass through large groups of enemies, and negate fall damage.</p>
3295<p>If there are a lot of monsters, remember to <strong>leave</strong> from there! It's really important for survival. Most enemies <strong>don't do body damage</strong>. Not even the body of the <a href="http://riskofrain.wikia.com/wiki/Magma_Worm">Magma Worm</a> or the <a href="http://riskofrain.wikia.com/wiki/Wandering_Vagrant">Wandering Vagrant</a> (just dodge the head and projectiles respectively).</p>
3296<p>The first thing you must do is always <strong>rush for the teleporter</strong>. Completing the levels quick will make the game easier. But make sure to take note of <strong>where the chests are</strong>! When you have time (even when the countdown finishes), go back for them and buy as many as you can. Generally, prefer <a href="http://riskofrain.wikia.com/wiki/Chest">chests</a> over <a href="http://riskofrain.wikia.com/wiki/Shrine">shrines</a> since they may eat all your money.</p>
3297<p>Completing the game on <a href="http://riskofrain.wikia.com/wiki/Difficulty">Drizzle</a> is really easy if you follow these tips.</p>
3298<h2 id="requisites">Requisites</h2>
3299<p>Before breaking the game, you must obtain several <a href="http://riskofrain.wikia.com/wiki/Item#Artifacts">artifacts</a>. We are interested in particular in the following:</p>
3300<ul>
3301<li><a href="http://riskofrain.wikia.com/wiki/Sacrifice">Sacrifice</a>. You really need this one, and may be a bit hard to get. With it, you will be able to farm the first level for 30 minutes and kill the final boss in 30 seconds.</li>
3302<li><a href="http://riskofrain.wikia.com/wiki/Command">Command</a>. You need this unless you want to grind for hours to get enough of the items you really need for the rest of the game. Getting this one is easy.</li>
3303<li><a href="http://riskofrain.wikia.com/wiki/Glass">Glass</a>. Your life will be very small (at the beginning…), but you will be able to one-shot everything easily.</li>
3304<li><a href="http://riskofrain.wikia.com/wiki/Kin">Kin</a> (optional). It makes it easier to obtain a lot of boxes if you restart the first level until you get <a href="http://riskofrain.wikia.com/wiki/Lemurian">lemurians</a> or <a href="http://riskofrain.wikia.com/wiki/Jellyfish">jellyfish</a> as the monster, since they're cheap to spawn.</li>
3305</ul>
3306<p>With those, the game becomes trivial. Playing as <a href="http://riskofrain.wikia.com/wiki/Huntress">Huntress</a> is excellent since she can move at high speed while killing everything on screen.</p>
3307<h2 id="breaking-the-game">Breaking the Game</h2>
3308<p>The rest is easy! With the command artifact you want the following items.</p>
3309<h3 id="common-items"><a href="http://riskofrain.wikia.com/wiki/Category:Common_Items">Common Items</a></h3>
3310<ul>
3311<li><a href="http://riskofrain.wikia.com/wiki/Soldier&#x27;s_Syringe">Soldier's Syringe</a>. <strong>Stack 13</strong> of these and you will triple your attack speed. You can get started with 4 or so.</li>
3312<li><a href="http://riskofrain.wikia.com/wiki/Paul&#x27;s_Goat_Hoof">Paul's Goat Hoof</a>. <strong>Stack +30</strong> of these and your movement speed will be insane. You can get a very good speed with 8 or so.</li>
3313<li><a href="http://riskofrain.wikia.com/wiki/Crowbar">Crowbar</a>. <strong>Stack +20</strong> to guarantee you can one-shot bosses.</li>
3314</ul>
3315<p>If you want to be safer:</p>
3316<ul>
3317<li><a href="http://riskofrain.wikia.com/wiki/Hermit&#x27;s_Scarf">Hermit's Scarf</a>. <strong>Stack 6</strong> of these to dodge 1/3 of the attacks.</li>
3318<li><a href="http://riskofrain.wikia.com/wiki/Monster_Tooth">Monster Tooth</a>. <strong>Stack 9</strong> of these to recover 50 life on kill. This is plenty, since you will be killing <em>a lot</em>.</li>
3319</ul>
3320<p>If you don't have enough and want more fun, get one of these:</p>
3321<ul>
3322<li><a href="http://riskofrain.wikia.com/wiki/Gasoline">Gasoline</a>. Burn the ground on kill, and more will die!</li>
3323<li><a href="http://riskofrain.wikia.com/wiki/Headstompers">Headstompers</a>. They make a pleasing sound on fall, and hurt.</li>
3324<li><a href="http://riskofrain.wikia.com/wiki/Lens-Maker&#x27;s_Glasses">Lens-Maker's Glasses</a>. <strong>Stack 14</strong> and you will always deal a critical strike for double the damage.</li>
3325</ul>
3326<h3 id="uncommon-items"><a href="http://riskofrain.wikia.com/wiki/Category:Uncommon_Items">Uncommon Items</a></h3>
3327<ul>
3328<li><a href="http://riskofrain.wikia.com/wiki/Infusion">Infusion</a>. You only really need one of this. Your life will skyrocket after a while, since this gives you 1HP per kill.</li>
3329<li><a href="http://riskofrain.wikia.com/wiki/Hopoo_Feather">Hopoo Feather</a>. <strong>Stack +10</strong> of these. You will pretty much be able to fly with so many jumps.</li>
3330<li><a href="http://riskofrain.wikia.com/wiki/Guardian&#x27;s_Heart">Guardian's Heart</a>. Not really necessary, but useful for early and late game, since it will absorb infinite damage the first hit.</li>
3331</ul>
3332<p>If, again, you want more fun, get one of these:</p>
3333<ul>
3334<li><a href="http://riskofrain.wikia.com/wiki/Ukulele">Ukelele</a>. Spazz your enemies!</li>
3335<li><a href="http://riskofrain.wikia.com/wiki/Will-o&#x27;-the-wisp">Will-o'-the-wisp</a>. Explode your enemies!</li>
3336<li><a href="http://riskofrain.wikia.com/wiki/Chargefield_Generator">Chargefield Generator</a>. It should cover your entire screen after a bit, hurting all enemies without moving a finger.</li>
3337<li><a href="http://riskofrain.wikia.com/wiki/Golden_Gun">Golden Gun</a>. You will be rich, so this gives you +40% damage.</li>
3338<li><a href="http://riskofrain.wikia.com/wiki/Predatory_Instincts">Predatory Instincts</a>. If you got 14 glasses, you will always be doing critical strikes, and this will give even more attack speed.</li>
3339<li><a href="http://riskofrain.wikia.com/wiki/56_Leaf_Clover">56 Leaf Clover</a>. More drops, in case you didn't have enough.</li>
3340</ul>
3341<h3 id="rare-items"><a href="http://riskofrain.wikia.com/wiki/Category:Rare_Items">Rare Items</a></h3>
3342<ul>
3343<li><a href="http://riskofrain.wikia.com/wiki/Ceremonial_Dagger">Ceremonial Dagger</a>. <strong>Stack +3</strong>, then killing one thing kills another thing and makes a chain reaction.</li>
3344<li><a href="http://riskofrain.wikia.com/wiki/Alien_Head">Alien Head</a>. <strong>Stack 3</strong>, and you will be able to use your abilities more often.</li>
3345</ul>
3346<p>For more fun:</p>
3347<ul>
3348<li><a href="http://riskofrain.wikia.com/wiki/Brilliant_Behemoth">Brilliant Behemoth</a>. Boom boom.</li>
3349</ul>
3350<h2 id="closing-words">Closing Words</h2>
3351<p>You can now beat the game in Monsoon solo with any character. Have fun! And be careful with the sadly common crashes.</p>
3352</content>
3353 </entry>
3354 <entry xml:lang="en">
3355 <title>WorldEdit Commands</title>
3356 <published>2018-07-11T00:00:00+00:00</published>
3357 <updated>2018-07-11T00:00:00+00:00</updated>
3358 <link href="https://lonami.dev/blog/world-edit/" type="text/html"/>
3359 <id>https://lonami.dev/blog/world-edit/</id>
3360 <content type="html"><p><a href="https://dev.bukkit.org/projects/worldedit">WorldEdit</a> is an extremely powerful tool for modifying entire worlds within <a href="https://minecraft.net">Minecraft</a>, which can be used as either a mod for your single-player worlds or as a plugin for your <a href="https://getbukkit.org/">Bukkit</a> servers.</p>
3361<p>This command guide was written for Minecraft 1.12.1, version <a href="https://dev.bukkit.org/projects/worldedit/files/2460562">6.1.7.3</a>, but should work for newer versions too. All WorldEdit commands can be used with a double slash (<code>//</code>) so they don't conlict with built-in commands. This means you can get a list of all commands with <code>//help</code>. Let's explore different categories!</p>
3362<h2 id="movement">Movement</h2>
3363<p>In order to edit a world properly you need to learn how to move in said world properly. There are several straightforward commands that let you move:</p>
3364<ul>
3365<li><code>//ascend</code> goes up one floor.</li>
3366<li><code>//descend</code> goes down one floor.</li>
3367<li><code>//thru</code> let's you pass through walls.</li>
3368<li><code>//jumpto</code> to go wherever you are looking.</li>
3369</ul>
3370<h2 id="information">Information</h2>
3371<p>Knowing your world properly is as important as knowing how to move within it, and will also let you change the information in said world if you need to.</p>
3372<ul>
3373<li><code>//biomelist</code> shows all known biomes.</li>
3374<li><code>//biomeinfo</code> shows the current biome.</li>
3375<li><code>//setbiome</code> lets you change the biome.</li>
3376</ul>
3377<h2 id="blocks">Blocks</h2>
3378<p>You can act over all blocks in a radius around you with quite a few commands. Some won't actually act over the entire range you specify, so 100 is often a good number.</p>
3379<h3 id="filling">Filling</h3>
3380<p>You can fill pools with <code>//fill water 100</code> or caves with <code>//fillr water 100</code>, both of which act below your feet.</p>
3381<h3 id="fixing">Fixing</h3>
3382<p>If the water or lava is buggy use <code>//fixwater 100</code> or <code>//fixlava 100</code> respectively.</p>
3383<p>Some creeper removed the snow or the grass? Fear not, you can use <code>//snow 10</code> or <code>//grass 10</code>.</p>
3384<h3 id="emptying">Emptying</h3>
3385<p>You can empty a pool completely with <code>//drain 100</code>, remove the snow with <code>//thaw 10</code>, and remove fire with <code>//ex 10</code>.</p>
3386<h3 id="removing">Removing</h3>
3387<p>You can remove blocks above and below you in some area with the <code>//removeabove N</code> and <code>//removebelow N</code>. You probably want to set a limit though, or you could fall off the world with <code>//removebelow 1 10</code> for radius and depth. You can also remove near blocks with <code>//removenear block 10</code>.</p>
3388<h3 id="shapes">Shapes</h3>
3389<p>Making a cylinder (or circle) can be done with through <code>//cyl stone 10</code>, a third argument for the height. The radius can be comma-separated to make a ellipses instead, such as <code>//cyl stone 5,10</code>.</p>
3390<p>Spheres are done with <code>//sphere stone 5</code>. This will build one right at your center, so you can raise it to be on your feet with <code>//sphere stone 5 yes</code>. Similar to cylinders, you can comma separate the radius <code>x,y,z</code>.</p>
3391<p>Pyramids can be done with <code>//pyramic stone 5</code>.</p>
3392<p>All these commands can be prefixed with &quot;h&quot; to make them hollow. For instance, <code>//hsphere stone 10</code>.</p>
3393<h2 id="regions">Regions</h2>
3394<h3 id="basics">Basics</h3>
3395<p>Operating over an entire region is really important, and the first thing you need to work comfortably with them is a tool to make selections. The default wooden-axe tool can be obtained with <code>//wand</code>, but you must be near the blocks to select. You can use a different tool, like a golden axe, to use as your &quot;far wand&quot; (wand usable over distance). Once you have one in your hand type <code>//farwand</code> to use it as your &quot;far wand&quot;. You can select the two corners of your region with left and right click. If you have selected the wrong tool, use <code>//none</code> to clear it.</p>
3396<p>If there are no blocks but you want to use your current position as a corner, use <code>//pos1</code> or 2.</p>
3397<p>If you made a region too small, you can enlarge it with <code>//expand 10 up</code>, or <code>//expand vert</code> for the entire vertical range, etc., or make it smaller with <code>//contract 10 up</code> etc., or <code>//inset</code> it to contract in both directions. You can use short-names for the cardinal directions (NSEW).</p>
3398<p>Finally, if you want to move your selection, you can <code>//shift 1 north</code> it to wherever you need.</p>
3399<h3 id="information-1">Information</h3>
3400<p>You can get the <code>//size</code> of the selection or even <code>//count torch</code> in some area. If you want to count all blocks, get their distribution <code>//distr</code>.</p>
3401<h3 id="filling-1">Filling</h3>
3402<p>With a region selected, you can <code>//set</code> it to be any block! For instance, you can use <code>//set air</code> to clear it entirely. You can use more than one block evenly by separting them with a comma <code>//set stone,dirt</code>, or with a custom chance <code>//set 20%stone,80%dirt</code>.</p>
3403<p>You can use <code>//replace from to</code> instead if you don't want to override all blocks in your selection.</p>
3404<p>You can make an hollow set with <code>//faces</code>, and if you just want the walls, use <code>//walls</code>.</p>
3405<h3 id="cleaning">Cleaning</h3>
3406<p>If someone destroyed your wonderful snow landscape, fear not, you can use <code>//overlay snow</code> over it (although for this you actually have <code>//snow N</code> and its opposite <code>//thaw</code>).</p>
3407<p>If you set some rough area, you can always <code>//smooth</code> it, even more than one time with <code>//smooth 3</code>. You can get your dirt and stone back with <code>//naturalize</code> and put some plants with <code>//flora</code> or <code>//forest</code>, both of which support a density or even the type for the trees. If you already have the dirt use <code>//green</code> instead. If you want some pumpkins, with <code>//pumpkins</code>.</p>
3408<h3 id="moving">Moving</h3>
3409<p>You can repeat an entire selection many times by stacking them with <code>//stack N DIR</code>. This is extremely useful to make things like corridors or elevators. For instance, you can make a small section of the corridor, select it entirely, and then repeat it 10 times with <code>//stack 10 north</code>. Or you can make the elevator and then <code>//stack 10 up</code>. If you need to also copy the air use <code>//stackair</code>.</p>
3410<p>Finally, if you don't need to repeat it and simply move it just a bit towards the right direction, you can use <code>//move N</code>. The default direction is &quot;me&quot; (towards where you are facing) but you can set one with <code>//move 1 up</code> for example.</p>
3411<h3 id="selecting">Selecting</h3>
3412<p>You can not only select cuboids. You can also select different shapes, or even just points:</p>
3413<ul>
3414<li><code>//sel cuboid</code> is the default.</li>
3415<li><code>//sel extend</code> expands the default.</li>
3416<li><code>//sel poly</code> first point with left click and right click to add new points.</li>
3417<li><code>//sel ellipsoid</code> first point to select the center and right click to select the different radius.</li>
3418<li><code>//sel sphere</code> first point to select the center and one more right click for the radius.</li>
3419<li><code>//sel cyl</code> for cylinders, first click being the center.</li>
3420<li><code>//sel convex</code> for convex shapes. This one is extremely useful for <code>//curve</code>.</li>
3421</ul>
3422<h2 id="brushes">Brushes</h2>
3423<p>Brushes are a way to paint in 3D without first bothering about making a selection, and there are spherical and cylinder brushes with e.g. <code>//brush sphere stone 2</code>, or the shorter form <code>//br s stone</code>. For cylinder, one must use <code>cyl</code> instead <code>sphere</code>.</p>
3424<p>There also exists a brush to smooth the terrain which can be enabled on the current item with <code>//br smooth</code>, which can be used with right-click like any other brush.</p>
3425<h2 id="clipboard">Clipboard</h2>
3426<p>Finally, you can copy and cut things around like you would do with normal text with <code>//copy</code> and <code>//cut</code>. The copy is issued from wherever you issue the command, so when you use <code>//paste</code>, remember that if you were 4 blocks apart when copying, it will be 4 blocks apart when pasting.</p>
3427<p>The contents of the clipboard can be flipped to wherever you are looking via <code>//flip</code>, and can be rotated via the <code>//rotate 90</code> command (in degrees).</p>
3428<p>To remove the copy use <code>//clearclipboard</code>.</p>
3429</content>
3430 </entry>
3431 <entry xml:lang="en">
3432 <title>An Introduction to Asyncio</title>
3433 <published>2018-06-13T00:00:00+00:00</published>
3434 <updated>2020-10-03T00:00:00+00:00</updated>
3435 <link href="https://lonami.dev/blog/asyncio/" type="text/html"/>
3436 <id>https://lonami.dev/blog/asyncio/</id>
3437 <content type="html"><h2 id="index">Index</h2>
3438<ul>
3439<li><a href="https://lonami.dev/blog/asyncio/#background">Background</a></li>
3440<li><a href="https://lonami.dev/blog/asyncio/#input_output">Input / Output</a></li>
3441<li><a href="https://lonami.dev/blog/asyncio/#diving_in">Diving In</a></li>
3442<li><a href="https://lonami.dev/blog/asyncio/#a_toy_example">A Toy Example</a></li>
3443<li><a href="https://lonami.dev/blog/asyncio/#a_real_example">A Real Example</a></li>
3444<li><a href="https://lonami.dev/blog/asyncio/#extra_material">Extra Material</a></li>
3445</ul>
3446<h2 id="background">Background</h2>
3447<p>After seeing some friends struggle with <code>asyncio</code> I decided that it could be a good idea to write a blog post using my own words to explain how I understand the world of asynchronous IO. I will focus on Python's <code>asyncio</code> module but this post should apply to any other language easily.</p>
3448<p>So what is <code>asyncio</code> and what makes it good? Why don't we just use the old and known threads to run several parts of the code concurrently, at the same time?</p>
3449<p>The first reason is that <code>asyncio</code> makes your code easier to reason about, as opposed to using threads, because the amount of ways in which your code can run grows exponentially. Let's see that with an example. Imagine you have this code:</p>
3450<pre><code class="language-python" data-lang="python">def method():
3451 line 1
3452 line 2
3453 line 3
3454 line 4
3455 line 5
3456</code></pre>
3457<p>And you start two threads to run the method at the same time. What is the order in which the lines of code get executed? The answer is that you can't know! The first thread can run the entire method before the second thread even starts. Or it could be the first thread that runs after the second thread. Perhaps both run the &quot;line 1&quot;, and then the line 2. Maybe the first thread runs lines 1 and 2, and then the second thread only runs the line 1 before the first thread finishes.</p>
3458<p>As you can see, any combination of the order in which the lines run is possible. If the lines modify some global shared state, that will get messy quickly.</p>
3459<p>Second, in Python, threads <em>won't</em> make your code faster most of the time. It will only increase the concurrency of your program (which is okay if it makes many blocking calls), allowing you to run several things at the same time.</p>
3460<p>If you have a lot of CPU work to do though, threads aren't a real advantage. Indeed, your code will probably run slower under the most common Python implementation, CPython, which makes use of a Global Interpreter Lock (GIL) that only lets a thread run at once. The operations won't run in parallel!</p>
3461<h2 id="input-output">Input / Output</h2>
3462<p>Before we go any further, let's first stop to talk about input and output, commonly known as &quot;IO&quot;. There are two main ways to perform IO operations, such as reading or writing from a file or a network socket.</p>
3463<p>The first one is known as &quot;blocking IO&quot;. What this means is that, when you try performing IO, the current application thread is going to <em>block</em> until the Operative System can tell you it's done. Normally, this is not a problem, since disks are pretty fast anyway, but it can soon become a performance bottleneck. And network IO will be much slower than disk IO!</p>
3464<pre><code class="language-python" data-lang="python">import socket
3465
3466# Setup a network socket and a very simple HTTP request.
3467# By default, sockets are open in blocking mode.
3468sock = socket.socket()
3469request = b'''HEAD / HTTP/1.0\r
3470Host: example.com\r
3471\r
3472'''
3473
3474# &quot;connect&quot; will block until a successful TCP connection
3475# is made to the host &quot;example.com&quot; on port 80.
3476sock.connect(('example.com', 80))
3477
3478# &quot;sendall&quot; will repeatedly call &quot;send&quot; until all the data in &quot;request&quot; is
3479# sent to the host we just connected, which blocks until the data is sent.
3480sock.sendall(request)
3481
3482# &quot;recv&quot; will try to receive up to 1024 bytes from the host, and block until
3483# there is any data to receive (or empty if the host closes the connection).
3484response = sock.recv(1024)
3485
3486# After all those blocking calls, we got out data! These are the headers from
3487# making a HTTP request to example.com.
3488print(response.decode())
3489</code></pre>
3490<p>Blocking IO offers timeouts, so that you can get control back in your code if the operation doesn't finish. Imagine that the remote host doesn't want to reply, your code would be stuck for as long as the connection remains alive!</p>
3491<p>But wait, what if we make the timeout small? Very, very small? If we do that, we will never block waiting for an answer. That's how asynchronous IO works, and it's the opposite of blocking IO (you can also call it non-blocking IO if you want to).</p>
3492<p>How does non-blocking IO work if the IO device needs a while to answer with the data? In that case, the operative system responds with &quot;not ready&quot;, and your application gets control back so it can do other stuff while the IO device completes your request. It works a bit like this:</p>
3493<pre><code>&lt;app&gt; Hey, I would like to read 16 bytes from this file
3494&lt;OS&gt; Okay, but the disk hasn't sent me the data yet
3495&lt;app&gt; Alright, I will do something else then
3496(a lot of computer time passes)
3497&lt;app&gt; Do you have my 16 bytes now?
3498&lt;OS&gt; Yes, here they are! &quot;Hello, world !!\n&quot;
3499</code></pre>
3500<p>In reality, you can tell the OS to notify you when the data is ready, as opposed to polling (constantly asking the OS whether the data is ready yet or not), which is more efficient.</p>
3501<p>But either way, that's the difference between blocking and non-blocking IO, and what matters is that your application gets to run more without ever needing to wait for data to arrive, because the data will be there immediately when you ask, and if it's not yet, your app can do more things meanwhile.</p>
3502<h2 id="diving-in">Diving In</h2>
3503<p>Now we've seen what blocking and non-blocking IO is, and how threads make your code harder to reason about, but they give concurrency (yet not more speed). Is there any other way to achieve this concurrency that doesn't involve threads? Yes! The answer is <code>asyncio</code>.</p>
3504<p>So how does <code>asyncio</code> help? First we need to understand a very crucial concept before we can dive any deeper, and I'm talking about the <em>event loop</em>. What is it and why do we need it?</p>
3505<p>You can think of the event loop as a <em>loop</em> that will be responsible for calling your <code>async</code> functions:</p>
3506<p><img src="https://lonami.dev/blog/asyncio/eventloop.svg" alt="The Event Loop" /></p>
3507<p>That's silly you may think. Now not only we run our code but we also have to run some &quot;event loop&quot;. It doesn't sound beneficial at all. What are these events? Well, they are the IO events we talked about before!</p>
3508<p><code>asyncio</code>'s event loop is responsible for handling those IO events, such as file is ready, data arrived, flushing is done, and so on. As we saw before, we can make these events non-blocking by setting their timeout to 0.</p>
3509<p>Let's say you want to read from 10 files at the same time. You will ask the OS to read data from 10 files, and at first none of the reads will be ready. But the event loop will be constantly asking the OS to know which are done, and when they are done, you will get your data.</p>
3510<p>This has some nice advantages. It means that, instead of waiting for a network request to send you a response or some file, instead of blocking there, the event loop can decide to run other code meanwhile. Whenever the contents are ready, they can be read, and your code can continue. Waiting for the contents to be received is done with the <code>await</code> keyword, and it tells the loop that it can run other code meanwhile:</p>
3511<p><img src="https://lonami.dev/blog/asyncio/awaitkwd1.svg" alt="Step 1, await keyword" /></p>
3512<p><img src="https://lonami.dev/blog/asyncio/awaitkwd2.svg" alt="Step 2, await keyword" /></p>
3513<p>Start reading the code of the event loop and follow the arrows. You can see that, in the beginning, there are no events yet, so the loop calls one of your functions. The code runs until it has to <code>await</code> for some IO operation to complete, such as sending a request over the network. The method is &quot;paused&quot; until an event occurs (for example, an &quot;event&quot; occurs when the request has been sent completely).</p>
3514<p>While the first method is busy, the event loop can enter the second method, and run its code until the first <code>await</code>. But it can happen that the event of the second query occurs before the request on the first method, so the event loop can re-enter the second method because it has already sent the query, but the first method isn't done sending the request yet.</p>
3515<p>Then, the second method <code>await</code>'s for an answer, and an event occurs telling the event loop that the request from the first method was sent. The code can be resumed again, until it has to <code>await</code> for a response, and so on. Here's an explanation with pseudo-code for this process if you prefer:</p>
3516<pre><code class="language-python" data-lang="python">async def method(request):
3517 prepare request
3518 await send request
3519
3520 await receive request
3521
3522 process request
3523 return result
3524
3525run in parallel (
3526 method with request 1,
3527 method with request 2,
3528)
3529</code></pre>
3530<p>This is what the event loop will do on the above pseudo-code:</p>
3531<pre><code>no events pending, can advance
3532
3533enter method with request 1
3534 prepare request
3535 await sending request
3536pause method with request 1
3537
3538no events ready, can advance
3539
3540enter method with request 2
3541 prepare request
3542 await sending request
3543pause method with request 2
3544
3545both requests are paused, cannot advance
3546wait for events
3547event for request 2 arrives (sending request completed)
3548
3549enter method with request 2
3550 await receiving response
3551pause method with request 2
3552
3553event for request 1 arrives (sending request completed)
3554
3555enter method with request 1
3556 await receiving response
3557pause method with request 1
3558
3559...and so on
3560</code></pre>
3561<p>You may be wondering &quot;okay, but threads work for me, so why should I change?&quot;. There are some important things to note here. The first is that we only need one thread to be running! The event loop decides when and which methods should run. This results in less pressure for the operating system. The second is that we know when it may run other methods. Those are the <code>await</code> keywords! Whenever there is one of those, we know that the loop is able to run other things until the resource (again, like network) becomes ready (when a event occurs telling us it's ready to be used without blocking or it has completed).</p>
3562<p>So far, we already have two advantages. We are only using a single thread so the cost for switching between methods is low, and we can easily reason about where our program may interleave operations.</p>
3563<p>Another advantage is that, with the event loop, you can easily schedule when a piece of code should run, such as using the method <a href="https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.call_at"><code>loop.call_at</code></a>, without the need for spawning another thread at all.</p>
3564<p>To tell the <code>asyncio</code> to run the two methods shown above, we can use <a href="https://docs.python.org/3/library/asyncio-future.html#asyncio.ensure_future"><code>asyncio.ensure_future</code></a>, which is a way of saying &quot;I want the future of my method to be ensured&quot;. That is, you want to run your method in the future, whenever the loop is free to do so. This method returns a <code>Future</code> object, so if your method returns a value, you can <code>await</code> this future to retrieve its result.</p>
3565<p>What is a <code>Future</code>? This object represents the value of something that will be there in the future, but might not be there yet. Just like you can <code>await</code> your own <code>async def</code> functions, you can <code>await</code> these <code>Future</code>'s.</p>
3566<p>The <code>async def</code> functions are also called &quot;coroutines&quot;, and Python does some magic behind the scenes to turn them into such. The coroutines can be <code>await</code>'ed, and this is what you normally do.</p>
3567<h2 id="a-toy-example">A Toy Example</h2>
3568<p>That's all about <code>asyncio</code>! Let's wrap up with some example code. We will create a server that replies with the text a client sends, but reversed. First, we will show what you could write with normal synchronous code, and then we will port it.</p>
3569<p>Here is the <strong>synchronous version</strong>:</p>
3570<pre><code class="language-python" data-lang="python"># server.py
3571import socket
3572
3573
3574def server_method():
3575 # create a new server socket to listen for connections
3576 server = socket.socket()
3577
3578 # bind to localhost:6789 for new connections
3579 server.bind(('localhost', 6789))
3580
3581 # we will listen for one client at most
3582 server.listen(1)
3583
3584 # *block* waiting for a new client
3585 client, _ = server.accept()
3586
3587 # *block* waiting for some data
3588 data = client.recv(1024)
3589
3590 # reverse the data
3591 data = data[::-1]
3592
3593 # *block* sending the data
3594 client.sendall(data)
3595
3596 # close client and server
3597 server.close()
3598 client.close()
3599
3600
3601if __name__ == '__main__':
3602 # block running the server
3603 server_method()
3604</code></pre>
3605<pre><code class="language-python" data-lang="python"># client.py
3606import socket
3607
3608
3609def client_method():
3610 message = b'Hello Server!\n'
3611 client = socket.socket()
3612
3613 # *block* trying to stabilish a connection
3614 client.connect(('localhost', 6789))
3615
3616 # *block* trying to send the message
3617 print('Sending', message)
3618 client.sendall(message)
3619
3620 # *block* until we receive a response
3621 response = client.recv(1024)
3622 print('Server replied', response)
3623
3624 client.close()
3625
3626
3627if __name__ == '__main__':
3628 client_method()
3629</code></pre>
3630<p>From what we've seen, this code will block on all the lines with a comment above them saying that they will block. This means that for running more than one client or server, or both in the same file, you will need threads. But we can do better, we can rewrite it into <code>asyncio</code>!</p>
3631<p>The first step is to mark all your <code>def</code>initions that may block with <code>async</code>. This marks them as coroutines, which can be <code>await</code>ed on.</p>
3632<p>Second, since we're using low-level sockets, we need to make use of the methods that <code>asyncio</code> provides directly. If this was a third-party library, this would be just like using their <code>async def</code>initions.</p>
3633<p>Here is the <strong>asynchronous version</strong>:</p>
3634<pre><code class="language-python" data-lang="python"># server.py
3635import asyncio
3636import socket
3637
3638# get the default &quot;event loop&quot; that we will run
3639loop = asyncio.get_event_loop()
3640
3641
3642# notice our new &quot;async&quot; before the definition
3643async def server_method():
3644 server = socket.socket()
3645 server.bind(('localhost', 6789))
3646 server.listen(1)
3647
3648 # await for a new client
3649 # the event loop can run other code while we wait here!
3650 client, _ = await loop.sock_accept(server)
3651
3652 # await for some data
3653 data = await loop.sock_recv(client, 1024)
3654 data = data[::-1]
3655
3656 # await for sending the data
3657 await loop.sock_sendall(client, data)
3658
3659 server.close()
3660 client.close()
3661
3662
3663if __name__ == '__main__':
3664 # run the loop until &quot;server method&quot; is complete
3665 loop.run_until_complete(server_method())
3666</code></pre>
3667<pre><code class="language-python" data-lang="python"># client.py
3668import asyncio
3669import socket
3670
3671loop = asyncio.get_event_loop()
3672
3673
3674async def client_method():
3675 message = b'Hello Server!\n'
3676 client = socket.socket()
3677
3678 # await to stabilish a connection
3679 await loop.sock_connect(client, ('localhost', 6789))
3680
3681 # await to send the message
3682 print('Sending', message)
3683 await loop.sock_sendall(client, message)
3684
3685 # await to receive a response
3686 response = await loop.sock_recv(client, 1024)
3687 print('Server replied', response)
3688
3689 client.close()
3690
3691
3692if __name__ == '__main__':
3693 loop.run_until_complete(client_method())
3694</code></pre>
3695<p>That's it! You can place these two files separately and run, first the server, then the client. You should see output in the client.</p>
3696<p>The big difference here is that you can easily modify the code to run more than one server or clients at the same time. Whenever you <code>await</code> the event loop will run other of your code. It seems to &quot;block&quot; on the <code>await</code> parts, but remember it's actually jumping to run more code, and the event loop will get back to you whenever it can.</p>
3697<p>In short, you need an <code>async def</code> to <code>await</code> things, and you run them with the event loop instead of calling them directly. So this…</p>
3698<pre><code class="language-python" data-lang="python">def main():
3699 ... # some code
3700
3701
3702if __name__ == '__main__':
3703 main()
3704</code></pre>
3705<p>…becomes this:</p>
3706<pre><code class="language-python" data-lang="python">import asyncio
3707
3708
3709async def main():
3710 ... # some code
3711
3712
3713if __name__ == '__main__':
3714 asyncio.get_event_loop().run_until_complete(main)
3715</code></pre>
3716<p>This is pretty much how most of your <code>async</code> scripts will start, running the main method until its completion.</p>
3717<h2 id="a-real-example">A Real Example</h2>
3718<p>Let's have some fun with a real library. We'll be using <a href="https://github.com/LonamiWebs/Telethon">Telethon</a> to broadcast a message to our three best friends, all at the same time, thanks to the magic of <code>asyncio</code>. We'll dive right into the code, and then I'll explain our new friend <code>asyncio.wait(...)</code>:</p>
3719<pre><code class="language-python" data-lang="python"># broadcast.py
3720import asyncio
3721import sys
3722
3723from telethon import TelegramClient
3724
3725# (you need your own values here, check Telethon's documentation)
3726api_id = 123
3727api_hash = '123abc'
3728friends = [
3729 '@friend1__username',
3730 '@friend2__username',
3731 '@bestie__username'
3732]
3733
3734# we will have to await things, so we need an async def
3735async def main(message):
3736 # start is a coroutine, so we need to await it to run it
3737 client = await TelegramClient('me', api_id, api_hash).start()
3738
3739 # wait for all three client.send_message to complete
3740 await asyncio.wait([
3741 client.send_message(friend, message)
3742 for friend in friends
3743 ])
3744
3745 # and close our client
3746 await client.disconnect()
3747
3748
3749if __name__ == '__main__':
3750 if len(sys.argv) != 2:
3751 print('You must pass the message to broadcast!')
3752 quit()
3753
3754 message = sys.argv[1]
3755 asyncio.get_event_loop().run_until_complete(main(message))
3756</code></pre>
3757<p>Wait… how did that send a message to all three of
3758my friends? The magic is done here:</p>
3759<pre><code class="language-python" data-lang="python">[
3760 client.send_message(friend, message)
3761 for friend in friends
3762]
3763</code></pre>
3764<p>This list comprehension creates another list with three
3765coroutines, the three <code>client.send_message(...)</code>.
3766Then we just pass that list to <code>asyncio.wait</code>:</p>
3767<pre><code class="language-python" data-lang="python">await asyncio.wait([...])
3768</code></pre>
3769<p>This method, by default, waits for the list of coroutines to run until they've all finished. You can read more on the Python <a href="https://docs.python.org/3/library/asyncio-task.html#asyncio.wait">documentation</a>. Truly a good function to know about!</p>
3770<p>Now whenever you have some important news for your friends, you can simply <code>python3 broadcast.py 'I bought a car!'</code> to tell all your friends about your new car! All you need to remember is that you need to <code>await</code> on coroutines, and you will be good. <code>asyncio</code> will warn you when you forget to do so.</p>
3771<h2 id="extra-material">Extra Material</h2>
3772<p>If you want to understand how <code>asyncio</code> works under the hood, I recommend you to watch this hour-long talk <a href="https://youtu.be/M-UcUs7IMIM">Get to grips with asyncio in Python 3</a> by Robert Smallshire. In the video, they will explain the differences between concurrency and parallelism, along with others concepts, and how to implement your own <code>asyncio</code> &quot;scheduler&quot; from scratch.</p>
3773</content>
3774 </entry>
3775 <entry xml:lang="en">
3776 <title>Atemporal Blog Posts</title>
3777 <published>2018-02-03T00:00:00+00:00</published>
3778 <updated>2021-02-19T00:00:00+00:00</updated>
3779 <link href="https://lonami.dev/blog/posts/" type="text/html"/>
3780 <id>https://lonami.dev/blog/posts/</id>
3781 <content type="html"><p>These are some interesting posts and links I've found around the web. I believe they are quite interesting and nice reads, so if you have the time, I encourage you to check some out.</p>
3782<h2 id="algorithms">Algorithms</h2>
3783<ul>
3784<li><a href="http://www.tannerhelland.com/4660/dithering-eleven-algorithms-source-code/">Image Dithering: Eleven Algorithms and Source Code</a>. What does it mean and how to achieve it?</li>
3785<li><a href="https://cristian.io/post/bloom-filters/">Idempotence layer on bloom filters</a>. What are they and how can they help?</li>
3786<li><a href="https://en.wikipedia.org/wiki/Huffman_coding">Huffman coding</a>. This encoding is a simple yet interesting way of compressing information.</li>
3787<li><a href="https://github.com/mxgmn/WaveFunctionCollapse">Wave Function Collapse</a>. Bitmap &amp; tilemap generation from a single example with the help of ideas from quantum mechanics.</li>
3788<li><a href="https://blog.nelhage.com/2015/02/regular-expression-search-with-suffix-arrays/">Regular Expression Search with Suffix Arrays</a>. A way to efficiently search large amounts of text.</li>
3789</ul>
3790<h2 id="culture">Culture</h2>
3791<ul>
3792<li><a href="https://www.wired.com/story/ideas-joi-ito-robot-overlords/">Why Westerners Fear Robots and the Japanese Do Not</a>. Explains some possible reasons for this case.</li>
3793<li><a href="http://catb.org/%7Eesr/faqs/smart-questions.html">How To Ask Questions The Smart Way</a>. Some bits of hacker culture and amazing tips on how to ask a question.</li>
3794<li><a href="http://apenwarr.ca/log/?m=201809#14">XML, blockchains, and the strange shapes of progress</a>. Some of history about XML and blockchain.</li>
3795<li><a href="https://czep.net/17/legion-of-lobotomized-unices.html">Legion of lobotomized unices</a>. A time where computers are treated a lot more nicely.</li>
3796<li><a href="https://eli.thegreenplace.net/2016/the-expression-problem-and-its-solutions/">The Expression Problem and its solutions</a>. What is it and what can we do to solve it?</li>
3797<li><a href="http://allendowney.blogspot.com/2015/08/the-inspection-paradox-is-everywhere.html">The Inspection Paradox is Everywhere</a>. Interesting and very common phenomena.</li>
3798<li><a href="https://github.com/ChrisKnott/Algojammer">An experimental code editor for writing algorithms</a>. Contains several links to different tools for reverse debugging.</li>
3799<li><a href="http://habitatchronicles.com/2017/05/what-are-capabilities/">What Are Capabilities?</a> Good ideas with great security implications.</li>
3800<li><a href="https://blog.aurynn.com/2015/12/16-contempt-culture">Contempt Culture</a>. Or why you should not speak crap about your non-favourite programming languages.</li>
3801<li><a href="https://www.lesswrong.com/posts/tscc3e5eujrsEeFN4/well-kept-gardens-die-by-pacifism">Well-Kept Gardens Die By Pacifism</a>. Risks any online community can run into.</li>
3802<li><a href="https://ncase.me/">It's Nicky Case!</a> They make some cool things worth checking out, I really like &quot;we become what we behold&quot;.</li>
3803</ul>
3804<h2 id="debate">Debate</h2>
3805<ul>
3806<li><a href="https://steemit.com/opensource/@crell/open-source-is-awful">Open Source is awful</a>. Has some points about why is it bad and how it could improve.</li>
3807<li><a href="http://www.mondo2000.com/2018/01/17/pink-lexical-goop-dark-side-autocorrect/">Pink Lexical Goop: The Dark Side of Autocorrect</a>. It can shape how you think.</li>
3808<li><a href="http://blog.ploeh.dk/2015/08/03/idiomatic-or-idiosyncratic/">Idiomatic or idiosyncratic?</a> Can porting code constructs from other languages have a positive effect?</li>
3809<li><a href="https://gamasutra.com/view/news/169296/Indepth_Functional_programming_in_C.php">In-depth: Functional programming in C++</a>. Is it useful to bother with functional concepts in a language like C++?</li>
3810<li><a href="https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/">Notes on structured concurrency, or: Go statement considered harmful</a>.</li>
3811<li><a href="https://queue.acm.org/detail.cfm?id=3212479">C Is Not a Low-level Language</a>. Could there be alternative programming models designed for more specialized CPUs?</li>
3812</ul>
3813<h2 id="food-for-thought">Food for Thought</h2>
3814<ul>
3815<li><a href="https://www.hillelwayne.com/post/divide-by-zero/">1/0 = 0</a>. Explores why it makes sense to redefine mathemathics under some circumstances, and why it is possible to do so.</li>
3816<li><a href="https://jeremykun.com/2018/04/13/for-mathematicians-does-not-mean-equality/">For mathematicians, = does not mean equality</a>. What other definitions does the equal sign have?</li>
3817<li><a href="https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-thoughts">Cached Thoughts</a>. How is it possible that our brains work at all?</li>
3818<li><a href="http://tonsky.me/blog/disenchantment/">Software disenchantment</a>. Faster hardware and slower software is a trend.
3819<ul>
3820<li><a href="https://blackhole12.com/blog/software-engineering-is-bad-but-it-s-not-that-bad/">Software Engineering Is Bad, But That's Not Why</a>. This post has some good counterpoints to Software disenchantment.</li>
3821</ul>
3822</li>
3823<li><a href="http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/">What Color is Your Function?</a>. Spoiler: can we approach asynchronous IO better?</li>
3824<li><a href="https://hackernoon.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5">I'm harvesting credit card numbers and passwords from your site</a>. A word of warning when mindlessly adding dependencies.</li>
3825<li><a href="https://medium.com/message/everything-is-broken-81e5f33a24e1">Everything Is Broken</a>. Some of the (probable) truths about our world.</li>
3826<li><a href="http://johnsalvatier.org/blog/2017/reality-has-a-surprising-amount-of-detail">Reality has a surprising amount of detail</a>.</li>
3827</ul>
3828<h2 id="funny">Funny</h2>
3829<ul>
3830<li><a href="http://thedailywtf.com/articles/We-Use-BobX">We Use BobX</a>. BobX.</li>
3831<li><a href="http://thedailywtf.com/articles/the-inner-json-effect">The Inner JSON Effect</a>. For some reason, custom languages are in.</li>
3832<li><a href="https://thedailywtf.com/articles/exponential-backup">Exponential Backup</a>. Far better than git.</li>
3833<li><a href="https://thedailywtf.com/articles/ITAPPMONROBOT">ITAPPMONROBOT</a>. Solving software problems with hardware.</li>
3834<li><a href="https://thedailywtf.com/articles/a-tapestry-of-threads">A Tapestry of Threads</a>. More threads must mean faster code, right?</li>
3835<li><a href="https://medium.com/commitlog/a-brief-totally-accurate-history-of-programming-languages-cd93ec806124">A Brief Totally Accurate History Of Programming Languages</a>. Don't take offense for it!</li>
3836</ul>
3837<h2 id="graphics">Graphics</h2>
3838<ul>
3839<li><a href="http://shaunlebron.github.io/visualizing-projections/">Visualizing Projections</a>. Small post about different projection methods.</li>
3840<li><a href="http://www.iquilezles.org/www/index.htm">Inigo Quilez :: fractals, computer graphics, mathematics, shaders, demoscene and more</a> A <em>lot</em> of useful and quality articles regarding computer graphics.</li>
3841</ul>
3842<h2 id="history">History</h2>
3843<ul>
3844<li><a href="https://twobithistory.org/2018/08/18/ada-lovelace-note-g.html">What Did Ada Lovelace's Program Actually Do?</a>. And other characters that took part in the beginning's of programming.</li>
3845<li><a href="https://chrisdown.name/2018/01/02/in-defence-of-swap.html">In defence of swap: common misconceptions</a>. Swap is still an useful concept.</li>
3846<li><a href="https://www.pacifict.com/Story/">The Graphing Calculator Story</a>. A great classic Apple tale.</li>
3847<li><a href="https://twobithistory.org/2018/10/14/lisp.html">How Lisp Became God's Own Programming Language</a>. Lisp as a foundational programming language.</li>
3848</ul>
3849<h2 id="motivational">Motivational</h2>
3850<ul>
3851<li><a href="https://www.joelonsoftware.com/2002/01/06/fire-and-motion/">Fire And Motion</a>. What does actually take to get things done?</li>
3852<li><a href="https://realmensch.org/2017/08/25/the-parable-of-the-two-programmers/">The Parable of the Two Programmers</a>. This tale is about two different types of programmer and their respective endings in a company, illustrating how the one you wouldn't expect to actually ends in a better situation.</li>
3853<li><a href="https://byorgey.wordpress.com/2018/05/06/conversations-with-a-six-year-old-on-functional-programming/">Conversations with a six-year-old on functional programming</a>. Little kids today can be really interested in technological topics.</li>
3854<li><a href="https://bulletproofmusician.com/how-many-hours-a-day-should-you-practice/">How Many Hours a Day Should You Practice?</a>. While the article is about music, it applies to any other areas.</li>
3855<li><a href="http://nathanmarz.com/blog/suffering-oriented-programming.html">Suffering-oriented programming</a>. A possibly new approach on how you could tackle your new projects.</li>
3856<li><a href="https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/">Things You Should Never Do, Part I</a>. There is no need to rewrite your code.</li>
3857</ul>
3858<h2 id="optimization">Optimization</h2>
3859<ul>
3860<li><a href="http://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html">What Every C Programmer Should Know About Undefined Behavior #1/3</a>. Explains what undefined behaviour is and why it makes sense.</li>
3861<li><a href="http://ridiculousfish.com/blog/posts/labor-of-division-episode-i.html">Labor of Division (Episode I)</a>. Some tricks to divide without division.</li>
3862<li><a href="http://blog.moertel.com/posts/2013-12-14-great-old-timey-game-programming-hack.html">A Great Old-Timey Game-Programming Hack</a>. Abusing instructions to make games playable even on the slowest hardware.</li>
3863<li><a href="https://web.archive.org/web/20191213224640/https://people.eecs.berkeley.edu/%7Esangjin/2012/12/21/epoll-vs-kqueue.html">Scalable Event Multiplexing: epoll vs kqueue</a>. How good OS primitives can really help performance and scability.</li>
3864<li><a href="https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html">Command-line Tools can be 235x Faster than your Hadoop Cluster</a>. Or how to use the right tool for the right job.</li>
3865<li><a href="https://nullprogram.com/blog/2018/05/27/">When FFI Function Calls Beat Native C</a>. How lua beat C at it and the explanation behind it.</li>
3866<li><a href="http://igoro.com/archive/gallery-of-processor-cache-effects/">Gallery of Processor Cache Effects</a>. Knowing a few things about the cache can make a big difference.</li>
3867</ul>
3868</content>
3869 </entry>
3870 <entry xml:lang="en">
3871 <title>Graphs</title>
3872 <published>2017-06-02T00:00:00+00:00</published>
3873 <updated>2017-06-02T00:00:00+00:00</updated>
3874 <link href="https://lonami.dev/blog/graphs/" type="text/html"/>
3875 <id>https://lonami.dev/blog/graphs/</id>
3876 <content type="html"><p><noscript>There are a few things which won't render unless you enable
3877JavaScript. No tracking, I promise!</noscript></p>
3878<blockquote>
3879<p>Don't know English? <a href="https://lonami.dev/blog/graphs/spanish.html">Read the Spanish version instead</a>.</p>
3880</blockquote>
3881<p>Let's imagine we have 5 bus stations, which we'll denote by ((s_i)):</p>
3882<div class="matrix">
3883 ' s_1 ' s_2 ' s_3 ' s_4 ' s_5 \\
3884s_1 ' ' V ' ' ' \\
3885s_2 ' V ' ' ' ' V \\
3886s_3 ' ' ' ' V ' \\
3887s_4 ' ' V ' V ' ' \\
3888s_5 ' V ' ' ' V '
3889</div>
3890<p>This is known as a &quot;table of direct interconnections&quot;.
3891The ((V)) represent connected paths. For instance, on the first
3892row starting at ((s_1)), reaching the ((V)),
3893allows us to turn up to get to ((s_2)).</p>
3894<p>We can see the above table represented in a more graphical way:</p>
3895<p><img src="https://lonami.dev/blog/graphs/example1.svg" alt="Table 1 as a Graph" /></p>
3896<p>This type of graph is called, well, a graph, and it's a directed
3897graph (or digraph), since the direction on which the arrows go does
3898matter. It's made up of vertices, joined together by edges (also known as
3899lines or directed arcs).</p>
3900<p>One can walk from a node to another through different paths. For
3901example, ((s_4 $rightarrow s_2 $rightarrow s_5)) is an indirect path of order
3902two, because we must use two edges to go from ((s_4)) to
3903((s_5)).</p>
3904<p>Let's now represent its adjacency matrix called A which represents the
3905same table, but uses 1 instead V to represent
3906a connection:</p>
3907<div class="matrix">
3908 0 ' 1 ' 0 ' 0 ' 0 \\
3909 1 ' 0 ' 0 ' 0 ' 1 \\
3910 0 ' 0 ' 0 ' 1 ' 0 \\
3911 0 ' 1 ' 1 ' 0 ' 0 \\
3912 1 ' 0 ' 0 ' 1 ' 0
3913</div>
3914<p>This way we can see how the ((a_{2,1})) element represents the
3915connection ((s_2 $rightarrow s_1)), and the ((a_{5,1})) element the
3916((s_5 $rightarrow s_1)) connection, etc.</p>
3917<p>In general, ((a_{i,j})) represents a connection from
3918((s_i $rightarrow s_j))as long as ((a_{i,j}$geq 1)).</p>
3919<p>Working with matrices allows us to have a computable representation of
3920any graph, which is very useful.</p>
3921<hr />
3922<p>Graphs have a lot of interesting properties besides being representable
3923by a computer. What would happen if, for instance, we calculated
3924((A^2))? We obtain the following matrix:</p>
3925<div class="matrix">
39261 ' 0 ' 0 ' 0 ' 1 \\
39271 ' 1 ' 0 ' 1 ' 0 \\
39280 ' 1 ' 1 ' 0 ' 0 \\
39291 ' 0 ' 0 ' 1 ' 1 \\
39300 ' 2 ' 1 ' 0 ' 0
3931</div>
3932<p>We can interpret this as the paths of order two.
3933But what does the element ((a_{5,2}=2)) represent? It indicates
3934the amount of possible ways to go from ((s_5 $rightarrow s_i $rightarrow s_2)).</p>
3935<p>One can manually multiply the involved row and column to determine which
3936element is the one we need to pass through, this way we have the row
3937(([1 0 0 1 0])) and the column (([1 0 0 1 0])) (on
3938vertical). The elements ((s_i$geq 1)) are ((s_1)) and
3939((s_4)). This is, we can go from ((s_5)) to
3940((s_2)) via ((s_5 $rightarrow s_1 $rightarrow s_2)) or via
3941((s_5 $rightarrow s_4 $rightarrow s_2)):
3942<img src="example2.svg" /></p>
3943<p>It's important to note that graphs to not consider self-connections, this
3944is, ((s_i $rightarrow s_i)) is not allowed; neither we work with multigraphs
3945here (those which allow multiple connections, for instance, an arbitrary
3946number ((n)) of times).</p>
3947<div class="matrix">
39481 ' 1 ' 0 ' 1 ' 0 \\
39491 ' 2 ' \textbf{1} ' 0 ' 1 \\
39501 ' 0 ' 0 ' 1 ' 1 \\
39511 ' 2 ' 1 ' 1 ' 0 \\
39522 ' 0 ' 0 ' 1 ' 2
3953</div>
3954<p>We can see how the first ((1)) just appeared on the element
3955((a_{2,3})), which means that the shortest path to it is at least
3956of order three.</p>
3957<hr />
3958<p>A graph is said to be strongly connected as long as there is a
3959way to reach all its elements.</p>
3960<p>We can see all the available paths until now by simply adding up all the
3961direct and indirect ways to reach a node, so for now, we can add
3962((A+A^2+A^3)) in such a way that:</p>
3963<div class="matrix">
39642 ' 2 ' 0 ' 1 ' 1 \\
39653 ' 3 ' 1 ' 1 ' 3 \\
39661 ' 1 ' 1 ' 2 ' 1 \\
39672 ' 3 ' 2 ' 2 ' 1 \\
39683 ' 2 ' 1 ' 2 ' 2
3969</div>
3970<p>There isn't a connection between ((s_1)) and ((s_3)) yet.
3971If we were to calculate ((A^4)):</p>
3972<div class="matrix">
39731 ' 2 ' 1 ' ' \\
3974 ' ' ' ' \\
3975 ' ' ' ' \\
3976 ' ' ' ' \\
3977 ' ' ' '
3978</div>
3979<p>We don't need to calculate anymore. We now know that the graph is
3980strongly connected!</p>
3981<hr />
3982<p>Congratulations! You've completed this tiny introduction to graphs.
3983Now you can play around with them and design your own connections.</p>
3984<p>Hold the left mouse button on the above area and drag it down to create
3985a new node, or drag a node to this area to delete it.</p>
3986<p>To create new connections, hold the right mouse button on the node you
3987want to start with, and drag it to the node you want it to be connected to.</p>
3988<p>To delete the connections coming from a specific node, middle click it.</p>
3989<table><tr><td style="width:100%;">
3990 <button onclick="resetConnections()">Reset connections</button>
3991 <button onclick="clearNodes()">Clear all the nodes</button>
3992 <br />
3993 <br />
3994 <label for="matrixOrder">Show matrix of order:</label>
3995 <input id="matrixOrder" type="number" min="1" max="5"
3996 value="1" oninput="updateOrder()">
3997 <br />
3998 <label for="matrixAccum">Show accumulated matrix</label>
3999 <input id="matrixAccum" type="checkbox" onchange="updateOrder()">
4000 <br />
4001 <br />
4002 <div>
4003 <table id="matrixTable"></table>
4004 </div>
4005</td><td>
4006 <canvas id="canvas" width="400" height="400" oncontextmenu="return false;">
4007 Looks like your browser won't let you see this fancy example :(
4008 </canvas>
4009 <br />
4010</td></tr></table>
4011<script src="tinyparser.js"></script>
4012<script src="enhancements.js"></script>
4013<script src="graphs.js"></script>
4014</content>
4015 </entry>
4016 <entry xml:lang="en">
4017 <title>Installing NixOS</title>
4018 <published>2017-05-13T00:00:00+00:00</published>
4019 <updated>2019-02-16T00:00:00+00:00</updated>
4020 <link href="https://lonami.dev/blog/installing-nixos/" type="text/html"/>
4021 <id>https://lonami.dev/blog/installing-nixos/</id>
4022 <content type="html"><h2 id="update">Update</h2>
4023<p><em>Please see <a href="../installing_nixos_2/index.html">my followup post with NixOS</a> for a far better experience with it</em></p>
4024<hr />
4025<p>Today I decided to install <a href="http://nixos.org/">NixOS</a> as a recommendation, a purely functional Linux distribution, since <a href="https://xubuntu.org/">Xubuntu</a> kept crashing. Here's my journey, and how I managed to install it from a terminal for the first time in my life. Steps aren't hard, but they may not seem obvious at first.</p>
4026<ul>
4027<li>
4028<p>Grab the Live CD, burn it on a USB stick and boot. I recommend using <a href="https://etcher.io/">Etcher</a>.</p>
4029</li>
4030<li>
4031<p>Type <code>systemctl start display-manager</code> and wait.<sup class="footnote-reference"><a href="#1">1</a></sup></p>
4032</li>
4033<li>
4034<p>Open both the manual and the <code>konsole</code>.</p>
4035</li>
4036<li>
4037<p>Connect to the network using the GUI.</p>
4038</li>
4039<li>
4040<p>Create the disk partitions by using <code>fdisk</code>.</p>
4041<p>You can list them with <code>fdisk -l</code>, modify a certain drive with <code>fdisk /dev/sdX</code> (for instance, <code>/dev/sda</code>) and follow the instructions.</p>
4042<p>To create the file system, use <code>mkfs.ext4 -L &lt;label&gt; /dev/sdXY</code> and swap with <code>mkswap -L &lt;label&gt; /dev/sdXY</code>.</p>
4043<p>The EFI partition should be done with <code>mkfs.vfat</code>.</p>
4044</li>
4045<li>
4046<p>Mount the target to <code>/mnt</code> e.g. if the label was <code>nixos</code>, <code>mount /dev/disk/by-label/nixos /mnt</code></p>
4047</li>
4048<li>
4049<p><code>mkdir /mnt/boot</code> and then mount your EFI partition to it.</p>
4050</li>
4051<li>
4052<p>Generate a configuration template with <code>nixos-generate-config --root /mnt</code>, and modify it with <code>nano /etc/nixos/configuration.nix</code>.</p>
4053</li>
4054<li>
4055<p>While modifying the configuration, make sure to add <code>boot.loader.grub.device = &quot;/dev/sda&quot;</code></p>
4056</li>
4057<li>
4058<p>More useful configuration things are:</p>
4059<ul>
4060<li>Uncomment the whole <code>i18n</code> block.</li>
4061<li>Add some essential packages like <code>environment.systemPackages = with pkgs; [wget git firefox pulseaudio networkmanagerapplet];</code>.</li>
4062<li>If you want to use XFCE, add <code>services.xserver.desktopManager.xfce.enable = true;</code>, otherwise, you don't need <code>networkmanagerapplet</code> either. Make sure to add <code>networking.networkmanager.enable = true;</code> too.</li>
4063<li>Define some user for yourself (modify <code>guest</code> name) and use a UID greater than 1000. Also, add yourself to <code>extraGroups = [&quot;wheel&quot; &quot;networkmanager&quot;];</code> (the first to be able to <code>sudo</code>, the second to use network related things).</li>
4064</ul>
4065</li>
4066<li>
4067<p>Run <code>nixos-install</code>. If you ever modify that file again, to add more packages for instance (this is how they're installed), run <code>nixos-rebuild switch</code> (or use <code>test</code> to test but don't boot to it, or <code>boot</code> not to switch but to use on next boot.</p>
4068</li>
4069<li>
4070<p><code>reboot</code>.</p>
4071</li>
4072<li>
4073<p>Login as <code>root</code>, and set a password for your user with <code>passwd &lt;user&gt;</code>. Done!</p>
4074</li>
4075</ul>
4076<p>I enjoyed the process of installing it, and it's really cool that it has versioning and is so clean to keep track of which packages you install. But not being able to run arbitrary binaries by default is something very limitting in my opinion, though they've done a good job.</p>
4077<p>I'm now back to Xubuntu, with a fresh install.</p>
4078<h2 id="update-1">Update</h2>
4079<p>It is not true that &quot;they don't allow running arbitrary binaries by default&quot;, as pointed out in their <a href="https://nixos.org/nixpkgs/manual/#sec-fhs-environments">manual, buildFHSUserEnv</a>:</p>
4080<blockquote>
4081<p><code>buildFHSUserEnv</code> provides a way to build and run FHS-compatible lightweight sandboxes. It creates an isolated root with bound <code>/nix/store</code>, so its footprint in terms of disk space needed is quite small. This allows one to run software which is hard or unfeasible to patch for NixOS -- 3rd-party source trees with FHS assumptions, games distributed as tarballs, software with integrity checking and/or external self-updated binaries. It uses Linux namespaces feature to create temporary lightweight environments which are destroyed after all child processes exit, without root user rights requirement.</p>
4082</blockquote>
4083<p>Thanks to <a href="https://github.com/bb010g">@bb010g</a> for pointing this out.</p>
4084<h2 id="notes">Notes</h2>
4085<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
4086<p>The keyboard mapping is a bit strange. On my Spanish keyboard, the keys were as follows:</p>
4087</div>
4088<table><thead><tr><th>Keyboard</th><th>Maps to</th><th>Shift</th></tr></thead><tbody>
4089<tr><td>'</td><td>-</td><td>_</td></tr>
4090<tr><td>´</td><td>'</td><td>&quot;</td></tr>
4091<tr><td>`</td><td>[</td><td></td></tr>
4092<tr><td>+</td><td>]</td><td></td></tr>
4093<tr><td>¡</td><td>=</td><td></td></tr>
4094<tr><td>-</td><td>/</td><td></td></tr>
4095<tr><td>ñ</td><td>;</td><td></td></tr>
4096</tbody></table>
4097</content>
4098 </entry>
4099</feed>