This should be a quick one! No crazy bugs, just an interesting little target I hadn't seen anyone dig into, and a few helpful tricks for quick reverse engineering.

A little bit ago I re-installed the racing game Trackmania, and I noticed I got product ads displayed at me in-game alongside the racetrack. Where were those coming from?

A bit of digging revealed a DLL loaded into the game, anzu.dll. Finding the company’s site, anzu.io helped to peel back the covers. These in-game ads were part of a complete ad-breakfast, including targeting, analytics, ad bidding, and more. Their site even claimed you could target ads based on in-game metrics). Their site also had many fun pictures of smiling people holding controllers in teal hoodies.

Digging into the DLL I wanted to see how this library was being used. The DLL was stripped of symbol information, hiding function and variable names that would have made reverse engineering easier. The exceptions were the exported symbols, and there were quite a few exports. We could just start walking through in Binary Ninja or another decompiler, but having a concrete look at the way the game was using the library would make things go much quicker.

Trampoline Tricks

I wanted to start with just tracking what Anzu_ functions were called, and how. Which functions exported by the library even mattered? What did the arguments look like? To quickly gather call traces I generated a "trampoline" DLL that exported all of the same functions as the original, but each function would simply output to a log file before jumping into the original DLL.

To do this we couldn't just export a C function with the same name, like so:

int Anzu_CampaignMetricGet(...) {  
    printf("Called Anzu_CampaignMetricGet\n");  
    return real_Anzu_CampaignMetricGet();  
}

This would not work as we didn't know how many arguments these calls took, or what types the arguments should have been. I solved this by generating a small assembly stub for each export that would:

Save the register state to the stack
Call the logging function with the export's name
Restore the register state
jmp to the original function

If you try this, just make sure you save the SIMD registers as well, or you will clobber your float arguments and end up sending a bunch of strange requests to the server. (Oops!)

EXTERN real_Anzu_CampaignMetricGet : QWORD

Anzu_CampaignMetricGet proc EXPORT  
    //; save volatile registers  
    push rax  
    push rcx  
    //;...snipped

    //; call log  
    lea rcx, [Anzu_CampaignMetricGet_name]  
    call log_proc

    //; restore  
    //;...snipped  
    pop rcx  
    pop rax

    //; go to original  
    jmp [real_Anzu_CampaignMetricGet]  
Anzu_CampaignMetricGet ENDP

Anzu_CampaignMetricGet_name db "Anzu_CampaignMetricGet", 0

Example Trampoline

Our generated trampoline logic would now load the original DLL and save off pointers to all the original functions. Our exports were these small assembly stubs passing all the arguments on to the original functions. (I also double checked that all the exports seemed to be functions, exported data pointers would need some extra care.)

You can see my messy little code generator here in the repo gen_stubs.py. It outputs:

A declaration section to:
- Export the trampoline stub with __declspec
- Create a global to hold a pointer to the original function
Logic to grab the original function pointer using GetProcAddress
The assembly stub for each needed export

This all took only a few minutes and a python script to be able to gather some traces showing how Trackmania was using the Anzu library. (Thankfully Trackmania didn't do any sort of signature checks on the DLL, and would just load any DLL at the correct path.)

Interception DLL Loaded  
Init complete  
@ Anzu_GetVersionFloat  
@ Anzu_InternalDebugging  
@ Anzu_SetGDPRConsent  
@ Anzu_InternalDebugging  
@ Anzu_ApplicationActive  
@ Anzu_InternalDebugging  
@ Anzu_SetLogLevel  
@ Anzu_RegisterLogCallback  
@ Anzu_RegisterMessageCallback  
@ Anzu_RegisterTextureUpdateCallback  
@ Anzu_RegisterTextureInitCallback  
@ Anzu_RegisterTextureImpressionCallback  
@ Anzu_InternalDebugging  
@ Anzu_InternalDebugging  
@ Anzu_RegisterTexturePlaybackCompleteCallback  
@ Anzu_InternalDebugging  
@ Anzu_InternalDebugging  
...snipped

Initial Library Call Trace

Of course, we could have done all this easily with a scriptable debugger like Frida, x64dbg, or WinDbg. A simple .logopen and pattern breakpoint (bm anzu!* ".printf \"%y\\n\", @rip; gc") in WinDbg would get us a similar trace.

If we could have accomplished all this in a debugger, why go through the trouble of building our own DLL? I chose the DLL route to give us a nice way to start iterating. Our initial traces showed us the functions we cared to start digging into. As we delved into those with our disassembler we started defining nicer stubs than our generic assembly trampolines. We started logging arguments and even changing functionality. Below is a stub that enabled more verbose logging in Anzu, even when the game asked for only error logging.

__declspec(dllexport) void Anzu_SetLogLevel(int32_t level) {

    fprintf(g_logf, "@ Anzu_SetLogLevel(%x) (forcing 0 and injecting debug setup)\n", level);  
    real_Anzu_SetLogLevel(0);

    fprintf(g_logf, "DEBUG: Force registering log file:" LOG_NAME "\\n");  
    real_Anzu_InternalDebugging(0xc0de5b05, LOG_NAME);  
}

Forcing Verbose Logging

Interception DLL Loaded  
Init complete  
@ Anzu_GetVersionFloat  
@ Anzu_InternalDebugging(c0de5b11, 0000000008000000) => 0000000000000000  
@ Anzu_SetGDPRConsent  
@ Anzu_InternalDebugging(c0de5b10, 00000165857AD171) => 0000000000000000  
@ Anzu_ApplicationActive(d6eb0001)  
@ Anzu_InternalDebugging(c0de5b03, 0000000000000001) => 0000000000000000  
@ Anzu_SetLogLevel(4) (forcing 0 and injecting debug setup)  
DEBUG: Force registering log file:log.log  
@ Anzu_RegisterMessageCallback(00007FF783725A50, 0000000000000000) (redirecting)  
DEBUG: Force registering net callback  
@ Anzu_RegisterMessageCallback(00007FF61B645B70, 0000000000000000) (redirecting)  
@ Anzu_RegisterTextureUpdateCallback  
@ Anzu_RegisterTextureInitCallback  
@ Anzu_RegisterTextureImpressionCallback  
@ Anzu_InternalDebugging(c0de5b31, 00007FF61B646840) => 0000000000000000  
@ Anzu_InternalDebugging(c0de5b04, 00007FF61B6469E0) => 0000000000000000  
@ Anzu_RegisterTexturePlaybackCompleteCallback  
@ Anzu_InternalDebugging(c0de5aff, 0000000000000001) => 0000000000000000  
@ Anzu_InternalDebugging(c0de5b00, 0000000000000001) => 0000000000000000  
@ Anzu_InternalDebugging(c0de5b16, 00007FF61C21E1E8) => 0000000000000000  
@ Anzu_InternalDebugging(c0de5b26, 00007FF61C21E3F8) => 0000000000000000  
@ Anzu_InternalDebugging(c0de5b17, 00007FF61C0BEEAC) => 0000000000000000  
@ Anzu_InternalDebugging(c0de5b27, 00007FF61C1ECE80) => 0000000000000000  
@ Anzu_Initialize  
@ Log CB 0: Registering SDK exit handler ...  
@ Message Callback: {"data":"ok","subtype":"initialize","type":"status"}  
@ Log CB 0: Cache capacity: 134217728, size: 12322  
@ Log CB 0: Cleaning cache entries...  
@ Log CB 1: Anzu SDK 5.41 is being initialized, CPUs = 24  
...snipped

Updated Trace A Few DLL Iterations Later

This codebase also gave us a jumping off point for building our own harness. With a harness for anzu.dll we could replace Trackmania, giving us faster resets and a simpler debugging environment. The code we wrote as we iterated on our middleware DLL, along with the gathered traces, made it easy to spin up a harness that used the Anzu library the same way Trackmania did. Having some harness code that can evolve alongside your reverse engineering effort can be worth more than a bunch of comments in your disassembler.

A few interesting things started popping out as we dug into the surface layer interactions with the Anzu DLL. First was their Anzu_InternalDebugging function exposed a lot of interesting functionality that was being used. By cross-referencing data used in their logging functions we found one of these internal commands would redirect the debug output to a log file.

*Layout of* ***Anzu_InternalDebugging***

There were lots of interesting callback functions we added to our trampoline DLL and harness. Log callbacks, network callbacks, texture update callbacks, etc.

We also saw raycasts reporting ad visibility to the library (Anzu__Texture_SetVisibilityScore). These were used for reporting when users actually saw the ad, and how much of it. The logging callbacks were also showing us some of the network connections that this library was making as it called home to load the latest configuration and logic from the mothership.

This lead us to our next quick RE trick!

Network Harness

While we could have continued to dive into our disassemblers, we cut to the good stuff quicker by getting in-between the network and the DLL, similar to how we got between the game and the library.

For quickly getting between the network comms of an application that doesn't support a proxy, mitmproxy has some neat tools. Using the local mode, we could redirect traffic for our processes by PID or process name. (mitmproxy uses WinDivert behind the scenes on Windows). Making an mitmproxy plugin is simple, and lets us do things like we did in our trampoline-DLL:

modifying certain responses
logging interesting calls
iterating our tooling as we better understand our target
etc.

mitmproxy also handled the certificate generation for us and TLS-shenanigans behind the scenes. This made it pretty plug-and-play to start messing with the Anzu traffic.

Anzu thankfully made this interception simple for us because it used the system's trust store for verifying TLS certificates. We just had to add our own root certificate to our machine's store and let mitmproxy use it. After that Anzu happily let us do some mischief in the middle, accepting our TLS certificate for it's servers. Besides the lack of pinning, the TLS verification logic seemed tight.

Trackmania itself seemed to do some kind of certificate pinning for its communications and would not accept our intercepted TLS connections. If the Anzu DLL had done the same we would have needed to go find and subvert the verification logic before we could easily intervene in the communications. As is, all we had to do was tell mitmproxy to only intercept connections to the Anzu backend, letting the Trackmania connections continue unimpeded.

mitmproxy --mode local:trackmania -s .\server\server.py --allow-hosts '^(gateway\.prod\.anzu\-us\.com|3\.231\.86\.84|3\.234\.186\.225|52\.55\.69\.208|3\.230\.88\.142|35\.171\.212\.145|3\.91\.163\.159):443$'

mitmproxy using --allow-hosts

Having the harness made it easy to identify what domains were associted with Anzu and interceptable. We also could have used --ignore-hosts similarly to just cut out the Trackmania specific traffic.

Going through the network communications we found it downloading JavaScript that contained the core logic for managing the ad campaigns. With a little bit of beautifying and code reveiw, we figured out how to craft responses to serve our own ads. See the repo for the full plugin.

def tls_start_client(self, tlsdata):  
        self.lg(f"TLS Start: {tlsdata.conn.sni}")

    def request(self, flow):  
        self.lg(f"Request: {flow.request.method} {flow.request.pretty_url}")

        if flow.request.path.startswith("/multi_gateway/"):  
            # give our own response  
            self.update_campaigns()  
            resp = self.ads  
            self.lg(f"Responding")

            flow.response = http.Response.make(  
                200,  
                bytes(json.dumps(resp), "utf8"),  
            )

mitmproxy Plugin for Serving Our Own Ad Campaign

*Injected Response for a Custom Ad Campaign*

For the longest time I thought Anzu had shadow-banned me, as I was served no ads after I started developing my tooling. It wasn't until I had already gotten my own ad campaigns going that I did eventually capture traffic of an actual ad campaign. The real ad campaigns were interesting though, as they supported some interesting features. They could contain JavaScript filters used to better target ads from on the client’s machine, as well as tons of options for tracking callbacks at various points of ad playback. On the ad campaign I was served they had over twenty callback URLs registered to report ad impressions going to a variety of tracking domains.

Our plugin makes it easy to respond with our own campaign when the system asks for ads from the gateway. In my ads, I served some images straight from localhost. It is left as an exercise for the reader to hook this up to cataas.com

There are still some fun questions left open. What in-game metrics are other games actually reporting? Besides the visibility metrics, I didn't see any other game behavior mentioned. Do you remember the funny Pranking My Roommate with Targeted Facebook Ads article; can we replicate that? Can I advertise only to those stuck in losers queue? And what does it look like when the MQTT library is used? (I was never served a configuration that used it.) AWS put out a case study about Anzu, even hinting at some possible future LLM use in contextual advertising, has that been rolled out yet? Will Anzu ever respond to my email and let me bid on a few ads too? (I'll be nice!)

There are interesting hidden systems like Anzu all over the place; it is a lot of fun digging into them. Even though I didn't find any major security issues in the client, it was still fun to be able to whip up some harnesses and peer behind the curtain a bit. Feel free to contact me (@jordan9001) and tell me about your own little reverse engineering projects! If you own an interesting system you want us at Atredis to help secure, please contact us via our contact form; we would love to work with you.

A Peek into an In-Game Ad Client

Trampoline Tricks

Network Harness

Contact Us

A Peek into an In-Game Ad Client

Trampoline Tricks

Network Harness

3D Printing Flying Probe Test Harnesses: Can you?

Contact Us