Production Firefighting

A lot of the job is being pulled into something broken and owning it until it works, and just as often spotting the gaps and needs along the way and turning them into validators and tooling so they stop happening. Here are some examples.

Cook / Mobile

glTF avatar normals breaking on mobile

Symptom In Colony, players unlock the Fabricator building to create and share their own cosmetic items, helmets and weapons, much like in-game skins. Those items are produced through Atlas, the AI-native 3D content pipeline Parallel partnered with, and come into the engine as glTF assets. Their normal maps looked correct on iOS but rendered wrong and flipped on Android, so the models came out visibly broken on that platform.

Trace To isolate it I rebuilt the project under both mobile texture formats: it broke only under ETC2, never ASTC. ETC2 (LA_NORMALMAPS = 0) reconstructs the normal from .rg, and the plugin was zeroing exactly those red and green channels — applying the engine's cook-time ASTC normal swizzle (R=0, G=0, B=B, A=G) to runtime-built textures that never go through that cook path. Narrowed it to a single file, glTFRuntimeParserMaterials.cpp.

Fix Low on engineering resources, I paired with Claude to confirm the root cause against Unreal's engine source and shape a one-file fix: guard for uncompressed RGBA8 normal-map mips, copy RGB through unchanged, and duplicate green into alpha so .rg (ETC2) and .ra (ASTC LA) both reconstruct (X, Y). Scoped tightly enough that no other format or platform is touched.

Cook / Mobile

Nanite stripped on Android

Symptom Meshes that looked optimized in-editor were heavy on device, and over a few commits the cooked build grew sharply.

Trace A custom cooked-content audit I added to the validation framework, check_asset_size_cooked.py, flagged the worst offenders on cook and took me straight to them. They all had Nanite enabled, and since Nanite is stripped at cook when it is off project-wide, each fell back to full source geometry on device.

Fix Disabled Nanite on those meshes and let the optimized LODs do their job. The editor validators had flagged it too, but the assets were pushed straight through Perforce, so I added server-side Perforce checks to block submits from outside the editor.

Materials / Mobile

Time-animated materials breaking over long sessions

Symptom On low-end devices, time-driven materials (panners and similar) degraded the longer the game ran, getting visibly worse as elapsed time grew.

Trace The game was CPU bound, so I first suspected thermal throttling, but after testing a few devices and cooling them down the issue persisted, ruling that out. The materials animate off the engine's ever-increasing absolute Time. On lower-precision mobile GPUs that value eventually grows too large for the float to hold, so per-frame increments stop resolving and the animation quantizes, stutters, then falls apart.

Fix All came from one master material with a couple of instances, so the fix was contained: enabled full (high) float precision on the master, and every instance inherited it.

Pipeline Tool

Repairing broken asset import sources without reimporting

Symptom Hundreds of .uasset files had import metadata pointing at the wrong place (e.g. a local C:/P4V_Workspaces/... path instead of the shared P:/RawData source), which the validators flagged but were tedious and risky to fix by hand.

Fix Built a Python tool with three modes so artists never touch import settings manually: Move (source is on P: at the wrong path → relocate and reimport), Export (source missing on P: → write the asset out to the correct path, then reimport), and Reimport (source is correct but the stored path is stale → rewrite the metadata in place). Most assets were repaired by editing import data directly, no reimport needed.

Import path fixer tool: lists assets with broken import metadata and repairs them with Move, Export, or Reimport modes

The import-path fixer: scan flagged assets and repair them in place (Move / Export / Reimport).

Pipeline Tool

Clearing stuck exclusive checkouts (Perforce)

Symptom Exclusive-checkout assets (uassets, maps) routinely got stuck locked by someone who was out or had a dead workspace, blocking everyone else from editing them.

Fix Shipped a shared P4V custom tool plus a PowerShell installer so admins and supervisors can clear another user's lock in one click — wrapping p4 revert -C <client> -k <path> so it releases the lock without touching the other person's workspace files.

Perforce custom tool that clears another user's stuck exclusive checkout in one click

The P4V custom tool: clear another user's stuck exclusive checkout in one click.

Codenames and internal paths shown here are generalized. The work is on Parallel's Colony (mobile) and Sanctuary (open world).

Day-to-Day Production Fixes