OpenRA3/docs/REVERSE_ENGINEERING.md

# Reverse engineering workflow

OpenRA3 is reconstructed from two sources:

1. **Architecture** — the GPLv3 SAGE 1.0 tree,
   [`electronicarts/CnC_Generals_Zero_Hour`](https://github.com/electronicarts/CnC_Generals_Zero_Hour).
   RA3 runs SAGE 2.0, so class names, message flow and subsystem boundaries
   carry over even though the code does not.
2. **Facts** — the retail `ra3_1.12.game` binary (image base `0x400000`),
   analysed in Ghidra and, when needed, observed live.

Nothing in OpenRA3 should assert a structure or constant that is not either
copied from the reference or cited to a retail address.

## Fetching the reference

```bash
tools/fetch_reference.sh            # sparse-clone Code/GameEngine into reference/
```

The checkout is git-ignored (large, and GPLv3 terms differ from this repo's).

## Ghidra

The retail module is loaded into Ghidra as `ra3_1.12.game`
(`x86:LE:32:default`, 34k+ functions). The analysis is driven through the
Ghidra MCP bridge, so every recovered fact can be re-derived:

| Question | Tool call |
| --- | --- |
| What does an address do? | `decompile_function(address=0x…)` |
| What is this function? | `get_function_by_address(address=0x…)` |
| Who touches a global? | `get_xrefs_to(address=0x…)` |
| What are the vtable slots? | `list_class_members` / `analyze_data_region` |
| Where is a string referenced? | `search_strings` + `get_xrefs_to` |

### Recovery loop

1. Pick a subsystem from the reference tree (e.g. `MessageStream`).
2. Find its RTTI/vtable in the binary via `search_strings` and xrefs.
3. Decompile the constructor to recover object size and field init order.
4. Decompile the hot methods to recover field meaning.
5. Write the OpenRA3 module with a comment citing the address.
6. Add a smoke test that pins the behaviour.

## Recovered symbol map (retail `ra3_1.12.game`)

### Engine singletons

Each holds an object pointer (0 when the subsystem is down).

| Global | Address | Object vtable |
| --- | --- | --- |
| `TheGameLogic` | `0x00cd8ce4` | `0x00beb630` |
| `ThePlayerList` | `0x00ce8c9c` | `0x00c5b9e0` |
| `ThePartitionManager` | `0x00ce2f9c` | `0x00c6af98` |
| `TheShroudManager` | `0x00ce2fa0` | `0x00c6aefc` |
| `ThePlacementGrid` | `0x00cd8d0c` | `0x00bea560` |
| `TheRecorder` | `0x00ce2fd0` | `0x00c10544` |
| `TheGlobalObjectRegistry` | `0x00cd8d08` | `0x00bea644` |
| `ThePlayerTemplateStore` | `0x00ce8ca0` | `0x00c5bc70` |
| `TheGameState` | `0x00cdbbc4` | `0x00bef0c0` |
| `GlobalData` | `0x00ce2fa8` | `0x00c0d8e4` |
| `TheMessageStream` | `0x00ce2fb8` | `0x00c0ecd4` |

### Reconstructed layouts

- **`GameLogic`** — tick counter at `+0x50` (incremented once per 30 Hz
  simulation step); `starting` guard flag at `+0xa7`, raised while a new match
  initialises.
- **`MessageStream`** — intrusive doubly linked list; head at `+0x24`, tail at
  `+0x28`. `appendMessage` (`0x0060c4a0`) allocates a `0x74`-byte node:
  `+0x00` next, `+0x04` prev, `+0x08` owner stream, `+0x0c` message type,
  `+0x10` player index, `+0x18` capacity, `+0x1c` data pointer (`node + 0x20`).
- **`Player`** — money via `std::vector<Money*>` at `+0xe4`; power at `+0x74`;
  team at `+0xac`; relation maps at `+0xfc` / `+0x100`.
- **Skirmish setup** (`SkirmishGameInfo`, pointer at `[0x00ce3a78]`) — starting
  cash `+0x64`, player slots `+0xfc` (stride `0x5c`, 6 slots), faction at
  `slot + 0x18` (`Empire=2`, `Allied=4`, `Soviet=8`, `Random=7`).

### Tactical view (camera)

The in-game camera is the `TheTacticalView` object, held in the global at
`0x00cdb7b4`. Its vtable accessors return zoom, pitch (current/target), yaw,
world position and FOV; the debug overlay `FUN_005ef0a0` prints them through the
format string at `0x00c0b900`. Mouse state is the singleton at `0x00ce9284`
(cursor position vtable slot `+0x3c`, button down `+0x48`); the keyboard manager
is `0x00ce927c` (modifier mask `+0x38`).

Per-map camera tuning is a named-field table in `.rdata` (around
`0x00c11a54`): `cameraMinHeight`, `cameraMaxHeight`, `cameraPitchAngle`,
`cameraYawAngle`, `cameraScrollSpeedScalar`, `cameraGroundMinHeight`,
`cameraGroundMaxHeight`. The map-load chunk `CHUNK_TacticalView`
(`0x00beea14`, consumed near `0x00548a00`) seeds the view from the map; the
tutorial actions `LOCK_CAMERA_SCROLL` / `LOCK_CAMERA_ZOOM` / `LOCK_CAMERA_ROTATION`
gate the controls.

OpenRA3 has no 3D terrain yet, so `ra3::render::view_camera` reproduces the
*controls* - wheel zoom, screen-edge scroll, clamped pan and opening on the
player's start - over the 2D map overview, not the retail perspective camera.

### Match start

The BEGIN button calls `SkirmishGameOptionsMenu::start` (`0x00b28d60`), which
copies the map, calls `GameInfo::startGame(0)`, seeds the logic random and
appends `MSG_NEW_GAME` (`0x2`). `startNewGame` itself is `0x00623e40`. OpenRA3
mirrors this two-phase `prepare_new_game` / `start_new_game` split.

## Container and map formats

Recovered by inspection of `Data\*.big` (see `ra3.fs` / `ra3.map`).

### `BIG4` archive

All integers little-endian except where noted:

```
offset 0   magic      "BIG4"
offset 4   fileSize   u32 LE   total archive size
offset 8   fileCount  u32 BE   number of entries
offset 12  indexSize  u32
offset 16  entries    fileCount * { offset u32 BE, size u32 BE, name cstring }
```

Entry offsets are absolute; payloads are RefPack-compressed.

### RefPack

EA's `10 FB` stream. `ra3.fs::refpack_decompress` implements the 2/3/4-byte
commands and the long-literal/stop opcodes; `refpack_output_size` reads the
declared output length without decompressing.

### Map file (`.map`)

Two layers of compression:

```
BIG4 payload          = RefPack  ->  "EAR\0" + u32 unpacked_size + RefPack  ->  "CkMp" ...
```

The `CkMp` payload is the compiled SAGE map: a type/field name table followed by
chunk data. Player start positions appear as waypoints named
`Player_1_Start`, `Player_2_Start`, ... Each waypoint record carries a `Coord3D`
(three little-endian `float`s) shortly after the name; `ra3.map` scans forward
from the name for the first plausible `(x, y, 0)` triple. Maps that store starts
in `MPPositionList` instead yield no waypoints and fall back.

Verified example (`map_mp_2_feasel4`): `Player_1_Start` = `(1338.9, 1940.5, 0)`,
`Player_2_Start` = `(1290.8, 1404.9, 0)`.

### Terrain (`ra3.terrain`)

The `CkMp` tree is a flat chunk list: `"CkMp"`, `u32 assetCount`, the name
table (`{ u8 len, name, u32 index }`, index descending from `assetCount`), then
`{ u32 index, u16 version, u32 size, data }` per chunk. On
`map_mp_2_feasel4` the terrain is `HeightMapData` v6 (540 x 600, border 20,
`u16` elevations, scale `0.0390625`) and `BlendTileData` v27.

`BlendTileData` opens with `NumTiles`, the `u16` tile grid, then the
blend/three-way/cliff tables (`u16` for v27); the passability flag arrays that
follow are not needed for rendering, so the texture table is located by
scanning for its `{ cellStart, cellCount, cellSize, magic }` + `u16`-prefixed
name entries. A tile value is `(cellIndex << 2) | variant`, and `cellIndex`
indexes the global `TextureCellCount`-cell table (each texture owning
`cellSize^2` 64 px cells). The textures themselves are
`art\terrain\<stem>.tga` in `Terrain.big` / `Core11.big` (RefPack + 256x256
TGA). Rendered top-down, `map_mp_2_feasel4` correlates 0.94 with the official
`_art.tga` overview.

### Terrain blending (`BlendTileData` tail)

After the tile grid, `BlendTileData` stores three per-cell `u16` tables —
`Blends`, `ThreeWayBlends` and `CliffTextures` — then `TextureCellCount`,
`BlendsCount`, the texture table, two magic words and `BlendsCount - 1` blend
descriptions (18 bytes each: `u32 secondaryTile`, four direction bytes,
`u8 flags`, `u8 twoSided`, `u32 0xFFFFFFFF`, `u32 0x7ADA0000`). A non-zero
`Blends[cell]` is a 1-based index into the descriptions; `secondaryTile` is a
packed tile value (`secondaryTile >> 2` is its cell).

The retail `Terrain.fx` (compiled `terrain.fxo`, parameters `Terrain.BaseTexture`,
`Terrain.MacroTexture`, `MapCellSize`, `IsTerrainAtlasEnabled`; technique
`TerrainTile`) cross-fades a cell's base tile into `secondaryTile` with a linear
ramp selected by `BlendDirection`: `1` right, `2` top, `4` top-right, `8`
top-left, where `flags` bit 0 flips the axis and bit 1 marks a two-sided
diagonal. OpenSAGE's `Terrain.frag` reconstructs the exact
`CalculateBlendFactor`; `ra3::terrain::blend_factor` and `shaders/terrain.frag`
mirror it. The row axis is *not* inverted (73% of long-axis blends point at a
neighbour of the same texture, versus 25% inverted). On `map_mp_2_feasel4`,
33828 of 324000 cells carry a blend and there are 7094 descriptions.

The GPU atlas pads every 64 px tile with a 2-texel replicated gutter. Packed
edge-to-edge, bilinear/mipmap filtering averaged two unrelated tiles at every
cell border — that cross-tile bleed was the visible grid line the hardware path
drew. (`shaders/terrain.frag` samples `cell_stride = cell_texels + 2 * gutter`.)

### Open question: per-cell tile sampling

Measured on `map_mp_2_feasel4`, sampling each cell as its own 64 px block and
restarting the UV every cell leaves a 1.39x edge spike at cell boundaries
(43.4 vs 31.1 mean gradient at 8 px/cell). Two candidate mappings reduce it and
need a visual decision against the retail art:

| Mapping | Boundary/interior |
| --- | --- |
| per-cell 64 px block (current) | 1.39 |
| OpenSAGE `BlendTileTextureIndex` Morton layout, 32 px block | 1.18 |
| continuous `uv / (cellSize * 2)`, per OpenSAGE `Terrain.frag` | 1.11 |

The retail `Terrain.frag` (OpenSAGE) samples the tile texture *continuously*
(`uv / (CellSize * 2)`), so adjacent cells never restart the texture; our
per-cell restart is the remaining source of grid lines. Switching the atlas
from 64 px blocks to continuous 32 px regions (or the Morton 8x8 layout) is the
next step, pending an art correlation check.

### Compiled art (BinaryAssetBuilder) and map objects

Retail RA3 ships no `.w3x`/`.w3d` files: BinaryAssetBuilder bakes every model,
texture and script into a *binary asset stream* — a `.manifest` index plus a
`.bin` of relocatable instance data (and optional `.relo`/`.imp` fixups). The
layout (little-endian) is:

```
ManifestHeader (48 B)  isBigEndian u8, isLinked u8, version u16,
                       streamChecksum, allTypesHash, assetCount u32,
                       totalInstanceDataSize, maxInstance/maxRelocation/
                       maxImportsChunkSize, assetReferenceBufferSize,
                       referenceManifestNameBufferSize, assetNameBufferSize,
                       sourceFileNameBufferSize
AssetEntry (48 B) * count
                       typeId, instanceId, typeHash, instanceHash,
                       assetReferenceOffset i32, assetReferenceCount i32,
                       nameOffset i32, sourceFileNameOffset i32,
                       instanceDataSize i32, relocationDataSize i32,
                       importsDataSize i32, tokenized u32
then the reference / referenced-name / asset-name / source-name buffers
```

Asset names are `Type:Instance` (e.g. `W3DMesh:BB_GRASS02`). Instance pointers
are stored as offsets from the start of the instance data (which begins at byte
4 of `.bin`, after the stream checksum), so a slice is readable without the
relocation stream.

Each multiplayer map carries its own stream (`data\maps\official\<id>\map.bin`)
but it is **linked**: only map-specific assets (the terrain texture atlas,
scripts, `GameMap`) have data; the rendered props are imported and therefore
have `instanceDataSize == 0`. The complete prop art (meshes + textures) lives in
`Data\WBData.big`'s `data\worldbuilder.bin`, which is **uncompressed** (first
four bytes are the stream checksum, not `10 FB`), so `ra3.models` reads the
1.1 GB stream lazily — the manifest is parsed and only the needed instance
slices are read.

`W3DMesh` compiled layout (offsets from the instance start):

```
+4   vertexBufferPtr        +52 triangleCount      +56 triangleItemPtr
+60  shaderNameLength       +64 shaderNamePtr
vertexBuffer: +0 numVertices, +4 stride, +8 elementDataPtr,
              +12 declarationBytes, +16 declarationPtr
declaration:  text "p0:00:3f32 n0:0C:3f32 t0:1C:2f32" (D3D9 usage:index:offset:type)
triangles:    triangleCount * { u32 indexCount, u32 indexPtr } (24 B each), u32 indices
```

The diffuse texture is found through the mesh's `FXShaderConstant`s
(`+76` count, `+80` items): a texture-valued constant (TypeId `0xA59096A6`)
names its role (`DiffuseTexture`, `NormalMap`, `SpecMap`) and points at a
1-based index into the mesh's cross-asset references, which resolve by
`(typeId, instanceId)`. The `Texture` instance embeds a standard DDS file (at
`u32@+4`, or scan for `"DDS "`); `ra3.models::decode_dds` decodes DXT1/3/5 and
uncompressed 16/24/32-bit.

**Vertices are stored in bone space, not object space.** A mesh whose vertex
declaration carries blend data (`i0:..:4u8 w0:..:4u8n`, e.g. buildings and
vehicles) must be skinned; props without it (`BB_GRASS02`, `IF_STREETSEGMENT01`)
are already in object space. The skeleton is a `W3DHierarchy` asset named after
the mesh's instance prefix (`W3DMesh:FI_STRUCTURE_02.NEWSKIN_CIV01` →
`W3DHierarchy:FI_STRUCTURE_02`). Compiled layout:

```
W3DHierarchy: u32 pad, u32 boneCount, u32 headerBytes, then boneCount records
100 B each:
    u32 nameHash, i32 parent (-1 = root), f32 translation[3],
    f32 quaternion[4] (x, y, z, w), f32 matrix[12]
```

The default (bind) pose is rebuilt by composing each bone's local
translation/quaternion down the parent chain, then `skinnedPos = Σ wᵢ ·
(Rᵢ·p + Tᵢ)` (and the normal by the rotation only). `ra3.models` does this before
placing the mesh at the map object's `(x, y, angle)`.

The compiled shader (below) binds **one joint per vertex** — `WorldBones` holds
64 bones as 2 `float4` each (quaternion `c[128+2j]`, translation `c[129+2j]`) —
so the skin is rigid: `blendindices.x` selects the joint, remapped through the
mesh's per-model **bone table** (vertex-descriptor `+0x14` = bone count,
`+0x18` = `u16` bone indices into the `W3DHierarchy`). Applying the raw blend
index without that remap tears models apart (`FI_BUILDING01`'s table is
`[0,14,15,16,17,18]`, not `[0..5]`).

The map objects that lie flat on the ground (sidewalks, deck pieces) are
coplanar with the terrain; retail biases their depth in the shader so they do
not z-fight. The object pass reproduces that with a small negative depth bias
(Vulkan `depthBiasConstant/SlopeFactor`, and a half-unit bias in the software
rasteriser).

Meshes whose material has no diffuse texture (`DefaultW3D.fx`, `BasicW3D.fx` —
`FXLIGHTS`/ambient helper billboards) are not opaque geometry and are skipped;
drawing them fills the frame with garbage triangles. Likewise the
`BuildingsGenericDamageFill.fx` **damage-fill** sub-meshes are skipped: they are
the wrecked-interior shell (e.g. `CBBuilding_Wood`, an orange plank texture) that
retail only reveals through damage holes, but our opaque pass would paint it over
the main shell and tint whole buildings warm.

### Official shader behaviour (`Shaders.big` → `*.fxo`)

The compiled D3D9 effects in `Data\Shaders.big` name their parameters, so the
model pipeline is recoverable. `buildingsgeneric.fxo` (`BuildingsGeneric.fx`)
vertex stage: skinning (above), `World`/`ViewProjection`, and vertex color
`c0` multiplied into the lit color
(`(Ambient·AmbientColor + Σ DLᵢ.Color·max(N·DLᵢ,0)) · DiffuseColor · vertexColor`).
Pixel stage samples `DiffuseTexture`/`NormalMap`/`SpecMap`/`DamagedTexture`/
`CloudTexture` (all at **UV0**, except `DamagedTexture` at `v0.wz` = transposed
UV1), then `final.rgb *= TintColor` and `*= ShroudTexture.rgb`. `DiffuseVelocity`
is not used for static structures. So the diffuse texture is UV0 and is tinted by
vertex color and `TintColor`; `basicw3d.fxo` instead modulates a single
macro/lightmap with `(vertexColor + additive) * diffuse * 2` and has no normal
map. (Recovered by disassembling the embedded `vs_3_0`/`ps_3_0` bytecode.)

### Water / ocean (`Ocean.fx`, `OceanDisplacement.fx`, `RiverWater.fx`, `UnderwaterDeferred.fx`)

The SAGE water surface is reconstructed by OpenSAGE as
`Assets/Shaders/Water.vert`+`Water.frag` (same family as retail `Ocean.fx`). The
surface is a world-space mesh drawn with alpha blending, fed by two render
targets rendered before it: a **reflection** map (scene from the mirrored camera
about `GlobalWaterSettings.ReflectionPlaneZ`) and a **refraction** map + depth.
The fragment model (`Water.frag`) is:

- `waterUV = worldPos.xy / 320`; a scrolling `WaterTexture` supplies both a
  flow distortion (`(tex.xy*2-1)*0.05`) and the flow layer; a `BumpTexture`
  supplies the surface `worldNormal`.
- `fresnelFactor = dot(viewVector, +Z)`; reflection/refraction are sampled in
  screen space (`gl_FragCoord / ViewportSize`), each displaced by the distortion.
- `linearWaterDepth = linearize(RefractionDepth) - linearize(gl_FragCoord.z)`;
  `alpha = clamp((linearWaterDepth/2)/TransparentWaterDepth, 0, TransparentWaterMinOpacity)`.
- `final = diffuseColor * textureColor * cloudColor`, then mixed with
  `mix(reflectionColor, refractionColor, fresnelFactor)` (both maps on) or just
  one of them, per `IsRenderReflection` / `IsRenderRefraction`.
- Per-time-of-day `WaterSet`: `WaterTexture`, `UScrollPerMS`/`VScrollPerMS`,
  `DiffuseColor`, `TransparentDiffuseColor`; `WaterTransparency` supplies
  `TransparentWaterDepth`/`TransparentWaterMinOpacity`, the skybox faces,
  `RiverTransparencyMultiplier`, `ReflectionPlaneZ`/`ReflectionOn`.

OpenRA3 has no water mesh or reflection/refraction targets (its terrain is
raymarched), so the model is folded into the terrain pass (`shaders/terrain.frag`
and its HLSL/GLSL-ES/WGSL twins): the ray's water-plane hit takes a scrolling
wave normal built from the retail bump map combined with a de-gridded procedural
wave, Schlick fresnel (F0 = 0.02), a sky reflection and a depth-graded
refraction, SAGE diffuse + specular lighting, a depth-based transparency fade,
and — when the camera is below `ReflectionPlaneZ` — an underwater tint/fog
(`UnderwaterDeferred.fx`). The two retail maps
`art/terrain/ra3_deepocean.tga` (flow/distortion) and `ra3_deepocean_nrm.tga`
(bump normal) are appended as the **last two layers of the terrain atlas**
(`ra3::terrain::build_gpu_terrain`, `water_flow = layer_count - 2`,
`water_normal = layer_count - 1`), so every backend samples them with the
existing atlas binding; absent maps fall back to a neutral layer. True
reflection/refraction render targets are still the next step.
`GPUParticleOceanDisplacement.fx` drives wave displacement from a GPU particle
buffer and has no analogue here.


The map's `ObjectsList` chunk is a list of nested `Object` assets —
`Coord3D`, Z `angle`, `RoadType` u32, a `u16`-prefixed type-name and a property
list whose keys index the shared name table (`ra3.map::parse_objects`). Each
type resolves to the `W3DMesh` parts whose instance name equals it or starts
with `<type>.`.

### Roads / sidewalks (`Road` assets)

The flat sidewalk and road decals are not meshes: the map places them as
**consecutive pairs** of objects — a `RoadType::Start` (`2`) and a
`RoadType::End` (`4`) at the segment's two ends (`Angled` `8`, `TightCurve`
`64` and `EndCap` `128` are curve/cap flags; `BridgeStart/End` `16/32` are
bridges). The road's type-name (`IslandFortressSidewalk01`, ...) resolves to a
compiled `Road` asset in the same art stream:

```
Road: u32 version, u32 pad, u32 textureCount, f32 roadWidth, f32 pad,
      f32 ???, then textureCount texture references (diffuse, normal)
```

`ra3.models` pairs the objects in file order, emits a flat quad ribbon of
`roadWidth` along each segment (overlapping the joins by half a width), sampled
with the diffuse texture, and lifts it onto the terrain. Retail gives roads a
small depth offset toward the screen so they do not clip into the ground; the
object pass reproduces that with the same negative depth bias used for the other
ground decals (`depthBiasConstant/SlopeFactor` on Vulkan, a half-unit bias in the
software rasteriser).

### Map display names

The skirmish map list labels live in `Data\English.big`'s
`data\gamestrings.csf` (the newest `Lang-English*.big` wins) under
`MAP:<UPPERCASE_ID>`, e.g. `MAP:MAP_MP_2_FEASEL4` = "Battlebase Beta". CSF
values are UTF-16 code units whose low byte is XORed with `0xFF`
(`ra3::map::parse_map_names`). `openra3 extract` writes the decoded table to
`maps/map_names.tsv`; `openra3 menu` shows them instead of the raw map id.


### Still to recover

- `MPPositionList` layout (per-player starts for maps without waypoints).
- Cliff textures and the `CliffTextureMapping` UV remap (`CliffTextures` is
  parsed but not yet drawn).
- The `Road` network mesher (the map's sidewalk/road objects reference `Road`
  templates, not `W3DMesh` assets).
- W3D container/hierarchy assembly and animation (props are drawn as their
  static mesh parts; skinned/animated in-game models are not).
- Compiled asset blobs (`global.bin`, `static.*.bin`) and the `.manifest`
  schema used to deserialise them.