OpenRA3 v0.0.1: C++26 modules, BIG4/RefPack reader, minimal skirmish
This commit is contained in:
EnderTheCoder
@@ -0,0 +1,78 @@
|
||||
# Architecture
|
||||
|
||||
OpenRA3 is organised as a stack of C++ modules. Each layer may import the ones
|
||||
below it, never the ones above.
|
||||
|
||||
```
|
||||
┌───────────────────────────────┐
|
||||
applications │ openra3 (apps/openra3) │
|
||||
└───────────────┬───────────────┘
|
||||
│ import ra3
|
||||
┌───────────────▼───────────────┐
|
||||
umbrella │ ra3 (re-exports everything) │
|
||||
└───────────────┬───────────────┘
|
||||
┌───────────────┬────────┴────────┬───────────────┐
|
||||
▼ ▼ ▼ ▼
|
||||
ra3.skirmish ra3.client ra3.game ra3.map
|
||||
match rules display/loop RA3 sides map catalog
|
||||
│ │ │ │
|
||||
└───────┬───────┴────────┬────────┘ │
|
||||
▼ ▼ ▼
|
||||
ra3.logic ra3.core ra3.fs
|
||||
simulation types/math/random BIG4 + RefPack
|
||||
```
|
||||
|
||||
## Module responsibilities
|
||||
|
||||
### `ra3.core`
|
||||
The vocabulary every other module shares, mirroring SAGE's `GameEngine/Common`:
|
||||
`real`/`int32`/`uint32`, `coord3d`/`coord2d`/`rgb_color`, `ascii_string` +
|
||||
`make_name_key`, the deterministic `random` stream, and the `message_stream`
|
||||
command bus (node layout derived from retail `MessageStream::appendMessage`,
|
||||
`0x0060c4a0`).
|
||||
|
||||
### `ra3.logic`
|
||||
The deterministic simulation, mirroring SAGE's `GameLogic`: `thing` → `object`
|
||||
with pluggable `update_module`s, `player`/`player_list`, the `partition_manager`
|
||||
spatial grid, and the 30 Hz `game_logic` driver (`prepare_new_game` /
|
||||
`start_new_game` / `update`).
|
||||
|
||||
### `ra3.client`
|
||||
The presentation boundary: an abstract `display` with a `headless_display`
|
||||
implementation, and `game_client`, the seam a future W3D/D3D9 renderer plugs
|
||||
into.
|
||||
|
||||
### `ra3.game`
|
||||
Red Alert 3 data that SAGE keeps in `PlayerTemplate`: the three sides
|
||||
(`faction` flags `Empire=2`, `Allied=4`, `Soviet=8`, `Random=7`, recovered from
|
||||
the retail skirmish setup) and skirmish defaults.
|
||||
|
||||
### `ra3.fs`
|
||||
Reading the user's installation. Implements the `BIG4` archive container and
|
||||
EA's RefPack codec, plus `find_game_dir` (`--game-dir` / `$RA3_GAME_DIR` /
|
||||
`C:\Red Alert 3`). Only the archive index is held in memory; payloads are read
|
||||
on demand.
|
||||
|
||||
### `ra3.map`
|
||||
Map discovery and loading: scans `MapsMultiplayer.big` for main map entries,
|
||||
unwraps the two compression layers (`BIG4` RefPack → `EAR\0` wrapper → RefPack →
|
||||
`CkMp`), and recovers `Player_N_Start` waypoint coordinates. Degenerate
|
||||
extractions are rejected so the caller can fall back.
|
||||
|
||||
### `ra3.skirmish`
|
||||
The minimal match: two players, unit classes (harvester/infantry/tank/base),
|
||||
passive + harvester income, a simple build AI, movement and combat on a fixed
|
||||
30 Hz step, and a base-destruction win condition. Fully deterministic.
|
||||
|
||||
## Design rules
|
||||
|
||||
1. **No raw owning pointers.** Ownership is `std::unique_ptr`; the partition
|
||||
manager holds non-owning `thing *` views only.
|
||||
2. **Portable simulation.** No platform APIs in `ra3.core` / `ra3.logic`. All
|
||||
platform concerns live behind `display`.
|
||||
3. **Determinism.** Anything that can diverge between runs (random, iteration
|
||||
order) is explicit and seedable.
|
||||
4. **RE-traceable.** Where a structure or constant comes from the retail
|
||||
binary, the address is cited in the comment.
|
||||
5. **No bundled assets, no online.** Game data is read from the user's install
|
||||
and never committed; there is no networking or online service.
|
||||
@@ -0,0 +1,135 @@
|
||||
# Reverse engineering workflow
|
||||
|
||||
OpenRA3 is reconstructed from two sources:
|
||||
|
||||
1. **Architecture** — the GPLv3 SAGE 1.0 tree,
|
||||
[`electronicarts/CnC_Generals_Zero_Hour`](https://github.com/electronicarts/CnC_Generals_Zero_Hour).
|
||||
RA3 runs SAGE 2.0, so class names, message flow and subsystem boundaries
|
||||
carry over even though the code does not.
|
||||
2. **Facts** — the retail `ra3_1.12.game` binary (image base `0x400000`),
|
||||
analysed in Ghidra and, when needed, observed live.
|
||||
|
||||
Nothing in OpenRA3 should assert a structure or constant that is not either
|
||||
copied from the reference or cited to a retail address.
|
||||
|
||||
## Fetching the reference
|
||||
|
||||
```bash
|
||||
tools/fetch_reference.sh # sparse-clone Code/GameEngine into reference/
|
||||
```
|
||||
|
||||
The checkout is git-ignored (large, and GPLv3 terms differ from this repo's).
|
||||
|
||||
## Ghidra
|
||||
|
||||
The retail module is loaded into Ghidra as `ra3_1.12.game`
|
||||
(`x86:LE:32:default`, 34k+ functions). The analysis is driven through the
|
||||
Ghidra MCP bridge, so every recovered fact can be re-derived:
|
||||
|
||||
| Question | Tool call |
|
||||
| --- | --- |
|
||||
| What does an address do? | `decompile_function(address=0x…)` |
|
||||
| What is this function? | `get_function_by_address(address=0x…)` |
|
||||
| Who touches a global? | `get_xrefs_to(address=0x…)` |
|
||||
| What are the vtable slots? | `list_class_members` / `analyze_data_region` |
|
||||
| Where is a string referenced? | `search_strings` + `get_xrefs_to` |
|
||||
|
||||
### Recovery loop
|
||||
|
||||
1. Pick a subsystem from the reference tree (e.g. `MessageStream`).
|
||||
2. Find its RTTI/vtable in the binary via `search_strings` and xrefs.
|
||||
3. Decompile the constructor to recover object size and field init order.
|
||||
4. Decompile the hot methods to recover field meaning.
|
||||
5. Write the OpenRA3 module with a comment citing the address.
|
||||
6. Add a smoke test that pins the behaviour.
|
||||
|
||||
## Recovered symbol map (retail `ra3_1.12.game`)
|
||||
|
||||
### Engine singletons
|
||||
|
||||
Each holds an object pointer (0 when the subsystem is down).
|
||||
|
||||
| Global | Address | Object vtable |
|
||||
| --- | --- | --- |
|
||||
| `TheGameLogic` | `0x00cd8ce4` | `0x00beb630` |
|
||||
| `ThePlayerList` | `0x00ce8c9c` | `0x00c5b9e0` |
|
||||
| `ThePartitionManager` | `0x00ce2f9c` | `0x00c6af98` |
|
||||
| `TheShroudManager` | `0x00ce2fa0` | `0x00c6aefc` |
|
||||
| `ThePlacementGrid` | `0x00cd8d0c` | `0x00bea560` |
|
||||
| `TheRecorder` | `0x00ce2fd0` | `0x00c10544` |
|
||||
| `TheGlobalObjectRegistry` | `0x00cd8d08` | `0x00bea644` |
|
||||
| `ThePlayerTemplateStore` | `0x00ce8ca0` | `0x00c5bc70` |
|
||||
| `TheGameState` | `0x00cdbbc4` | `0x00bef0c0` |
|
||||
| `GlobalData` | `0x00ce2fa8` | `0x00c0d8e4` |
|
||||
| `TheMessageStream` | `0x00ce2fb8` | `0x00c0ecd4` |
|
||||
|
||||
### Reconstructed layouts
|
||||
|
||||
- **`GameLogic`** — tick counter at `+0x50` (incremented once per 30 Hz
|
||||
simulation step); `starting` guard flag at `+0xa7`, raised while a new match
|
||||
initialises.
|
||||
- **`MessageStream`** — intrusive doubly linked list; head at `+0x24`, tail at
|
||||
`+0x28`. `appendMessage` (`0x0060c4a0`) allocates a `0x74`-byte node:
|
||||
`+0x00` next, `+0x04` prev, `+0x08` owner stream, `+0x0c` message type,
|
||||
`+0x10` player index, `+0x18` capacity, `+0x1c` data pointer (`node + 0x20`).
|
||||
- **`Player`** — money via `std::vector<Money*>` at `+0xe4`; power at `+0x74`;
|
||||
team at `+0xac`; relation maps at `+0xfc` / `+0x100`.
|
||||
- **Skirmish setup** (`SkirmishGameInfo`, pointer at `[0x00ce3a78]`) — starting
|
||||
cash `+0x64`, player slots `+0xfc` (stride `0x5c`, 6 slots), faction at
|
||||
`slot + 0x18` (`Empire=2`, `Allied=4`, `Soviet=8`, `Random=7`).
|
||||
|
||||
### Match start
|
||||
|
||||
The BEGIN button calls `SkirmishGameOptionsMenu::start` (`0x00b28d60`), which
|
||||
copies the map, calls `GameInfo::startGame(0)`, seeds the logic random and
|
||||
appends `MSG_NEW_GAME` (`0x2`). `startNewGame` itself is `0x00623e40`. OpenRA3
|
||||
mirrors this two-phase `prepare_new_game` / `start_new_game` split.
|
||||
|
||||
## Container and map formats
|
||||
|
||||
Recovered by inspection of `Data\*.big` (see `ra3.fs` / `ra3.map`).
|
||||
|
||||
### `BIG4` archive
|
||||
|
||||
All integers little-endian except where noted:
|
||||
|
||||
```
|
||||
offset 0 magic "BIG4"
|
||||
offset 4 fileSize u32 LE total archive size
|
||||
offset 8 fileCount u32 BE number of entries
|
||||
offset 12 indexSize u32
|
||||
offset 16 entries fileCount * { offset u32 BE, size u32 BE, name cstring }
|
||||
```
|
||||
|
||||
Entry offsets are absolute; payloads are RefPack-compressed.
|
||||
|
||||
### RefPack
|
||||
|
||||
EA's `10 FB` stream. `ra3.fs::refpack_decompress` implements the 2/3/4-byte
|
||||
commands and the long-literal/stop opcodes; `refpack_output_size` reads the
|
||||
declared output length without decompressing.
|
||||
|
||||
### Map file (`.map`)
|
||||
|
||||
Two layers of compression:
|
||||
|
||||
```
|
||||
BIG4 payload = RefPack -> "EAR\0" + u32 unpacked_size + RefPack -> "CkMp" ...
|
||||
```
|
||||
|
||||
The `CkMp` payload is the compiled SAGE map: a type/field name table followed by
|
||||
chunk data. Player start positions appear as waypoints named
|
||||
`Player_1_Start`, `Player_2_Start`, ... Each waypoint record carries a `Coord3D`
|
||||
(three little-endian `float`s) shortly after the name; `ra3.map` scans forward
|
||||
from the name for the first plausible `(x, y, 0)` triple. Maps that store starts
|
||||
in `MPPositionList` instead yield no waypoints and fall back.
|
||||
|
||||
Verified example (`map_mp_2_feasel4`): `Player_1_Start` = `(1338.9, 1940.5, 0)`,
|
||||
`Player_2_Start` = `(1290.8, 1404.9, 0)`.
|
||||
|
||||
### Still to recover
|
||||
|
||||
- `MPPositionList` layout (per-player starts for maps without waypoints).
|
||||
- Map dimensions / `HeightMapData` / `BlendTileData`.
|
||||
- Compiled asset blobs (`map.bin`, `global.bin`, `static.*.bin`) and the
|
||||
`.manifest` schema used to deserialise them.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Roadmap
|
||||
|
||||
OpenRA3 is a very large undertaking. This roadmap is deliberately honest about
|
||||
scope: reconstructing a 2008 RTS engine from a decompiler plus a related open
|
||||
engine is a multi-year, multi-person effort. The milestones below are ordered so
|
||||
that each one produces something that builds and runs.
|
||||
|
||||
## Done
|
||||
|
||||
- [x] **v0.0.1 — skeleton + minimal skirmish.** C++26 modules, GCC 16,
|
||||
CMake/Ninja, Docker `dev`/`deploy`, GitLab CI. Reads `BIG4`/RefPack data
|
||||
from a local install, recovers map start waypoints, and runs a
|
||||
deterministic headless two-player skirmish to a decision.
|
||||
|
||||
## Next
|
||||
|
||||
- [ ] **v0.1.0 — complete map parsing.** Parse `MPPositionList` for maps that do
|
||||
not use `Player_N_Start` waypoints; recover map dimensions, terrain
|
||||
heightmap and placement grid; place real starting structures/units.
|
||||
- [ ] **v0.2.0 — data & file formats.** Parse compiled gameplay assets
|
||||
(`GameObject`, `WeaponTemplate`, `ArmorTemplate`, `LocomotorTemplate`) so
|
||||
units use the real balance numbers instead of OpenRA3's stand-ins.
|
||||
- [ ] **v0.3.0 — deterministic simulation.** Real update modules, locomotor
|
||||
movement, weapons/damage/armour resolution, build queues and the tech
|
||||
tree, pathfinding and shroud.
|
||||
- [ ] **v0.4.0 — AI.** Skirmish AI: build states, team composition, attack
|
||||
waves (the reference tree's `AI*` modules).
|
||||
- [ ] **v0.5.0 — renderer.** A W3D/D3D9 (or portable Vulkan/OpenGL) client
|
||||
backend implementing `ra3::client::display`, plus input.
|
||||
- [ ] **v0.6.0 — content.** Load real maps, units, powers and strings; play a
|
||||
skirmish end-to-end with a UI.
|
||||
|
||||
## Cross-cutting tracks
|
||||
|
||||
- **RE depth** — keep recovering retail layouts (see
|
||||
[`REVERSE_ENGINEERING.md`](REVERSE_ENGINEERING.md)); every structure gets an
|
||||
address citation and a test.
|
||||
- **Determinism & replay** — the logic random stream and frame ordering must be
|
||||
reproducible; replay format and a golden-replay test suite.
|
||||
- **Offline only** — no online mode. Multiplayer, if pursued, is LAN lockstep on
|
||||
the message stream, never an online service.
|
||||
Reference in New Issue
Block a user