AnotherReplayReader/WIP.md

AI Analysis WIP

User Need

The application is adding AI analysis for Red Alert 3 replay operation logs. The current flow sends player information and a compacted operation log to a chat-completion-compatible LLM, then displays the analysis in AIChatPanel.

The main goals are:

• Improve the current system prompt.
• Make the system prompt configurable by the user.
• Reduce AI analysis errors, especially errors caused by general-world assumptions or overconfident UnitId guesses.
• Add a validation path for LLM output so wrong claims can be detected and corrected without wasting all prior reasoning.

Current Code Areas

• Utils/AIAnalyze.cs: builds the system prompt, user prompts, segment prompts, final summary prompts, and performs OpenAI-compatible chat completion calls.
• AIChatPanel.xaml.cs: runs the analysis workflow, displays streaming chunks, retries failed segments, and now logs machine-readable claim validation results.
• Utils/AiSettings.cs: stores AI provider/model settings and now prompt settings.
• AIProviderSettingsControl.xaml(.cs): edits provider/model settings and now prompt settings.
• EventDump.xaml.cs: generates the replay operation text and starts AI analysis.
• Utils/AIAnalysisValidation.cs: new validation model and parser for machine-readable AI claims.
• CONTEXT.md: glossary for the AI analysis domain.

Discussion Notes

Prompt Problems

The current prompt has a lot of useful game knowledge, but the LLM can still:

• Use common sense that is wrong for the game or mod.
• Assume infantry, helicopters, transports, amphibious movement, and water placement work like they do in other RTS games.
• Treat one observed skill as conclusive evidence when multiple units share that skill.
• Overstate UnitId guesses.

Examples discussed:

• Only units explicitly marked amphibious can move on both land and water.
• Only units explicitly marked as passenger transports can transport infantry.
• Building water placement depends on game rules, not common assumptions.
• SpecialPower_UnpackReplaceSelf does not uniquely identify an Allied MCV because Allied miners can also unpack into a command hub.
• A UnitId claimed as an aircraft should be challenged if the same UnitId is observed using an unpack/deploy skill.
• A UnitId claimed as a bomber should be challenged if it is operated before the player starts producing their first bomber.

Documented Example: MCV vs Miner Ambiguity (Allied)

Observed replay sequence:

1. UnitId 246 (confirmed main base) → PackReplaceSelf
2. Player selects UnitId 587
3. UnitId 587 → UnpackReplaceSelf
4. AI claims: 587 = AlliedMCV, evidenceLevel: confirmed

Why this cannot be definitively resolved:

• After base 246 packs, the engine creates a new MCV (UnitId A). The miner (UnitId B) also exists on the map.
• When the player selects 587, we cannot prove 587 = A vs 587 = B.
• After UnpackReplaceSelf, 587 is replaced by yet another UnitId (C if MCV→base, D if miner→command hub).
• Even if we later see C building things (开始建造建筑 [UnitId]C(建造者)), there is no replay-observable link connecting C back to 587.
• Allied MCV in mobile form has no unique observable ability that would distinguish it from a miner.

Conclusion: There is no deterministic validation rule that can confirm an Allied MCV claim from replay operations alone. The upper bound for any such claim is possible, and an alternative (miner command hub) must always be listed.

Contrast with other factions:

• Soviet/Japan/神州 MCVs may have different observable behaviors (e.g., unique deploy animations, different upgrade paths) — each faction needs independent analysis.

Validation rule (negative check only):

• If a claim says confirmed or highly likely for AlliedMCV based only on PackReplaceSelf → UnpackReplaceSelf sequence, flag as overconfident (WeakEvidence). Downgrade recommendation: possible with miner command hub as alternative.

Prompt Decisions

The default prompt should explicitly require:

• Evidence-first analysis.
• No use of external common sense over replay facts and supplied game rules.
• UnitId guesses with evidence levels.
• Multiple candidates when a behavior has several possible sources.
• Support evidence and possible counter-evidence for important claims.
• Correction or abandonment of claims contradicted by replay facts.

Evidence levels currently used:

• confirmed
• highly likely
• possible
• uncertain
• ruled out

The default provider temperature was lowered from 0.75 to 0.35 because this task is closer to audit/reconstruction than creative writing.

Prompt Configuration Decisions

The prompt is now configurable through AI settings.

The design has two prompt layers:

• A built-in dynamic system prompt, still assembled from replay/mod/faction/map context.
• User prompt settings:
  • optional full custom system prompt
  • additional rules appended to the final system prompt

This keeps the normal path safe while allowing advanced users to override the whole prompt.

Validation Philosophy

LLM natural-language analysis should not be treated as directly valid. The plan is to validate structured claims emitted by the LLM.

Important decision:

• Do not immediately throw away a whole analysis when a problem is found.
• Do not show the user two competing analyses or apology text such as "sorry, my previous answer was wrong."
• Prefer a hidden revision pass: send the draft, validation issues, and relevant replay facts back to the AI, asking it to output a clean corrected version without mentioning the revision.
• Limit retries/revisions. If the model still cannot resolve a claim, downgrade confidence or mark it uncertain instead of looping forever.

Severity model:

• Info: useful diagnostic only.
• WeakEvidence: claim may be plausible but lacks enough support.
• Warning: malformed or questionable claim that should be logged or possibly revised.
• Contradiction: claim conflicts with replay facts or game rules and should trigger revision.
• Fatal: output cannot be used for the current phase, such as empty or unparseable required output.

JSON Format Decision

We discussed whether to require JSON or use a simpler line-based format.

Decision:

• Use JSON for machine-readable claims.
• Keep the schema small.
• Make the parser tolerant.

Reasoning:

• JSON can naturally represent evidence arrays, alternatives, and needed confirmations.
• A custom line format would be easier for a trivial parser but would become fragile once nested data is needed.
• The app can tolerate partial or missing JSON by logging validation issues instead of failing the whole analysis.

Evidence Format Decision (2026-07-06)

We decided to move from free-form evidence text to a structured pipe-delimited format:

type|time|param1|param2|...

Supported types: build, place, produce, sell, select, move, power.

Reasoning:

• Free-form text could not be programmatically validated without NLP.
• Structured evidence can be parsed deterministically with a simple regex.
• Enables deterministic validation rules like unpack-ambiguity checking.
• The format is simple enough for AI models to follow reliably.

Current expected shape (all three claim types now have schema definitions in the prompt):

{
  "unitClaims": [
    {
      "unitId": 123,
      "player": "PlayerA",
      "claim": "AlliedMCV",
      "evidenceLevel": "possible",
      "evidence": ["8:30 使用 SpecialPower_UnpackReplaceSelf"],
      "alternatives": ["AlliedMiner 展开后的指挥中心"],
      "needsConfirmation": ["是否曾使用 SpecialPower_PackReplaceSelf", "后续是否作为建造者出现"]
    }
  ],
  "eventClaims": [
    {
      "claim": "PlayerA 主基地打包并开始迁移",
      "evidenceLevel": "confirmed",
      "evidence": ["1:24.00 SpecialPower_PackReplaceSelf", "后续移动和展开操作"]
    }
  ],
  "timelineClaims": [
    {
      "claim": "PlayerA 在开局 2 分钟内完成了基地迁移",
      "evidenceLevel": "confirmed",
      "evidence": ["1:24.00 打包", "1:41.00 展开"]
    }
  ]
}

The prompt asks the AI to output:

[机器可读声明]
```json
{ ... }

The parser first looks for the last fenced JSON block near `[机器可读声明]`, then falls back to the last `{...}`.

**Known format issue (fixed):** The original prompt only defined `unitClaims` entries; `eventClaims` and `timelineClaims` were shown as empty arrays. The AI therefore invented its own fields (e.g., `"event"` / `"time"` instead of `"claim"`), which the parser silently ignored. Fixed by:
1. Adding full schema definitions for all three claim types in the prompt.
2. Making the parser accept `"event"` as a fallback for `"claim"` in `eventClaims`.

**Claim count limits added:** Prompt instructs the AI to limit output (unitClaims ≤ 10, eventClaims ≤ 5, timelineClaims ≤ 3). The parser enforces these caps and emits Info-level issues if the AI exceeds them.

## Validation We Can Do

### Implemented Now

Format validation:

- Missing machine-readable claims.
- JSON parse failure.
- Root value is not an object.
- Claim count limits with truncation warnings.
- `eventClaims` accepts both `"claim"` and `"event"` as field names.
- Unknown `evidenceLevel` values logged as Info issue, fallback to `Uncertain`.

Self-consistency validation:

- Unit claim missing `unitId`.
- Unit claim missing `claim`.
- High-confidence UnitId guess without evidence.
- Low-confidence UnitId guess without alternatives or needed confirmation.

### Evidence Format (Structured)

The `evidence` field now uses a structured pipe-delimited format instead of free-form text:

build|time|assetName|builderUnitId place|time|assetName|builderUnitId|x,y,z produce|time|unitName|producerUnitId sell|time|unitId select|time|unitId move|time|x,y,z power|time|powerName|unitId

This format is parsed by `ParseStructuredEvidence()` into a `StructuredEvidence` record with typed `AIEvidenceType` enum. Parsing uses a single regex and is fully deterministic.

**Backward compatibility:** The parser silently returns `Unknown` type for strings that don't match the structured format. No validation rules currently fire on unknown-typed evidence, so it degrades gracefully but invisibly.

### Near-Term Validations

These need replay facts extracted from `CommandChunk` or an intermediate fact index:

- UnitId production timeline contradictions (e.g., "bomber" claimed before first bomber production — needs game knowledge of which unit names are bombers).
- Claims that use game knowledge not present in rules, such as transport/amphibious/building-placement abilities.
- **Overconfidence detection:** Claims with `confirmed`/`highly likely` that lack sufficient evidence given what is knowable from replay data alone (e.g., claiming AlliedMCV as `confirmed`).

### Implemented via Fact Index

The `ReplayFactIndex` now powers these checks:

- **Special power contradiction:** Evidence `power|...|SomePower|unitId` is cross-checked against the actual special powers observed for that UnitId. If the power was never used, a `Contradiction` issue is emitted.
- **UnitId existence:** Warns if a claim references a UnitId never seen in any replay command.
- **Builder consistency:** If a claim describes a unit as MCV/builder but the UnitId was never observed as a builder, emits `WeakEvidence`.

### Suggested Fact Index

Useful derived facts:

- `UnitId -> first observed time`
- `UnitId -> observed special powers`
- `UnitId -> observed as builder`
- `UnitId -> observed as production structure`
- `Player -> first production time by asset id`
- `Player -> selected UnitIds over time`
- `Player -> tech/protocol choices`
- `Player -> building placements by asset and position`

## Current Progress

Implemented:

- Added `CONTEXT.md` glossary.
- Added prompt settings:
  - `AiPromptSettings`
  - `UseCustomSystemPrompt`
  - `CustomSystemPrompt`
  - `AdditionalRules`
- Added prompt editing UI to `AIProviderSettingsControl`.
- Connected prompt settings from `EventDump` to `AIChatPanel` to `AIAnalyze`.
- Split default prompt construction from prompt composition.
- Strengthened default prompt with evidence-first and uncertainty rules.
- Added machine-readable JSON claim instructions to system and segment prompts.
- Added `Utils/AIAnalysisValidation.cs` with:
  - evidence level enum
  - machine-readable claim records
  - validation issue records
  - JSON extraction and parsing
  - initial self-consistency checks
- Added per-segment validation logging in `AIChatPanel`.
- Fixed inconsistent prompt ↔ parser schema for `eventClaims`/`timelineClaims`:
  - Added full schema definitions for all three claim types in the system prompt.
  - Parser now accepts `"event"` as fallback for `"claim"` in `eventClaims`.
  - Both prompt and parser enforce claim count limits (10 unit, 5 event, 3 timeline) with truncation warnings.
- Unknown `evidenceLevel` values now produce an Info-level validation issue (fallback to `Uncertain`).
- Created ADR 0001 documenting the hidden revision pass design decision.
- Recorded MCV vs Miner ambiguity as a documented validation scenario.
- Evidence format changed from free-form text to structured pipe-delimited format:
  - 7 evidence types: `build`, `place`, `produce`, `sell`, `select`, `move`, `power`.
  - Prompt updated to require structured format only.
  - Added `StructuredEvidence` record and `ParseStructuredEvidence()` parser.
  - Added `ParseAllEvidence()` to convert all evidence strings for a claim.
- Added first validation rule `ValidateUnpackAmbiguity()`:
  - Flags `confirmed`/`highly likely` claims that use `UnpackReplaceSelf` without matching `PackReplaceSelf`.
  - Emits `WeakEvidence`/`MissingAlternative` — the unpack could be MCV deploy or miner command hub deploy.
  - If `PackReplaceSelf` IS present in the same claim's evidence, the chain is consistent and no flag.
- Created `Utils/ReplayFactIndex.cs` — builds a fact index from raw `CommandChunk` data:
  - `UnitIdFirstObservedTime`: first time each UnitId appears in any command.
  - `UnitIdSpecialPowers`: set of special powers used by each UnitId.
  - `BuilderUnitIds`: UnitIds that appeared as builder in construction commands.
  - `ProducerUnitIds`: UnitIds that appeared as production structures.
  - `PlayerFirstProductionTime`: per player, first production time for each unit asset name.
  - `PlayerSelectedUnitIds`: which UnitIds each player has selected.
- Plumbed `ReplayFactIndex` through the analysis pipeline:
  - Built in `EventDump.ShowPlainText()` from `CommandChunk` + string hash table.
  - Passed to `AIChatPanel.StartAnalysisAsync()` as new parameter.
  - Forwarded to `AIAnalysisValidation.ValidateMachineReadableClaims()`.
- Added `ValidateTimelineConsistency()` — three checks using fact index:
  1. **UnitId existence check:** Warns if a claim references a UnitId never seen in the replay.
  2. **Special power verification:** Cross-references `power|...` evidence entries against actual special powers observed for that UnitId; emits `Contradiction` if the claim says a UnitId used a power it never used.
  3. **Builder consistency check:** If a claim describes a unit as MCV/builder/Nanocore but that UnitId was never observed as a builder, emits `WeakEvidence`.

Build status:

- `dotnet build AnotherReplayReader.csproj --no-restore` succeeds.
- Remaining warnings are existing nullable warnings in `AIAnalyze.cs` stream response handling and a `System.Text.Encoding.CodePages` support warning for `net461`.

## Open Questions

- Should AI natural-language output continue streaming live, or should content be buffered until validation and possible revision are complete?
- Should reasoning chunks remain visible during hidden revision, or should only final content be shown?
- How strict should missing machine-readable claims be?
  - Current behavior: warning log only.
  - Possible future behavior: one hidden repair request asking the model to append valid claims.
- Should validation issues be visible by default, or only in an advanced/debug foldout?
- How much game-unit knowledge should live in code versus prompt text?
- Should the first verifier use hardcoded RA3/Corona knowledge, or should it load a small unit capability table from data files?
- How should free-form evidence strings be programmatically validated? (See "Known Limitation" above.)

## Suggested Next Steps

1. ✅ Build a replay fact index from `CommandChunk` — done (`ReplayFactIndex`).
2. ✅ Add first deterministic validation rules:
   - ✅ ambiguous Allied unpack — done (`ValidateUnpackAmbiguity`).
   - ✅ pack/unpack consistency — covered by unpack rule.
   - ✅ UnitId used as builder — done (builder consistency check in `ValidateTimelineConsistency`).
   - ✅ special power contradictions — done (special power verification in `ValidateTimelineConsistency`).
   - ⬜ first production time vs first operation time — needs game knowledge of unit type names (e.g., "which names are bombers").
3. Decide whether to buffer per-segment content before display.
4. Add one hidden revision pass for `Contradiction` issues.
5. Add validation summary UI, such as "验证器发现并修正 N 个问题".