AnotherReplayReader/WIP.md

# AI Analysis WIP

## User Need

The application is adding AI analysis for Red Alert 3 replay operation logs. The current flow sends player information and a compacted operation log to a chat-completion-compatible LLM, then displays the analysis in `AIChatPanel`.

The main goals are:

- Improve the current system prompt.
- Make the system prompt configurable by the user.
- Reduce AI analysis errors, especially errors caused by general-world assumptions or overconfident UnitId guesses.
- Add a validation path for LLM output so wrong claims can be detected and corrected without wasting all prior reasoning.

## Current Code Areas

- `Utils/AIAnalyze.cs`: builds the system prompt, user prompts, segment prompts, final summary prompts, and performs OpenAI-compatible chat completion calls.
- `AIChatPanel.xaml.cs`: runs the analysis workflow, displays streaming chunks, retries failed segments, and now logs machine-readable claim validation results.
- `Utils/AiSettings.cs`: stores AI provider/model settings and now prompt settings.
- `AIProviderSettingsControl.xaml(.cs)`: edits provider/model settings and now prompt settings.
- `EventDump.xaml.cs`: generates the replay operation text and starts AI analysis.
- `Utils/AIAnalysisValidation.cs`: new validation model and parser for machine-readable AI claims.
- `CONTEXT.md`: glossary for the AI analysis domain.

## Discussion Notes

### Prompt Problems

The current prompt has a lot of useful game knowledge, but the LLM can still:

- Use common sense that is wrong for the game or mod.
- Assume infantry, helicopters, transports, amphibious movement, and water placement work like they do in other RTS games.
- Treat one observed skill as conclusive evidence when multiple units share that skill.
- Overstate UnitId guesses.

Examples discussed:

- Only units explicitly marked amphibious can move on both land and water.
- Only units explicitly marked as passenger transports can transport infantry.
- Building water placement depends on game rules, not common assumptions.
- `SpecialPower_UnpackReplaceSelf` does not uniquely identify an Allied MCV because Allied miners can also unpack into a command hub.
- A UnitId claimed as an aircraft should be challenged if the same UnitId is observed using an unpack/deploy skill.
- A UnitId claimed as a bomber should be challenged if it is operated before the player starts producing their first bomber.

### Documented Example: MCV vs Miner Ambiguity (Allied)

Observed replay sequence:
1. UnitId 246 (confirmed main base) → `PackReplaceSelf`
2. Player selects UnitId 587
3. UnitId 587 → `UnpackReplaceSelf`
4. AI claims: `587 = AlliedMCV`, evidenceLevel: `confirmed`

**Why this cannot be definitively resolved:**
- After base 246 packs, the engine creates a new MCV (UnitId A). The miner (UnitId B) also exists on the map.
- When the player selects 587, we cannot prove 587 = A vs 587 = B.
- After `UnpackReplaceSelf`, 587 is replaced by yet another UnitId (C if MCV→base, D if miner→command hub).
- Even if we later see C building things (`开始建造建筑 [UnitId]C(建造者)`), there is no replay-observable link connecting C back to 587.
- Allied MCV in mobile form has no unique observable ability that would distinguish it from a miner.

**Conclusion:** There is **no deterministic validation rule** that can confirm an Allied MCV claim from replay operations alone. The upper bound for any such claim is `possible`, and an alternative (miner command hub) must always be listed.

**Contrast with other factions:**
- Soviet/Japan/神州 MCVs may have different observable behaviors (e.g., unique deploy animations, different upgrade paths) — each faction needs independent analysis.

**Validation rule (negative check only):**
- If a claim says `confirmed` or `highly likely` for AlliedMCV based only on `PackReplaceSelf → UnpackReplaceSelf` sequence, flag as **overconfident** (WeakEvidence). Downgrade recommendation: `possible` with miner command hub as alternative.

### Prompt Decisions

The default prompt should explicitly require:

- Evidence-first analysis.
- No use of external common sense over replay facts and supplied game rules.
- UnitId guesses with evidence levels.
- Multiple candidates when a behavior has several possible sources.
- Support evidence and possible counter-evidence for important claims.
- Correction or abandonment of claims contradicted by replay facts.

Evidence levels currently used:

- confirmed
- highly likely
- possible
- uncertain
- ruled out

The default provider temperature was lowered from `0.75` to `0.35` because this task is closer to audit/reconstruction than creative writing.

### Prompt Configuration Decisions

The prompt is now configurable through AI settings.

The design has two prompt layers:

- A built-in dynamic system prompt, still assembled from replay/mod/faction/map context.
- User prompt settings:
  - optional full custom system prompt
  - additional rules appended to the final system prompt

This keeps the normal path safe while allowing advanced users to override the whole prompt.

### Validation Philosophy

LLM natural-language analysis should not be treated as directly valid. The plan is to validate structured claims emitted by the LLM.

Important decision:

- Do not immediately throw away a whole analysis when a problem is found.
- Do not show the user two competing analyses or apology text such as "sorry, my previous answer was wrong."
- Prefer a hidden revision pass: send the draft, validation issues, and relevant replay facts back to the AI, asking it to output a clean corrected version without mentioning the revision.
- Limit retries/revisions. If the model still cannot resolve a claim, downgrade confidence or mark it uncertain instead of looping forever.

Severity model:

- `Info`: useful diagnostic only.
- `WeakEvidence`: claim may be plausible but lacks enough support.
- `Warning`: malformed or questionable claim that should be logged or possibly revised.
- `Contradiction`: claim conflicts with replay facts or game rules and should trigger revision.
- `Fatal`: output cannot be used for the current phase, such as empty or unparseable required output.

### JSON Format Decision

We discussed whether to require JSON or use a simpler line-based format.

Decision:

- Use JSON for machine-readable claims.
- Keep the schema small.
- Make the parser tolerant.

Reasoning:

- JSON can naturally represent evidence arrays, alternatives, and needed confirmations.
- A custom line format would be easier for a trivial parser but would become fragile once nested data is needed.
- The app can tolerate partial or missing JSON by logging validation issues instead of failing the whole analysis.

### Evidence Format Decision (2026-07-06)

We decided to move from free-form evidence text to a **structured pipe-delimited format**:

```
type|time|param1|param2|...
```

Supported types: `build`, `place`, `produce`, `sell`, `select`, `move`, `power`.

Reasoning:
- Free-form text could not be programmatically validated without NLP.
- Structured evidence can be parsed deterministically with a simple regex.
- Enables deterministic validation rules like unpack-ambiguity checking.
- The format is simple enough for AI models to follow reliably.

Current expected shape (all three claim types now have schema definitions in the prompt):

```json
{
  "unitClaims": [
    {
      "unitId": 123,
      "player": "PlayerA",
      "claim": "AlliedMCV",
      "evidenceLevel": "possible",
      "evidence": ["8:30 使用 SpecialPower_UnpackReplaceSelf"],
      "alternatives": ["AlliedMiner 展开后的指挥中心"],
      "needsConfirmation": ["是否曾使用 SpecialPower_PackReplaceSelf", "后续是否作为建造者出现"]
    }
  ],
  "eventClaims": [
    {
      "claim": "PlayerA 主基地打包并开始迁移",
      "evidenceLevel": "confirmed",
      "evidence": ["1:24.00 SpecialPower_PackReplaceSelf", "后续移动和展开操作"]
    }
  ],
  "timelineClaims": [
    {
      "claim": "PlayerA 在开局 2 分钟内完成了基地迁移",
      "evidenceLevel": "confirmed",
      "evidence": ["1:24.00 打包", "1:41.00 展开"]
    }
  ]
}
```

The prompt asks the AI to output:

```text
[机器可读声明]
```json
{ ... }
```
```

The parser first looks for the last fenced JSON block near `[机器可读声明]`, then falls back to the last `{...}`.

**Known format issue (fixed):** The original prompt only defined `unitClaims` entries; `eventClaims` and `timelineClaims` were shown as empty arrays. The AI therefore invented its own fields (e.g., `"event"` / `"time"` instead of `"claim"`), which the parser silently ignored. Fixed by:
1. Adding full schema definitions for all three claim types in the prompt.
2. Making the parser accept `"event"` as a fallback for `"claim"` in `eventClaims`.

**Claim count limits added:** Prompt instructs the AI to limit output (unitClaims ≤ 10, eventClaims ≤ 5, timelineClaims ≤ 3). The parser enforces these caps and emits Info-level issues if the AI exceeds them.

## Validation We Can Do

### Implemented Now

Format validation:

- Missing machine-readable claims.
- JSON parse failure.
- Root value is not an object.
- Claim count limits with truncation warnings.
- `eventClaims` accepts both `"claim"` and `"event"` as field names.
- Unknown `evidenceLevel` values logged as Info issue, fallback to `Uncertain`.

Self-consistency validation:

- Unit claim missing `unitId`.
- Unit claim missing `claim`.
- High-confidence UnitId guess without evidence.
- Low-confidence UnitId guess without alternatives or needed confirmation.

### Evidence Format (Structured)

The `evidence` field now uses a structured pipe-delimited format instead of free-form text:

```
build|time|assetName|builderUnitId
place|time|assetName|builderUnitId|x,y,z
produce|time|unitName|producerUnitId
sell|time|unitId
select|time|unitId
move|time|x,y,z
power|time|powerName|unitId
```

This format is parsed by `ParseStructuredEvidence()` into a `StructuredEvidence` record with typed `AIEvidenceType` enum. Parsing uses a single regex and is fully deterministic.

**Backward compatibility:** The parser silently returns `Unknown` type for strings that don't match the structured format. No validation rules currently fire on unknown-typed evidence, so it degrades gracefully but invisibly.

### Near-Term Validations

These need replay facts extracted from `CommandChunk` or an intermediate fact index:

- UnitId production timeline contradictions (e.g., "bomber" claimed before first bomber production — needs game knowledge of which unit names are bombers).
- Claims that use game knowledge not present in rules, such as transport/amphibious/building-placement abilities.
- **Overconfidence detection:** Claims with `confirmed`/`highly likely` that lack sufficient evidence given what is knowable from replay data alone (e.g., claiming AlliedMCV as `confirmed`).

### Implemented via Fact Index

The `ReplayFactIndex` now powers these checks:

- **Special power contradiction:** Evidence `power|...|SomePower|unitId` is cross-checked against the actual special powers observed for that UnitId. If the power was never used, a `Contradiction` issue is emitted.
- **UnitId existence:** Warns if a claim references a UnitId never seen in any replay command.
- **Builder consistency:** If a claim describes a unit as MCV/builder but the UnitId was never observed as a builder, emits `WeakEvidence`.

### Suggested Fact Index

Useful derived facts:

- `UnitId -> first observed time`
- `UnitId -> observed special powers`
- `UnitId -> observed as builder`
- `UnitId -> observed as production structure`
- `Player -> first production time by asset id`
- `Player -> selected UnitIds over time`
- `Player -> tech/protocol choices`
- `Player -> building placements by asset and position`

## Current Progress

Implemented:

- Added `CONTEXT.md` glossary.
- Added prompt settings:
  - `AiPromptSettings`
  - `UseCustomSystemPrompt`
  - `CustomSystemPrompt`
  - `AdditionalRules`
- Added prompt editing UI to `AIProviderSettingsControl`.
- Connected prompt settings from `EventDump` to `AIChatPanel` to `AIAnalyze`.
- Split default prompt construction from prompt composition.
- Strengthened default prompt with evidence-first and uncertainty rules.
- Added machine-readable JSON claim instructions to system and segment prompts.
- Added `Utils/AIAnalysisValidation.cs` with:
  - evidence level enum
  - machine-readable claim records
  - validation issue records
  - JSON extraction and parsing
  - initial self-consistency checks
- Added per-segment validation logging in `AIChatPanel`.
- Fixed inconsistent prompt ↔ parser schema for `eventClaims`/`timelineClaims`:
  - Added full schema definitions for all three claim types in the system prompt.
  - Parser now accepts `"event"` as fallback for `"claim"` in `eventClaims`.
  - Both prompt and parser enforce claim count limits (10 unit, 5 event, 3 timeline) with truncation warnings.
- Unknown `evidenceLevel` values now produce an Info-level validation issue (fallback to `Uncertain`).
- Created ADR 0001 documenting the hidden revision pass design decision.
- Recorded MCV vs Miner ambiguity as a documented validation scenario.
- Evidence format changed from free-form text to structured pipe-delimited format:
  - 7 evidence types: `build`, `place`, `produce`, `sell`, `select`, `move`, `power`.
  - Prompt updated to require structured format only.
  - Added `StructuredEvidence` record and `ParseStructuredEvidence()` parser.
  - Added `ParseAllEvidence()` to convert all evidence strings for a claim.
- Added first validation rule `ValidateUnpackAmbiguity()`:
  - Flags `confirmed`/`highly likely` claims that use `UnpackReplaceSelf` without matching `PackReplaceSelf`.
  - Emits `WeakEvidence`/`MissingAlternative` — the unpack could be MCV deploy or miner command hub deploy.
  - If `PackReplaceSelf` IS present in the same claim's evidence, the chain is consistent and no flag.
- Created `Utils/ReplayFactIndex.cs` — builds a fact index from raw `CommandChunk` data:
  - `UnitIdFirstObservedTime`: first time each UnitId appears in any command.
  - `UnitIdSpecialPowers`: set of special powers used by each UnitId.
  - `BuilderUnitIds`: UnitIds that appeared as builder in construction commands.
  - `ProducerUnitIds`: UnitIds that appeared as production structures.
  - `PlayerFirstProductionTime`: per player, first production time for each unit asset name.
  - `PlayerSelectedUnitIds`: which UnitIds each player has selected.
- Plumbed `ReplayFactIndex` through the analysis pipeline:
  - Built in `EventDump.ShowPlainText()` from `CommandChunk` + string hash table.
  - Passed to `AIChatPanel.StartAnalysisAsync()` as new parameter.
  - Forwarded to `AIAnalysisValidation.ValidateMachineReadableClaims()`.
- Added `ValidateTimelineConsistency()` — three checks using fact index:
  1. **UnitId existence check:** Warns if a claim references a UnitId never seen in the replay.
  2. **Special power verification:** Cross-references `power|...` evidence entries against actual special powers observed for that UnitId; emits `Contradiction` if the claim says a UnitId used a power it never used.
  3. **Builder consistency check:** If a claim describes a unit as MCV/builder/Nanocore but that UnitId was never observed as a builder, emits `WeakEvidence`.

Build status:

- `dotnet build AnotherReplayReader.csproj --no-restore` succeeds.
- Remaining warnings are existing nullable warnings in `AIAnalyze.cs` stream response handling and a `System.Text.Encoding.CodePages` support warning for `net461`.

## Open Questions

- Should AI natural-language output continue streaming live, or should content be buffered until validation and possible revision are complete?
- Should reasoning chunks remain visible during hidden revision, or should only final content be shown?
- How strict should missing machine-readable claims be?
  - Current behavior: warning log only.
  - Possible future behavior: one hidden repair request asking the model to append valid claims.
- Should validation issues be visible by default, or only in an advanced/debug foldout?
- How much game-unit knowledge should live in code versus prompt text?
- Should the first verifier use hardcoded RA3/Corona knowledge, or should it load a small unit capability table from data files?
- How should free-form evidence strings be programmatically validated? (See "Known Limitation" above.)

## Suggested Next Steps

1. ✅ Build a replay fact index from `CommandChunk` — done (`ReplayFactIndex`).
2. ✅ Add first deterministic validation rules:
   - ✅ ambiguous Allied unpack — done (`ValidateUnpackAmbiguity`).
   - ✅ pack/unpack consistency — covered by unpack rule.
   - ✅ UnitId used as builder — done (builder consistency check in `ValidateTimelineConsistency`).
   - ✅ special power contradictions — done (special power verification in `ValidateTimelineConsistency`).
   - ⬜ first production time vs first operation time — needs game knowledge of unit type names (e.g., "which names are bombers").
3. Decide whether to buffer per-segment content before display.
4. Add one hidden revision pass for `Contradiction` issues.
5. Add validation summary UI, such as "验证器发现并修正 N 个问题".