AnotherReplayReader/WIP.md

# AI Analysis WIP

## User Need

The application is adding AI analysis for Red Alert 3 replay operation logs. The current flow sends player information and a compacted operation log to a chat-completion-compatible LLM, then displays the analysis in `AIChatPanel`.

The main goals are:

- Improve the current system prompt.
- Make the system prompt configurable by the user.
- Reduce AI analysis errors, especially errors caused by general-world assumptions or overconfident UnitId guesses.
- Add a validation path for LLM output so wrong claims can be detected and corrected without wasting all prior reasoning.

## Current Code Areas

- `Utils/AIAnalyze.cs`: builds the system prompt, user prompts, segment prompts, final summary prompts, and performs OpenAI-compatible chat completion calls.
- `AIChatPanel.xaml.cs`: runs the analysis workflow, displays streaming chunks, retries failed segments, and now logs machine-readable claim validation results.
- `Utils/AiSettings.cs`: stores AI provider/model settings and now prompt settings.
- `AIProviderSettingsControl.xaml(.cs)`: edits provider/model settings and now prompt settings.
- `EventDump.xaml.cs`: generates the replay operation text and starts AI analysis.
- `Utils/AIAnalysisValidation.cs`: new validation model and parser for machine-readable AI claims.
- `CONTEXT.md`: glossary for the AI analysis domain.

## Discussion Notes

### Prompt Problems

The current prompt has a lot of useful game knowledge, but the LLM can still:

- Use common sense that is wrong for the game or mod.
- Assume infantry, helicopters, transports, amphibious movement, and water placement work like they do in other RTS games.
- Treat one observed skill as conclusive evidence when multiple units share that skill.
- Overstate UnitId guesses.

Examples discussed:

- Only units explicitly marked amphibious can move on both land and water.
- Only units explicitly marked as passenger transports can transport infantry.
- Building water placement depends on game rules, not common assumptions.
- `SpecialPower_UnpackReplaceSelf` does not uniquely identify an Allied MCV because Allied miners can also unpack into a command hub.
- A UnitId claimed as an aircraft should be challenged if the same UnitId is observed using an unpack/deploy skill.
- A UnitId claimed as a bomber should be challenged if it is operated before the player starts producing their first bomber.

### Prompt Decisions

The default prompt should explicitly require:

- Evidence-first analysis.
- No use of external common sense over replay facts and supplied game rules.
- UnitId guesses with evidence levels.
- Multiple candidates when a behavior has several possible sources.
- Support evidence and possible counter-evidence for important claims.
- Correction or abandonment of claims contradicted by replay facts.

Evidence levels currently used:

- confirmed
- highly likely
- possible
- uncertain
- ruled out

The default provider temperature was lowered from `0.75` to `0.35` because this task is closer to audit/reconstruction than creative writing.

### Prompt Configuration Decisions

The prompt is now configurable through AI settings.

The design has two prompt layers:

- A built-in dynamic system prompt, still assembled from replay/mod/faction/map context.
- User prompt settings:
  - optional full custom system prompt
  - additional rules appended to the final system prompt

This keeps the normal path safe while allowing advanced users to override the whole prompt.

### Validation Philosophy

LLM natural-language analysis should not be treated as directly valid. The plan is to validate structured claims emitted by the LLM.

Important decision:

- Do not immediately throw away a whole analysis when a problem is found.
- Do not show the user two competing analyses or apology text such as "sorry, my previous answer was wrong."
- Prefer a hidden revision pass: send the draft, validation issues, and relevant replay facts back to the AI, asking it to output a clean corrected version without mentioning the revision.
- Limit retries/revisions. If the model still cannot resolve a claim, downgrade confidence or mark it uncertain instead of looping forever.

Severity model:

- `Info`: useful diagnostic only.
- `WeakEvidence`: claim may be plausible but lacks enough support.
- `Warning`: malformed or questionable claim that should be logged or possibly revised.
- `Contradiction`: claim conflicts with replay facts or game rules and should trigger revision.
- `Fatal`: output cannot be used for the current phase, such as empty or unparseable required output.

### JSON Format Decision

We discussed whether to require JSON or use a simpler line-based format.

Decision:

- Use JSON for machine-readable claims.
- Keep the schema small.
- Make the parser tolerant.

Reasoning:

- JSON can naturally represent evidence arrays, alternatives, and needed confirmations.
- A custom line format would be easier for a trivial parser but would become fragile once nested data is needed.
- The app can tolerate partial or missing JSON by logging validation issues instead of failing the whole analysis.

Current expected shape:

```json
{
  "unitClaims": [
    {
      "unitId": 123,
      "player": "PlayerA",
      "claim": "AlliedMCV",
      "evidenceLevel": "possible",
      "evidence": ["8:30 使用 SpecialPower_UnpackReplaceSelf"],
      "alternatives": ["AlliedMiner 展开后的指挥中心"],
      "needsConfirmation": ["是否曾使用 SpecialPower_PackReplaceSelf", "后续是否作为建造者出现"]
    }
  ],
  "eventClaims": [],
  "timelineClaims": []
}
```

The prompt asks the AI to output:

```text
[机器可读声明]
```json
{ ... }
```
```

The parser first looks for the last fenced JSON block near `[机器可读声明]`, then falls back to the last `{...}` block.

## Validation We Can Do

### Implemented Now

Format validation:

- Missing machine-readable claims.
- JSON parse failure.
- Root value is not an object.

Self-consistency validation:

- Unit claim missing `unitId`.
- Unit claim missing `claim`.
- High-confidence UnitId guess without evidence.
- Low-confidence UnitId guess without alternatives or needed confirmation.

### Near-Term Validations

These need replay facts extracted from `CommandChunk` or an intermediate fact index:

- UnitId special power contradictions.
- UnitId production timeline contradictions.
- UnitId used as builder vs claimed as non-builder unit.
- Claims that use game knowledge not present in rules, such as transport/amphibious/building-placement abilities.
- Missing alternatives for ambiguous skills such as Allied unpack.

### Suggested Fact Index

Useful derived facts:

- `UnitId -> first observed time`
- `UnitId -> observed special powers`
- `UnitId -> observed as builder`
- `UnitId -> observed as production structure`
- `Player -> first production time by asset id`
- `Player -> selected UnitIds over time`
- `Player -> tech/protocol choices`
- `Player -> building placements by asset and position`

## Current Progress

Implemented:

- Added `CONTEXT.md` glossary.
- Added prompt settings:
  - `AiPromptSettings`
  - `UseCustomSystemPrompt`
  - `CustomSystemPrompt`
  - `AdditionalRules`
- Added prompt editing UI to `AIProviderSettingsControl`.
- Connected prompt settings from `EventDump` to `AIChatPanel` to `AIAnalyze`.
- Split default prompt construction from prompt composition.
- Strengthened default prompt with evidence-first and uncertainty rules.
- Added machine-readable JSON claim instructions to system and segment prompts.
- Added `Utils/AIAnalysisValidation.cs` with:
  - evidence level enum
  - machine-readable claim records
  - validation issue records
  - JSON extraction and parsing
  - initial self-consistency checks
- Added per-segment validation logging in `AIChatPanel`.

Build status:

- `dotnet build AnotherReplayReader.csproj --no-restore` succeeds.
- Remaining warnings are existing nullable warnings in `AIAnalyze.cs` stream response handling and a `System.Text.Encoding.CodePages` support warning for `net461`.

## Open Questions

- Should AI natural-language output continue streaming live, or should content be buffered until validation and possible revision are complete?
- Should reasoning chunks remain visible during hidden revision, or should only final content be shown?
- How strict should missing machine-readable claims be?
  - Current behavior: warning log only.
  - Possible future behavior: one hidden repair request asking the model to append valid claims.
- Should validation issues be visible by default, or only in an advanced/debug foldout?
- How much game-unit knowledge should live in code versus prompt text?
- Should the first verifier use hardcoded RA3/Corona knowledge, or should it load a small unit capability table from data files?

## Suggested Next Steps

1. Build a replay fact index from `CommandChunk`.
2. Add first deterministic validation rules:
   - ambiguous Allied unpack
   - pack/unpack consistency
   - UnitId used as builder
   - first production time vs first operation time
3. Decide whether to buffer per-segment content before display.
4. Add one hidden revision pass for `Contradiction` issues.
5. Add validation summary UI, such as "验证器发现并修正 N 个问题".