AnotherReplayReader/WIP.md

AI Analysis WIP

User Need

The application is adding AI analysis for Red Alert 3 replay operation logs. The current flow sends player information and a compacted operation log to a chat-completion-compatible LLM, then displays the analysis in AIChatPanel.

The main goals are:

• Improve the current system prompt.
• Make the system prompt configurable by the user.
• Reduce AI analysis errors, especially errors caused by general-world assumptions or overconfident UnitId guesses.
• Add a validation path for LLM output so wrong claims can be detected and corrected without wasting all prior reasoning.

Current Code Areas

• Utils/AIAnalyze.cs: builds the system prompt, user prompts, segment prompts, final summary prompts, and performs OpenAI-compatible chat completion calls.
• AIChatPanel.xaml.cs: runs the analysis workflow, displays streaming chunks, retries failed segments, and now logs machine-readable claim validation results.
• Utils/AiSettings.cs: stores AI provider/model settings and now prompt settings.
• AIProviderSettingsControl.xaml(.cs): edits provider/model settings and now prompt settings.
• EventDump.xaml.cs: generates the replay operation text and starts AI analysis.
• Utils/AIAnalysisValidation.cs: new validation model and parser for machine-readable AI claims.
• CONTEXT.md: glossary for the AI analysis domain.

Discussion Notes

Prompt Problems

The current prompt has a lot of useful game knowledge, but the LLM can still:

• Use common sense that is wrong for the game or mod.
• Assume infantry, helicopters, transports, amphibious movement, and water placement work like they do in other RTS games.
• Treat one observed skill as conclusive evidence when multiple units share that skill.
• Overstate UnitId guesses.

Examples discussed:

• Only units explicitly marked amphibious can move on both land and water.
• Only units explicitly marked as passenger transports can transport infantry.
• Building water placement depends on game rules, not common assumptions.
• SpecialPower_UnpackReplaceSelf does not uniquely identify an Allied MCV because Allied miners can also unpack into a command hub.
• A UnitId claimed as an aircraft should be challenged if the same UnitId is observed using an unpack/deploy skill.
• A UnitId claimed as a bomber should be challenged if it is operated before the player starts producing their first bomber.

Prompt Decisions

The default prompt should explicitly require:

• Evidence-first analysis.
• No use of external common sense over replay facts and supplied game rules.
• UnitId guesses with evidence levels.
• Multiple candidates when a behavior has several possible sources.
• Support evidence and possible counter-evidence for important claims.
• Correction or abandonment of claims contradicted by replay facts.

Evidence levels currently used:

• confirmed
• highly likely
• possible
• uncertain
• ruled out

The default provider temperature was lowered from 0.75 to 0.35 because this task is closer to audit/reconstruction than creative writing.

Prompt Configuration Decisions

The prompt is now configurable through AI settings.

The design has two prompt layers:

• A built-in dynamic system prompt, still assembled from replay/mod/faction/map context.
• User prompt settings:
  • optional full custom system prompt
  • additional rules appended to the final system prompt

This keeps the normal path safe while allowing advanced users to override the whole prompt.

Validation Philosophy

LLM natural-language analysis should not be treated as directly valid. The plan is to validate structured claims emitted by the LLM.

Important decision:

• Do not immediately throw away a whole analysis when a problem is found.
• Do not show the user two competing analyses or apology text such as "sorry, my previous answer was wrong."
• Prefer a hidden revision pass: send the draft, validation issues, and relevant replay facts back to the AI, asking it to output a clean corrected version without mentioning the revision.
• Limit retries/revisions. If the model still cannot resolve a claim, downgrade confidence or mark it uncertain instead of looping forever.

Severity model:

• Info: useful diagnostic only.
• WeakEvidence: claim may be plausible but lacks enough support.
• Warning: malformed or questionable claim that should be logged or possibly revised.
• Contradiction: claim conflicts with replay facts or game rules and should trigger revision.
• Fatal: output cannot be used for the current phase, such as empty or unparseable required output.

JSON Format Decision

We discussed whether to require JSON or use a simpler line-based format.

Decision:

• Use JSON for machine-readable claims.
• Keep the schema small.
• Make the parser tolerant.

Reasoning:

• JSON can naturally represent evidence arrays, alternatives, and needed confirmations.
• A custom line format would be easier for a trivial parser but would become fragile once nested data is needed.
• The app can tolerate partial or missing JSON by logging validation issues instead of failing the whole analysis.

Current expected shape:

{
  "unitClaims": [
    {
      "unitId": 123,
      "player": "PlayerA",
      "claim": "AlliedMCV",
      "evidenceLevel": "possible",
      "evidence": ["8:30 使用 SpecialPower_UnpackReplaceSelf"],
      "alternatives": ["AlliedMiner 展开后的指挥中心"],
      "needsConfirmation": ["是否曾使用 SpecialPower_PackReplaceSelf", "后续是否作为建造者出现"]
    }
  ],
  "eventClaims": [],
  "timelineClaims": []
}

The prompt asks the AI to output:

[机器可读声明]
```json
{ ... }

The parser first looks for the last fenced JSON block near `[机器可读声明]`, then falls back to the last `{...}` block.

## Validation We Can Do

### Implemented Now

Format validation:

- Missing machine-readable claims.
- JSON parse failure.
- Root value is not an object.

Self-consistency validation:

- Unit claim missing `unitId`.
- Unit claim missing `claim`.
- High-confidence UnitId guess without evidence.
- Low-confidence UnitId guess without alternatives or needed confirmation.

### Near-Term Validations

These need replay facts extracted from `CommandChunk` or an intermediate fact index:

- UnitId special power contradictions.
- UnitId production timeline contradictions.
- UnitId used as builder vs claimed as non-builder unit.
- Claims that use game knowledge not present in rules, such as transport/amphibious/building-placement abilities.
- Missing alternatives for ambiguous skills such as Allied unpack.

### Suggested Fact Index

Useful derived facts:

- `UnitId -> first observed time`
- `UnitId -> observed special powers`
- `UnitId -> observed as builder`
- `UnitId -> observed as production structure`
- `Player -> first production time by asset id`
- `Player -> selected UnitIds over time`
- `Player -> tech/protocol choices`
- `Player -> building placements by asset and position`

## Current Progress

Implemented:

- Added `CONTEXT.md` glossary.
- Added prompt settings:
  - `AiPromptSettings`
  - `UseCustomSystemPrompt`
  - `CustomSystemPrompt`
  - `AdditionalRules`
- Added prompt editing UI to `AIProviderSettingsControl`.
- Connected prompt settings from `EventDump` to `AIChatPanel` to `AIAnalyze`.
- Split default prompt construction from prompt composition.
- Strengthened default prompt with evidence-first and uncertainty rules.
- Added machine-readable JSON claim instructions to system and segment prompts.
- Added `Utils/AIAnalysisValidation.cs` with:
  - evidence level enum
  - machine-readable claim records
  - validation issue records
  - JSON extraction and parsing
  - initial self-consistency checks
- Added per-segment validation logging in `AIChatPanel`.

Build status:

- `dotnet build AnotherReplayReader.csproj --no-restore` succeeds.
- Remaining warnings are existing nullable warnings in `AIAnalyze.cs` stream response handling and a `System.Text.Encoding.CodePages` support warning for `net461`.

## Open Questions

- Should AI natural-language output continue streaming live, or should content be buffered until validation and possible revision are complete?
- Should reasoning chunks remain visible during hidden revision, or should only final content be shown?
- How strict should missing machine-readable claims be?
  - Current behavior: warning log only.
  - Possible future behavior: one hidden repair request asking the model to append valid claims.
- Should validation issues be visible by default, or only in an advanced/debug foldout?
- How much game-unit knowledge should live in code versus prompt text?
- Should the first verifier use hardcoded RA3/Corona knowledge, or should it load a small unit capability table from data files?

## Suggested Next Steps

1. Build a replay fact index from `CommandChunk`.
2. Add first deterministic validation rules:
   - ambiguous Allied unpack
   - pack/unpack consistency
   - UnitId used as builder
   - first production time vs first operation time
3. Decide whether to buffer per-segment content before display.
4. Add one hidden revision pass for `Contradiction` issues.
5. Add validation summary UI, such as "验证器发现并修正 N 个问题".