codex wip
This commit is contained in:
@@ -0,0 +1,233 @@
|
||||
# AI Analysis WIP
|
||||
|
||||
## User Need
|
||||
|
||||
The application is adding AI analysis for Red Alert 3 replay operation logs. The current flow sends player information and a compacted operation log to a chat-completion-compatible LLM, then displays the analysis in `AIChatPanel`.
|
||||
|
||||
The main goals are:
|
||||
|
||||
- Improve the current system prompt.
|
||||
- Make the system prompt configurable by the user.
|
||||
- Reduce AI analysis errors, especially errors caused by general-world assumptions or overconfident UnitId guesses.
|
||||
- Add a validation path for LLM output so wrong claims can be detected and corrected without wasting all prior reasoning.
|
||||
|
||||
## Current Code Areas
|
||||
|
||||
- `Utils/AIAnalyze.cs`: builds the system prompt, user prompts, segment prompts, final summary prompts, and performs OpenAI-compatible chat completion calls.
|
||||
- `AIChatPanel.xaml.cs`: runs the analysis workflow, displays streaming chunks, retries failed segments, and now logs machine-readable claim validation results.
|
||||
- `Utils/AiSettings.cs`: stores AI provider/model settings and now prompt settings.
|
||||
- `AIProviderSettingsControl.xaml(.cs)`: edits provider/model settings and now prompt settings.
|
||||
- `EventDump.xaml.cs`: generates the replay operation text and starts AI analysis.
|
||||
- `Utils/AIAnalysisValidation.cs`: new validation model and parser for machine-readable AI claims.
|
||||
- `CONTEXT.md`: glossary for the AI analysis domain.
|
||||
|
||||
## Discussion Notes
|
||||
|
||||
### Prompt Problems
|
||||
|
||||
The current prompt has a lot of useful game knowledge, but the LLM can still:
|
||||
|
||||
- Use common sense that is wrong for the game or mod.
|
||||
- Assume infantry, helicopters, transports, amphibious movement, and water placement work like they do in other RTS games.
|
||||
- Treat one observed skill as conclusive evidence when multiple units share that skill.
|
||||
- Overstate UnitId guesses.
|
||||
|
||||
Examples discussed:
|
||||
|
||||
- Only units explicitly marked amphibious can move on both land and water.
|
||||
- Only units explicitly marked as passenger transports can transport infantry.
|
||||
- Building water placement depends on game rules, not common assumptions.
|
||||
- `SpecialPower_UnpackReplaceSelf` does not uniquely identify an Allied MCV because Allied miners can also unpack into a command hub.
|
||||
- A UnitId claimed as an aircraft should be challenged if the same UnitId is observed using an unpack/deploy skill.
|
||||
- A UnitId claimed as a bomber should be challenged if it is operated before the player starts producing their first bomber.
|
||||
|
||||
### Prompt Decisions
|
||||
|
||||
The default prompt should explicitly require:
|
||||
|
||||
- Evidence-first analysis.
|
||||
- No use of external common sense over replay facts and supplied game rules.
|
||||
- UnitId guesses with evidence levels.
|
||||
- Multiple candidates when a behavior has several possible sources.
|
||||
- Support evidence and possible counter-evidence for important claims.
|
||||
- Correction or abandonment of claims contradicted by replay facts.
|
||||
|
||||
Evidence levels currently used:
|
||||
|
||||
- confirmed
|
||||
- highly likely
|
||||
- possible
|
||||
- uncertain
|
||||
- ruled out
|
||||
|
||||
The default provider temperature was lowered from `0.75` to `0.35` because this task is closer to audit/reconstruction than creative writing.
|
||||
|
||||
### Prompt Configuration Decisions
|
||||
|
||||
The prompt is now configurable through AI settings.
|
||||
|
||||
The design has two prompt layers:
|
||||
|
||||
- A built-in dynamic system prompt, still assembled from replay/mod/faction/map context.
|
||||
- User prompt settings:
|
||||
- optional full custom system prompt
|
||||
- additional rules appended to the final system prompt
|
||||
|
||||
This keeps the normal path safe while allowing advanced users to override the whole prompt.
|
||||
|
||||
### Validation Philosophy
|
||||
|
||||
LLM natural-language analysis should not be treated as directly valid. The plan is to validate structured claims emitted by the LLM.
|
||||
|
||||
Important decision:
|
||||
|
||||
- Do not immediately throw away a whole analysis when a problem is found.
|
||||
- Do not show the user two competing analyses or apology text such as "sorry, my previous answer was wrong."
|
||||
- Prefer a hidden revision pass: send the draft, validation issues, and relevant replay facts back to the AI, asking it to output a clean corrected version without mentioning the revision.
|
||||
- Limit retries/revisions. If the model still cannot resolve a claim, downgrade confidence or mark it uncertain instead of looping forever.
|
||||
|
||||
Severity model:
|
||||
|
||||
- `Info`: useful diagnostic only.
|
||||
- `WeakEvidence`: claim may be plausible but lacks enough support.
|
||||
- `Warning`: malformed or questionable claim that should be logged or possibly revised.
|
||||
- `Contradiction`: claim conflicts with replay facts or game rules and should trigger revision.
|
||||
- `Fatal`: output cannot be used for the current phase, such as empty or unparseable required output.
|
||||
|
||||
### JSON Format Decision
|
||||
|
||||
We discussed whether to require JSON or use a simpler line-based format.
|
||||
|
||||
Decision:
|
||||
|
||||
- Use JSON for machine-readable claims.
|
||||
- Keep the schema small.
|
||||
- Make the parser tolerant.
|
||||
|
||||
Reasoning:
|
||||
|
||||
- JSON can naturally represent evidence arrays, alternatives, and needed confirmations.
|
||||
- A custom line format would be easier for a trivial parser but would become fragile once nested data is needed.
|
||||
- The app can tolerate partial or missing JSON by logging validation issues instead of failing the whole analysis.
|
||||
|
||||
Current expected shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"unitClaims": [
|
||||
{
|
||||
"unitId": 123,
|
||||
"player": "PlayerA",
|
||||
"claim": "AlliedMCV",
|
||||
"evidenceLevel": "possible",
|
||||
"evidence": ["8:30 使用 SpecialPower_UnpackReplaceSelf"],
|
||||
"alternatives": ["AlliedMiner 展开后的指挥中心"],
|
||||
"needsConfirmation": ["是否曾使用 SpecialPower_PackReplaceSelf", "后续是否作为建造者出现"]
|
||||
}
|
||||
],
|
||||
"eventClaims": [],
|
||||
"timelineClaims": []
|
||||
}
|
||||
```
|
||||
|
||||
The prompt asks the AI to output:
|
||||
|
||||
```text
|
||||
[机器可读声明]
|
||||
```json
|
||||
{ ... }
|
||||
```
|
||||
```
|
||||
|
||||
The parser first looks for the last fenced JSON block near `[机器可读声明]`, then falls back to the last `{...}` block.
|
||||
|
||||
## Validation We Can Do
|
||||
|
||||
### Implemented Now
|
||||
|
||||
Format validation:
|
||||
|
||||
- Missing machine-readable claims.
|
||||
- JSON parse failure.
|
||||
- Root value is not an object.
|
||||
|
||||
Self-consistency validation:
|
||||
|
||||
- Unit claim missing `unitId`.
|
||||
- Unit claim missing `claim`.
|
||||
- High-confidence UnitId guess without evidence.
|
||||
- Low-confidence UnitId guess without alternatives or needed confirmation.
|
||||
|
||||
### Near-Term Validations
|
||||
|
||||
These need replay facts extracted from `CommandChunk` or an intermediate fact index:
|
||||
|
||||
- UnitId special power contradictions.
|
||||
- UnitId production timeline contradictions.
|
||||
- UnitId used as builder vs claimed as non-builder unit.
|
||||
- Claims that use game knowledge not present in rules, such as transport/amphibious/building-placement abilities.
|
||||
- Missing alternatives for ambiguous skills such as Allied unpack.
|
||||
|
||||
### Suggested Fact Index
|
||||
|
||||
Useful derived facts:
|
||||
|
||||
- `UnitId -> first observed time`
|
||||
- `UnitId -> observed special powers`
|
||||
- `UnitId -> observed as builder`
|
||||
- `UnitId -> observed as production structure`
|
||||
- `Player -> first production time by asset id`
|
||||
- `Player -> selected UnitIds over time`
|
||||
- `Player -> tech/protocol choices`
|
||||
- `Player -> building placements by asset and position`
|
||||
|
||||
## Current Progress
|
||||
|
||||
Implemented:
|
||||
|
||||
- Added `CONTEXT.md` glossary.
|
||||
- Added prompt settings:
|
||||
- `AiPromptSettings`
|
||||
- `UseCustomSystemPrompt`
|
||||
- `CustomSystemPrompt`
|
||||
- `AdditionalRules`
|
||||
- Added prompt editing UI to `AIProviderSettingsControl`.
|
||||
- Connected prompt settings from `EventDump` to `AIChatPanel` to `AIAnalyze`.
|
||||
- Split default prompt construction from prompt composition.
|
||||
- Strengthened default prompt with evidence-first and uncertainty rules.
|
||||
- Added machine-readable JSON claim instructions to system and segment prompts.
|
||||
- Added `Utils/AIAnalysisValidation.cs` with:
|
||||
- evidence level enum
|
||||
- machine-readable claim records
|
||||
- validation issue records
|
||||
- JSON extraction and parsing
|
||||
- initial self-consistency checks
|
||||
- Added per-segment validation logging in `AIChatPanel`.
|
||||
|
||||
Build status:
|
||||
|
||||
- `dotnet build AnotherReplayReader.csproj --no-restore` succeeds.
|
||||
- Remaining warnings are existing nullable warnings in `AIAnalyze.cs` stream response handling and a `System.Text.Encoding.CodePages` support warning for `net461`.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should AI natural-language output continue streaming live, or should content be buffered until validation and possible revision are complete?
|
||||
- Should reasoning chunks remain visible during hidden revision, or should only final content be shown?
|
||||
- How strict should missing machine-readable claims be?
|
||||
- Current behavior: warning log only.
|
||||
- Possible future behavior: one hidden repair request asking the model to append valid claims.
|
||||
- Should validation issues be visible by default, or only in an advanced/debug foldout?
|
||||
- How much game-unit knowledge should live in code versus prompt text?
|
||||
- Should the first verifier use hardcoded RA3/Corona knowledge, or should it load a small unit capability table from data files?
|
||||
|
||||
## Suggested Next Steps
|
||||
|
||||
1. Build a replay fact index from `CommandChunk`.
|
||||
2. Add first deterministic validation rules:
|
||||
- ambiguous Allied unpack
|
||||
- pack/unpack consistency
|
||||
- UnitId used as builder
|
||||
- first production time vs first operation time
|
||||
3. Decide whether to buffer per-segment content before display.
|
||||
4. Add one hidden revision pass for `Contradiction` issues.
|
||||
5. Add validation summary UI, such as "验证器发现并修正 N 个问题".
|
||||
Reference in New Issue
Block a user