430 lines
15 KiB
Plaintext
430 lines
15 KiB
Plaintext
# RimWorld AI Agent 开发文档
|
||
|
||
> 本文档用于移交给 Codex 继续开发
|
||
|
||
---
|
||
|
||
## 1. 项目概述
|
||
|
||
### 目标
|
||
创建一个**完全自主的 AI Agent**，能够自动玩 RimWorld 游戏。用户只需给出开放式指令（如"帮我挖点铁"或"帮我玩10分钟"），AI 即可独立决策并操作游戏。
|
||
|
||
### 技术栈
|
||
- **语言**: C# (.NET Framework 4.8)
|
||
- **框架**: RimWorld Mod (Verse/RimWorld API)
|
||
- **AI 后端**: 阿里云百炼 (DashScope) API
|
||
- **VLM 模型**: Qwen-VL / Qwen-Omni-Realtime
|
||
|
||
### 核心设计
|
||
```
|
||
用户指令 → AIIntelligenceCore → [被动模式 | 主动模式] → 工具执行 → 游戏操作
|
||
```
|
||
|
||
---
|
||
|
||
## 2. 架构
|
||
|
||
### 2.1 模式切换设计
|
||
|
||
```
|
||
┌────────────────────────────────────────────┐
|
||
│ AIIntelligenceCore │
|
||
│ ┌─────────────┐ ┌─────────────────┐ │
|
||
│ │ 被动模式 │◄──►│ 主动模式 │ │
|
||
│ │ (聊天对话) │ │ (Agent循环) │ │
|
||
│ └─────────────┘ └────────┬────────┘ │
|
||
└──────────────────────────────┼────────────┘
|
||
▼
|
||
┌─────────────────────┐
|
||
│ AutonomousAgentLoop │
|
||
│ Observe → Think │
|
||
│ → Act │
|
||
└─────────────────────┘
|
||
```
|
||
|
||
**模式切换触发条件（待实现）**:
|
||
- 用户说"帮我玩X分钟" → 切换到主动模式
|
||
- 主动模式任务完成 → 自动切回被动模式
|
||
- 用户说"停止" → 强制切回被动模式
|
||
|
||
### 2.2 文件结构
|
||
|
||
```
|
||
Source/WulaFallenEmpire/EventSystem/AI/
|
||
├── AIIntelligenceCore.cs # 核心AI控制器（已有）
|
||
├── SimpleAIClient.cs # HTTP API 客户端（已有）
|
||
├── ScreenCaptureUtility.cs # 截屏工具（已有）
|
||
│
|
||
├── Agent/ # ★ 新增目录
|
||
│ ├── AutonomousAgentLoop.cs # 主动模式循环
|
||
│ ├── StateObserver.cs # 游戏状态收集器
|
||
│ ├── GameStateSnapshot.cs # 状态数据结构
|
||
│ ├── VisualInteractionTools.cs # 视觉交互工具(10个)
|
||
│ ├── MouseSimulator.cs # 鼠标模拟
|
||
│ └── OmniRealtimeClient.cs # WebSocket流式连接
|
||
│
|
||
├── Tools/ # AI工具
|
||
│ ├── AITool.cs # 工具基类（已有）
|
||
│ ├── Tool_GetGameState.cs # ★ 新增
|
||
│ ├── Tool_DesignateMine.cs # ★ 新增
|
||
│ ├── Tool_DraftPawn.cs # ★ 新增
|
||
│ ├── Tool_VisualClick.cs # ★ 新增
|
||
│ └── ... (其他原有工具)
|
||
│
|
||
└── UI/
|
||
├── Dialog_AIConversation.cs # 对话UI（已有）
|
||
└── Overlay_WulaLink.cs # 悬浮UI（已有）
|
||
```
|
||
|
||
---
|
||
|
||
## 3. 已完成组件
|
||
|
||
### 3.1 StateObserver (状态观察器)
|
||
|
||
**路径**: [Agent/StateObserver.cs](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/StateObserver.cs)
|
||
|
||
**功能**: 收集当前游戏状态，生成给 VLM 的文本描述
|
||
|
||
**API**:
|
||
```csharp
|
||
public static class StateObserver
|
||
{
|
||
// 捕获当前游戏状态快照
|
||
public static GameStateSnapshot CaptureState();
|
||
}
|
||
```
|
||
|
||
**收集内容**:
|
||
- 时间（小时、季节、年份）
|
||
- 环境（生物群系、温度、天气）
|
||
- 殖民者（名字、健康、心情、当前工作、位置）
|
||
- 资源（钢铁、银、食物、医药等）
|
||
- 建筑进度（蓝图、建造框架）
|
||
- 威胁（敌对派系、距离）
|
||
- 最近消息
|
||
|
||
---
|
||
|
||
### 3.2 GameStateSnapshot (状态数据结构)
|
||
|
||
**路径**: [Agent/GameStateSnapshot.cs](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/GameStateSnapshot.cs)
|
||
|
||
**功能**: 存储游戏状态数据
|
||
|
||
**关键方法**:
|
||
```csharp
|
||
public class GameStateSnapshot
|
||
{
|
||
public List<PawnSnapshot> Colonists;
|
||
public Dictionary<string, int> Resources;
|
||
public List<ThreatSnapshot> Threats;
|
||
|
||
// 生成给VLM的文本描述
|
||
public string ToPromptText();
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 3.3 VisualInteractionTools (视觉交互工具集)
|
||
|
||
**路径**: [Agent/VisualInteractionTools.cs](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs)
|
||
|
||
**功能**: 10个纯视觉交互工具，使用 Windows API 模拟输入
|
||
|
||
| 方法 | 功能 | 参数 |
|
||
|------|------|------|
|
||
| [MouseClick(x, y, button, clicks)](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs#49-89) | 鼠标点击 | 0-1比例坐标 |
|
||
| [TypeText(x, y, text)](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs#90-118) | 输入文本 | 通过剪贴板 |
|
||
| [ScrollWindow(x, y, direction, amount)](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs#119-144) | 滚动 | up/down |
|
||
| [MouseDrag(sx, sy, ex, ey, duration)](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs#145-187) | 拖拽 | 起止坐标 |
|
||
| [Wait(seconds)](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs#188-203) | 等待 | 秒数 |
|
||
| [PressEnter()](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs#204-220) | 按回车 | 无 |
|
||
| [PressEscape()](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs#221-237) | 按ESC | 无 |
|
||
| [DeleteText(x, y, count)](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs#238-262) | 删除 | 字符数 |
|
||
| [PressHotkey(x, y, hotkey)](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs#263-306) | 快捷键 | 如"ctrl+c" |
|
||
| [CloseWindow(x, y)](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/VisualInteractionTools.cs#307-329) | 关闭窗口 | Alt+F4 |
|
||
|
||
---
|
||
|
||
### 3.4 AutonomousAgentLoop (自主Agent循环)
|
||
|
||
**路径**: [Agent/AutonomousAgentLoop.cs](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/AutonomousAgentLoop.cs)
|
||
|
||
**功能**: 主动模式的核心循环
|
||
|
||
**状态**:
|
||
- `IsRunning`: 是否运行中
|
||
- `CurrentObjective`: 当前目标
|
||
- `DecisionCount`: 已执行决策次数
|
||
|
||
**关键API**:
|
||
```csharp
|
||
public class AutonomousAgentLoop : GameComponent
|
||
{
|
||
public static AutonomousAgentLoop Instance;
|
||
|
||
// 开始执行目标
|
||
public void StartObjective(string objective);
|
||
|
||
// 停止Agent
|
||
public void Stop();
|
||
|
||
// 事件
|
||
public event Action<string> OnDecisionMade;
|
||
public event Action<string> OnObjectiveComplete;
|
||
}
|
||
```
|
||
|
||
**待完成**: [ExecuteDecision()](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/AutonomousAgentLoop.cs#244-269) 方法需要整合工具执行逻辑
|
||
|
||
---
|
||
|
||
### 3.5 原生API工具
|
||
|
||
| 工具 | 路径 | 功能 | 参数格式 |
|
||
|------|------|------|----------|
|
||
| [Tool_GetGameState](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Tools/Tool_GetGameState.cs#8-45) | Tools/ | 获取游戏状态 | `<get_game_state/>` |
|
||
| [Tool_DesignateMine](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Tools/Tool_DesignateMine.cs#12-139) | Tools/ | 采矿指令 | `<designate_mine><x>数字</x><z>数字</z><radius>可选</radius></designate_mine>` |
|
||
| [Tool_DraftPawn](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Tools/Tool_DraftPawn.cs#10-105) | Tools/ | 征召殖民者 | `<draft_pawn><pawn_name>名字</pawn_name><draft>true/false</draft></draft_pawn>` |
|
||
|
||
---
|
||
|
||
## 4. 待完成任务
|
||
|
||
### 4.1 模式切换整合 (高优先级)
|
||
|
||
**目标**: 在 [AIIntelligenceCore](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/AIIntelligenceCore.cs#16-1364) 中实现被动/主动模式切换
|
||
|
||
**实现思路**:
|
||
```csharp
|
||
// AIIntelligenceCore 中添加
|
||
private bool _isAgentMode = false;
|
||
|
||
public void ProcessUserMessage(string message)
|
||
{
|
||
// 检测是否触发主动模式
|
||
if (IsAgentTrigger(message, out string objective, out float duration))
|
||
{
|
||
_isAgentMode = true;
|
||
AutonomousAgentLoop.Instance.StartObjective(objective);
|
||
// 设置定时器，duration后自动停止
|
||
}
|
||
else
|
||
{
|
||
// 正常对话处理
|
||
RunConversation(message);
|
||
}
|
||
}
|
||
|
||
private bool IsAgentTrigger(string msg, out string obj, out float dur)
|
||
{
|
||
// 匹配模式：
|
||
// "帮我玩10分钟" → obj="管理殖民地", dur=600
|
||
// "帮我挖点铁" → obj="采集铁矿", dur=0(无限)
|
||
// ...
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 4.2 工具执行整合 (高优先级)
|
||
|
||
**目标**: 让 `AutonomousAgentLoop.ExecuteDecision()` 能够执行工具
|
||
|
||
**当前状态**: 方法体是空的 TODO
|
||
|
||
**实现思路**:
|
||
```csharp
|
||
private void ExecuteDecision(string decision)
|
||
{
|
||
// 1. 检查特殊标记
|
||
if (decision.Contains("<no_action")) return;
|
||
if (decision.Contains("<objective_complete"))
|
||
{
|
||
OnObjectiveComplete?.Invoke(_currentObjective);
|
||
Stop();
|
||
return;
|
||
}
|
||
|
||
// 2. 获取工具实例
|
||
var core = AIIntelligenceCore.Instance;
|
||
var tools = core.GetAvailableTools();
|
||
|
||
// 3. 解析XML工具调用（复用AIIntelligenceCore的逻辑）
|
||
foreach (var tool in tools)
|
||
{
|
||
if (decision.Contains($"<{tool.Name}"))
|
||
{
|
||
string result = tool.Execute(decision);
|
||
WulaLog.Debug($"Tool {tool.Name}: {result}");
|
||
_lastToolResult = result;
|
||
break;
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 4.3 AI待办清单 (中优先级)
|
||
|
||
**目标**: AI 维护自己的待办清单，持久化到游戏存档
|
||
|
||
**设计**:
|
||
```csharp
|
||
public class AgentTodoList : IExposable
|
||
{
|
||
public List<TodoItem> Items = new List<TodoItem>();
|
||
|
||
public void ExposeData()
|
||
{
|
||
Scribe_Collections.Look(ref Items, "todoItems", LookMode.Deep);
|
||
}
|
||
}
|
||
|
||
public class TodoItem : IExposable
|
||
{
|
||
public string Description;
|
||
public bool IsComplete;
|
||
public int Priority;
|
||
public int CreatedTick;
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 4.4 Qwen-Omni-Realtime 测试 (低优先级)
|
||
|
||
**目标**: 测试 WebSocket 流式连接
|
||
|
||
**已完成**: [OmniRealtimeClient.cs](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/OmniRealtimeClient.cs) 基础实现
|
||
|
||
**待测试**:
|
||
- WebSocket 连接建立
|
||
- 图片发送 (`input_image_buffer.append`)
|
||
- 文本接收 (`response.text.delta`)
|
||
|
||
---
|
||
|
||
## 5. 关键接口参考
|
||
|
||
### 5.1 AITool 基类
|
||
|
||
```csharp
|
||
public abstract class AITool
|
||
{
|
||
public abstract string Name { get; }
|
||
public abstract string Description { get; }
|
||
public abstract string UsageSchema { get; }
|
||
public abstract string Execute(string args);
|
||
|
||
// 解析XML参数
|
||
protected Dictionary<string, string> ParseXmlArgs(string xmlContent);
|
||
}
|
||
```
|
||
|
||
### 5.2 SimpleAIClient API
|
||
|
||
```csharp
|
||
public class SimpleAIClient
|
||
{
|
||
public SimpleAIClient(string apiKey, string baseUrl, string model);
|
||
|
||
// 文本对话
|
||
public Task<string> GetChatCompletionAsync(
|
||
string systemPrompt,
|
||
List<(string role, string message)> messages,
|
||
int maxTokens = 1024,
|
||
float temperature = 0.7f
|
||
);
|
||
|
||
// VLM 视觉分析
|
||
public Task<string> GetVisionCompletionAsync(
|
||
string systemPrompt,
|
||
string userPrompt,
|
||
string base64Image,
|
||
int maxTokens = 1024,
|
||
float temperature = 0.7f
|
||
);
|
||
}
|
||
```
|
||
|
||
### 5.3 设置 (WulaFallenEmpireSettings)
|
||
|
||
```csharp
|
||
public class WulaFallenEmpireSettings : ModSettings
|
||
{
|
||
// 主模型
|
||
public string apiKey;
|
||
public string baseUrl = "https://dashscope.aliyuncs.com/compatible-mode/v1";
|
||
public string model = "qwen-turbo";
|
||
|
||
// VLM 模型
|
||
public string vlmApiKey;
|
||
public string vlmBaseUrl;
|
||
public string vlmModel = "qwen-vl-max";
|
||
public bool enableVlmFeatures = false;
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 6. 开发指南
|
||
|
||
### 6.1 添加新工具
|
||
|
||
1. 在 [Tools/](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/AIIntelligenceCore.cs#310-337) 创建新类继承 [AITool](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Tools/AITool.cs#8-41)
|
||
2. 实现 [Name](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/AIIntelligenceCore.cs#636-642), [Description](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Agent/StateObserver.cs#156-187), `UsageSchema`, [Execute](file:///C:/Steam/steamapps/common/RimWorld/Mods/3516260226/Source/WulaFallenEmpire/EventSystem/AI/Tools/Tool_VisualClick.cs#133-165)
|
||
3. 在 `AIIntelligenceCore.InitializeTools()` 中注册
|
||
|
||
```csharp
|
||
// 示例：Tool_BuildWall.cs
|
||
public class Tool_BuildWall : AITool
|
||
{
|
||
public override string Name => "build_wall";
|
||
public override string Description => "在指定位置放置墙壁蓝图";
|
||
public override string UsageSchema => "<build_wall><x>X</x><z>Z</z><stuff>材料</stuff></build_wall>";
|
||
|
||
public override string Execute(string args)
|
||
{
|
||
var dict = ParseXmlArgs(args);
|
||
// 实现建造逻辑
|
||
// GenConstruct.PlaceBlueprintForBuild(...)
|
||
return "Success: 墙壁蓝图已放置";
|
||
}
|
||
}
|
||
```
|
||
|
||
### 6.2 调试技巧
|
||
|
||
- 使用 `WulaLog.Debug()` 输出日志
|
||
- 检查 RimWorld 的 `Player.log` 文件
|
||
- 在开发者模式下 `Prefs.DevMode = true` 显示更多信息
|
||
|
||
### 6.3 常见问题
|
||
|
||
**Q: .NET Framework 4.8 兼容性问题**
|
||
- 不支持 `TakeLast()` → 使用 `Skip(list.Count - n)`
|
||
- 不支持 `string.Contains(x, StringComparison)` → 使用 `IndexOf`
|
||
|
||
**Q: Unity 主线程限制**
|
||
- 异步操作结果需要回到主线程执行
|
||
- 使用 `LongEventHandler.ExecuteWhenFinished(() => { ... })`
|
||
|
||
---
|
||
|
||
## 7. 参考资源
|
||
|
||
- [RimWorld Modding Wiki](https://rimworldwiki.com/wiki/Modding)
|
||
- [Harmony Patching](https://harmony.pardeike.net/)
|
||
- [阿里云百炼 API](https://help.aliyun.com/zh/model-studio/)
|
||
- [Qwen-Omni 文档](https://help.aliyun.com/zh/model-studio/user-guide/qwen-omni)
|
||
|
||
---
|
||
|
||
**文档版本**: v1.0
|
||
**更新时间**: 2025-12-27
|
||
**作者**: Gemini AI Agent
|