Compare commits
5
Commits
130f8b4c1d
..
0.1.1
| Author | SHA1 | Date | |
|---|---|---|---|
 lanyi
|
44b2eaf273 | ||
|
lanyi
|
4cc07b9abc | ||
|
lanyi
|
45fa2b9a27 | ||
|
lanyi
|
429ea367f3 | ||
|
lanyi
|
656fab349d |
@@ -3,3 +3,4 @@ out/
|
||||
dist/
|
||||
*.vsix
|
||||
OpenSAGE/
|
||||
debug.log
|
||||
@@ -4,19 +4,28 @@
|
||||
|
||||
## 功能
|
||||
|
||||
- **语法高亮**：在普通 XML 高亮之上叠加领域标记（`$DEFINE` 常量、`inheritFrom`、`xai:joinAction`、结构标签）。
|
||||
- **语法高亮**：在普通 XML 高亮之上叠加领域标记（`$DEFINE` 常量、`inheritFrom`、`xai:joinAction`、结构标签）；XML 语法异常（如未闭合引号）期间由语义 token 兜底，标签/属性/值着色不中断。
|
||||
- **自动补全**：
|
||||
- 元素名：按当前父元素的 XSD 模型补全子元素；顶层资产（`AssetDeclaration` 内）补全 `GameObject`、`WeaponTemplate` 等 99+ 类型。
|
||||
- 元素名：按当前父元素的 XSD 模型补全子元素；顶层资产（`AssetDeclaration` 内）补全 `GameObject`、`WeaponTemplate` 等 295 种类型。
|
||||
- 属性名：必填属性优先，附带类型/文档/默认值；自动提示 `xai:joinAction` 与 `xmlns:xai`。
|
||||
- 属性值：
|
||||
- 引用型属性（如 `CommandSet`、`Weapon`）按 `xas:refType` 补全对应类型的资产 ID（**同名 ID 只补全匹配类型**）；
|
||||
- `inheritFrom` 补全可继承的资产 ID；
|
||||
- 枚举（如 `Include type`）、布尔值、`$DEFINE` 常量；
|
||||
- 枚举与位标志列表（如 `Include type`、`LocomotorTemplate@Surfaces`、`KindOf`；列表值支持空格后继续补全下一项）；
|
||||
- 布尔值、`$DEFINE` 常量；
|
||||
- `<Include source>` 补全可解析的 `DATA:` / `ART:` / `AUDIO:` 与项目相对路径。
|
||||
- **悬停提示**：元素/属性显示 XSD 文档、类型、必填/默认值；引用值显示定义位置；`$DEFINE` 显示值与定义位置。
|
||||
- **引用导航**：从引用值（`CommandSet="..."`、`Weapon="..."`、`inheritFrom`）跳转到定义（严格按引用类型过滤）；`Ctrl+点击` Include 打开目标文件；Find All References 搜索整个工作区；文档大纲列出顶层资产与 `$DEFINE`。
|
||||
- **错误检查**：XML 格式错误、未知元素/属性、顶层资产缺 `id`、重复 ID、未解析引用（含类型不匹配）、Include 找不到、`$DEFINE` 未定义。
|
||||
- **悬停提示**：元素/属性显示 XSD 文档、类型、必填/默认值；引用值显示定义位置；`$DEFINE` 显示值与定义位置；`Include source` / `xi:include href` 显示解析后的目标文件；`xi:include` 元素与属性给出 XInclude 说明。
|
||||
- **引用导航**：从引用值（`CommandSet="..."`、`Weapon="..."`、`inheritFrom`）跳转到定义（严格按引用类型过滤，候选由 `ra3modxml.definitionMode` 控制：`all` 列出 mod + 原版、`project-only` 优先项目内定义）；`Ctrl+点击` Include / `xi:include href` 打开目标文件；Find All References 搜索整个工作区；文档大纲列出顶层资产与 `$DEFINE`。
|
||||
- **错误检查**：XML 格式错误、未知元素/属性（`xi:` 等外来命名空间不误报）、顶层资产缺 `id`、重复 ID、未解析引用（含类型不匹配）、Include / 嵌套 `xi:include` 目标找不到、`$DEFINE` 未定义。
|
||||
- **manifest 支持**：`<Include type="reference">` 指向的 `static/global/audio.manifest`（SDK `builtmods`）会被解析，manifest 中的原版资产 ID 可用于补全/悬停/导航/诊断。
|
||||
- **美术资产（`.w3x`）**：`W3X.xml` / `ART:` include 链中的 `.w3x` 模型文件会被
|
||||
索引（`W3DContainer` / `W3DMesh` / `W3DHierarchy` 等顶层资产），因此
|
||||
`Model@Name`、`Hierarchy`、`Mesh` 等引用可以解析、悬停与跳转。超大模型
|
||||
（几十 MB 的顶点/三角形数据）采用浅扫描——只提取顶层资产记录、不建 DOM 树，
|
||||
结果在 workspace 级缓存并跨重建复用，保存文件触发的重建不会重读未变化的模型文件。
|
||||
- **大项目性能**：索引记录（资产 / Define / Include / 行号）与 include 解析结果
|
||||
跨重建缓存，保存触发的重建零 stat、零重读（Corona 实测约 2 秒）；DOM 树只按需
|
||||
保留并设元素预算，避免内存膨胀。
|
||||
|
||||
## 使用
|
||||
|
||||
@@ -32,6 +41,7 @@
|
||||
| `ra3modxml.indexSageXml` | `true` | 是否索引 SDK 的 `SageXml` 原版源码 |
|
||||
| `ra3modxml.reportUnresolvedReferences` | `warning` | 未解析引用诊断级别（`warning`/`information`/`none`） |
|
||||
| `ra3modxml.diagnoseUnknownElements` | `true` | 是否报告未知元素/属性（自定义 XSD 项目可关闭） |
|
||||
| `ra3modxml.definitionMode` | `all` | 跳转候选：`all` 列出 mod 定义与原版定义（mod 优先）；`project-only` 仅在项目内已有定义时直接跳转 mod 定义 |
|
||||
| `ra3modxml.additionalDataSearchPaths` | `[]` | 追加的 `DATA:` 搜索目录 |
|
||||
|
||||
### 命令
|
||||
@@ -57,15 +67,25 @@ npm run package # 生成可安装的 .vsix
|
||||
src/
|
||||
extension.ts 激活入口与 provider 注册
|
||||
workspace.ts 项目检测、索引生命周期、状态栏
|
||||
language/xmlParser.ts 带源码偏移的轻量 XML 解析器
|
||||
language/context.ts 补全上下文分析
|
||||
model/schemaModel.ts XSD 模型运行时（由 tools 生成 JSON 驱动）
|
||||
settings.ts 配置读取（sdkPath、definitionMode 等）
|
||||
language/
|
||||
xmlParser.ts 带源码偏移的轻量 XML 解析器（容错、行尾恢复）
|
||||
context.ts 补全上下文分析
|
||||
typeContext.ts 上下文感知元素类型解析
|
||||
semanticTokens.ts 语义 token 兜底高亮（纯 TS）
|
||||
model/
|
||||
schemaModel.ts XSD 模型运行时（schema-model.json / asset-types.json 由 tools 生成）
|
||||
indexer/
|
||||
includeResolver.ts Include 路径解析（纯 TS，移植 check_duplicate_ids.py）
|
||||
manifestParser.ts .manifest 二进制解析（移植 OpenSAGE ManifestFile.cs）
|
||||
fileScanner.ts 目录扫描与 Include source 候选
|
||||
refs.ts 引用目标解析（按引用类型过滤）
|
||||
indexer.ts 工作区索引器（后台、缓存、增量重建）
|
||||
features/ completion / hover / navigation / diagnostics
|
||||
shallowScan.ts .w3x 等大体积美术资产顶层浅扫描（纯 TS，不建 DOM）
|
||||
records.ts 每文件紧凑索引记录（资产/Define/Include/xi + 行号）
|
||||
caches.ts 跨重建持久缓存（DocumentCache / IndexRecordsCache /
|
||||
IncludeResolveCache）
|
||||
indexer.ts 工作区索引器（后台、缓存、记录驱动重建）
|
||||
features/ completion / hover / navigation / diagnostics / semanticTokens
|
||||
syntaxes/ TextMate 注入语法
|
||||
tools/ XSD → 模型、AssetType 枚举提取
|
||||
```
|
||||
@@ -76,4 +96,5 @@ tools/ XSD → 模型、AssetType 枚举提取
|
||||
|
||||
- 领域说明与需求：`docs/requirements.md`
|
||||
- 调研与设计决策：`docs/plan.md`
|
||||
- 问题分析与修复记录：`docs/analysis-issues.md`
|
||||
- Manifest 格式参考：OpenSAGE `src/OpenSage.Game/Data/StreamFS/ManifestFile.cs`（本仓库 `OpenSAGE/` 子目录，commit `d45d361`）
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
[0801/032923.150:ERROR:third_party\crashpad\crashpad\util\win\registration_protocol_win.cc:108] CreateFile: 拒绝访问。 (0x5)
|
||||
@@ -240,3 +240,525 @@ CommandSet 引用仍能解析到 LogicCommandSet 定义（无回归）
|
||||
- AttachTest 实机：三个原始场景全部按预期（Locomotor 命中 LocomotorTemplate、_SKN 命中 W3DContainer、cannon 双候选均可精确定位）。
|
||||
|
||||
`.vsix` 已重新打包（13:33）。D 盘恢复后照旧可补跑 GenEvoTest / Corona 回归。
|
||||
|
||||
---
|
||||
|
||||
## 九、问题分析（第四轮，2026-08-01）：模块 `id` 被误报为未解析引用
|
||||
|
||||
### 问题：`id="ModuleTag_Draw"` 误报 `Unresolved reference`
|
||||
|
||||
**现象**：AttachTest `Allied Vehicle\Guardian Tank\GameObject.xml` 第 45 行
|
||||
|
||||
```xml
|
||||
<TruckDraw
|
||||
id="ModuleTag_Draw"
|
||||
...>
|
||||
```
|
||||
|
||||
报 `Unresolved reference "ModuleTag_Draw" (not found in the current index)`，
|
||||
hover 同时显示 `No matching definition of the expected declared type...`。
|
||||
但该 id 是 TruckDraw（GameObject 模块）自身的标识，只在所属 `<GameObject />` 内部有效，
|
||||
此处就是定义处，全局资产索引中不存在（也不应存在）它的定义。
|
||||
|
||||
**根因（两层叠加）**：
|
||||
|
||||
1. **模型生成器丢失属性级 `xas:refType`**：`ModuleData@id` 在 XSD 中声明为
|
||||
|
||||
```xml
|
||||
<xs:complexType name="ModuleData" xas:isPolymorphic="true">
|
||||
<xs:attribute name="id" type="Poid" xas:refType="ModuleData" />
|
||||
</xs:complexType>
|
||||
```
|
||||
|
||||
refType 写在 `<xs:attribute>` 节点上，而 `tools/xsd-to-model.mjs` 只从 simple type
|
||||
描述符读取 refType，属性级声明被丢弃。`Poid` 本身带 `xas:isWeakRef="true"`（“管线对象
|
||||
ID”），于是该 id 在模型里变成 `{ type: "Poid", refType: null, isRef: true }`——一个
|
||||
“无类型引用”，导致**所有继承自 ModuleData 的模块类型（以及 ObjectFilter、MapObject、
|
||||
GameScript、AIStateTactic 等共 430 个类型）的 `id` 都被当作全局引用检查**。
|
||||
|
||||
2. **`id` 的语义是“定义点”而非“引用”**：即便 refType 正确，嵌套元素（模块、nugget、
|
||||
地图对象）的 `id` 也只是其局部标识，检查全局 unresolved 必然误报。反过来，XSD 里确有一类
|
||||
真正引用其他资产类型的 `id`（如 `RoadObject@id` 为 `AssetReference` + `xas:refType="Road"`），
|
||||
这类检查必须保留。
|
||||
|
||||
**验证**：
|
||||
|
||||
- 用未修改的生成器对当前 SDK XSD 重新生成模型，与仓库内模型逐字段一致（0 差异），
|
||||
证明修复后重新生成的 diff 只落在属性级 refType 上，无无关噪音。
|
||||
- 修复后 `W3DTruckDrawModuleData@id` → `refType: ModuleData`；
|
||||
`isReferenceAttributeOfType(...) === false`；空索引下 `TruckDraw@id` 不再产生诊断；
|
||||
全文件扫描 0 个 `id` 误报，61 处真实引用属性（`inheritFrom`、`CommandSet`、`Side`、
|
||||
`Locomotor`、`TrackMarks` 等）行为不变。
|
||||
|
||||
**修复**：
|
||||
|
||||
1. **生成器**（`tools/xsd-to-model.mjs`）：`collectAttributes` 改为
|
||||
`attr["@_refType"] ?? desc?.refType`（属性级优先、simple type 兜底），重新生成
|
||||
`schema-model.json`——共恢复 444 处 refType（含继承传播），`isRef` 与其他字段零变化。
|
||||
附带收益：`Locomotor → LocomotorTemplate`、`Armor → ArmorTemplate`、
|
||||
`ThingTemplate → GameObject` 等此前被当作“无类型引用”的属性恢复真实类型
|
||||
（第三轮中 Locomotor“无 refType”的结论实为该生成器 bug 的误判）。
|
||||
2. **局部引用规则**（`src/indexer/refs.ts` 新增 `isLocalReferenceAttribute`）：
|
||||
- `id`：无 refType，或 refType 与元素自身类型兼容（`isAssignableTo`，如
|
||||
`W3DTruckDrawModuleData → ModuleData`）→ 定义点，不做全局引用检查；
|
||||
refType 指向不同类型（`RoadObject@id → Road`）→ 保留真实引用检查；
|
||||
- 非 `id` 且类型为 `Poid` 的属性（`ModuleId`、`AutoResolveBody`、`SoundRef`、
|
||||
`AttachModuleId`…）→ 管线局部引用，全局索引无法判定，不检查。
|
||||
`isReferenceAttributeOfType` 与 `resolveReferenceTargetsForType` 同步使用该守卫，
|
||||
诊断 / hover / 跳转 / 补全行为一致。
|
||||
3. **补全**（`src/features/completion.ts`）：`id` 与 Poid 属性不再按 refType 提供
|
||||
全局资产补全（模块 id 是局部的，全局资产列表是错误建议）。
|
||||
|
||||
**测试（31 → 37，全部通过）**：
|
||||
|
||||
- `refs.test.mjs`：TruckDraw 实景结构（GameObject → Draws → TruckDraw）下 `id` 不再是
|
||||
引用且不解析；`RoadObject@id` 仍是引用并能解析到 Road；`AttachModuleId` 等 Poid 属性
|
||||
不误报；Locomotor 改为严格类型引用（同名 GameObject 不再匹配）；
|
||||
- `schemaModel.test.mjs`：`ModuleData@id` / `MapObject@id` / `ThingTemplate` /
|
||||
`RoadObject@id` / `AttachModuleId` 的属性级 refType 断言；
|
||||
- 回归：原有 31 个用例全部保持通过。
|
||||
|
||||
> **后续可做**：GameObject 内模块 id 的“局部作用域”解析——`AttachModuleId`、
|
||||
> `ModuleId` 等模块引用指向同一 GameObject 内的兄弟模块，但部分引用（如武器上的
|
||||
> `AttachModuleId`）目标 GameObject 跨文件无法静态确定，本轮先统一不检查；待局部
|
||||
> 作用域建模落地后再启用这些引用的解析与诊断。
|
||||
|
||||
---
|
||||
|
||||
## 十、问题分析（第五轮，2026-08-01）：`xi:include` 的 `href`/`xpointer` 误报未知属性
|
||||
|
||||
### 问题
|
||||
|
||||
AttachTest `Allied Vehicle\Guardian Tank\GameObject.xml` 第 249 行附近：
|
||||
|
||||
```xml
|
||||
<xi:include
|
||||
href="DATA:Includes/HeadlightDraw2.xml"
|
||||
xpointer="xmlns(n=uri:ea.com:eala:asset) xpointer(/n:HeadlightDraw2/child::*)"/>
|
||||
```
|
||||
|
||||
报两条 `Unknown attribute "href" / "xpointer" for <include>`（`unknown-attribute`），
|
||||
hover 同时显示 `Unknown attribute for this element.`。
|
||||
|
||||
这个元素属于 **W3C XInclude 命名空间**（`xmlns:xi="http://www.w3.org/2001/XInclude"`），
|
||||
并不是 EA `uri:ea.com:eala:asset` XSD 的一部分。同一行在第二轮“问题 C”处理过
|
||||
（嵌套 `xi:include` 的索引与导航），但那轮没有覆盖 unknown-attribute 诊断，属于遗留缺口。
|
||||
|
||||
### 根因
|
||||
|
||||
诊断的属性校验没有像元素校验那样排除外来命名空间：
|
||||
|
||||
- 元素校验已有 `!el.name.startsWith("xi:")` 守卫（所以 `<include>` 本身不报 unknown element）；
|
||||
- 属性校验只跳过 `xmlns*` / `xai:` / `xi:` 前缀的属性名，而 `href`、`xpointer` 是不带
|
||||
前缀的普通属性名；
|
||||
- `<xi:include>` 解析类型为 null（XSD 模型不含该元素），knownAttrs 为空 → 任何属性
|
||||
都被判为 unknown。
|
||||
|
||||
### 修复
|
||||
|
||||
1. `schemaModel` 新增两个纯函数：
|
||||
- `isXsdElementName`：`xi:` 前缀元素不属于 EA XSD 模型；
|
||||
- `isXsdAttributeName`：EA XSD 属性不带命名空间前缀，带前缀（`xai:`、`xi:`、
|
||||
`xlink:`、`xml:`、`xsi:`、`xmlns:*`）的都是命名空间机制，不做 schema 校验。
|
||||
2. `diagnostics`：`xi:` 前缀元素整体跳过 schema 校验（元素与属性都不再误报）；
|
||||
前缀属性名统一跳过。
|
||||
3. `hover`：`xi:include` 元素/属性给出 XInclude 说明；`href` 值悬停像
|
||||
`<Include source>` 一样解析目标文件（Ctrl+点击跳转此前已可用）。
|
||||
|
||||
### 验证
|
||||
|
||||
- 真实文件全量扫描：0 未知元素、0 未知属性（修复前 `href`/`xpointer` 两条必现）；
|
||||
- 新增测试：`isXsdElementName` / `isXsdAttributeName` 断言；`xi:include` 解析类型为
|
||||
null 且不参与校验；全量 39/39 通过。
|
||||
|
||||
### 后续（架构方向，待确认）
|
||||
|
||||
用户提出“先展开 `xi:include`（类比 C++ 宏展开），再处理 mod XML 解析”。该方向与第二轮
|
||||
遗留的“虚拟合并”开放项一致，设计要点：
|
||||
|
||||
- 构建**逻辑树**而非文本拼接：把目标文件选中内容（`xpointer` 子集）作为子节点拼入父
|
||||
元素，节点保留源文件与原始偏移，避免文本级拼接导致的偏移断裂；
|
||||
- 展开范围：`xi:include` 与 EA `<Include type="all">`（内容合并）；`instance` /
|
||||
`reference` 是可见性 / 编译产物语义，不拼树；`inheritFrom` + `joinAction` 是属性级
|
||||
继承合并，不是宏展开；
|
||||
- 收益：跨 include 的上下文类型解析、包含内容的结构校验、以及后续“GameObject 内模块
|
||||
id 局部作用域”（HeadlightDraw2 的模块也是该 GameObject 的模块）；
|
||||
- 风险：include 环 / 深度限制、大文件性能、`xpointer` 仅支持现有子集形式
|
||||
（`/n:Name/child::*`）。
|
||||
|
||||
---
|
||||
|
||||
## 十、问题分析（第五轮，2026-08-01）：`xi:include` 的 `href` / `xpointer` 被误报为未知属性
|
||||
|
||||
### 问题
|
||||
|
||||
同一 GameObject.xml 第 246–249 行：
|
||||
|
||||
```xml
|
||||
<!-- include Headlight draw module. -->
|
||||
<xi:include
|
||||
href="DATA:Includes/HeadlightDraw2.xml"
|
||||
xpointer="xmlns(n=uri:ea.com:eala:asset) xpointer(/n:HeadlightDraw2/child::*)"/>
|
||||
```
|
||||
|
||||
报两条 `Unknown attribute "href" / "xpointer" for <include>`（unknown-attribute），
|
||||
hover 显示 `Unknown attribute for this element.`。
|
||||
|
||||
**与第二轮的关系**：第二轮“问题 C”处理的正是同一行的嵌套 `xi:include`——但那一轮修的是
|
||||
**索引器**（嵌套 include 不再被静默忽略、缺失目标产生 include-not-found、目标内容进索引），
|
||||
本轮这处 **unknown-attribute 诊断**是当时未覆盖的遗留问题。
|
||||
|
||||
### 根因
|
||||
|
||||
`xi:include` 属于 W3C XInclude 命名空间（`http://www.w3.org/2001/XInclude`），
|
||||
**不是 RA3 XSD（`uri:ea.com:eala:asset`）定义的元素**：
|
||||
|
||||
- 未知元素检查已通过 `el.name.startsWith("xi:")` 跳过，所以没有 unknown-element 误报；
|
||||
- 但属性检查没有同类守卫：`xi:include` 解析类型为 null → `knownAttrs` 为空 →
|
||||
`href`、`xpointer` 两个非 `xi:` 前缀的属性名全部落入 unknown-attribute 分支。
|
||||
|
||||
复现证据（真实文件）：
|
||||
|
||||
```
|
||||
xi:include found: true | parent: Draws
|
||||
resolved element type: null
|
||||
known attribute names: (none)
|
||||
attr href: known=false -> would flag unknown-attribute: true
|
||||
```
|
||||
|
||||
### 修复
|
||||
|
||||
1. **模型层新增命名空间守卫**（`schemaModel.ts`）：
|
||||
- `isXsdElementName(name)`：`xi:` 前缀（XInclude）等外来命名空间元素不属于 XSD 模型；
|
||||
- `isXsdAttributeName(name)`：EA XSD 属性一律无前缀，带前缀的属性
|
||||
（`xai:`、`xi:`、`xlink:`、`xml:`、`xsi:`、`xmlns:*`）都是命名空间机制，不做未知属性校验。
|
||||
2. **诊断**（`diagnostics.ts`）：外来命名空间元素的未知元素/未知属性检查整体跳过
|
||||
（`href`、`xpointer` 不再误报）；属性名带前缀的一律跳过校验（比原先只跳过
|
||||
`xmlns`/`xai:`/`xi:` 更完整）。
|
||||
3. **hover**（`hover.ts`）：`xi:include` 的元素/属性悬停显示 XInclude 说明；
|
||||
`xi:include@href` 与 `Include@source` 一样显示解析后的目标文件（与第二轮已可用的
|
||||
Ctrl+点击跳转对齐）。
|
||||
|
||||
**测试（37 → 39，全部通过）**：
|
||||
|
||||
- `schemaModel.test.mjs`：`isXsdElementName` / `isXsdAttributeName` 判定
|
||||
（`xi:include` 非 XSD 元素；`href`/`xpointer` 是合法属性名形态；
|
||||
`xai:joinAction`、`xlink:href`、`xmlns:xi` 等带前缀属性不校验）；
|
||||
- `refs.test.mjs`：GameObject → Draws → `xi:include` 实景结构解析类型为 null、
|
||||
元素被判定为外来命名空间，`href`/`xpointer` 不会进入未知属性分支；
|
||||
- 实机复验：整份 GameObject.xml 0 个 unknown-attribute 残留。
|
||||
|
||||
> **架构讨论（用户提议）**：把 XML 先“宏展开”成不含 `xi:include` 的版本再解析。
|
||||
> 这与 BAB 编译时的实际行为一致（`defaultscript.cs` 把整个 Mod 合并成一份大 XML），
|
||||
> 也是实现“GameObject 内模块 id 局部作用域解析”（第四轮遗留）的正确地基——展开后一个
|
||||
> GameObject 连同 include 进来的兄弟模块都在同一棵树里，`AttachModuleId` 等模块引用才能
|
||||
> 静态判定。设计备忘（现状 / 逻辑树方案 / 展开范围 / 落地点 / 检查清单）已整理在
|
||||
> `docs/plan.md` 第六节，等待确认后作为下一阶段实现。
|
||||
|
||||
---
|
||||
|
||||
## 十一、问题分析（第六轮，2026-08-01）：`Surfaces="` 未闭合引号导致枚举补全失效
|
||||
|
||||
### 现象
|
||||
|
||||
在 `<Locomotor ...>` 的起始标签里输入 `Surfaces="`（引号尚未闭合）时：
|
||||
|
||||
1. 光标处不出 `LocomotorSurfaceBitFlags` 的枚举补全（GROUND、WATER 等）；
|
||||
2. 整个文件高亮退化（XML 变成“不合法”的观感），直到补上第二个引号才恢复。
|
||||
|
||||
### 根因（两个独立缺陷叠加）
|
||||
|
||||
**A. 上下文分析不认未闭合的引号（`src/language/context.ts`）**
|
||||
|
||||
复现证据（编译产物直接执行）：
|
||||
|
||||
```
|
||||
闭合引号: <Locomotor id="x" Surfaces="GROUND">…
|
||||
ctx.kind = attribute-value, attr = Surfaces, valuePrefix = "GROUND"
|
||||
|
||||
未闭合引号: <Locomotor id="x" Surfaces="GROUND>…
|
||||
Surfaces: quoteStart=27, quoteEnd=-1 ← 解析器吞掉整个文件
|
||||
ctx.kind = attribute-name ← 补全走错分支
|
||||
```
|
||||
|
||||
`analyzeStartTag` 判断属性值上下文的条件是 `offset >= quoteStart && offset <= quoteEnd`，
|
||||
未闭合时 `quoteEnd = -1` 永远不成立，于是回退成 attribute-name。
|
||||
|
||||
**B. 模型生成器不支持 `xs:list`（`tools/xsd-to-model.mjs`）**
|
||||
|
||||
XSD 中 `LocomotorSurfaceBitFlags` 是：
|
||||
|
||||
```xml
|
||||
<xs:simpleType name="LocomotorSurfaceBitFlags">
|
||||
<xs:list itemType="Surface"></xs:list>
|
||||
</xs:simpleType>
|
||||
```
|
||||
|
||||
而 `Surface` 才是真正带 11 个枚举值（GROUND、WATER、CLIFF、AIR…）的类型。生成器
|
||||
只读取 `restriction.enumeration`，list 层把枚举全部丢掉——所以**即使引号闭合，模型里
|
||||
该属性也没有任何候选值**。影响面：SDK XSD 共 79 个 `xs:list` 简单类型、317 处属性
|
||||
声明使用它们（`KindOfBitFlags`、`ObjectStatusBitFlags`、`WeaponFlagsBitFlags`、
|
||||
`ModelConditionBitFlags`、`BuildPlacementTypeBitFlags` 等），展开到继承后的模型条目
|
||||
共 890 个属性受影响。
|
||||
|
||||
### 关于高亮丢失
|
||||
|
||||
这是 TextMate XML 语法对“未闭合字符串”的正常行为：后续内容被当作字符串吞掉，直到
|
||||
遇到下一个引号或 EOF。与插件注入语法无关（注入部分只有 `$DEFINE`、`inheritFrom` 等
|
||||
少量规则），**即使改成 LSP 也不会自动消失**。真正的解法是语义 token
|
||||
（`DocumentSemanticTokensProvider`），本次未实施，列为可选后续。
|
||||
|
||||
### 修复（第 1–4 项）
|
||||
|
||||
1. **解析器行尾恢复**（`src/language/xmlParser.ts`）：起始标签扫描到 EOF 且引号仍未
|
||||
闭合时，把标签在第一个换行处截断并继续解析主循环。未闭合引号只影响当前行，后面
|
||||
的元素照常进入解析树，补全 / hover / 诊断不中断；解析错误仍照常上报
|
||||
（`Unterminated start tag`）。
|
||||
2. **未闭合引号上下文**（`src/language/context.ts`）：`quoteEnd < 0` 时，
|
||||
`offset >= quoteStart` 即视为 attribute-value 上下文，`valuePrefix` 照常取引号后
|
||||
到光标处文本。
|
||||
3. **模型支持 `xs:list`**（`tools/xsd-to-model.mjs`）：`resolveTypeDescriptor` 解析
|
||||
`xs:list` 的 `itemType`（属性形式或内联 simpleType），继承其枚举值 / refType /
|
||||
isRef / allowsDefine，新增 `isList` 标记；重新生成 `schema-model.json`
|
||||
（`LocomotorSurfaceBitFlags` 恢复 11 个枚举值，`ModelConditionBitFlags` 457 个
|
||||
值与 XSD 一致）。`AttributeInfo` / `SimpleTypeInfo` 接口同步新增 `isList`。
|
||||
4. **多值补全按“最后一段”过滤**（`src/features/completion.ts` +
|
||||
`context.splitListValuePrefix`）：list 属性只取当前空格段做前缀过滤，替换范围只
|
||||
覆盖该段——`Surfaces="GROUND ` 之后输入 `W` 也能提示 WATER / WALL_RAILING，而不是
|
||||
用整段前缀匹配失败。
|
||||
|
||||
### 验证
|
||||
|
||||
- 未闭合引号复现场景：仅报 1 条 `Unterminated start tag`，`<Other/>` 等后续元素仍被
|
||||
解析；`ctx.kind = attribute-value`、`attr = Surfaces`、`valuePrefix = "GROUND"`。
|
||||
- 模拟补全过滤：前缀 `G` → `GROUND`；前缀 `W` → `WATER, WALL_RAILING`；
|
||||
`GROUND ` 后输入 `W` → `WATER, WALL_RAILING`。
|
||||
- `GameObject@KindOf`：`isList=true`、284 个枚举值。
|
||||
- 单元测试 **39 → 50 全部通过**（新增 `test/context.test.mjs` 与带 vscode stub 的
|
||||
`test/completion.test.mjs` 集成用例；xmlParser 新增未闭合引号恢复 / EOF 用例；
|
||||
schemaModel 新增 list 枚举与 `isList` 用例）。
|
||||
- `tsc` / `esbuild` 构建通过。
|
||||
|
||||
> 补充：真实文件中该场景的元素名是 `<LocomotorTemplate ...>`（AttachTest
|
||||
> `Locomotor.xml` 实测，`Surfaces="GROUND CRUSHABLE_OBSTACLE"` 正是多值 list）；
|
||||
> SDK XSD 中没有名为 `Locomotor` 的元素，补全集成测试按真实写法夹具。
|
||||
|
||||
### 后续可做（未列入本次）
|
||||
|
||||
- `AssetIdList` 等“任意资产 ID 列表”的引用语义建模（list 补全框架已就绪，但这类
|
||||
属性在 XSD 里没有 refType，需要另行定义过滤规则）。
|
||||
|
||||
> 语义 token 兜底高亮已在第七轮实现（见下节）。
|
||||
|
||||
---
|
||||
|
||||
## 十二、问题分析（第七轮，2026-08-01）：语义 token 兜底高亮
|
||||
|
||||
### 目标
|
||||
|
||||
未闭合引号期间 TextMate 把后续内容当字符串吞掉、整个文件高亮退化，这是 XML 语法
|
||||
固有的行为（任何 XML 编辑器皆然），也无法靠注入 grammar 修复。本轮的解法是语义
|
||||
token：文档出现解析错误时，由插件用自己的容错解析树继续给标签 / 属性 / 值着色。
|
||||
|
||||
### 设计
|
||||
|
||||
- **纯 TS 核心**（`src/language/semanticTokens.ts`）：`buildSemanticTokenRanges(doc, text)`
|
||||
把解析树转换成按位置排序的 `{ line, startChar, length, tokenType }`；token 类型只用
|
||||
标准 `type` / `property` / `string`，所有主题自带配色，无需额外贡献样式。
|
||||
- 元素名：起始标签与闭合标签各一个 `type` token；
|
||||
- 属性名：`property` token；
|
||||
- 属性值：`string` token，闭合时含两端引号，未闭合时从开引号到行尾恢复点
|
||||
（如 `"GROUND>`）。
|
||||
- **provider**（`src/features/semanticTokens.ts`）：`parseXml` 后若
|
||||
`doc.errors.length === 0` 直接返回空——合法文件观感与纯 TextMate 完全一致；
|
||||
有解析错误时才用 `SemanticTokensBuilder` 编码输出。
|
||||
- **注册**：`extension.ts` 对 `xml` 语言注册 `DocumentSemanticTokensProvider`，
|
||||
legend 与 provider 共用同一实例。
|
||||
|
||||
### 验证
|
||||
|
||||
- malformed（`Surfaces="GROUND>` 未闭合）：`AssetDeclaration` / `LocomotorTemplate` /
|
||||
`<Other/>` 标签名、`id` / `Surfaces` 属性名、`"x"` 与 `"GROUND>` 值均有 token，
|
||||
且按位置升序排列；
|
||||
- 合法文档：返回空 token 数组；
|
||||
- 单元测试 **50 → 53 全部通过**（新增 `test/semanticTokens.test.mjs`，纯函数 + vscode
|
||||
stub 的 provider 集成）；`tsc` 通过。
|
||||
|
||||
### 边界与取舍
|
||||
|
||||
- 语义 token 只在解析报错时启用，且使用主题对 `type` / `property` / `string` 的默认
|
||||
配色，可能与 TextMate XML 配色略有差异——只在打字过程中出现，可接受；
|
||||
- 未实现 `provideDocumentSemanticTokensEdits`（delta 版本），每次全量计算；单文件
|
||||
解析在 KB 级，开销可忽略。
|
||||
|
||||
---
|
||||
|
||||
## 十三、问题分析（第八轮，2026-08-02）：`.w3x` 美术资产未被索引（AUGunship_SKN）
|
||||
|
||||
### 问题
|
||||
|
||||
AttachTest `Harbinger Gunship\GameObject.xml` 中 `<Model Name="AUGunship_SKN"/>`
|
||||
报两条错误：
|
||||
|
||||
1. Problems 面板诊断：`Unresolved reference "AUGunship_SKN" (not found in the current index)`
|
||||
（`unresolved-reference`，来自 `features/diagnostics.ts`）；
|
||||
2. 悬停提示：`No matching definition of type BaseRenderAssetType in the current index
|
||||
(may exist in a compiled manifest or vanilla data).`（来自 `features/hover.ts`）。
|
||||
|
||||
两条是同一缺失定义的两种呈现（诊断 + hover），不是两个独立 bug。
|
||||
`W3DContainer:AUGunship_SKN` 确实由 mod 定义——但定义在
|
||||
`Harbinger Gunship\W3X\AUGUNSHIP_SKN.w3x` 里，而索引器只解析 `.xml` / `.manifestxml`。
|
||||
|
||||
### 关键事实（实测）
|
||||
|
||||
1. **`.w3x` 是文本 XML**：文件头即 `<?xml ...?>` + `<AssetDeclaration>`，内容为
|
||||
`<W3DContainer id="AUGUNSHIP_SKN" Hierarchy="AUGUNSHIP_SKL">` + `<SubObject>` 子树。
|
||||
附带 UTF-8 BOM（抽样 292 个 Corona w3x：0 个 UTF-16，40 个带 BOM）。
|
||||
2. **w3x 通过 `<Include type="all">` 链进索引**：`Mod.xml → … → W3X.xml → W3X/*.w3x`，
|
||||
也常用 `ART:xxx.w3x`（SDK 根、项目 `Art` 等搜索路径）。索引器此前只把 w3x 登记为
|
||||
文件（`stream.files`），从不解析，因此其顶层资产不在 `assetsById` 中。
|
||||
3. **类型匹配本来是对的**：`Model@Name` 的 refType 是 `BaseRenderAssetType`，
|
||||
`W3DContainer → BaseRenderAssetType` 可赋值（`isAssignableTo` = true）。缺的只是定义。
|
||||
4. **manifest / SageXml 支持早已存在**（第二轮/第三轮），但该资产是 mod 自己的 w3x，
|
||||
不在 `builtmods/*.manifest` 也不在 SageXml——hover 的 "may exist in a compiled manifest
|
||||
or vanilla data" 只是通用兜底文案。
|
||||
|
||||
### 规模调查（Corona，D: 盘已连接）
|
||||
|
||||
- **3788 个 w3x，共 2.64 GB**；最大 22.8 MB；163 个超过原 4 MB 解析上限，11 个超 10 MB。
|
||||
- 大文件结构符合"建模软件导出"模式：顶层是少量固定 W3D 资产
|
||||
（`W3DHierarchy` + 若干 `W3DMesh` / `W3DContainer` / `W3DCollisionBox`，
|
||||
有的还带 `<Includes>` 引用 `ART:*.xml`）；体积大头是 `W3DMesh` 内的
|
||||
`Vertices/V`、`Normals/N`、`TexCoords/T`、`Triangles/T` 等 `maxOccurs="unbounded"`
|
||||
数值元素。例如 21.7 MB 的 `CBRefinery_BLD.W3X` 只有 22 个顶层记录，
|
||||
Vertices+Triangles 块占约 12 MB（55%）。
|
||||
- 全量建 DOM 的代价（实测 6.3 MB Aegis 文件）：407 ms、193,651 个元素、
|
||||
**堆内存 +109 MB（约 17 倍文本体积）**；22 MB 文件外推约 1.5 s + ~380 MB/份。
|
||||
浅扫描（只取顶层记录）：6.3 MB ~200 ms、22 MB ~600 ms，保留内存近似为零。
|
||||
- **读整个文件不可避免，但建树不是**：浅扫描仍然是线性扫描全文（要知道顶层元素边界
|
||||
就必须扫完），只是不分配子节点对象——所以"事后优化"完全有意义，且是必要项。
|
||||
|
||||
### 修复
|
||||
|
||||
1. **新增浅扫描模块** `src/indexer/shallowScan.ts`（纯 TS）：
|
||||
单次线性扫描，只提取顶层元素 `name + id`（含精确 offset）、顶层 `<Includes>`
|
||||
的 `Include@type/source`、任意层级 `<xi:include>` 的 `href/xpointer`、`<Defines>`
|
||||
常量；注释 / CDATA / DOCTYPE / PI 整体跳过；属性解析兼容引号内 `>` 与 `/`。
|
||||
2. **索引模式三分**：`.xml`/`.manifestxml` 全量解析（4 MB 上限不变）；
|
||||
`.w3x` 一律浅扫描；未知扩展名先嗅探文件头（512 字节，BOM/空白后以 `<` 开头且无
|
||||
NUL 字节 → 按 XML 浅扫描，否则按二进制仅登记）。w3x 自身的 `<Includes>` 与
|
||||
嵌套 `xi:include` 会继续被 walk（BAB 语义）。
|
||||
3. **持久缓存**：`DocumentCache` / `ShallowScanCache` 移到 `src/indexer/caches.ts`，
|
||||
由 `ModWorkspace` 持有并传入每次新建的 `ModIndexer`；读取时按 `mtimeMs + size`
|
||||
校验，未变化不重读。这是 w3x 方案在 Corona 上可用的前提（否则每次保存都重读 2.6 GB）。
|
||||
4. **BOM 剥离**：`xmlParser` 新增 `stripBom()`，全量解析与浅扫描前统一剥离，
|
||||
保证第一行偏移与编辑器一致。
|
||||
5. **Include source 补全候选**：DATA 目录与项目相对路径候选加入 `.w3x`
|
||||
（`fileScanner`），`W3X/xxx.w3x`、`ART/xxx.w3x` 可补全。
|
||||
6. 索引报告/状态栏新增 `shallowScannedFiles` / `shallowCacheHits` 统计。
|
||||
|
||||
### 验证
|
||||
|
||||
- 单元测试 **53 → 63 全绿**：新增 `test/shallowScan.test.mjs`（顶层资产 / Includes /
|
||||
xi:include / Defines / 引号内 `>` / CDATA / 未闭合标签 / 数值载荷不产生记录）；
|
||||
indexer 新增 w3x 链、`.w3d` 嗅探、二进制 `.dds` 跳过、跨重建缓存命中、
|
||||
BOM 偏移断言；xmlParser 新增 `stripBom` 用例。
|
||||
- AttachTest 实机：
|
||||
- 首次构建 1.2 s，浅扫 62 个文件；`Model@Name=AUGunship_SKN` →
|
||||
`W3DContainer @ …\W3X\AUGUNSHIP_SKN.w3x:3`；`Hierarchy=AUGunship_SKL`、
|
||||
`AUGunship_FP` 同样解析；资产数 35,546 → 35,607。
|
||||
- 第二次构建 354 ms，0 次重扫，62 次缓存命中。
|
||||
- Corona 实机（D: 盘）：
|
||||
- 首次构建 241 s：8,976 个文件、**浅扫 4,829 个**、资产 64,868
|
||||
（manifest 35,322）、3 个流、183 个 Define。
|
||||
- 第二次构建 38 s：0 次重扫、4,829 次缓存命中、资产数一致。
|
||||
- `CBREFINERY_BLD` 正确解析到 `W3DHierarchy @ cb/CBRefinery_BLD.w3x:16` 与
|
||||
`W3DContainer @ …:776412`。
|
||||
|
||||
### 边界与后续
|
||||
|
||||
- 首次建索引较慢（Corona ~4 分钟，机械盘）：读 2.6 GB 是下限，浅扫描本身 ~30-90 s；
|
||||
后续可做并行扫描或把 w3x 顶层记录做成独立小缓存文件。
|
||||
- 第二次构建仍有 ~38 s（主要是对 ~9k 文件逐文件 stat + 重建 Map，机械盘随机读）；
|
||||
后续可做"文件清单 + stat 快照"级缓存。
|
||||
- 浅扫描对 w3x 不做 XSD 校验（编辑器特性不注册 w3x 语言）；若用户手动把 `*.w3x`
|
||||
关联为 xml，完整解析仍会发生在打开的文档上（VS Code 自身行为）。
|
||||
- UTF-16 XML 的 w3x 会被嗅探判定为二进制（文件头含 NUL）；实测生态中不存在，
|
||||
如遇可再扩展解码。
|
||||
|
||||
---
|
||||
|
||||
## 十四、问题分析（第九轮，2026-08-02）：Corona 重建性能与内存
|
||||
|
||||
### 目标与基线
|
||||
|
||||
第八轮后 Corona 的实测基线：
|
||||
|
||||
| 场景 | 耗时 |
|
||||
|---|---|
|
||||
| 首次全量建索引 | ~180-290s（机械盘波动） |
|
||||
| 信任二次构建（保存触发） | 8-21s |
|
||||
| 强制 reindex | 26-38s |
|
||||
| 首建后保留堆 | ~2.5GB（原因不明） |
|
||||
|
||||
### 调查 1：二次构建为何仍有 8-21s
|
||||
|
||||
插桩 `fs.statSync` 后发现：**每次构建（含信任重建）都会执行约 11 万次同步 statSync**，
|
||||
全部来自 `resolveSource` 的 include 目标存在性检查（BAB 搜索路径逐 base `statSync`）。
|
||||
机械盘上这就是 10-20s 的来源——即使文件内容全部命中缓存，include 解析仍在逐路径 stat。
|
||||
|
||||
修复：新增 `IncludeResolveCache`（workspace 级、跨重建复用）：
|
||||
- 键 = 当前文件目录 + source 字符串；命中直接返回，不 stat；
|
||||
- 内容编辑不影响"文件是否存在"，因此保存触发的重建**不清除**解析缓存；
|
||||
- 文件创建/删除（watcher `onDidCreate` / `onDidDelete`）与 `ra3modxml.reindex`
|
||||
（强制校验）清除缓存；
|
||||
- `reference` 的 manifest 查找（`builtmods/*.manifest` 存在性）同样缓存；
|
||||
- 配置变更（搜索路径 / builtmods 目录）时清除。
|
||||
|
||||
### 调查 2：首建后保留堆 2.5GB 是什么
|
||||
|
||||
逐级清空测量（构建 → 清 DOM 缓存 → 清 records → 清 walker → 清索引 Map）：
|
||||
|
||||
- `DocumentCache`（元素预算 1M、64 条）：实际只保留 64 条 / 1788 个元素 ≈ 1MB；
|
||||
- `IndexRecordsCache`（8976 个文件的紧凑记录）：约 11MB；
|
||||
- 目录 walker：约 4MB；
|
||||
- 全部索引 Map（manifests + assets + assetsById + defines + files + streams +
|
||||
candidates + diagnostics）：约 75MB；
|
||||
- 全部清空并强制 GC 后，堆回到基线（0MB 保留）。
|
||||
|
||||
结论：2.5GB 是**构建期可回收垃圾**（60MB XML 的 DOM 瞬态 + 2.6GB w3x 扫描文本/行映射
|
||||
的分配压力），不是常驻泄漏；VSCode 正常 GC 压力下会回收。扩展常驻内存约为
|
||||
基线 + ~100MB。顺带修复了一个潜在常驻风险：w3x 的 `LineMap`（Corona 全量约 700MB）
|
||||
不再随 `ShallowScanCache` 保留，浅扫描结果以"带行号的紧凑记录"形式缓存。
|
||||
|
||||
### 其他改动
|
||||
|
||||
1. **`IndexRecordsCache` 取代 DOM 依赖的重建**：新增 `src/indexer/records.ts`，
|
||||
每个文件解析时提取紧凑索引记录（顶层资产 / Define / Include / xi:include +
|
||||
1-based 行号），跨重建缓存；信任重建完全不接触 DOM。
|
||||
2. **`DocumentCache` 双重淘汰**：条数（LRU）+ 元素预算（超预算先淘汰最大树），
|
||||
把 DOM 常驻内存封顶；DOM 只服务于按需特性（跳转精确范围等）。
|
||||
3. **候选目录扫描并行化**（`collectSourceCandidates` 各目录 `Promise.all`）。
|
||||
4. **阶段计时**：`stats.candidatesMs` / `stats.walkMs` / `stats.resolveCalls` /
|
||||
`stats.resolveCacheHits` 进入索引报告，便于后续定位耗时。
|
||||
5. 版本 0.1.0 → **0.1.1**。
|
||||
|
||||
### 实测（Corona，D: 机械盘）
|
||||
|
||||
| 场景 | 优化前 | 优化后 |
|
||||
|---|---|---|
|
||||
| 首次全量建索引 | ~180-290s | ~250s（含 2.6GB w3x 读取+浅扫描、~7.6 万次冷 statSync） |
|
||||
| 信任二次构建 | 8-21s | **2.0s**（statSync 0 次，resolveHits 15,333） |
|
||||
| 强制 reindex | 26-38s | ~5s（保留解析缓存时）；显式命令会清解析缓存，约 15-25s |
|
||||
| 首建后保留堆 | ~2.5GB（疑为泄漏） | 确认是可回收垃圾；常驻 ~100MB |
|
||||
|
||||
单元测试 **63 → 73 全绿**：新增 `records.test.mjs`（DOM→记录提取、浅扫描→记录）、
|
||||
`caches.test.mjs`（元素预算淘汰、LRU、records/resolve 缓存），indexer 测试补充
|
||||
resolve 缓存命中与"信任重建 0 次重解析"断言。
|
||||
|
||||
### 遗留
|
||||
|
||||
- 首次全量建索引仍受限于机械盘读 2.6GB + 冷 statSync（~4 分钟）；后续可考虑
|
||||
并行浅扫描（worker）或把 w3x 顶层记录持久化到磁盘缓存。
|
||||
- `ra3modxml.reindex` 出于正确性会清空解析缓存（可能 ~15-25s）；如接受 watcher
|
||||
可靠性可改为保留。
|
||||
|
||||
+158
-17
@@ -1,6 +1,6 @@
|
||||
# 调研结论与实施计划（已按最新代码同步更新）
|
||||
|
||||
> 说明：本文档随实现演进持续同步。最近一次同步（2026-08-01）对齐了实现过程中新增的模块与设计变更：BAB 精确搜索路径、manifest 类型/ID 推导、上下文感知元素类型、无类型引用、精确跳转范围、嵌套 `xi:include`、注入式语法高亮等。
|
||||
> 说明：本文档随实现演进持续同步。最近一次同步（2026-08-01）对齐了实现过程中新增的模块与设计变更：BAB 精确搜索路径、manifest 类型/ID 推导、上下文感知元素类型、属性级 refType / Poid 局部引用（`id` 定义点）、精确跳转范围、嵌套 `xi:include`、注入式语法高亮等。
|
||||
|
||||
## 一、调研结论（带证据）
|
||||
|
||||
@@ -26,7 +26,8 @@
|
||||
- 根元素 `AssetDeclaration`：`Tags` / `Includes` / `Defines` + **295 个顶层资产元素**（含内联声明）。
|
||||
- 每个元素名对应一个 `complexType`，子元素用 `xs:sequence` / `xs:choice` 定义，属性用 `xs:attribute` 定义；复杂类型通过 `xs:extension` 继承（如 `BaseInheritableAsset` 提供 `inheritFrom`）。
|
||||
- `Includes/Ref.xsd` 定义了大量带 `xas:refType="<资产类型>"` 的引用类型（如 `CommandSet` 引用 `LogicCommandSet`）→ 补全/导航按引用类型过滤的依据。
|
||||
- `XmlEdit:Default` 提供默认值；`xs:enumeration` 提供枚举值；`xas:isRef="true"` 无 `refType` 时为**无类型引用**（匹配任意已声明资产）。
|
||||
- `XmlEdit:Default` 提供默认值；`xs:enumeration` 提供枚举值。`xas:refType` 可声明在 simple type 上，也可声明在 `<xs:attribute>` 节点上（模型生成器两者都读、属性级优先）。
|
||||
- `Poid`（"Pipeline Object Id"，`xas:isWeakRef="true"`）表示**管线局部标识**：`id` 属性定义元素自身（如 `ModuleData@id` → refType `ModuleData`）；`ModuleId`、`AutoResolveBody`、`SoundRef` 等 Poid 属性引用同一资产/子树内的模块、子对象、材质——它们都不对全局资产索引做 resolved 判定。
|
||||
- **同名元素在不同父节点下类型不同**（如 `<Weapon>` 在武器槽下是 `WeaponSlot_WeaponData`，在别处可能是 `WeaponRef`）→ 需要上下文感知解析。
|
||||
|
||||
### 4. 领域特有约定
|
||||
@@ -60,7 +61,7 @@
|
||||
- **核心与编辑器解耦**：`language/`、`model/`、`indexer/` 为纯 TS 模块（不 import `vscode`），可单测与复用（呼应 P1 需求 7）。
|
||||
- **esbuild 打包**，产物 `dist/extension.js`。
|
||||
- **XSD → JSON 模型**：开发期工具 `tools/xsd-to-model.mjs` 把 821 个 XSD 解析成 `schema-model.json`（元素树、属性、文档、枚举、引用类型映射），随插件发布；运行时不再解析 XSD。
|
||||
- **运行时 XML 解析**：自研带源码偏移的轻量解析器 `language/xmlParser.ts`（标签/属性/值均记录起止偏移，容错解析以支持输入中的补全与诊断）。`fast-xml-parser` 仅用于开发期 XSD 生成。
|
||||
- **运行时 XML 解析**：自研带源码偏移的轻量解析器 `language/xmlParser.ts`（标签/属性/值均记录起止偏移，容错解析以支持输入中的补全与诊断；未闭合引号在行尾恢复，避免吞掉整个文档）。`fast-xml-parser` 仅用于开发期 XSD 生成。
|
||||
- **AssetType 哈希表**：`tools/extract-asset-types.mjs` 从 OpenSAGE `AssetType.cs` 提取 `asset-types.json`；哈希未知时以 manifest 名称前缀推导类型。
|
||||
|
||||
### 架构
|
||||
@@ -74,6 +75,7 @@ src/
|
||||
xmlParser.ts 带源码偏移的轻量 XML 解析器（格式错误定位、容错）
|
||||
context.ts 补全上下文分析（元素名/属性名/属性值/内容）
|
||||
typeContext.ts 上下文感知元素类型解析（resolveElementType 沿解析树逐层解析）
|
||||
semanticTokens.ts 语义 token 兜底高亮（纯 TS：标签/属性/值范围，仅 malformed 时启用）
|
||||
model/
|
||||
schemaModel.ts schema-model.json 的类型/属性/子元素查询 + 类型名规范化（纯 TS）
|
||||
schema-model.json 由 tools/xsd-to-model.mjs 生成
|
||||
@@ -83,36 +85,85 @@ src/
|
||||
manifestParser.ts .manifest 二进制解析 + 类型/ID 推导（纯 TS）
|
||||
fileScanner.ts 目录遍历缓存 + Include source 候选收集
|
||||
refs.ts 引用目标解析（按 refType / isRef / inheritFrom 过滤，纯 TS）
|
||||
indexer.ts 工作区索引器（资产/Define/流/manifest 合并，LRU 解析缓存）
|
||||
shallowScan.ts 大体积美术资产（.w3x 等）顶层浅扫描（纯 TS，不建 DOM）
|
||||
records.ts 每文件紧凑索引记录（资产/Define/Include/xi + 行号）
|
||||
caches.ts 跨重建持久缓存（DocumentCache / IndexRecordsCache /
|
||||
IncludeResolveCache）
|
||||
indexer.ts 工作区索引器（资产/Define/流/manifest/w3x 合并）
|
||||
types.ts 共享类型
|
||||
features/
|
||||
completion.ts 补全 provider（元素/属性/值，上下文感知）
|
||||
completion.ts 补全 provider（元素/属性/值，上下文感知；xs:list 多值按当前段过滤）
|
||||
hover.ts hover provider
|
||||
navigation.ts 定义/引用/文档链接/大纲
|
||||
diagnostics.ts 实时诊断
|
||||
semanticTokens.ts 语义 token provider（文档有解析错误时接管着色）
|
||||
syntaxes/
|
||||
ra3modxml.tmLanguage.json 注入 source.xml 的领域高亮（纯注入，不替换 XML 主语法）
|
||||
tools/
|
||||
xsd-to-model.mjs XSD → schema-model.json
|
||||
xsd-to-model.mjs XSD → schema-model.json（含 xs:list：继承 item 枚举/引用语义，isList 标记）
|
||||
extract-asset-types.mjs OpenSAGE AssetType.cs → asset-types.json
|
||||
test/
|
||||
fixtures/minimod 样例 Mod（include 各种情形、同名 ID、嵌套 xi:include、manifest 回退）
|
||||
*.test.mjs 8 个测试文件（xmlParser / includeResolver / manifestParser /
|
||||
indexer / schemaModel / refs / typeContext / manifestTypes）
|
||||
*.test.mjs 11 个测试文件（xmlParser / context / completion / semanticTokens /
|
||||
includeResolver / manifestParser / indexer / schemaModel / refs /
|
||||
typeContext / manifestTypes）
|
||||
```
|
||||
|
||||
### 关键设计决策
|
||||
|
||||
1. **语言激活范围**：不劫持 `*.xml`。通过 `workspaceContains:**/Data/Mod.xml`、`**/*.babproj` 激活；语法高亮为**纯注入** grammar（不声明 `language`，避免覆盖内置 XML 语法）。
|
||||
2. **索引范围与默认值**：索引“项目 Data + additionalmaps + 沿 include 可达的 SageXml 原版源码”；SDK 路径默认 `C:\Apps\RA3-MODSDK-X`（可配置）。`reference` include 解析为 `builtmods` 下对应 manifest（惰性解析、按文件缓存），manifest 缺失/无效时回退到占位 XML。
|
||||
**美术资产（.w3x）**：`<Include type="all">` / `ART:` 指向的 `.w3x`（及内容嗅探为
|
||||
XML 的未知扩展名文件）按其顶层资产入库（`W3DContainer` / `W3DMesh` /
|
||||
`W3DHierarchy` / `W3DCollisionBox` 等），使 `Model@Name`、`Hierarchy`、`Mesh`
|
||||
等引用可解析；大模型文件**浅扫描**（不建 DOM），结果缓存在 workspace 级、
|
||||
跨重建复用（详见设计决策 14）。
|
||||
3. **manifest 资产建模**：类型优先用哈希表，未知时从名称前缀推导；可引用 ID 取最后冒号段；类型名统一走大小写规范化（`W3dContainer` ↔ `W3DContainer`），类型匹配严格遵循 XSD 继承链。
|
||||
4. **上下文感知元素类型**：同名元素按父元素类型解析（`resolveElementType` 沿解析树逐层 `childTypeOf`，失败回退全局映射），保证 `<Weapon>` 等元素的属性/引用判定正确。
|
||||
5. **引用判定与解析**：`refType` 或 `isRef` 均视为引用；带 `refType` 时严格按类型过滤（同名 ID 不串类型）；无类型引用匹配任意声明 ID；`inheritFrom` 按可继承类型过滤。
|
||||
5. **引用判定与解析**：`refType` 或 `isRef` 均视为引用；带 `refType` 时严格按类型过滤（同名 ID 不串类型）；`inheritFrom` 按可继承类型过滤。**局部作用域例外**（`isLocalReferenceAttribute`）：`id` 是元素自身的定义点——无 refType 或 refType 与自身类型兼容时不检查、不解析（`RoadObject@id→Road` 这类跨类型 id 引用保留检查）；Poid 类型属性是管线局部引用，全局索引无法判定，不检查、不解析。
|
||||
6. **重复 ID 诊断**：与 `check_duplicate_ids.py` 一致——SageXml 不参与冲突判定，mod 覆盖原版视为正常。
|
||||
7. **未解析引用诊断**：按设置严重级别报告（默认 warning）；类型不匹配时给出明确文案（"有同名 ID 但类型不匹配"）。`definitionMode` 设置控制跳转候选：`all`（mod + 原版全部列出，mod 优先）或 `project-only`。
|
||||
8. **跳转精度**：XML 定义跳转到 `id` 属性值的精确 Range；manifest 定义映射到源码文件（如 SageXml）时也在文件内精确定位；找不到再回退到记录行。
|
||||
9. **嵌套 `xi:include`**：任意层级处理——目标缺失产生诊断、目标存在则纳入索引；根级 `xpointer` 容器内容按顶层资产索引。
|
||||
10. **性能**：索引在后台执行；解析结果 LRU 缓存（约 64 个文档）；文件保存后防抖全量重建（1.5s），重建期间的新请求标记脏并在完成后重跑；状态栏显示进度与统计。
|
||||
11. **`xs:list` 建模与多值补全**：list 简单类型继承 itemType 的枚举 / refType / isRef /
|
||||
allowsDefine 并标记 `isList`（`LocomotorSurfaceBitFlags`、`KindOfBitFlags` 等 79 个
|
||||
类型、317 处属性声明受益）；补全只对“最后一个空格段”过滤，替换范围只覆盖当前段，
|
||||
支持 `Surfaces="GROUND ` 之后继续输入 `W` 提示 `WATER`。
|
||||
12. **未闭合引号的行尾恢复**：起始标签扫描到 EOF 且引号未闭合时，在第一个换行处截断
|
||||
标签并继续解析，未闭合只影响当前行（仍上报 `Unterminated start tag`），后续元素
|
||||
的补全 / hover / 诊断不中断。
|
||||
13. **语义 token 兜底高亮**：TextMate 对未闭合引号会把后续内容当字符串吞掉（任何
|
||||
XML 编辑器皆然）；扩展注册 `DocumentSemanticTokensProvider`，仅当解析报错时用
|
||||
语义 token 覆盖标签名 / 属性名 / 属性值（标准 token 类型 `type` / `property` /
|
||||
`string`，主题自带配色）。合法文件返回空，观感与纯 TextMate 完全一致。
|
||||
14. **大体积美术资产浅扫描 + 跨重建持久缓存**（第八轮，2026-08-02）：
|
||||
- 实测 Corona：3788 个 w3x / 2.64 GB，163 个超过 4 MB，最大 22.8 MB；大文件是
|
||||
`W3DHierarchy` + 若干 `W3DMesh`（`Vertices/V`、`Triangles/T` 等 unbounded 数值
|
||||
载荷），顶层记录通常只有几个到二十几个。
|
||||
- 全量解析内存放大约 17 倍（6.3 MB 文本 → +109 MB DOM），不可接受；`scanXmlShallow`
|
||||
单次线性扫描只提取顶层 `name+id`、`<Includes>`、`<xi:include>`、`<Defines>`，
|
||||
22 MB 文件 ~600 ms、保留内存≈0。
|
||||
- 索引按扩展名三分：`.xml`/`.manifestxml` 全量解析（4 MB 上限不变）；`.w3x` 浅扫描；
|
||||
未知扩展名嗅探文件头（`<` 开头、无 NUL）决定按 XML 浅扫描或二进制登记。
|
||||
- `DocumentCache` / `ShallowScanCache` 由 `ModWorkspace` 持有，每次重建传入新的
|
||||
`ModIndexer`；按 `mtimeMs + size` 校验，未变化不重读。Corona 第二次构建
|
||||
w3x 重扫数为 0（4,829 次缓存命中）。
|
||||
- 读取整个文件不可避免（顶层边界需要全量扫描），但建 DOM 不是；优化的是
|
||||
"不分配子节点对象"与"跨重建不重读"，两者叠加后方案可行。
|
||||
15. **重建零 stat + 记录驱动索引**（第九轮，2026-08-02，v0.1.1）：
|
||||
- 插桩发现每次重建（含信任重建）都有约 11 万次同步 `statSync`
|
||||
（`resolveSource` 的 include 存在性检查），机械盘上占 10-20s；
|
||||
`IncludeResolveCache` 按（目录 + source）缓存解析结果，内容编辑不清、
|
||||
创建/删除文件与强制 reindex 才清。信任重建 statSync 降到 0。
|
||||
- `IndexRecordsCache` 缓存每文件紧凑索引记录（顶层资产 / Define / Include /
|
||||
xi:include + 1-based 行号），信任重建完全不接触 DOM；`DocumentCache`
|
||||
双重淘汰（条数 LRU + 元素预算，超预算先淘汰最大树）把 DOM 常驻内存封顶。
|
||||
- w3x 缓存不再保留 LineMap（Corona 全量约 700MB），浅扫描直接产出带行号的记录。
|
||||
- 实测：Corona 信任二次构建 21s → **2.0s**（statSync 0）；首建后 2.5GB
|
||||
堆保留确认为构建期可回收垃圾，常驻 ~100MB；强制 reindex ~5-25s。
|
||||
- 候选目录扫描并行化；`stats` 新增 `candidatesMs` / `walkMs` / `resolveCalls` /
|
||||
`resolveCacheHits` 供索引报告定位耗时。
|
||||
|
||||
## 三、实施步骤
|
||||
|
||||
@@ -121,24 +172,114 @@ test/
|
||||
3. [x] 生成模型：`schema-model.json`（XSD，295 顶层元素 / 1851 类型）与 `asset-types.json`（79 个 AssetType 哈希）。
|
||||
4. [x] 纯 TS 核心：include 解析、manifest 解析、XML 解析封装、索引器、引用解析。
|
||||
5. [x] 功能层：补全、hover、导航、诊断、大纲、高亮 grammar。
|
||||
6. [x] 单测（fixture Mod，31 个用例全绿）+ 编译 + `vsce package` 打包（ra3-mod-xml-0.1.0.vsix，约 259KB）。
|
||||
7. [x] 在 AttachTest / GenEvoTest / Corona 上做冒烟验证，并按真实项目反馈修复问题（详见 `docs/analysis-issues.md` 三轮分析）。
|
||||
6. [x] 单测（fixture Mod，73 个用例全绿：含 xs:list 枚举、未闭合引号恢复、list 多值
|
||||
分段、带 vscode stub 的补全集成、语义 token 兜底）+ 编译 + `vsce package` 打包
|
||||
（ra3-mod-xml-0.1.1.vsix，约 499KB）。
|
||||
7. [x] 在 AttachTest / GenEvoTest / Corona 上做冒烟验证，并按真实项目反馈修复问题（详见 `docs/analysis-issues.md` 八轮分析）。
|
||||
8. [x] w3x 美术资产索引（第八轮）：新增 `shallowScan.ts` / `caches.ts`，w3x 与内容嗅探
|
||||
XML 走浅扫描，Include source 补全候选加入 w3x，BOM 剥离，缓存跨重建持久化；
|
||||
AttachTest 报错场景（`AUGunship_SKN`）修复，Corona 首次/二次构建实测验证。
|
||||
9. [x] Corona 性能与内存优化（第九轮，v0.1.1）：`records.ts` 记录驱动索引、
|
||||
`IncludeResolveCache` 零 stat 重建、DOM 元素预算淘汰、w3x LineMap 移除、
|
||||
候选并行扫描、阶段计时；Corona 信任重建 2.0s；确认首建 2.5GB 为可回收垃圾。
|
||||
|
||||
## 四、验证结果（实测）
|
||||
|
||||
| 项目 | 规模 | 索引耗时 | 资产数 | 说明 |
|
||||
|---|---|---|---|---|
|
||||
| AttachTest | 71 文件 | ~0.6s | 35,546（manifest 35,322） | 2 个流（static + mapmetadata） |
|
||||
| GenEvoTest | 24 文件 | ~1.8s | 35,392（manifest 35,322） | 项目 ID（alliedmcv 等）正确收录 |
|
||||
| Corona | 3,448 文件（+ 非 XML 资产路径） | ~54.5s | 55,305（manifest 35,322） | 3 个流、183 个 Define、0 诊断 |
|
||||
| AttachTest | 88 文件 | ~1.2s | 35,607（manifest 35,322，w3x 浅扫 62） | 2 个流（static + mapmetadata）；二次构建 354ms / 62 缓存命中 |
|
||||
| GenEvoTest | 66 文件 | 首次 ~2.6s / 二次 ~0.4s | 35,502（manifest 35,322，w3x 浅扫 38） | 2 个流、73 个 Define；二次构建 0 重扫 / 38 缓存命中 |
|
||||
| Corona | 8,976 文件 | 首次 ~250s / 信任二次 ~2s / 强制 ~5-25s | 64,868（manifest 35,322，w3x 浅扫 4,829） | 3 个流、183 个 Define、0 诊断；二次构建 statSync 0、resolveHits 15,333 |
|
||||
|
||||
单元测试覆盖：XML 解析（自闭合/容错/偏移）、include 解析（BAB 顺序、SDK 根优先于 SageXml）、manifest 二进制解析（合成 v5 样本、类型/ID 推导）、索引器（资产/Define/流/缺失 include/嵌套 xi:include）、XSD 模型（上下文类型、`childTypeOf`、大小写规范化）、引用过滤（`Weapon="X"` 只跳 `WeaponTemplate`、无类型引用、`Side="Allies"` 命中 manifest 的 `PlayerTemplate`）。
|
||||
单元测试覆盖：XML 解析（自闭合/容错/偏移/未闭合引号行尾恢复）、补全上下文（未闭合引号仍为 attribute-value、list 多值分段）、补全集成（vscode stub 下 `LocomotorTemplate@Surfaces` 未闭合引号枚举补全、空格后第二段过滤与替换范围）、语义 token（标签/属性/值范围、合法文档返回空、malformed 返回兜底 token）、include 解析（BAB 顺序、SDK 根优先于 SageXml）、manifest 二进制解析（合成 v5 样本、类型/ID 推导）、索引器（资产/Define/流/缺失 include/嵌套 xi:include）、XSD 模型（上下文类型、`childTypeOf`、大小写规范化、属性级 refType、外来命名空间判定、`xs:list` 枚举继承与 `isList` 标记）、引用过滤（`Weapon="X"` 只跳 `WeaponTemplate`、模块 `id` 定义点、Poid 局部引用、`xi:include` 不校验、`Side="Allies"` 命中 manifest 的 `PlayerTemplate`）。
|
||||
|
||||
> 注：`D:\Mods\CoronaMod` 位于移动硬盘，当前未连接；GenEvoTest / Corona 的回归需在 D: 盘可用时补跑（用户会另行通知）。
|
||||
> 注：D: 盘移动硬盘已恢复连接；Corona 已在第八 / 九轮按上述新数据回归。
|
||||
|
||||
## 五、假设与开放问题
|
||||
|
||||
- 假设 SDK 路径默认 `C:\Apps\RA3-MODSDK-X`（与 prompts 一致），可在设置中修改。
|
||||
- 假设补全/导航以“文本语义分析”为主，不做完整 XSD 校验（BAB 才是权威校验器）。
|
||||
- 开放：是否发布到 VS Code Marketplace（需要 publisher）——本期先保证本地 `vsce package` 可安装。
|
||||
- 开放：嵌套 `xi:include` 内容的"虚拟合并"进父文档（用于父文档内的补全/诊断感知被内联内容）——当前仅保证目标文件可索引、可导航、缺失可诊断。
|
||||
- 开放：**“宏展开”式虚拟合并**（用户提议，方向已确认）：解析前先把 `xi:include`
|
||||
（以及顶层 `<Include type="all">`、`inheritFrom` 继承合并）展开成不含 include 的
|
||||
文档树，再对展开后的树做 XSD 校验、补全与诊断——与 BAB 编译时把整个 Mod 合并成
|
||||
一份大 XML 的行为一致。展开树需携带**来源追溯**（错误/跳转仍定位到原始文件），
|
||||
并处理 xpointer 子集解析与 include 循环。当前仅做到：目标文件可索引、可导航、
|
||||
缺失可诊断；`xi:include` 本身不参与 XSD 校验（第五轮）。
|
||||
该功能也是“GameObject 内模块 id 局部作用域解析”（第四轮遗留）的地基。
|
||||
详细设计备忘见下一节。
|
||||
|
||||
## 六、include 展开设计备忘（2026-08-01，待实施）
|
||||
|
||||
> 目的：集中记录 include 处理相关的现状、结论与设计，下次遇到 include 问题时从这里继续，
|
||||
> 并在实施后把结果回写本节。
|
||||
|
||||
### 1. 现状（截至第五轮，已实现）
|
||||
|
||||
| 能力 | 状态 |
|
||||
|---|---|
|
||||
| `<Include type="all">` / `instance` 递归索引（顶层资产、流、Define） | 已实现（indexer `walk`） |
|
||||
| `reference` → builtmods manifest 解析 / 缺失回退占位 XML | 已实现 |
|
||||
| 嵌套 `xi:include`（任意层级）：目标可索引、缺失报 `include-not-found`、Ctrl+点击跳转、`href` hover 解析目标 | 已实现（第二轮 + 第五轮） |
|
||||
| `xi:include` 及其属性不参与 XSD 校验（外来命名空间守卫 `isXsdElementName` / `isXsdAttributeName`） | 已实现（第五轮） |
|
||||
| include 目标内容“虚拟合并”进父文档的逻辑树 | **未实现**（本文档主题） |
|
||||
|
||||
### 2. 已确认的方向
|
||||
|
||||
先展开成不含 include 的文档树（类比 C++ 宏展开），再对展开后的树做 mod XML 解析。
|
||||
BAB（`defaultscript.cs`）编译时正是这样把整个 Mod 合并成一份大 XML 的。
|
||||
|
||||
### 3. 关键设计决策：逻辑树拼接，不做文本拼接
|
||||
|
||||
- **不要**把 include 目标展开成文本再整体重新解析：源码偏移会断裂，诊断 / 跳转 / hover /
|
||||
补全全部无法映射回原始文件。
|
||||
- **要做**的是：解析器逐文件解析（现状不变）；展开器把目标文件选中节点按 `xpointer`
|
||||
子集挂进父元素，节点保留各自的源文件与原始偏移（来源追溯）。后续分析跑在逻辑树上，
|
||||
范围映射按节点 `sourceFile` 回到对应文件的 lineMap。
|
||||
- 现有 `parseXml` 已记录标签 / 属性 / 值的起止偏移，`XmlElement` 结构可直接复用；拼接时
|
||||
用浅拷贝节点壳并重建 parent 链，避免破坏目标文件缓存树自身的 parent 指针。
|
||||
|
||||
### 4. 展开范围
|
||||
|
||||
| 构造 | 拼入逻辑树 | 理由 |
|
||||
|---|---|---|
|
||||
| `xi:include` | ✅ | 内容并入父元素（HeadlightDraw2 场景） |
|
||||
| EA `<Include type="all">` | ✅ | BAB“内容合并”，等价于复制进来 |
|
||||
| `type="instance"` | ❌ | 只影响编译可见性；拼树会把 BaseVehicle 的顶层资产错误塞进当前文档 |
|
||||
| `type="reference"` | ❌ | manifest 编译产物，无文本内容 |
|
||||
| `inheritFrom` + `xai:joinAction` | 单独一轮 | 元素级继承深合并（Replace/Remove），不是宏展开 |
|
||||
|
||||
### 5. 落地位置与接入点
|
||||
|
||||
- 新纯模块（与编辑器解耦，呼应 P1）：输入 `(parse 树, resolveSource 回调, readDocument 回调)`，
|
||||
输出逻辑树（root + elements 扁平列表，沿用 diagnostics 的遍历形态）。
|
||||
- 接入：diagnostics / hover / navigation / completion 目前各自 `parseXml(text)`；
|
||||
改为 parse 后过 expander 取逻辑树；范围映射按节点 `sourceFile` 选对应文件的 lineMap。
|
||||
- 按需展开（当前打开文档）+ 按文件缓存（复用 indexer `readDocument` 的 LRU）；
|
||||
环 / 深度守卫复用现有 `visitedAll` 与深度限制思路（建议最大深度 64）。
|
||||
- `xpointer`：仅支持 mod 实际使用的 `xmlns(n=...) xpointer(/n:Name/child::*)` 子集
|
||||
（现有 `findXPointerContainer` 正则已覆盖）；完整 XPath 暂不支持，遇到新形式先记录到本节。
|
||||
|
||||
### 6. 后续收益（承接第四 / 五轮遗留）
|
||||
|
||||
- **GameObject 内模块 id 局部作用域**：展开后 include 进来的兄弟模块（HeadlightDraw2）
|
||||
与本体模块同树，`AttachModuleId` / `ModuleId` / `AutoResolveBody` 等 Poid 引用
|
||||
才能静态解析与诊断（第四轮遗留）；
|
||||
- 跨 include 的上下文类型解析与结构校验；
|
||||
- 顶层 `type="all"` 合并后，当前文档视角的重复 id / 引用诊断更接近 BAB 结果。
|
||||
|
||||
### 7. 风险与边界
|
||||
|
||||
- 大文件性能（Corona 约 7500 文件 / 38MB）：只展开当前文档的可达链，不全局展开；
|
||||
- include 环 / 深度：visited 集合 + 最大深度；
|
||||
- 被包含内容的诊断上报位置：建议按节点源文件 URI 上报（与偏移一致），包含点的 `href`
|
||||
上只报“缺失 / 无法解析 / 环”类问题；
|
||||
- `inheritFrom` 深合并（`xai:joinAction` 的 Replace/Remove 语义）单独设计，别与宏展开混在一轮。
|
||||
|
||||
### 8. 下次遇到 include 问题的检查清单
|
||||
|
||||
1. 现象发生在哪一层：索引（indexer）、诊断（diagnostics）、导航 / hover、还是补全？
|
||||
2. 现状能力是否已覆盖（见第 1 节表格）；
|
||||
3. 涉及内容是否跨文件（需要逻辑树）还是本文件内（当前解析树即可）；
|
||||
4. 若要展开：先实现第 5 节的纯模块与单测（fixture 增加 HeadlightDraw2 场景），再接入 feature；
|
||||
5. 把新结论回写本节与 `docs/analysis-issues.md`。
|
||||
|
||||
@@ -70,6 +70,15 @@ XML 之间的组织靠 `<Include>` 标签，共有三种语义：
|
||||
- `TypeId` 哈希 → 类型名的映射来自 OpenSAGE 的 `AssetType` 枚举（本工作区可提取）；
|
||||
- 资产名与源文件名各自存放在独立的空字符结尾字符串缓冲区中。
|
||||
|
||||
**补充（美术资产 `.w3x` 索引）**：
|
||||
- `W3X.xml` / `ART:` include 链中的 `.w3x` 是文本 XML（建模工具导出，BAB 同样按
|
||||
XML 编译），其顶层资产（`W3DContainer` / `W3DMesh` / `W3DHierarchy` 等）应参与
|
||||
补全、悬停、导航与诊断——`Model@Name`、`Hierarchy`、`Mesh` 等引用依赖这些定义；
|
||||
- 大模型文件（实测 Corona 最大 22.8 MB，顶点/三角形数据占大头）采用**顶层浅扫描**
|
||||
（不建 DOM 树），结果在 workspace 级缓存并跨重建复用，避免每次保存都重读整个
|
||||
美术资产目录（Corona 约 2.6 GB）；索引记录与 include 解析结果同样跨重建缓存，
|
||||
保存触发的重建零 stat、零重读（Corona 实测约 2 秒）。
|
||||
|
||||
### P1：非近期目标（本期不做，但预留扩展点）
|
||||
|
||||
6. **高效搜索**：Mod 项目巨大（Corona 约 7500 个 XML、38MB）时直接全文搜索很慢，需要一个高效的 XML 内容索引机制。
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "ra3-mod-xml",
|
||||
"version": "0.1.0",
|
||||
"version": "0.1.1",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "ra3-mod-xml",
|
||||
"version": "0.1.0",
|
||||
"version": "0.1.1",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"fast-xml-parser": "^4.5.0"
|
||||
|
||||
+2
-2
@@ -2,7 +2,7 @@
|
||||
"name": "ra3-mod-xml",
|
||||
"displayName": "RA3 Mod XML",
|
||||
"description": "Red Alert 3 Mod XML tooling: syntax highlighting, completions, reference navigation and diagnostics for SAGE/BinaryAssetBuilder XML.",
|
||||
"version": "0.1.0",
|
||||
"version": "0.1.1",
|
||||
"publisher": "ra3-mod-xml",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -57,7 +57,7 @@
|
||||
"none"
|
||||
],
|
||||
"default": "warning",
|
||||
"description": "Severity for attribute references (CommandSet=..., inheritFrom=...) that cannot be resolved in the index. IDs that only exist in compiled manifests cannot be resolved yet."
|
||||
"description": "Severity for attribute references (CommandSet=..., inheritFrom=...) that cannot be resolved in the index. The index covers mod XML sources, SDK SageXml sources and compiled manifests."
|
||||
},
|
||||
"ra3modxml.diagnoseUnknownElements": {
|
||||
"type": "boolean",
|
||||
|
||||
+20
-3
@@ -9,6 +9,10 @@ import {
|
||||
Ra3ReferenceProvider,
|
||||
} from "./features/navigation";
|
||||
import { Ra3Diagnostics } from "./features/diagnostics";
|
||||
import {
|
||||
Ra3SemanticTokensProvider,
|
||||
RA3_SEMANTIC_TOKENS_LEGEND,
|
||||
} from "./features/semanticTokens";
|
||||
|
||||
const XML_SELECTOR: vscode.DocumentSelector = [{ language: "xml" }];
|
||||
|
||||
@@ -55,6 +59,13 @@ export function activate(context: vscode.ExtensionContext): void {
|
||||
new Ra3DocumentSymbolProvider(),
|
||||
),
|
||||
);
|
||||
context.subscriptions.push(
|
||||
vscode.languages.registerDocumentSemanticTokensProvider(
|
||||
XML_SELECTOR,
|
||||
new Ra3SemanticTokensProvider(),
|
||||
RA3_SEMANTIC_TOKENS_LEGEND,
|
||||
),
|
||||
);
|
||||
|
||||
const diagnostics = new Ra3Diagnostics(ws);
|
||||
context.subscriptions.push(diagnostics);
|
||||
@@ -98,18 +109,24 @@ export function activate(context: vscode.ExtensionContext): void {
|
||||
context.subscriptions.push(
|
||||
vscode.workspace.onDidSaveTextDocument((doc) => {
|
||||
if (doc.languageId !== "xml") return;
|
||||
ws.invalidate(doc.uri.fsPath);
|
||||
ws.scheduleRebuild();
|
||||
void diagnostics.update(doc);
|
||||
}),
|
||||
);
|
||||
context.subscriptions.push(
|
||||
vscode.workspace.onDidChangeConfiguration((e) => {
|
||||
if (e.affectsConfiguration("ra3modxml")) ws.scheduleRebuild();
|
||||
if (e.affectsConfiguration("ra3modxml")) {
|
||||
// Search paths / builtmods locations may have changed: cached include
|
||||
// resolutions and manifest lookups are no longer valid.
|
||||
ws.invalidateExistence();
|
||||
ws.scheduleRebuild();
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
context.subscriptions.push(
|
||||
vscode.commands.registerCommand("ra3modxml.reindex", () => ws.rebuild()),
|
||||
vscode.commands.registerCommand("ra3modxml.reindex", () => ws.rebuild(true)),
|
||||
);
|
||||
context.subscriptions.push(
|
||||
vscode.commands.registerCommand("ra3modxml.openIndexReport", () => {
|
||||
@@ -124,7 +141,7 @@ export function activate(context: vscode.ExtensionContext): void {
|
||||
void vscode.window.showInformationMessage(
|
||||
`RA3 Mod XML index\n` +
|
||||
`Project: ${s.projectDir}\n` +
|
||||
`Files: ${s.indexedFiles} (${s.parsedFiles} parsed)\n` +
|
||||
`Files: ${s.indexedFiles} (${s.parsedFiles} parsed, ${s.shallowScannedFiles} shallow-scanned, ${s.shallowCacheHits + s.recordsCacheHits} cache hits)\n` +
|
||||
`Assets: ${s.assetCount} (${s.manifestAssetCount} from ${s.manifestFiles} manifests)\n` +
|
||||
`Defines: ${s.defineCount} · Streams: ${s.streams} · Candidates: ${s.sourceCandidates}\n` +
|
||||
`Indexed in ${(s.elapsedMs / 1000).toFixed(1)}s`,
|
||||
|
||||
+35
-11
@@ -1,8 +1,13 @@
|
||||
import * as vscode from "vscode";
|
||||
import { parseXml, type XmlElement } from "../language/xmlParser";
|
||||
import { analyzeContext, type CompletionContext } from "../language/context";
|
||||
import {
|
||||
analyzeContext,
|
||||
splitListValuePrefix,
|
||||
type CompletionContext,
|
||||
} from "../language/context";
|
||||
import { resolveElementType } from "../language/typeContext";
|
||||
import * as model from "../model/schemaModel";
|
||||
import { isLocalReferenceAttribute } from "../indexer/refs";
|
||||
import type { ModWorkspace } from "../workspace";
|
||||
import type { ModIndex, AssetDef } from "../indexer/types";
|
||||
|
||||
@@ -177,12 +182,31 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider {
|
||||
const el = ctx.element;
|
||||
const attr = ctx.attr;
|
||||
if (!el || !attr) return [];
|
||||
const prefix = ctx.valuePrefix;
|
||||
const rawPrefix = ctx.valuePrefix;
|
||||
|
||||
const endOffset = attr.quoteEnd > attr.valueEnd ? attr.valueEnd : document.offsetAt(position);
|
||||
const attrName = attr.name.toLowerCase();
|
||||
const elType = resolveElementType(el);
|
||||
const attrInfo = model
|
||||
.attributesOfType(elType)
|
||||
.find((a) => a.name.toLowerCase() === attrName);
|
||||
|
||||
// xs:list values (bit flags such as Surfaces="GROUND WATER") are
|
||||
// whitespace-separated: only the token currently being edited is used for
|
||||
// filtering, and the replacement range covers that token instead of the
|
||||
// whole value.
|
||||
const seg = attrInfo?.isList
|
||||
? splitListValuePrefix(rawPrefix)
|
||||
: { token: rawPrefix, start: 0 };
|
||||
const prefix = seg.token;
|
||||
|
||||
const valueStartOffset =
|
||||
attr.valueStart >= 0 ? attr.valueStart : document.offsetAt(position);
|
||||
const endOffset =
|
||||
attr.quoteEnd > attr.valueEnd ? attr.valueEnd : document.offsetAt(position);
|
||||
const rangeStart = valueStartOffset + seg.start;
|
||||
const valueRange = new vscode.Range(
|
||||
document.positionAt(attr.valueStart),
|
||||
document.positionAt(Math.max(attr.valueStart, endOffset)),
|
||||
document.positionAt(rangeStart),
|
||||
document.positionAt(Math.max(rangeStart, endOffset)),
|
||||
);
|
||||
|
||||
const make = (
|
||||
@@ -200,7 +224,6 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider {
|
||||
};
|
||||
|
||||
const isInclude = el.name === "Include";
|
||||
const attrName = attr.name.toLowerCase();
|
||||
|
||||
// Include type / source
|
||||
if (isInclude && attrName === "type") {
|
||||
@@ -217,16 +240,17 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider {
|
||||
);
|
||||
}
|
||||
|
||||
const elType = resolveElementType(el);
|
||||
const attrInfo = model
|
||||
.attributesOfType(elType)
|
||||
.find((a) => a.name.toLowerCase() === attrName);
|
||||
|
||||
// inheritFrom: same element type first, then everything.
|
||||
if (attrName === "inheritfrom") {
|
||||
return this.assetIdItems(idx, el.name, null, prefix, make);
|
||||
}
|
||||
|
||||
// `id` attributes are definitions and Poid attributes are pipeline-local
|
||||
// references; offering global asset ids for them would be wrong.
|
||||
if (attrInfo && isLocalReferenceAttribute(elType, attr.name)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
if (attrInfo?.refType) {
|
||||
return this.assetIdItems(idx, null, attrInfo.refType, prefix, make);
|
||||
}
|
||||
|
||||
@@ -115,8 +115,12 @@ export class Ra3Diagnostics {
|
||||
}
|
||||
}
|
||||
|
||||
// Elements outside the EA asset namespace (e.g. XInclude <xi:include>)
|
||||
// are not part of the RA3 XSD model; skip their validation entirely.
|
||||
const isXsdElement = model.isXsdElementName(el.name);
|
||||
|
||||
// Unknown element.
|
||||
if (settings.diagnoseUnknownElements && !el.name.startsWith("xi:")) {
|
||||
if (settings.diagnoseUnknownElements && isXsdElement) {
|
||||
const knownType = model.elementTypeName(local);
|
||||
if (!knownType) {
|
||||
diags.push(
|
||||
@@ -131,12 +135,15 @@ export class Ra3Diagnostics {
|
||||
}
|
||||
|
||||
// Attributes.
|
||||
if (isXsdElement) {
|
||||
const elType = resolveElementType(el);
|
||||
const knownAttrs = model.attributesOfType(elType);
|
||||
const knownNames = new Set(knownAttrs.map((a) => a.name));
|
||||
for (const attr of el.attrs) {
|
||||
const aName = attr.name;
|
||||
if (aName.startsWith("xmlns") || aName.startsWith("xai:") || aName.startsWith("xi:")) {
|
||||
// Namespace declarations and prefixed attributes (xai:, xi:,
|
||||
// xlink:, xml:, xsi:, xmlns:*) are not defined by the EA XSD.
|
||||
if (aName.startsWith("xmlns") || !model.isXsdAttributeName(aName)) {
|
||||
continue;
|
||||
}
|
||||
if (settings.diagnoseUnknownElements && !knownNames.has(aName)) {
|
||||
@@ -164,6 +171,7 @@ export class Ra3Diagnostics {
|
||||
diags,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Include-specific checks.
|
||||
if (local === "Include") {
|
||||
|
||||
+24
-3
@@ -29,7 +29,7 @@ export class Ra3HoverProvider implements vscode.HoverProvider {
|
||||
// Attribute name.
|
||||
for (const attr of el.attrs) {
|
||||
if (offset >= attr.nameStart && offset <= attr.nameEnd) {
|
||||
return this.attributeHover(elType, attr.name);
|
||||
return this.attributeHover(el, elType, attr.name);
|
||||
}
|
||||
}
|
||||
// Attribute value.
|
||||
@@ -47,6 +47,14 @@ export class Ra3HoverProvider implements vscode.HoverProvider {
|
||||
}
|
||||
|
||||
private elementHover(name: string): vscode.Hover | null {
|
||||
if (name.startsWith("xi:")) {
|
||||
const md = new vscode.MarkdownString();
|
||||
md.appendCodeblock(`<${name}>`, "xml");
|
||||
md.appendMarkdown(
|
||||
"XInclude element (W3C XInclude namespace) — not part of the RA3 XSD model.",
|
||||
);
|
||||
return new vscode.Hover(md);
|
||||
}
|
||||
const type = model.elementTypeName(name);
|
||||
const info = type ? model.typeInfo(type) : undefined;
|
||||
const md = new vscode.MarkdownString();
|
||||
@@ -68,7 +76,11 @@ export class Ra3HoverProvider implements vscode.HoverProvider {
|
||||
return new vscode.Hover(md);
|
||||
}
|
||||
|
||||
private attributeHover(elementType: string | null, attrName: string): vscode.Hover | null {
|
||||
private attributeHover(
|
||||
el: { name: string },
|
||||
elementType: string | null,
|
||||
attrName: string,
|
||||
): vscode.Hover | null {
|
||||
const attrs = model.attributesOfType(elementType);
|
||||
const attr = attrs.find((a) => a.name === attrName);
|
||||
const md = new vscode.MarkdownString();
|
||||
@@ -78,6 +90,12 @@ export class Ra3HoverProvider implements vscode.HoverProvider {
|
||||
md.appendMarkdown(`Namespace/instance attribute.`);
|
||||
return new vscode.Hover(md);
|
||||
}
|
||||
if (!model.isXsdElementName(el.name)) {
|
||||
md.appendMarkdown(
|
||||
`XInclude attribute (W3C XInclude namespace) — not part of the RA3 XSD model.`,
|
||||
);
|
||||
return new vscode.Hover(md);
|
||||
}
|
||||
md.appendMarkdown("Unknown attribute for this element.");
|
||||
return new vscode.Hover(md);
|
||||
}
|
||||
@@ -117,7 +135,10 @@ export class Ra3HoverProvider implements vscode.HoverProvider {
|
||||
}
|
||||
|
||||
// Include source.
|
||||
if (el.name === "Include" && attrName === "source") {
|
||||
if (
|
||||
(el.name === "Include" && attrName === "source") ||
|
||||
(el.name === "xi:include" && attrName === "href")
|
||||
) {
|
||||
const resolved = idx
|
||||
? resolveSource(
|
||||
value,
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
import * as vscode from "vscode";
|
||||
import { parseXml } from "../language/xmlParser";
|
||||
import { buildSemanticTokenRanges } from "../language/semanticTokens";
|
||||
|
||||
const TOKEN_TYPES = ["type", "property", "string"] as const;
|
||||
|
||||
export const RA3_SEMANTIC_TOKENS_LEGEND = new vscode.SemanticTokensLegend([
|
||||
...TOKEN_TYPES,
|
||||
]);
|
||||
|
||||
/**
|
||||
* Highlighting fallback for malformed XML.
|
||||
*
|
||||
* While the document is well-formed, the built-in TextMate XML grammar colors
|
||||
* it as usual and this provider returns no tokens, so nothing changes. When
|
||||
* parsing reports errors (e.g. an attribute value whose closing quote has not
|
||||
* been typed yet), the TextMate structure is lost, and these semantic tokens
|
||||
* keep element names, attribute names and values colored.
|
||||
*/
|
||||
export class Ra3SemanticTokensProvider
|
||||
implements vscode.DocumentSemanticTokensProvider
|
||||
{
|
||||
async provideDocumentSemanticTokens(
|
||||
document: vscode.TextDocument,
|
||||
_token: vscode.CancellationToken,
|
||||
): Promise<vscode.SemanticTokens> {
|
||||
const text = document.getText();
|
||||
const doc = parseXml(text);
|
||||
if (doc.errors.length === 0) {
|
||||
return new vscode.SemanticTokens(new Uint32Array(0));
|
||||
}
|
||||
const ranges = buildSemanticTokenRanges(doc, text);
|
||||
const builder = new vscode.SemanticTokensBuilder(RA3_SEMANTIC_TOKENS_LEGEND);
|
||||
for (const r of ranges) {
|
||||
builder.push(
|
||||
new vscode.Range(r.line, r.startChar, r.line, r.startChar + r.length),
|
||||
r.tokenType,
|
||||
);
|
||||
}
|
||||
return builder.build();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
/**
|
||||
* Caches shared by the indexer.
|
||||
*
|
||||
* The workspace owns one instance of each cache and passes them into every
|
||||
* ModIndexer, so a rebuild (which creates a fresh indexer) does not re-read
|
||||
* files whose stat (mtime/size) is unchanged. This is what makes indexing
|
||||
* large art-asset corpora (Corona: ~3800 .w3x files, 2.6 GB) practical:
|
||||
* after the first build, save-triggered rebuilds only re-scan files that
|
||||
* actually changed.
|
||||
*/
|
||||
|
||||
import { resolve } from "node:path";
|
||||
import type { IndexedFile, ParsedFile } from "./types";
|
||||
import type { IndexRecords } from "./records";
|
||||
import type { ResolveResult } from "./includeResolver";
|
||||
|
||||
/** Case-insensitive absolute path key. */
|
||||
export function normKey(path: string): string {
|
||||
return resolve(path).toLowerCase();
|
||||
}
|
||||
|
||||
/**
|
||||
* LRU cache for fully parsed XML documents.
|
||||
*
|
||||
* Parse trees of huge mods can be memory-heavy (~17x the source text), so
|
||||
* retention is bounded twice: by entry count (least-recently-used eviction)
|
||||
* and by a total element budget (largest trees are evicted first). Evicted
|
||||
* entries are re-read from disk on demand.
|
||||
*/
|
||||
export class DocumentCache {
|
||||
private map = new Map<string, ParsedFile>();
|
||||
private totalElements = 0;
|
||||
|
||||
constructor(
|
||||
private capacity = 64,
|
||||
private elementBudget = 2_000_000,
|
||||
) {}
|
||||
|
||||
get(path: string): ParsedFile | undefined {
|
||||
const key = normKey(path);
|
||||
const hit = this.map.get(key);
|
||||
if (!hit) return undefined;
|
||||
this.map.delete(key);
|
||||
this.map.set(key, hit);
|
||||
return hit;
|
||||
}
|
||||
|
||||
/** Number of cached documents (for diagnostics). */
|
||||
get size(): number {
|
||||
return this.map.size;
|
||||
}
|
||||
|
||||
/** Total elements held by cached parse trees (for diagnostics). */
|
||||
get elements(): number {
|
||||
return this.totalElements;
|
||||
}
|
||||
|
||||
set(parsed: ParsedFile): void {
|
||||
const key = normKey(parsed.file.path);
|
||||
const prev = this.map.get(key);
|
||||
if (prev) this.totalElements -= elementCount(prev);
|
||||
this.map.delete(key);
|
||||
this.map.set(key, parsed);
|
||||
this.totalElements += elementCount(parsed);
|
||||
this.evict();
|
||||
}
|
||||
|
||||
invalidate(path: string): void {
|
||||
const key = normKey(path);
|
||||
const prev = this.map.get(key);
|
||||
if (prev) this.totalElements -= elementCount(prev);
|
||||
this.map.delete(key);
|
||||
}
|
||||
|
||||
clear(): void {
|
||||
this.map.clear();
|
||||
this.totalElements = 0;
|
||||
}
|
||||
|
||||
private evict(): void {
|
||||
while (this.map.size > this.capacity || this.totalElements > this.elementBudget) {
|
||||
if (this.map.size === 0) break;
|
||||
if (this.map.size > this.capacity) {
|
||||
// Over capacity: drop the least recently used entry.
|
||||
const oldest = this.map.keys().next().value;
|
||||
if (oldest === undefined) break;
|
||||
this.remove(oldest);
|
||||
} else {
|
||||
// Over the element budget: drop the largest tree, which frees the
|
||||
// most memory per eviction.
|
||||
let largestKey: string | undefined;
|
||||
let largest = -1;
|
||||
for (const [key, value] of this.map) {
|
||||
const n = elementCount(value);
|
||||
if (n > largest) {
|
||||
largest = n;
|
||||
largestKey = key;
|
||||
}
|
||||
}
|
||||
if (largestKey === undefined || largest <= 0) break;
|
||||
this.remove(largestKey);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private remove(key: string): void {
|
||||
const prev = this.map.get(key);
|
||||
if (prev) this.totalElements -= elementCount(prev);
|
||||
this.map.delete(key);
|
||||
}
|
||||
}
|
||||
|
||||
function elementCount(parsed: ParsedFile): number {
|
||||
return parsed.parse?.elements.length ?? 0;
|
||||
}
|
||||
|
||||
export interface IndexRecordsCacheEntry {
|
||||
stat: IndexedFile["stat"];
|
||||
records: IndexRecords;
|
||||
/** "shallow" for art-asset scans (.w3x), "full" for parsed XML. */
|
||||
kind: "shallow" | "full";
|
||||
}
|
||||
|
||||
/**
|
||||
* Cache for per-file index records (top-level assets, defines, includes,
|
||||
* xi:include targets). Records are tiny compared to DOM trees or line maps of
|
||||
* multi-megabyte model files, so the capacity comfortably covers a whole mod
|
||||
* (Corona: ~9k files) and rebuilds never re-read unchanged files.
|
||||
*/
|
||||
export class IndexRecordsCache {
|
||||
private map = new Map<string, IndexRecordsCacheEntry>();
|
||||
|
||||
constructor(private capacity = 16384) {}
|
||||
|
||||
get(path: string): IndexRecordsCacheEntry | undefined {
|
||||
const key = normKey(path);
|
||||
const hit = this.map.get(key);
|
||||
if (!hit) return undefined;
|
||||
this.map.delete(key);
|
||||
this.map.set(key, hit);
|
||||
return hit;
|
||||
}
|
||||
|
||||
/** Number of cached record sets (for diagnostics). */
|
||||
get size(): number {
|
||||
return this.map.size;
|
||||
}
|
||||
|
||||
set(path: string, entry: IndexRecordsCacheEntry): void {
|
||||
const key = normKey(path);
|
||||
this.map.delete(key);
|
||||
this.map.set(key, entry);
|
||||
if (this.map.size > this.capacity) {
|
||||
const oldest = this.map.keys().next().value;
|
||||
if (oldest !== undefined) this.map.delete(oldest);
|
||||
}
|
||||
}
|
||||
|
||||
invalidate(path: string): void {
|
||||
this.map.delete(normKey(path));
|
||||
}
|
||||
|
||||
clear(): void {
|
||||
this.map.clear();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Cache for Include/@source (and xi:include/@href) resolution results.
|
||||
*
|
||||
* Resolving a source performs synchronous statSync existence checks against
|
||||
* every search base; a Corona build issues ~110k of them (tens of seconds on
|
||||
* a mechanical drive). Content edits never change *existence*, so this cache
|
||||
* survives content rebuilds and is only cleared when files are created or
|
||||
* deleted (or on a forced reindex).
|
||||
*/
|
||||
export class IncludeResolveCache {
|
||||
private map = new Map<string, ResolveResult>();
|
||||
private manifestMap = new Map<string, string | null>();
|
||||
|
||||
constructor(private capacity = 262144) {}
|
||||
|
||||
get(key: string): ResolveResult | undefined {
|
||||
const hit = this.map.get(key);
|
||||
if (!hit) return undefined;
|
||||
this.map.delete(key);
|
||||
this.map.set(key, hit);
|
||||
return hit;
|
||||
}
|
||||
|
||||
set(key: string, result: ResolveResult): void {
|
||||
this.map.delete(key);
|
||||
this.map.set(key, result);
|
||||
if (this.map.size > this.capacity) {
|
||||
const oldest = this.map.keys().next().value;
|
||||
if (oldest !== undefined) this.map.delete(oldest);
|
||||
}
|
||||
}
|
||||
|
||||
getManifest(key: string): string | null | undefined {
|
||||
const hit = this.manifestMap.get(key);
|
||||
if (hit === undefined) return undefined;
|
||||
this.manifestMap.delete(key);
|
||||
this.manifestMap.set(key, hit);
|
||||
return hit;
|
||||
}
|
||||
|
||||
setManifest(key: string, path: string | null): void {
|
||||
this.manifestMap.delete(key);
|
||||
this.manifestMap.set(key, path);
|
||||
if (this.manifestMap.size > this.capacity) {
|
||||
const oldest = this.manifestMap.keys().next().value;
|
||||
if (oldest !== undefined) this.manifestMap.delete(oldest);
|
||||
}
|
||||
}
|
||||
|
||||
clear(): void {
|
||||
this.map.clear();
|
||||
this.manifestMap.clear();
|
||||
}
|
||||
|
||||
/** Number of cached resolutions (for diagnostics). */
|
||||
get size(): number {
|
||||
return this.map.size;
|
||||
}
|
||||
}
|
||||
+27
-13
@@ -1,7 +1,18 @@
|
||||
import { readdir, stat } from "node:fs/promises";
|
||||
import { join, relative, resolve } from "node:path";
|
||||
import { extname, join, relative, resolve } from "node:path";
|
||||
import type { FileWalker, SourceCandidate } from "./types";
|
||||
|
||||
/**
|
||||
* Extensions offered as Include/@source candidates in DATA directories:
|
||||
* regular XML plus art-asset XML (.w3x) that mods include via W3X.xml hubs.
|
||||
* ART/AUDIO directories already list every file.
|
||||
*/
|
||||
const DATA_CANDIDATE_EXTENSIONS = new Set([".xml", ".w3x"]);
|
||||
|
||||
function isDataCandidate(path: string): boolean {
|
||||
return DATA_CANDIDATE_EXTENSIONS.has(extname(path).toLowerCase());
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursive file list walker with a simple directory-mtime cache. Used to
|
||||
* enumerate candidate files for Include/@source completion. Directory scans
|
||||
@@ -76,29 +87,32 @@ export async function collectSourceCandidates(
|
||||
out.push({ source, path: resolve(path), prefix, baseDir: resolve(baseDir) });
|
||||
};
|
||||
|
||||
for (const dir of dataDirs) {
|
||||
const files = await walker.listFiles(dir);
|
||||
// List directories in parallel; the walker caches each directory by its
|
||||
// mtime, so rebuilds after the first are still cheap.
|
||||
const [dataLists, artLists, audioLists] = await Promise.all([
|
||||
Promise.all(dataDirs.map(async (dir) => ({ dir, files: await walker.listFiles(dir) }))),
|
||||
Promise.all(artDirs.map(async (dir) => ({ dir, files: await walker.listFiles(dir) }))),
|
||||
Promise.all(audioDirs.map(async (dir) => ({ dir, files: await walker.listFiles(dir) }))),
|
||||
]);
|
||||
|
||||
for (const { dir, files } of dataLists) {
|
||||
for (const f of files) {
|
||||
if (!f.toLowerCase().endsWith(".xml")) continue;
|
||||
if (!isDataCandidate(f)) continue;
|
||||
add(f, dir, "DATA");
|
||||
}
|
||||
if (samePath(dir, projectDataDir)) {
|
||||
// Also offer project-relative paths (what Mod.xml itself uses).
|
||||
for (const f of files) {
|
||||
if (f.toLowerCase().endsWith(".xml")) add(f, projectDataDir, null);
|
||||
if (isDataCandidate(f)) add(f, projectDataDir, null);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const dir of artDirs) {
|
||||
for (const f of await walker.listFiles(dir)) {
|
||||
add(f, dir, "ART");
|
||||
}
|
||||
}
|
||||
for (const dir of audioDirs) {
|
||||
for (const f of await walker.listFiles(dir)) {
|
||||
add(f, dir, "AUDIO");
|
||||
for (const { dir, files } of artLists) {
|
||||
for (const f of files) add(f, dir, "ART");
|
||||
}
|
||||
for (const { dir, files } of audioLists) {
|
||||
for (const f of files) add(f, dir, "AUDIO");
|
||||
}
|
||||
|
||||
return out;
|
||||
|
||||
+349
-145
@@ -11,13 +11,21 @@
|
||||
* outside the extension.
|
||||
*/
|
||||
|
||||
import { readFile, readdir, stat } from "node:fs/promises";
|
||||
import { open, readFile, readdir, stat } from "node:fs/promises";
|
||||
import type { Stats } from "node:fs";
|
||||
import { basename, dirname, extname, join, resolve } from "node:path";
|
||||
import { LineMap, parseXml, type XmlDocument, type XmlElement } from "../language/xmlParser";
|
||||
import {
|
||||
LineMap,
|
||||
parseXml,
|
||||
stripBom,
|
||||
type XmlDocument,
|
||||
type XmlElement,
|
||||
} from "../language/xmlParser";
|
||||
import {
|
||||
buildSearchPaths,
|
||||
manifestPathForReference,
|
||||
resolveSource,
|
||||
type ResolveResult,
|
||||
type SearchPaths,
|
||||
} from "./includeResolver";
|
||||
import {
|
||||
@@ -28,6 +36,14 @@ import {
|
||||
} from "./manifestParser";
|
||||
import { canonicalTypeName } from "../model/schemaModel";
|
||||
import { collectSourceCandidates } from "./fileScanner";
|
||||
import { DocumentCache, IncludeResolveCache, IndexRecordsCache, normKey } from "./caches";
|
||||
import type { IndexRecordsCacheEntry } from "./caches";
|
||||
import { scanXmlShallow } from "./shallowScan";
|
||||
import {
|
||||
extractIndexRecords,
|
||||
recordsFromShallow,
|
||||
type IndexRecordXi,
|
||||
} from "./records";
|
||||
import type {
|
||||
AssetDef,
|
||||
DefineDef,
|
||||
@@ -42,54 +58,35 @@ import type {
|
||||
const MAX_DEPTH = 300;
|
||||
/** Files above this size are never parsed (safety against binary blobs). */
|
||||
const MAX_PARSE_BYTES = 4 * 1024 * 1024;
|
||||
/** Only these extensions are treated as XML documents. */
|
||||
const XML_EXTENSIONS = new Set([".xml", ".manifestxml"]);
|
||||
|
||||
function normKey(path: string): string {
|
||||
return resolve(path).toLowerCase();
|
||||
}
|
||||
|
||||
/**
|
||||
* LRU cache for parsed documents. The parse trees of huge mods can be
|
||||
* memory-heavy, so only a bounded number of recent documents is retained;
|
||||
* evicted entries are re-read from disk on demand.
|
||||
* Fully parsed XML documents. `.xml` / `.manifestxml` files are small enough
|
||||
* that a full DOM is affordable.
|
||||
*/
|
||||
export class DocumentCache {
|
||||
private map = new Map<string, ParsedFile>();
|
||||
const FULL_XML_EXTENSIONS = new Set([".xml", ".manifestxml"]);
|
||||
/**
|
||||
* XML documents whose top-level structure is all the index needs (art-asset
|
||||
* files exported by modeling tools, e.g. .w3x). They are shallow-scanned so
|
||||
* multi-megabyte vertex/triangle payloads never become a DOM.
|
||||
*/
|
||||
const SHALLOW_XML_EXTENSIONS = new Set([".w3x"]);
|
||||
/** Bytes peeked when deciding whether an unknown extension is XML text. */
|
||||
const SNIFF_BYTES = 512;
|
||||
|
||||
constructor(private capacity = 64) {}
|
||||
|
||||
get(path: string): ParsedFile | undefined {
|
||||
const key = normKey(path);
|
||||
const hit = this.map.get(key);
|
||||
if (!hit) return undefined;
|
||||
this.map.delete(key);
|
||||
this.map.set(key, hit);
|
||||
return hit;
|
||||
}
|
||||
|
||||
set(parsed: ParsedFile): void {
|
||||
const key = normKey(parsed.file.path);
|
||||
this.map.delete(key);
|
||||
this.map.set(key, parsed);
|
||||
if (this.map.size > this.capacity) {
|
||||
const oldest = this.map.keys().next().value;
|
||||
if (oldest !== undefined) this.map.delete(oldest);
|
||||
}
|
||||
}
|
||||
|
||||
invalidate(path: string): void {
|
||||
this.map.delete(normKey(path));
|
||||
}
|
||||
|
||||
clear(): void {
|
||||
this.map.clear();
|
||||
}
|
||||
}
|
||||
type XmlMode = "full" | "shallow" | "binary";
|
||||
|
||||
export class ModIndexer {
|
||||
private searchPaths: SearchPaths;
|
||||
private docs = new DocumentCache();
|
||||
private docs: DocumentCache;
|
||||
private recordsCache: IndexRecordsCache;
|
||||
private resolveCache: IncludeResolveCache;
|
||||
private scanCounters = {
|
||||
shallowScannedFiles: 0,
|
||||
shallowCacheHits: 0,
|
||||
recordsCacheHits: 0,
|
||||
resolveCacheHits: 0,
|
||||
resolveCalls: 0,
|
||||
};
|
||||
private phase = { candidatesMs: 0, walkMs: 0 };
|
||||
private assets = new Map<string, Map<string, AssetDef[]>>();
|
||||
private assetsById = new Map<string, AssetDef[]>();
|
||||
private defines = new Map<string, DefineDef[]>();
|
||||
@@ -106,42 +103,221 @@ export class ModIndexer {
|
||||
this.searchPaths = buildSearchPaths(opts.sdkDir, opts.projectDir, {
|
||||
DATA: opts.additionalDataSearchPaths,
|
||||
});
|
||||
// Caches may be owned by the workspace so they survive rebuilds.
|
||||
this.docs = opts.documentCache ?? new DocumentCache();
|
||||
this.recordsCache = opts.recordsCache ?? new IndexRecordsCache();
|
||||
this.resolveCache = opts.resolveCache ?? new IncludeResolveCache();
|
||||
}
|
||||
|
||||
/** Re-reads and caches a document; null when unreadable. */
|
||||
/**
|
||||
* Returns a document for indexing/navigation:
|
||||
* - `.xml` / `.manifestxml` files are fully parsed (bounded by
|
||||
* MAX_PARSE_BYTES);
|
||||
* - `.w3x` (and unknown-extension files whose content looks like XML) are
|
||||
* shallow-scanned, so huge model files never become a DOM;
|
||||
* - everything else is registered as a file but never parsed.
|
||||
*
|
||||
* Cached entries are reused when the file stat is unchanged, which lets a
|
||||
* workspace-owned cache survive rebuilds.
|
||||
*/
|
||||
async readDocument(path: string): Promise<ParsedFile | null> {
|
||||
const hit = this.docs.get(path);
|
||||
if (hit) return hit;
|
||||
const key = normKey(path);
|
||||
const trust =
|
||||
this.opts.trustUnchanged === true && !this.opts.changedFiles?.has(key);
|
||||
|
||||
// Trusted fast path: a file that the watcher has not reported as changed
|
||||
// is reused without any stat / content sniff / read at all. This is what
|
||||
// makes save-triggered rebuilds cheap on huge corpora (Corona: ~9k files,
|
||||
// ~2.6 GB of art assets on a mechanical drive).
|
||||
if (trust) {
|
||||
const rec = this.recordsCache.get(key);
|
||||
if (rec) return this.recordsParsed(path, rec);
|
||||
const cached = this.docs.get(key);
|
||||
if (cached) {
|
||||
this.files.set(key, cached.file);
|
||||
return cached;
|
||||
}
|
||||
}
|
||||
|
||||
// Untrusted / cache miss: verify the stat against caches, then read.
|
||||
try {
|
||||
const [st, text] = await Promise.all([stat(path), readFile(path, "utf8")]);
|
||||
if (st.size > MAX_PARSE_BYTES) {
|
||||
const st = await stat(path);
|
||||
const rec = this.recordsCache.get(key);
|
||||
if (rec?.stat && rec.stat.mtimeMs === st.mtimeMs && rec.stat.size === st.size) {
|
||||
return this.recordsParsed(path, rec);
|
||||
}
|
||||
const hit = this.docs.get(key);
|
||||
if (
|
||||
hit?.file.stat &&
|
||||
hit.file.stat.mtimeMs === st.mtimeMs &&
|
||||
hit.file.stat.size === st.size
|
||||
) {
|
||||
this.files.set(key, hit.file);
|
||||
return hit;
|
||||
}
|
||||
const mode = await this.detectXmlMode(path);
|
||||
if (mode === "shallow") return this.scanShallow(path, st);
|
||||
if (mode === "binary") {
|
||||
const file: IndexedFile = { path: resolve(path), stat: { mtimeMs: st.mtimeMs, size: st.size } };
|
||||
const parsed: ParsedFile = { file, parse: null, lineMap: null };
|
||||
const parsed: ParsedFile = { file, parse: null, records: null, lineMap: null };
|
||||
this.docs.set(parsed);
|
||||
this.files.set(normKey(parsed.file.path), file);
|
||||
this.files.set(key, file);
|
||||
return parsed;
|
||||
}
|
||||
if (st.size > MAX_PARSE_BYTES) {
|
||||
const file: IndexedFile = { path: resolve(path), stat: { mtimeMs: st.mtimeMs, size: st.size } };
|
||||
const parsed: ParsedFile = { file, parse: null, records: null, lineMap: null };
|
||||
this.docs.set(parsed);
|
||||
this.files.set(key, file);
|
||||
return parsed;
|
||||
}
|
||||
const text = stripBom(await readFile(path, "utf8"));
|
||||
const lineMap = new LineMap(text);
|
||||
const parse = parseXml(text);
|
||||
const records = extractIndexRecords(parse, lineMap);
|
||||
const parsed: ParsedFile = {
|
||||
file: { path: resolve(path), stat: { mtimeMs: st.mtimeMs, size: st.size } },
|
||||
parse,
|
||||
lineMap: new LineMap(text),
|
||||
records,
|
||||
lineMap,
|
||||
};
|
||||
this.docs.set(parsed);
|
||||
this.files.set(normKey(parsed.file.path), parsed.file);
|
||||
this.recordsCache.set(key, { stat: parsed.file.stat, records, kind: "full" });
|
||||
this.files.set(key, parsed.file);
|
||||
return parsed;
|
||||
} catch {
|
||||
const parsed: ParsedFile = {
|
||||
file: { path: resolve(path), stat: null },
|
||||
parse: null,
|
||||
records: null,
|
||||
lineMap: null,
|
||||
};
|
||||
this.docs.set(parsed);
|
||||
this.files.set(normKey(parsed.file.path), parsed.file);
|
||||
this.files.set(key, parsed.file);
|
||||
return parsed;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Shallow-scans a large art-asset XML document (no DOM built) and caches
|
||||
* its compact index records. The transient LineMap used to compute record
|
||||
* lines is discarded, so the cache never retains megabytes of line offsets
|
||||
* for model files (Corona w3x alone would otherwise keep ~700 MB).
|
||||
*/
|
||||
private async scanShallow(path: string, st: Stats): Promise<ParsedFile | null> {
|
||||
const key = normKey(path);
|
||||
try {
|
||||
const text = stripBom(await readFile(path, "utf8"));
|
||||
const lineMap = new LineMap(text);
|
||||
const records = recordsFromShallow(scanXmlShallow(text), lineMap);
|
||||
const parsed: ParsedFile = {
|
||||
file: { path: resolve(path), stat: { mtimeMs: st.mtimeMs, size: st.size } },
|
||||
parse: null,
|
||||
records,
|
||||
lineMap: null,
|
||||
};
|
||||
this.recordsCache.set(key, { stat: parsed.file.stat, records, kind: "shallow" });
|
||||
this.files.set(key, parsed.file);
|
||||
this.scanCounters.shallowScannedFiles++;
|
||||
return parsed;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Builds a lean ParsedFile from cached index records (no DOM / line map). */
|
||||
private recordsParsed(path: string, entry: IndexRecordsCacheEntry): ParsedFile {
|
||||
if (entry.kind === "shallow") this.scanCounters.shallowCacheHits++;
|
||||
else this.scanCounters.recordsCacheHits++;
|
||||
const file: IndexedFile = { path: resolve(path), stat: entry.stat };
|
||||
this.files.set(normKey(path), file);
|
||||
return { file, parse: null, records: entry.records, lineMap: null };
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads a document and guarantees a DOM parse tree. Used only for
|
||||
* root-level <xi:include> xpointer selection (rare), where the target's
|
||||
* container children are needed.
|
||||
*/
|
||||
private async readDom(path: string): Promise<ParsedFile | null> {
|
||||
const key = normKey(path);
|
||||
const cached = this.docs.get(key);
|
||||
if (cached?.parse?.root) {
|
||||
this.files.set(key, cached.file);
|
||||
return cached;
|
||||
}
|
||||
try {
|
||||
const st = await stat(path);
|
||||
const hit = this.docs.get(key);
|
||||
if (
|
||||
hit?.parse?.root &&
|
||||
hit.file.stat &&
|
||||
hit.file.stat.mtimeMs === st.mtimeMs &&
|
||||
hit.file.stat.size === st.size
|
||||
) {
|
||||
this.files.set(key, hit.file);
|
||||
return hit;
|
||||
}
|
||||
if (st.size > MAX_PARSE_BYTES) return null;
|
||||
const text = stripBom(await readFile(path, "utf8"));
|
||||
const lineMap = new LineMap(text);
|
||||
const parse = parseXml(text);
|
||||
const records = extractIndexRecords(parse, lineMap);
|
||||
const parsed: ParsedFile = {
|
||||
file: { path: resolve(path), stat: { mtimeMs: st.mtimeMs, size: st.size } },
|
||||
parse,
|
||||
records,
|
||||
lineMap,
|
||||
};
|
||||
this.docs.set(parsed);
|
||||
this.recordsCache.set(key, { stat: parsed.file.stat, records, kind: "full" });
|
||||
this.files.set(key, parsed.file);
|
||||
return parsed;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Decides how a resolved include target should be consumed. */
|
||||
private async detectXmlMode(path: string): Promise<XmlMode> {
|
||||
const ext = extname(path).toLowerCase();
|
||||
if (FULL_XML_EXTENSIONS.has(ext)) return "full";
|
||||
if (SHALLOW_XML_EXTENSIONS.has(ext)) return "shallow";
|
||||
return (await looksLikeXml(path)) ? "shallow" : "binary";
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves an include source through the cross-rebuild cache. Existence
|
||||
* does not change on content edits, so trusted rebuilds pay zero statSync
|
||||
* for include resolution (Corona does ~110k checks per build otherwise).
|
||||
*/
|
||||
private resolveCached(source: string, currentDir: string | null): ResolveResult {
|
||||
const key = `${normKey(currentDir ?? "")}\u0000${source}`;
|
||||
const hit = this.resolveCache.get(key);
|
||||
if (hit) {
|
||||
this.scanCounters.resolveCacheHits++;
|
||||
return hit;
|
||||
}
|
||||
this.scanCounters.resolveCalls++;
|
||||
const result = resolveSource(source, currentDir, this.searchPaths);
|
||||
this.resolveCache.set(key, result);
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Cached manifest lookup for `reference` includes. */
|
||||
private manifestPathCached(source: string): string | null {
|
||||
const key = source.toLowerCase();
|
||||
const hit = this.resolveCache.getManifest(key);
|
||||
if (hit !== undefined) {
|
||||
this.scanCounters.resolveCacheHits++;
|
||||
return hit;
|
||||
}
|
||||
this.scanCounters.resolveCalls++;
|
||||
const path = manifestPathForReference(source, this.opts.builtmodsDirs);
|
||||
this.resolveCache.setManifest(key, path);
|
||||
return path;
|
||||
}
|
||||
|
||||
/** Returns the cached parse if present (does not read from disk). */
|
||||
cachedDocument(path: string): ParsedFile | undefined {
|
||||
return this.docs.get(path);
|
||||
@@ -155,6 +331,7 @@ export class ModIndexer {
|
||||
: null;
|
||||
|
||||
// ── Streams ──
|
||||
const walkStart = Date.now();
|
||||
const staticEntry = projectData ? join(projectData, "Mod.xml") : null;
|
||||
if (staticEntry) {
|
||||
const stream: StreamInfo = { name: "static", entry: staticEntry, files: new Set() };
|
||||
@@ -183,8 +360,10 @@ export class ModIndexer {
|
||||
await this.walk(entry, "all", stream, 0);
|
||||
}
|
||||
}
|
||||
this.phase.walkMs = Date.now() - walkStart;
|
||||
|
||||
// ── Source completion candidates ──
|
||||
const candidatesStart = Date.now();
|
||||
const dataDirs = [
|
||||
projectData ?? join(this.opts.projectDir, "Data"),
|
||||
join(this.opts.sdkDir, "SageXml"),
|
||||
@@ -229,6 +408,7 @@ export class ModIndexer {
|
||||
...sdkRootCandidates,
|
||||
...this.sourceCandidates,
|
||||
]);
|
||||
this.phase.candidatesMs = Date.now() - candidatesStart;
|
||||
|
||||
const manifestAssetCount = [...this.manifests.values()].reduce(
|
||||
(sum, m) => sum + m.assets.length,
|
||||
@@ -253,6 +433,13 @@ export class ModIndexer {
|
||||
parsedFiles: [...this.files.values()].filter(
|
||||
(f) => f.stat != null && f.stat.size <= MAX_PARSE_BYTES,
|
||||
).length,
|
||||
shallowScannedFiles: this.scanCounters.shallowScannedFiles,
|
||||
shallowCacheHits: this.scanCounters.shallowCacheHits,
|
||||
recordsCacheHits: this.scanCounters.recordsCacheHits,
|
||||
resolveCacheHits: this.scanCounters.resolveCacheHits,
|
||||
resolveCalls: this.scanCounters.resolveCalls,
|
||||
candidatesMs: this.phase.candidatesMs,
|
||||
walkMs: this.phase.walkMs,
|
||||
assetCount: [...this.assets.values()].reduce((sum, byId) => sum + byId.size, 0),
|
||||
defineCount: this.defines.size,
|
||||
manifestFiles: this.manifests.size,
|
||||
@@ -294,142 +481,135 @@ export class ModIndexer {
|
||||
|
||||
stream.files.add(key);
|
||||
|
||||
// Binary assets (w3x/dds/...) are referenced but never parsed.
|
||||
if (!isXmlPath(path)) return;
|
||||
|
||||
// readDocument returns compact index records for every indexable XML
|
||||
// document (full parse or shallow scan), or a bare file registration
|
||||
// for binary / unparseable targets.
|
||||
const parsed = await this.readDocument(path);
|
||||
if (!parsed?.parse?.root) return;
|
||||
const root = parsed.parse.root;
|
||||
|
||||
for (const child of root.children) {
|
||||
const local = localName(child.name);
|
||||
if (local === "Tags" || local === "Includes" || local === "Defines") continue;
|
||||
if (local === "include") {
|
||||
await this.handleXiInclude(child, parsed, stream, depth);
|
||||
continue;
|
||||
if (!parsed) return;
|
||||
if (parsed.records) {
|
||||
await this.applyRecords(parsed, stream, depth, mode === "instance");
|
||||
return;
|
||||
}
|
||||
const idAttr = child.attrs.find((a) => a.name === "id");
|
||||
if (idAttr) {
|
||||
}
|
||||
|
||||
/**
|
||||
* Applies a document's compact index records: top-level assets, defines,
|
||||
* <Includes>, nested <xi:include> and root-level <xi:include> targets.
|
||||
* Works identically for fully parsed XML and shallow-scanned art assets.
|
||||
*/
|
||||
private async applyRecords(
|
||||
parsed: ParsedFile,
|
||||
stream: StreamInfo,
|
||||
depth: number,
|
||||
viaInstance: boolean,
|
||||
): Promise<void> {
|
||||
const records = parsed.records;
|
||||
if (!records) return;
|
||||
const file = parsed.file.path;
|
||||
const origin = this.originOf(file);
|
||||
|
||||
for (const asset of records.assets) {
|
||||
this.addAsset({
|
||||
type: local,
|
||||
id: idAttr.value,
|
||||
file: parsed.file.path,
|
||||
line: lineOf(parsed, idAttr.valueStart),
|
||||
origin: this.originOf(parsed.file.path),
|
||||
type: asset.type,
|
||||
id: asset.id,
|
||||
file,
|
||||
line: asset.line,
|
||||
origin,
|
||||
stream: stream.name,
|
||||
viaInstance: mode === "instance",
|
||||
viaInstance,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
for (const child of root.children) {
|
||||
if (localName(child.name) !== "Defines") continue;
|
||||
for (const define of child.children) {
|
||||
if (localName(define.name) !== "Define") continue;
|
||||
const name = define.attrs.find((a) => a.name === "name")?.value;
|
||||
const value = define.attrs.find((a) => a.name === "value")?.value;
|
||||
if (!name) continue;
|
||||
for (const define of records.defines) {
|
||||
const entry: DefineDef = {
|
||||
name,
|
||||
value: value ?? "",
|
||||
file: parsed.file.path,
|
||||
line: lineOf(parsed, define.start),
|
||||
origin: this.originOf(parsed.file.path),
|
||||
name: define.name,
|
||||
value: define.value,
|
||||
file,
|
||||
line: define.line,
|
||||
origin,
|
||||
};
|
||||
const arr = this.defines.get(name.toLowerCase());
|
||||
const arr = this.defines.get(define.name.toLowerCase());
|
||||
if (arr) arr.push(entry);
|
||||
else this.defines.set(name.toLowerCase(), [entry]);
|
||||
}
|
||||
else this.defines.set(define.name.toLowerCase(), [entry]);
|
||||
}
|
||||
|
||||
const includesElem = root.children.find((c) => localName(c.name) === "Includes");
|
||||
if (includesElem) {
|
||||
for (const inc of includesElem.children) {
|
||||
if (localName(inc.name) !== "Include") continue;
|
||||
const type = inc.attrs.find((a) => a.name === "type")?.value;
|
||||
const source = inc.attrs.find((a) => a.name === "source")?.value;
|
||||
if (!source) continue;
|
||||
const resolved = resolveSource(source, dirname(parsed.file.path), this.searchPaths);
|
||||
for (const inc of records.includes) {
|
||||
const resolved = this.resolveCached(inc.source, dirname(file));
|
||||
if (!resolved.path) {
|
||||
this.diagnostics.push({
|
||||
file: parsed.file.path,
|
||||
line: lineOf(parsed, inc.start),
|
||||
message: `Include target not found: ${source}`,
|
||||
file,
|
||||
line: inc.line,
|
||||
message: `Include target not found: ${inc.source}`,
|
||||
severity: "warning",
|
||||
code: "include-not-found",
|
||||
});
|
||||
continue;
|
||||
}
|
||||
if (type === "all" || type === "instance") {
|
||||
await this.walk(resolved.path, type === "all" ? "all" : "instance", stream, depth + 1);
|
||||
} else if (type === "reference") {
|
||||
const manifestPath = manifestPathForReference(source, this.opts.builtmodsDirs);
|
||||
if (inc.type === "all" || inc.type === "instance") {
|
||||
await this.walk(
|
||||
resolved.path,
|
||||
inc.type === "all" ? "all" : "instance",
|
||||
stream,
|
||||
depth + 1,
|
||||
);
|
||||
} else if (inc.type === "reference") {
|
||||
const manifestPath = this.manifestPathCached(inc.source);
|
||||
if (manifestPath) {
|
||||
const loaded = await this.loadManifest(manifestPath, stream.name);
|
||||
if (!loaded && isXmlPath(resolved.path)) {
|
||||
// The manifest could not be parsed (missing/invalid): fall back
|
||||
// to the placeholder XML so its content is still available.
|
||||
if (!loaded) await this.walk(resolved.path, "instance", stream, depth + 1);
|
||||
} else {
|
||||
await this.walk(resolved.path, "instance", stream, depth + 1);
|
||||
}
|
||||
} else if (isXmlPath(resolved.path)) {
|
||||
// reference to a real XML file: treat its assets as available
|
||||
await this.walk(resolved.path, "instance", stream, depth + 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Nested <xi:include> anywhere in the tree (not just under the root):
|
||||
// the target content is inlined into the parent element. We make the
|
||||
// target file available and surface missing targets instead of ignoring
|
||||
// them silently.
|
||||
for (const el of parsed.parse.elements) {
|
||||
if (localName(el.name) !== "include") continue;
|
||||
if (el.parent === root) continue; // already handled in the loop above
|
||||
const href = el.attrs.find((a) => a.name === "href")?.value;
|
||||
if (!href) continue;
|
||||
const resolved = resolveSource(href, dirname(parsed.file.path), this.searchPaths);
|
||||
for (const xi of records.nestedXiIncludes) {
|
||||
const resolved = this.resolveCached(xi.href, dirname(file));
|
||||
if (!resolved.path) {
|
||||
this.diagnostics.push({
|
||||
file: parsed.file.path,
|
||||
line: lineOf(parsed, el.start),
|
||||
message: `xi:include target not found: ${href}`,
|
||||
file,
|
||||
line: xi.line,
|
||||
message: `xi:include target not found: ${xi.href}`,
|
||||
severity: "warning",
|
||||
code: "include-not-found",
|
||||
});
|
||||
continue;
|
||||
}
|
||||
stream.files.add(normKey(resolved.path));
|
||||
if (isXmlPath(resolved.path)) {
|
||||
await this.walk(resolved.path, "all", stream, depth + 1);
|
||||
}
|
||||
|
||||
for (const xi of records.rootXiIncludes) {
|
||||
await this.handleRootXiInclude(xi, file, stream, depth);
|
||||
}
|
||||
}
|
||||
|
||||
private async handleXiInclude(
|
||||
xi: XmlElement,
|
||||
parent: ParsedFile,
|
||||
/**
|
||||
* Root-level <xi:include> with an xpointer selects a named container's
|
||||
* children in the target document, so the target's DOM is needed. These
|
||||
* targets are rare, so they are parsed on demand (the tree is also cached).
|
||||
*/
|
||||
private async handleRootXiInclude(
|
||||
xi: IndexRecordXi,
|
||||
parentFile: string,
|
||||
stream: StreamInfo,
|
||||
depth: number,
|
||||
): Promise<void> {
|
||||
const href = xi.attrs.find((a) => a.name === "href")?.value;
|
||||
if (!href) return;
|
||||
const resolved = resolveSource(href, dirname(parent.file.path), this.searchPaths);
|
||||
const resolved = this.resolveCached(xi.href, dirname(parentFile));
|
||||
if (!resolved.path) {
|
||||
this.diagnostics.push({
|
||||
file: parent.file.path,
|
||||
line: lineOf(parent, xi.start),
|
||||
message: `xi:include target not found: ${href}`,
|
||||
file: parentFile,
|
||||
line: xi.line,
|
||||
message: `xi:include target not found: ${xi.href}`,
|
||||
severity: "warning",
|
||||
code: "include-not-found",
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (!isXmlPath(resolved.path)) return;
|
||||
const target = await this.readDocument(resolved.path);
|
||||
if (!target?.parse?.root) return;
|
||||
|
||||
const xpointer = xi.attrs.find((a) => a.name === "xpointer")?.value ?? "";
|
||||
const target = await this.readDom(resolved.path);
|
||||
if (target?.parse?.root) {
|
||||
const xpointer = xi.xpointer ?? "";
|
||||
let candidates: XmlElement[];
|
||||
if (xpointer) {
|
||||
const container = findXPointerContainer(target.parse, xpointer);
|
||||
@@ -452,8 +632,9 @@ export class ModIndexer {
|
||||
});
|
||||
}
|
||||
}
|
||||
stream.files.add(normKey(target.file.path));
|
||||
await this.walk(target.file.path, "all", stream, depth + 1);
|
||||
}
|
||||
stream.files.add(normKey(resolved.path));
|
||||
await this.walk(resolved.path, "all", stream, depth + 1);
|
||||
}
|
||||
|
||||
// ── Manifest loading ──────────────────────────────────────────────
|
||||
@@ -551,9 +732,32 @@ function lineOf(parsed: ParsedFile, offset: number): number {
|
||||
return parsed.lineMap.positionAt(offset).line + 1;
|
||||
}
|
||||
|
||||
function isXmlPath(path: string): boolean {
|
||||
const ext = extname(path).toLowerCase();
|
||||
return XML_EXTENSIONS.has(ext);
|
||||
/**
|
||||
* Content sniffing for include targets with unknown extensions (e.g. art
|
||||
* formats beyond .w3x): a small header that starts with "<" after an
|
||||
* optional UTF-8 BOM / whitespace and contains no NUL bytes is treated as
|
||||
* XML text; anything else is a binary asset (registered, never parsed).
|
||||
*/
|
||||
async function looksLikeXml(path: string): Promise<boolean> {
|
||||
try {
|
||||
const fh = await open(path, "r");
|
||||
try {
|
||||
const buf = Buffer.alloc(SNIFF_BYTES);
|
||||
const { bytesRead } = await fh.read(buf, 0, SNIFF_BYTES, 0);
|
||||
const head = buf.subarray(0, bytesRead);
|
||||
if (head.includes(0)) return false;
|
||||
let i = 0;
|
||||
if (head[0] === 0xef && head[1] === 0xbb && head[2] === 0xbf) i = 3;
|
||||
while (i < head.length && (head[i] === 0x20 || head[i] === 0x09 || head[i] === 0x0a || head[i] === 0x0d)) {
|
||||
i++;
|
||||
}
|
||||
return i < head.length && head[i] === 0x3c; // "<"
|
||||
} finally {
|
||||
await fh.close();
|
||||
}
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function findCaseInsensitiveDir(dir: string): Promise<string | null> {
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
/**
|
||||
* Compact per-file index records.
|
||||
*
|
||||
* A rebuild only needs each file's top-level assets, defines, includes and
|
||||
* xi:include targets — not the DOM. Extracting these records at parse time
|
||||
* and caching them across rebuilds lets Corona-scale rebuilds skip both the
|
||||
* DOM and the file I/O for unchanged files, while keeping the DOM cache small
|
||||
* for on-demand features (hover / navigation / outline).
|
||||
*
|
||||
* Pure TypeScript: no vscode dependency.
|
||||
*/
|
||||
|
||||
import type { LineMap, XmlDocument } from "../language/xmlParser";
|
||||
import type { ShallowDocument } from "./shallowScan";
|
||||
|
||||
export interface IndexRecordAsset {
|
||||
/** Top-level element name, e.g. "W3DContainer". */
|
||||
type: string;
|
||||
id: string;
|
||||
/** 1-based line of the id attribute value. */
|
||||
line: number;
|
||||
}
|
||||
|
||||
export interface IndexRecordDefine {
|
||||
name: string;
|
||||
value: string;
|
||||
/** 1-based line of the <Define> element. */
|
||||
line: number;
|
||||
}
|
||||
|
||||
export interface IndexRecordInclude {
|
||||
type: "all" | "instance" | "reference" | null;
|
||||
source: string;
|
||||
/** 1-based line of the <Include> element. */
|
||||
line: number;
|
||||
}
|
||||
|
||||
export interface IndexRecordXi {
|
||||
href: string;
|
||||
xpointer: string | null;
|
||||
/** 1-based line of the <xi:include> element. */
|
||||
line: number;
|
||||
}
|
||||
|
||||
export interface IndexRecords {
|
||||
assets: IndexRecordAsset[];
|
||||
defines: IndexRecordDefine[];
|
||||
includes: IndexRecordInclude[];
|
||||
/** <xi:include> elements that are direct children of the root. */
|
||||
rootXiIncludes: IndexRecordXi[];
|
||||
/** <xi:include> elements nested anywhere else in the document. */
|
||||
nestedXiIncludes: IndexRecordXi[];
|
||||
}
|
||||
|
||||
const INCLUDE_TYPES = new Set(["all", "instance", "reference"]);
|
||||
|
||||
function lineOf(lineMap: LineMap, offset: number): number {
|
||||
return lineMap.positionAt(offset).line + 1;
|
||||
}
|
||||
|
||||
function localName(tag: string): string {
|
||||
const idx = tag.lastIndexOf(":");
|
||||
return idx >= 0 ? tag.slice(idx + 1) : tag;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts the index records of a fully parsed document. Mirrors the walk
|
||||
* semantics of the indexer exactly: top-level assets (excluding
|
||||
* Tags/Includes/Defines), $DEFINE constants, the top-level <Includes> block
|
||||
* and root/nested <xi:include> elements.
|
||||
*/
|
||||
export function extractIndexRecords(parse: XmlDocument, lineMap: LineMap): IndexRecords {
|
||||
const assets: IndexRecordAsset[] = [];
|
||||
const defines: IndexRecordDefine[] = [];
|
||||
const includes: IndexRecordInclude[] = [];
|
||||
const rootXiIncludes: IndexRecordXi[] = [];
|
||||
const nestedXiIncludes: IndexRecordXi[] = [];
|
||||
const root = parse.root;
|
||||
if (!root) return { assets, defines, includes, rootXiIncludes, nestedXiIncludes };
|
||||
|
||||
for (const child of root.children) {
|
||||
const local = localName(child.name);
|
||||
if (local === "Tags" || local === "Includes" || local === "Defines") continue;
|
||||
if (local === "include") {
|
||||
const href = child.attrs.find((a) => a.name === "href")?.value;
|
||||
if (href) {
|
||||
rootXiIncludes.push({
|
||||
href,
|
||||
xpointer: child.attrs.find((a) => a.name === "xpointer")?.value ?? null,
|
||||
line: lineOf(lineMap, child.start),
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const idAttr = child.attrs.find((a) => a.name === "id");
|
||||
if (idAttr) {
|
||||
assets.push({ type: local, id: idAttr.value, line: lineOf(lineMap, idAttr.valueStart) });
|
||||
}
|
||||
}
|
||||
|
||||
for (const child of root.children) {
|
||||
if (localName(child.name) !== "Defines") continue;
|
||||
for (const define of child.children) {
|
||||
if (localName(define.name) !== "Define") continue;
|
||||
const name = define.attrs.find((a) => a.name === "name")?.value;
|
||||
if (!name) continue;
|
||||
defines.push({
|
||||
name,
|
||||
value: define.attrs.find((a) => a.name === "value")?.value ?? "",
|
||||
line: lineOf(lineMap, define.start),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const includesElem = root.children.find((c) => localName(c.name) === "Includes");
|
||||
if (includesElem) {
|
||||
for (const inc of includesElem.children) {
|
||||
if (localName(inc.name) !== "Include") continue;
|
||||
const source = inc.attrs.find((a) => a.name === "source")?.value;
|
||||
if (!source) continue;
|
||||
const type = inc.attrs.find((a) => a.name === "type")?.value;
|
||||
includes.push({
|
||||
type:
|
||||
type && INCLUDE_TYPES.has(type)
|
||||
? (type as "all" | "instance" | "reference")
|
||||
: null,
|
||||
source,
|
||||
line: lineOf(lineMap, inc.start),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
for (const el of parse.elements) {
|
||||
if (localName(el.name) !== "include") continue;
|
||||
if (el.parent === root) continue; // already handled above
|
||||
const href = el.attrs.find((a) => a.name === "href")?.value;
|
||||
if (!href) continue;
|
||||
nestedXiIncludes.push({
|
||||
href,
|
||||
xpointer: el.attrs.find((a) => a.name === "xpointer")?.value ?? null,
|
||||
line: lineOf(lineMap, el.start),
|
||||
});
|
||||
}
|
||||
|
||||
return { assets, defines, includes, rootXiIncludes, nestedXiIncludes };
|
||||
}
|
||||
|
||||
/** Converts a shallow scan (offsets) into index records (1-based lines). */
|
||||
export function recordsFromShallow(scan: ShallowDocument, lineMap: LineMap): IndexRecords {
|
||||
return {
|
||||
assets: scan.assets.map((a) => ({
|
||||
type: a.name,
|
||||
id: a.id,
|
||||
line: lineOf(lineMap, a.idValueStart),
|
||||
})),
|
||||
defines: scan.defines.map((d) => ({
|
||||
name: d.name,
|
||||
value: d.value,
|
||||
line: lineOf(lineMap, d.start),
|
||||
})),
|
||||
includes: scan.includes.map((i) => ({
|
||||
type: i.type,
|
||||
source: i.source,
|
||||
line: lineOf(lineMap, i.start),
|
||||
})),
|
||||
rootXiIncludes: scan.rootXiIncludes.map((x) => ({
|
||||
href: x.href,
|
||||
xpointer: x.xpointer,
|
||||
line: lineOf(lineMap, x.start),
|
||||
})),
|
||||
nestedXiIncludes: scan.nestedXiIncludes.map((x) => ({
|
||||
href: x.href,
|
||||
xpointer: x.xpointer,
|
||||
line: lineOf(lineMap, x.start),
|
||||
})),
|
||||
};
|
||||
}
|
||||
+39
-1
@@ -10,6 +10,37 @@ export interface ReferenceTarget {
|
||||
score: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* True when an attribute is a "pipeline-local" reference that the global
|
||||
* asset index cannot judge:
|
||||
*
|
||||
* - `id` attributes declare the element's own identity. When the attribute
|
||||
* has no refType (plain Poid / pipeline ids) or its refType is compatible
|
||||
* with the element's own type (e.g. ModuleData@id -> ModuleData), the
|
||||
* element itself is the definition site, so no global definition is
|
||||
* required. An `id` whose refType points at a *different* asset type
|
||||
* (e.g. RoadObject@id -> Road) is a real cross-asset reference and keeps
|
||||
* its reference semantics.
|
||||
* - Poid-typed attributes (ModuleId, AutoResolveBody, SoundRef, ...) are
|
||||
* "pipeline object id" references that are resolved within the same
|
||||
* asset/subtree (modules, sub-objects, pivots, shader materials...),
|
||||
* never against the global asset index.
|
||||
*/
|
||||
export function isLocalReferenceAttribute(
|
||||
typeName: string | null,
|
||||
attrName: string,
|
||||
): boolean {
|
||||
const attr = attributesOfType(typeName).find((a) => a.name === attrName);
|
||||
if (!attr) return false;
|
||||
const isId = attrName.toLowerCase() === "id";
|
||||
if (isId) {
|
||||
if (!attr.refType) return true;
|
||||
if (typeName && isAssignableTo(typeName, attr.refType)) return true;
|
||||
return false;
|
||||
}
|
||||
return attr.type === "Poid";
|
||||
}
|
||||
|
||||
/**
|
||||
* True when an attribute is a typed reference: either the instance
|
||||
* inheritance attribute `inheritFrom`, or an XSD attribute whose simple type
|
||||
@@ -27,7 +58,11 @@ export function isReferenceAttributeOfType(
|
||||
): boolean {
|
||||
if (attrName.toLowerCase() === "inheritfrom") return true;
|
||||
const attr = attributesOfType(typeName).find((a) => a.name === attrName);
|
||||
return attr != null && (attr.refType != null || attr.isRef);
|
||||
if (attr == null || !(attr.refType != null || attr.isRef)) return false;
|
||||
// Definitions (id) and pipeline-local references (Poid) are not references
|
||||
// to global assets, so they never need a definition in the global index.
|
||||
if (isLocalReferenceAttribute(typeName, attrName)) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -73,6 +108,9 @@ export function resolveReferenceTargetsForType(
|
||||
} else {
|
||||
const attr = attributesOfType(typeName).find((a) => a.name === attrName);
|
||||
if (!attr || (!attr.refType && !attr.isRef)) return [];
|
||||
// id definitions and Poid pipeline-local references are never resolved
|
||||
// against the global asset index.
|
||||
if (isLocalReferenceAttribute(typeName, attrName)) return [];
|
||||
refType = attr.refType;
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,274 @@
|
||||
/**
|
||||
* Shallow XML scanner for large art-asset documents (e.g. .w3x).
|
||||
*
|
||||
* Full XML parsing builds an element tree whose memory footprint is roughly
|
||||
* 17x the source text (measured on Corona model files). Model exports like
|
||||
* <W3DMesh> contain hundreds of thousands of tiny numeric elements
|
||||
* (Vertices/V, Triangles/T, ...) that the extension never needs: the index
|
||||
* only consumes top-level asset definitions (name + id), top-level
|
||||
* <Includes>, nested <xi:include> targets and <Defines> constants.
|
||||
*
|
||||
* This scanner performs a single linear pass without building child nodes,
|
||||
* so multi-megabyte model files can be indexed in linear time with ~zero
|
||||
* retained memory.
|
||||
*
|
||||
* Pure TypeScript: no vscode dependency, reusable outside the extension.
|
||||
*/
|
||||
|
||||
export interface ShallowAssetRecord {
|
||||
/** Top-level element name, e.g. "W3DContainer". */
|
||||
name: string;
|
||||
/** Value of the id attribute. */
|
||||
id: string;
|
||||
/** Offset of the first id value character. */
|
||||
idValueStart: number;
|
||||
/** Offset one past the last id value character. */
|
||||
idValueEnd: number;
|
||||
/** Offset of the element's "<". */
|
||||
start: number;
|
||||
/** Offset one past the ">" of the start tag. */
|
||||
startTagEnd: number;
|
||||
}
|
||||
|
||||
export interface ShallowIncludeRecord {
|
||||
/** "all" | "instance" | "reference", or null when absent/unknown. */
|
||||
type: "all" | "instance" | "reference" | null;
|
||||
source: string;
|
||||
/** Offset of the <Include> element. */
|
||||
start: number;
|
||||
}
|
||||
|
||||
export interface ShallowXiIncludeRecord {
|
||||
href: string;
|
||||
xpointer: string | null;
|
||||
/** Offset of the <xi:include> element. */
|
||||
start: number;
|
||||
}
|
||||
|
||||
export interface ShallowDefineRecord {
|
||||
name: string;
|
||||
value: string;
|
||||
/** Offset of the <Define> element. */
|
||||
start: number;
|
||||
}
|
||||
|
||||
export interface ShallowScanError {
|
||||
message: string;
|
||||
offset: number;
|
||||
}
|
||||
|
||||
export interface ShallowDocument {
|
||||
assets: ShallowAssetRecord[];
|
||||
includes: ShallowIncludeRecord[];
|
||||
/** <xi:include> elements that are direct children of the root. */
|
||||
rootXiIncludes: ShallowXiIncludeRecord[];
|
||||
/** <xi:include> elements nested anywhere else in the document. */
|
||||
nestedXiIncludes: ShallowXiIncludeRecord[];
|
||||
defines: ShallowDefineRecord[];
|
||||
errors: ShallowScanError[];
|
||||
}
|
||||
|
||||
interface AttrHit {
|
||||
value: string;
|
||||
valueStart: number;
|
||||
valueEnd: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Matches a complete XML tag while respecting quoted attribute values, so a
|
||||
* ">" or "/" inside a value never terminates the tag early.
|
||||
*/
|
||||
const TAG_RE = /<(?:"[^"]*"|'[^']*'|[^'"<>])*>/g;
|
||||
|
||||
export function scanXmlShallow(text: string): ShallowDocument {
|
||||
const errors: ShallowScanError[] = [];
|
||||
const assets: ShallowAssetRecord[] = [];
|
||||
const includes: ShallowIncludeRecord[] = [];
|
||||
const rootXiIncludes: ShallowXiIncludeRecord[] = [];
|
||||
const nestedXiIncludes: ShallowXiIncludeRecord[] = [];
|
||||
const defines: ShallowDefineRecord[] = [];
|
||||
|
||||
// Depth of the currently open element stack. The document root opens at
|
||||
// depth 0 -> 1, so its direct children open when depth === 1.
|
||||
let depth = 0;
|
||||
let inIncludes = false;
|
||||
let inDefines = false;
|
||||
let i = 0;
|
||||
|
||||
while (i < text.length) {
|
||||
const lt = text.indexOf("<", i);
|
||||
if (lt < 0) break;
|
||||
|
||||
// Non-element constructs: comments, CDATA, DOCTYPE and processing
|
||||
// instructions are skipped whole so their content never looks like tags.
|
||||
if (text.startsWith("<!--", lt)) {
|
||||
const close = text.indexOf("-->", lt + 4);
|
||||
if (close < 0) {
|
||||
errors.push({ message: "Unterminated comment", offset: lt });
|
||||
break;
|
||||
}
|
||||
i = close + 3;
|
||||
continue;
|
||||
}
|
||||
if (text.startsWith("<![CDATA[", lt)) {
|
||||
const close = text.indexOf("]]>", lt + 9);
|
||||
if (close < 0) {
|
||||
errors.push({ message: "Unterminated CDATA section", offset: lt });
|
||||
break;
|
||||
}
|
||||
i = close + 3;
|
||||
continue;
|
||||
}
|
||||
if (text.startsWith("<!", lt)) {
|
||||
const close = text.indexOf(">", lt + 2);
|
||||
if (close < 0) {
|
||||
errors.push({ message: "Unterminated DOCTYPE", offset: lt });
|
||||
break;
|
||||
}
|
||||
i = close + 1;
|
||||
continue;
|
||||
}
|
||||
if (text.startsWith("<?", lt)) {
|
||||
const close = text.indexOf("?>", lt + 2);
|
||||
if (close < 0) {
|
||||
errors.push({ message: "Unterminated processing instruction", offset: lt });
|
||||
break;
|
||||
}
|
||||
i = close + 2;
|
||||
continue;
|
||||
}
|
||||
|
||||
TAG_RE.lastIndex = lt;
|
||||
const m = TAG_RE.exec(text);
|
||||
if (!m) {
|
||||
errors.push({ message: "Unterminated tag", offset: lt });
|
||||
break;
|
||||
}
|
||||
const tag = m[0];
|
||||
const gt = m.index + tag.length - 1;
|
||||
const inner = tag.slice(1, -1);
|
||||
const closing = inner.startsWith("/");
|
||||
const selfClosing = !closing && /\/\s*$/.test(inner);
|
||||
const body = closing ? inner.slice(1) : inner;
|
||||
let nameEnd = 0;
|
||||
while (nameEnd < body.length && !/[\s/>]/.test(body[nameEnd])) nameEnd++;
|
||||
const name = body.slice(0, nameEnd);
|
||||
const base = lt + 1;
|
||||
|
||||
if (!closing) {
|
||||
// Top-level elements (direct children of the root).
|
||||
if (depth === 1) {
|
||||
inIncludes = name === "Includes";
|
||||
inDefines = name === "Defines";
|
||||
const idAttr = findAttr(inner, base, "id");
|
||||
if (idAttr && idAttr.value) {
|
||||
assets.push({
|
||||
name,
|
||||
id: idAttr.value,
|
||||
idValueStart: idAttr.valueStart,
|
||||
idValueEnd: idAttr.valueEnd,
|
||||
start: lt,
|
||||
startTagEnd: gt + 1,
|
||||
});
|
||||
}
|
||||
}
|
||||
// <Include> entries inside the top-level <Includes> block.
|
||||
if (inIncludes && depth === 2 && name === "Include") {
|
||||
const type = findAttr(inner, base, "type");
|
||||
const source = findAttr(inner, base, "source");
|
||||
if (source?.value) {
|
||||
includes.push({
|
||||
type:
|
||||
type?.value === "all" || type?.value === "instance" || type?.value === "reference"
|
||||
? type.value
|
||||
: null,
|
||||
source: source.value,
|
||||
start: lt,
|
||||
});
|
||||
}
|
||||
}
|
||||
// <Define> entries inside the top-level <Defines> block.
|
||||
if (inDefines && depth === 2 && name === "Define") {
|
||||
const nameAttr = findAttr(inner, base, "name");
|
||||
const valueAttr = findAttr(inner, base, "value");
|
||||
if (nameAttr?.value) {
|
||||
defines.push({ name: nameAttr.value, value: valueAttr?.value ?? "", start: lt });
|
||||
}
|
||||
}
|
||||
// Nested <xi:include> (or any *:include) anywhere in the document.
|
||||
if (localName(name) === "include") {
|
||||
const href = findAttr(inner, base, "href");
|
||||
const xpointer = findAttr(inner, base, "xpointer");
|
||||
if (href?.value) {
|
||||
const rec: ShallowXiIncludeRecord = {
|
||||
href: href.value,
|
||||
xpointer: xpointer?.value ?? null,
|
||||
start: lt,
|
||||
};
|
||||
if (depth === 1) rootXiIncludes.push(rec);
|
||||
else nestedXiIncludes.push(rec);
|
||||
}
|
||||
}
|
||||
if (!selfClosing) depth++;
|
||||
} else {
|
||||
depth--;
|
||||
}
|
||||
i = gt + 1;
|
||||
}
|
||||
|
||||
return { assets, includes, rootXiIncludes, nestedXiIncludes, defines, errors };
|
||||
}
|
||||
|
||||
/**
|
||||
* Scans the attributes of a tag (its inner text, without "<" and ">") for
|
||||
* the first occurrence of `want` and returns its value with absolute offsets
|
||||
* (base = offset one past the "<").
|
||||
*/
|
||||
function findAttr(inner: string, base: number, want: string): AttrHit | null {
|
||||
let i = 0;
|
||||
while (i < inner.length) {
|
||||
while (i < inner.length && /\s/.test(inner[i])) i++;
|
||||
// Self-closing marker or end of tag: no more attributes.
|
||||
if (i >= inner.length || inner[i] === "/" || inner[i] === ">") break;
|
||||
const nameStart = i;
|
||||
while (i < inner.length && !/[\s=/>]/.test(inner[i])) i++;
|
||||
const name = inner.slice(nameStart, i);
|
||||
while (i < inner.length && /\s/.test(inner[i])) i++;
|
||||
if (inner[i] === "=") {
|
||||
i++;
|
||||
while (i < inner.length && /\s/.test(inner[i])) i++;
|
||||
const q = inner[i];
|
||||
if (q === '"' || q === "'") {
|
||||
const vStart = i + 1;
|
||||
const vEnd = inner.indexOf(q, vStart);
|
||||
if (vEnd < 0) {
|
||||
return name === want ? { value: "", valueStart: -1, valueEnd: -1 } : null;
|
||||
}
|
||||
if (name === want) {
|
||||
return {
|
||||
value: inner.slice(vStart, vEnd),
|
||||
valueStart: base + vStart,
|
||||
valueEnd: base + vEnd,
|
||||
};
|
||||
}
|
||||
i = vEnd + 1;
|
||||
continue;
|
||||
}
|
||||
// Unquoted value (tolerated).
|
||||
const vs = i;
|
||||
while (i < inner.length && !/[\s>]/.test(inner[i])) i++;
|
||||
if (name === want) {
|
||||
return { value: inner.slice(vs, i), valueStart: base + vs, valueEnd: base + i };
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// Attribute without a value (rare but tolerated).
|
||||
if (name === want) return { value: "", valueStart: -1, valueEnd: -1 };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function localName(name: string): string {
|
||||
const idx = name.lastIndexOf(":");
|
||||
return idx >= 0 ? name.slice(idx + 1) : name;
|
||||
}
|
||||
@@ -1,6 +1,8 @@
|
||||
import type { XmlDocument } from "../language/xmlParser";
|
||||
import type { ManifestInfo } from "./manifestParser";
|
||||
import type { LineMap } from "../language/xmlParser";
|
||||
import type { IndexRecords } from "./records";
|
||||
import type { DocumentCache, IncludeResolveCache, IndexRecordsCache } from "./caches";
|
||||
|
||||
export type AssetOrigin = "project" | "sdk" | "manifest";
|
||||
|
||||
@@ -65,6 +67,20 @@ export interface IndexStats {
|
||||
sdkDir: string;
|
||||
indexedFiles: number;
|
||||
parsedFiles: number;
|
||||
/** Art-asset documents indexed via shallow scan (no DOM tree). */
|
||||
shallowScannedFiles: number;
|
||||
/** Shallow scans served from the persistent cache (unchanged files). */
|
||||
shallowCacheHits: number;
|
||||
/** Parsed XML files served from the persistent records cache. */
|
||||
recordsCacheHits: number;
|
||||
/** Include/xi:include resolutions served from the resolve cache. */
|
||||
resolveCacheHits: number;
|
||||
/** Include/xi:include resolutions performed during this build. */
|
||||
resolveCalls: number;
|
||||
/** Time spent enumerating Include source candidates (ms). */
|
||||
candidatesMs: number;
|
||||
/** Time spent walking the include graph (ms). */
|
||||
walkMs: number;
|
||||
assetCount: number;
|
||||
defineCount: number;
|
||||
manifestFiles: number;
|
||||
@@ -102,6 +118,23 @@ export interface IndexOptions {
|
||||
additionalDataSearchPaths: string[];
|
||||
/** Directory walker used to enumerate files for source completion. */
|
||||
walker: FileWalker;
|
||||
/** Optional parse-tree cache shared across rebuilds (owned by the workspace). */
|
||||
documentCache?: DocumentCache;
|
||||
/** Optional index-records cache shared across rebuilds (owned by the workspace). */
|
||||
recordsCache?: IndexRecordsCache;
|
||||
/** Optional include-resolution cache shared across rebuilds (owned by the workspace). */
|
||||
resolveCache?: IncludeResolveCache;
|
||||
/**
|
||||
* When true, cached documents whose path is not in `changedFiles` are used
|
||||
* without a per-file stat. The workspace enables this while a file watcher
|
||||
* invalidates caches for changed paths; a forced reindex passes false.
|
||||
*/
|
||||
trustUnchanged?: boolean;
|
||||
/**
|
||||
* Normalized paths (see `normKey`) known to have changed since the caches
|
||||
* were populated. Only consulted when `trustUnchanged` is true.
|
||||
*/
|
||||
changedFiles?: ReadonlySet<string>;
|
||||
}
|
||||
|
||||
export interface FileWalker {
|
||||
@@ -121,5 +154,10 @@ export interface ParseCache {
|
||||
export interface ParsedFile {
|
||||
file: IndexedFile;
|
||||
parse: XmlDocument | null;
|
||||
/**
|
||||
* Compact index records (assets/defines/includes/xi:include with lines).
|
||||
* Present for every indexable XML document; null for binary files.
|
||||
*/
|
||||
records: IndexRecords | null;
|
||||
lineMap: LineMap | null;
|
||||
}
|
||||
|
||||
+26
-1
@@ -76,7 +76,15 @@ function analyzeStartTag(
|
||||
|
||||
// Inside an attribute value?
|
||||
for (const attr of el.attrs) {
|
||||
if (attr.hasValue && offset >= attr.quoteStart && offset <= attr.quoteEnd) {
|
||||
// An unterminated value (quoteEnd < 0) happens while the user is typing
|
||||
// the opening quote of a new attribute value; it must still be treated as
|
||||
// an attribute-value context so enum/ref/define completions show up.
|
||||
if (
|
||||
attr.hasValue &&
|
||||
attr.quoteStart >= 0 &&
|
||||
offset >= attr.quoteStart &&
|
||||
(attr.quoteEnd < 0 || offset <= attr.quoteEnd)
|
||||
) {
|
||||
const start = attr.valueStart;
|
||||
const prefix = offset > start ? text.slice(start, offset) : "";
|
||||
return {
|
||||
@@ -128,3 +136,20 @@ function empty(kind: ContextKind): CompletionContext {
|
||||
existingAttrs: [],
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Splits the typed prefix of a whitespace-separated list value (xs:list, e.g.
|
||||
* bit flags such as Surfaces="GROUND WATER") into the token being edited and
|
||||
* the offset of that token inside the prefix.
|
||||
*
|
||||
* "GROUND WA" -> { token: "WA", start: 7 }
|
||||
* "GROUND " -> { token: "", start: 7 }
|
||||
* "WA" -> { token: "WA", start: 0 }
|
||||
*/
|
||||
export function splitListValuePrefix(prefix: string): { token: string; start: number } {
|
||||
let start = prefix.length;
|
||||
while (start > 0 && !/\s/.test(prefix[start - 1])) {
|
||||
start--;
|
||||
}
|
||||
return { token: prefix.slice(start), start };
|
||||
}
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
import { LineMap, type XmlDocument } from "./xmlParser";
|
||||
|
||||
/**
|
||||
* Semantic token types used by the highlighting fallback. They are standard
|
||||
* vscode token types, so every theme already has colors for them.
|
||||
*/
|
||||
export type SemanticTokenType = "type" | "property" | "string";
|
||||
|
||||
export interface SemanticTokenRange {
|
||||
line: number;
|
||||
startChar: number;
|
||||
length: number;
|
||||
tokenType: SemanticTokenType;
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds semantic token ranges from a tolerant parse tree.
|
||||
*
|
||||
* This is the highlighting fallback for malformed XML: while the TextMate
|
||||
* grammar loses structure (e.g. an attribute value whose closing quote has
|
||||
* not been typed yet turns the rest of the file into one string), semantic
|
||||
* tokens keep element names, attribute names and attribute values colored.
|
||||
* The ranges are sorted by position for the vscode encoder.
|
||||
*/
|
||||
export function buildSemanticTokenRanges(
|
||||
doc: XmlDocument,
|
||||
text: string,
|
||||
): SemanticTokenRange[] {
|
||||
const lineMap = new LineMap(text);
|
||||
const out: SemanticTokenRange[] = [];
|
||||
|
||||
const push = (offset: number, length: number, tokenType: SemanticTokenType) => {
|
||||
if (length <= 0 || offset < 0 || offset + length > text.length) return;
|
||||
const pos = lineMap.positionAt(offset);
|
||||
out.push({
|
||||
line: pos.line,
|
||||
startChar: pos.character,
|
||||
length,
|
||||
tokenType,
|
||||
});
|
||||
};
|
||||
|
||||
for (const el of doc.elements) {
|
||||
// Element name in the start tag.
|
||||
push(el.start + 1, el.name.length, "type");
|
||||
// Element name in the closing tag (when present).
|
||||
if (el.closeTagStart >= 0) {
|
||||
push(el.closeTagStart + 2, el.name.length, "type");
|
||||
}
|
||||
for (const attr of el.attrs) {
|
||||
push(attr.nameStart, attr.name.length, "property");
|
||||
if (!attr.hasValue) continue;
|
||||
// Include the surrounding quotes when available; for an unterminated
|
||||
// value quoteEnd is -1 and the token ends at the recovered value end.
|
||||
const start = attr.quoteStart >= 0 ? attr.quoteStart : attr.valueStart;
|
||||
const end = attr.quoteEnd >= 0 ? attr.quoteEnd : attr.valueEnd;
|
||||
push(start, end - start, "string");
|
||||
}
|
||||
}
|
||||
|
||||
out.sort((a, b) => a.line - b.line || a.startChar - b.startChar);
|
||||
return out;
|
||||
}
|
||||
@@ -66,6 +66,14 @@ export interface Position {
|
||||
character: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Removes a leading UTF-8 byte-order mark (U+FEFF) so source offsets match
|
||||
* the text as editors expose it (VS Code strips the BOM from document text).
|
||||
*/
|
||||
export function stripBom(text: string): string {
|
||||
return text.charCodeAt(0) === 0xfeff ? text.slice(1) : text;
|
||||
}
|
||||
|
||||
/** Precomputes line start offsets for offset <-> position conversion. */
|
||||
export class LineMap {
|
||||
private lineStarts: number[] = [0];
|
||||
@@ -336,7 +344,16 @@ export function parseXml(text: string): XmlDocument {
|
||||
const gt = findTagEnd(text, i + 1);
|
||||
if (gt < 0) {
|
||||
err("Unterminated start tag", i);
|
||||
const content = text.slice(i + 1);
|
||||
// Recovery while typing: an attribute value whose closing quote has not
|
||||
// been typed yet makes the scanner run to EOF. End the malformed start
|
||||
// tag at the first line break (or EOF) so the rest of the document is
|
||||
// still parsed and completion/hover keep working for the elements after
|
||||
// the broken tag. The missing quote/tag end is still reported above.
|
||||
let recoverTo = i + 1;
|
||||
while (recoverTo < text.length && text[recoverTo] !== "\n" && text[recoverTo] !== "\r") {
|
||||
recoverTo++;
|
||||
}
|
||||
const content = text.slice(i + 1, recoverTo);
|
||||
const raw = parseTag(content, i + 1);
|
||||
if (raw.name) {
|
||||
const el = buildElement(raw, stack.length);
|
||||
@@ -344,7 +361,8 @@ export function parseXml(text: string): XmlDocument {
|
||||
root = root ?? el;
|
||||
stack.push(el);
|
||||
}
|
||||
break;
|
||||
i = recoverTo + 1;
|
||||
continue;
|
||||
}
|
||||
const content = text.slice(i + 1, gt);
|
||||
const raw = parseTag(content, i + 1);
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -20,6 +20,8 @@ export interface AttributeInfo {
|
||||
/** True for reference-typed attributes whose simple type has no refType. */
|
||||
isRef: boolean;
|
||||
enumValues: string[];
|
||||
/** True for xs:list types (whitespace-separated bit flags / lists). */
|
||||
isList: boolean;
|
||||
allowsDefine: boolean;
|
||||
isBoolean: boolean;
|
||||
base: string | null;
|
||||
@@ -39,6 +41,7 @@ export interface SimpleTypeInfo {
|
||||
refType: string | null;
|
||||
isRef: boolean;
|
||||
enumValues: string[];
|
||||
isList: boolean;
|
||||
allowsDefine: boolean;
|
||||
doc: string;
|
||||
}
|
||||
@@ -212,3 +215,26 @@ export const STRUCTURAL_ELEMENTS = [
|
||||
"Defines",
|
||||
"Define",
|
||||
];
|
||||
|
||||
/**
|
||||
* Element names that live outside the RA3 XSD model's namespace. The model
|
||||
* only covers `uri:ea.com:eala:asset`; other namespaces (so far the W3C
|
||||
* XInclude `xi:`) must not be validated against it.
|
||||
*/
|
||||
const FOREIGN_ELEMENT_PREFIXES = ["xi:"];
|
||||
|
||||
/** True when an element name belongs to the RA3 XSD model's namespace. */
|
||||
export function isXsdElementName(name: string): boolean {
|
||||
const lower = name.toLowerCase();
|
||||
return !FOREIGN_ELEMENT_PREFIXES.some((p) => lower.startsWith(p));
|
||||
}
|
||||
|
||||
/**
|
||||
* True when an attribute name can be validated against the RA3 XSD model.
|
||||
* EA schema attributes are unprefixed; prefixed attributes (`xai:`,
|
||||
* `xi:`, `xlink:`, `xml:`, `xsi:`, `xmlns:*`) are namespace machinery and
|
||||
* are not defined by the XSD.
|
||||
*/
|
||||
export function isXsdAttributeName(name: string): boolean {
|
||||
return !name.includes(":");
|
||||
}
|
||||
|
||||
+78
-2
@@ -3,6 +3,7 @@ import { existsSync } from "node:fs";
|
||||
import { join, dirname } from "node:path";
|
||||
import { CachedDirectoryWalker } from "./indexer/fileScanner";
|
||||
import { ModIndexer } from "./indexer/indexer";
|
||||
import { DocumentCache, IncludeResolveCache, IndexRecordsCache } from "./indexer/caches";
|
||||
import type { ModIndex } from "./indexer/types";
|
||||
import { readSettings, type ExtensionSettings } from "./settings";
|
||||
|
||||
@@ -15,12 +16,21 @@ export class ModWorkspace {
|
||||
settings: ExtensionSettings;
|
||||
|
||||
private walker = new CachedDirectoryWalker();
|
||||
// Caches owned here survive rebuilds: a fresh ModIndexer reuses them and
|
||||
// only re-reads files whose stat changed (crucial for the ~2.6 GB of .w3x
|
||||
// art assets in a project like Corona).
|
||||
private documentCache = new DocumentCache();
|
||||
private recordsCache = new IndexRecordsCache();
|
||||
private resolveCache = new IncludeResolveCache();
|
||||
private context: vscode.ExtensionContext;
|
||||
private watchers: vscode.FileSystemWatcher[] = [];
|
||||
private statusBar: vscode.StatusBarItem;
|
||||
private rebuildTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
private building = false;
|
||||
private dirty = false;
|
||||
|
||||
constructor(context: vscode.ExtensionContext) {
|
||||
this.context = context;
|
||||
this.settings = readSettings();
|
||||
this.statusBar = vscode.window.createStatusBarItem(
|
||||
vscode.StatusBarAlignment.Left,
|
||||
@@ -51,11 +61,69 @@ export class ModWorkspace {
|
||||
this.statusBar.hide();
|
||||
return;
|
||||
}
|
||||
this.startWatching();
|
||||
this.statusBar.text = "$(sync~spin) RA3 XML: indexing…";
|
||||
this.statusBar.show();
|
||||
await this.rebuild();
|
||||
}
|
||||
|
||||
/**
|
||||
* Invalidates cached documents for a path (called by the file watcher and
|
||||
* on document save), so the next rebuild re-reads it instead of trusting
|
||||
* the cached copy.
|
||||
*/
|
||||
invalidate(path: string): void {
|
||||
if (!path) return;
|
||||
this.documentCache.invalidate(path);
|
||||
this.recordsCache.invalidate(path);
|
||||
}
|
||||
|
||||
/**
|
||||
* Called when files are created or deleted: include-resolution results
|
||||
* (which encode file existence) are no longer trustworthy.
|
||||
*/
|
||||
invalidateExistence(): void {
|
||||
this.resolveCache.clear();
|
||||
}
|
||||
|
||||
/**
|
||||
* Watches the project, SDK and extra DATA roots for file changes and
|
||||
* invalidates the corresponding cache entries. With caches invalidated
|
||||
* precisely, rebuilds can trust every other cached file and skip per-file
|
||||
* stats (huge win on mechanical drives; Corona rebuild dropped from ~38s
|
||||
* to a few seconds).
|
||||
*/
|
||||
private startWatching(): void {
|
||||
if (!this.projectRoot) return;
|
||||
const roots = new Set([
|
||||
this.projectRoot,
|
||||
this.settings.sdkPath,
|
||||
...this.settings.additionalDataSearchPaths,
|
||||
]);
|
||||
for (const root of roots) {
|
||||
if (!existsSync(root)) continue;
|
||||
try {
|
||||
const watcher = vscode.workspace.createFileSystemWatcher(
|
||||
new vscode.RelativePattern(root, "**/*"),
|
||||
);
|
||||
watcher.onDidCreate((uri) => {
|
||||
this.invalidate(uri.fsPath);
|
||||
this.invalidateExistence();
|
||||
});
|
||||
watcher.onDidChange((uri) => this.invalidate(uri.fsPath));
|
||||
watcher.onDidDelete((uri) => {
|
||||
this.invalidate(uri.fsPath);
|
||||
this.invalidateExistence();
|
||||
});
|
||||
this.watchers.push(watcher);
|
||||
this.context.subscriptions.push(watcher);
|
||||
} catch {
|
||||
// The root may be temporarily unavailable (e.g. removable drive);
|
||||
// indexing still works, just without watcher-based invalidation.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
scheduleRebuild(): void {
|
||||
if (!this.projectRoot) return;
|
||||
if (this.rebuildTimer) clearTimeout(this.rebuildTimer);
|
||||
@@ -64,12 +132,13 @@ export class ModWorkspace {
|
||||
}, REBUILD_DEBOUNCE_MS);
|
||||
}
|
||||
|
||||
async rebuild(): Promise<void> {
|
||||
async rebuild(force = false): Promise<void> {
|
||||
if (!this.projectRoot) return;
|
||||
if (this.building) {
|
||||
this.dirty = true;
|
||||
return;
|
||||
}
|
||||
if (force) this.resolveCache.clear();
|
||||
this.building = true;
|
||||
this.settings = readSettings();
|
||||
try {
|
||||
@@ -81,6 +150,12 @@ export class ModWorkspace {
|
||||
indexSageXml: this.settings.indexSageXml,
|
||||
additionalDataSearchPaths: this.settings.additionalDataSearchPaths,
|
||||
walker: this.walker,
|
||||
documentCache: this.documentCache,
|
||||
recordsCache: this.recordsCache,
|
||||
resolveCache: this.resolveCache,
|
||||
// Trust cache entries unless the user explicitly asked for a full
|
||||
// verification (ra3modxml.reindex).
|
||||
trustUnchanged: !force,
|
||||
});
|
||||
const started = Date.now();
|
||||
this.index = await indexer.build();
|
||||
@@ -90,7 +165,7 @@ export class ModWorkspace {
|
||||
this.statusBar.text = `$(symbol-misc) RA3 XML: ${formatCount(s.assetCount)} assets`;
|
||||
this.statusBar.tooltip =
|
||||
`${s.projectDir}\n` +
|
||||
`${s.indexedFiles} files indexed (${secs}s)\n` +
|
||||
`${s.indexedFiles} files indexed (${s.parsedFiles} parsed, ${s.shallowScannedFiles} art assets shallow-scanned, ${secs}s)\n` +
|
||||
`${s.assetCount} assets (${s.manifestAssetCount} from ${s.manifestFiles} manifests)\n` +
|
||||
`${s.defineCount} defines, ${s.streams} streams, ${s.sourceCandidates} include candidates`;
|
||||
} catch (err) {
|
||||
@@ -113,6 +188,7 @@ export class ModWorkspace {
|
||||
return {
|
||||
file: { path, stat: null },
|
||||
parse,
|
||||
records: null,
|
||||
lineMap: new LineMap(text),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import {
|
||||
DocumentCache,
|
||||
IncludeResolveCache,
|
||||
IndexRecordsCache,
|
||||
} from "../out/indexer/caches.js";
|
||||
|
||||
function parsed(path, elements) {
|
||||
return {
|
||||
file: { path, stat: { mtimeMs: 1, size: 1 } },
|
||||
parse: { root: { name: "r" }, elements: new Array(elements), errors: [] },
|
||||
records: null,
|
||||
lineMap: null,
|
||||
};
|
||||
}
|
||||
|
||||
test("DocumentCache evicts the largest tree when over the element budget", () => {
|
||||
const cache = new DocumentCache(64, 100);
|
||||
cache.set(parsed("a.xml", 60));
|
||||
cache.set(parsed("b.xml", 60));
|
||||
assert.equal(cache.get("a.xml"), undefined, "largest tree evicted first");
|
||||
assert.ok(cache.get("b.xml"));
|
||||
});
|
||||
|
||||
test("DocumentCache evicts least recently used when over capacity", () => {
|
||||
const cache = new DocumentCache(2, 1_000_000);
|
||||
cache.set(parsed("a.xml", 10));
|
||||
cache.set(parsed("b.xml", 10));
|
||||
cache.set(parsed("c.xml", 10));
|
||||
assert.equal(cache.get("a.xml"), undefined);
|
||||
assert.ok(cache.get("b.xml"));
|
||||
assert.ok(cache.get("c.xml"));
|
||||
});
|
||||
|
||||
test("DocumentCache invalidate frees budget", () => {
|
||||
const cache = new DocumentCache(64, 100);
|
||||
cache.set(parsed("a.xml", 60));
|
||||
cache.invalidate("a.xml");
|
||||
cache.set(parsed("b.xml", 60));
|
||||
assert.ok(cache.get("b.xml"), "invalidating a freed its budget share");
|
||||
cache.set(parsed("c.xml", 60));
|
||||
assert.equal(cache.get("a.xml"), undefined);
|
||||
assert.ok(cache.get("c.xml"));
|
||||
});
|
||||
|
||||
test("IndexRecordsCache stores and invalidates entries", () => {
|
||||
const cache = new IndexRecordsCache();
|
||||
const entry = {
|
||||
stat: { mtimeMs: 1, size: 1 },
|
||||
records: { assets: [], defines: [], includes: [], rootXiIncludes: [], nestedXiIncludes: [] },
|
||||
kind: "full",
|
||||
};
|
||||
cache.set("a.xml", entry);
|
||||
assert.equal(cache.get("a.xml"), entry);
|
||||
cache.invalidate("a.xml");
|
||||
assert.equal(cache.get("a.xml"), undefined);
|
||||
});
|
||||
|
||||
test("IncludeResolveCache stores sources and manifest lookups", () => {
|
||||
const cache = new IncludeResolveCache();
|
||||
const key = "dir|DATA:static.xml";
|
||||
const result = { path: "C:/sdk/static.xml", prefix: "DATA", raw: "DATA:static.xml" };
|
||||
assert.equal(cache.get(key), undefined);
|
||||
cache.set(key, result);
|
||||
assert.equal(cache.get(key), result);
|
||||
assert.equal(cache.getManifest("static.xml"), undefined);
|
||||
cache.setManifest("static.xml", "C:/sdk/builtmods/static.manifest");
|
||||
assert.equal(cache.getManifest("static.xml"), "C:/sdk/builtmods/static.manifest");
|
||||
cache.clear();
|
||||
assert.equal(cache.get(key), undefined);
|
||||
assert.equal(cache.getManifest("static.xml"), undefined);
|
||||
});
|
||||
@@ -0,0 +1,157 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { createRequire } from "node:module";
|
||||
|
||||
// Minimal vscode shim so the compiled completion provider can run under plain
|
||||
// node. Only the APIs used by the completion call path are implemented.
|
||||
const CompletionItemKind = {
|
||||
Field: 1,
|
||||
Property: 2,
|
||||
EnumMember: 3,
|
||||
Value: 4,
|
||||
Constant: 5,
|
||||
File: 6,
|
||||
};
|
||||
|
||||
class CompletionItem {
|
||||
constructor(label, kind) {
|
||||
this.label = label;
|
||||
this.kind = kind;
|
||||
}
|
||||
}
|
||||
|
||||
class Position {
|
||||
constructor(line, character) {
|
||||
this.line = line;
|
||||
this.character = character;
|
||||
}
|
||||
}
|
||||
|
||||
class Range {
|
||||
constructor(start, end) {
|
||||
this.start = start;
|
||||
this.end = end;
|
||||
}
|
||||
}
|
||||
|
||||
class MarkdownString {
|
||||
constructor(value) {
|
||||
this.value = value ?? "";
|
||||
}
|
||||
appendMarkdown(text) {
|
||||
this.value += text;
|
||||
return this;
|
||||
}
|
||||
appendCodeblock(text) {
|
||||
this.value += "\n```\n" + text + "\n```\n";
|
||||
return this;
|
||||
}
|
||||
}
|
||||
|
||||
class SnippetString {
|
||||
constructor(value) {
|
||||
this.value = value;
|
||||
}
|
||||
}
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const Module = require("module");
|
||||
const origResolve = Module._resolveFilename;
|
||||
Module._resolveFilename = function (request, ...args) {
|
||||
if (request === "vscode") return "vscode-stub";
|
||||
return origResolve.call(this, request, ...args);
|
||||
};
|
||||
require.cache["vscode-stub"] = {
|
||||
id: "vscode-stub",
|
||||
filename: "vscode-stub",
|
||||
loaded: true,
|
||||
exports: {
|
||||
CompletionItem,
|
||||
CompletionItemKind,
|
||||
Position,
|
||||
Range,
|
||||
MarkdownString,
|
||||
SnippetString,
|
||||
},
|
||||
};
|
||||
|
||||
const { Ra3CompletionProvider } = require("../out/features/completion.js");
|
||||
|
||||
function makeDocument(text) {
|
||||
const lineStarts = [0];
|
||||
for (let i = 0; i < text.length; i++) {
|
||||
if (text.charCodeAt(i) === 10) lineStarts.push(i + 1);
|
||||
}
|
||||
return {
|
||||
getText: () => text,
|
||||
offsetAt: (pos) => lineStarts[pos.line] + pos.character,
|
||||
positionAt: (offset) => {
|
||||
let lo = 0;
|
||||
let hi = lineStarts.length - 1;
|
||||
while (lo < hi) {
|
||||
const mid = (lo + hi + 1) >> 1;
|
||||
if (lineStarts[mid] <= offset) lo = mid;
|
||||
else hi = mid - 1;
|
||||
}
|
||||
return new Position(lo, offset - lineStarts[lo]);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// Enum completions do not consult the index, but provideCompletionItems only
|
||||
// routes value contexts when the workspace index is present.
|
||||
const provider = new Ra3CompletionProvider({ index: {} });
|
||||
const token = { isCancellationRequested: false };
|
||||
|
||||
test("Surfaces enum completion works with an unclosed quote", async () => {
|
||||
const text =
|
||||
`<AssetDeclaration>\n <LocomotorTemplate id="x" Surfaces="G>\n <Other/>\n</LocomotorTemplate>\n</AssetDeclaration>`;
|
||||
const line1 = text.split("\n")[1];
|
||||
const line1Start = text.indexOf("\n") + 1;
|
||||
const valueStart = text.indexOf('Surfaces="') + 'Surfaces="'.length;
|
||||
const pos = new Position(1, line1.indexOf("G") + 1);
|
||||
|
||||
const items = await provider.provideCompletionItems(makeDocument(text), pos, token);
|
||||
const labels = items.map((i) => i.label);
|
||||
assert.ok(labels.includes("GROUND"));
|
||||
assert.ok(!labels.includes("WATER"));
|
||||
assert.ok(items.every((i) => i.kind === CompletionItemKind.EnumMember));
|
||||
|
||||
// Replacement range covers only the value (start..cursor).
|
||||
const ground = items.find((i) => i.label === "GROUND");
|
||||
assert.equal(ground.range.start.character, valueStart - line1Start);
|
||||
assert.equal(ground.range.end.character, pos.character);
|
||||
});
|
||||
|
||||
test("list values filter on the token after whitespace", async () => {
|
||||
const text =
|
||||
`<AssetDeclaration>\n <LocomotorTemplate id="x" Surfaces="GROUND W>\n <Other/>\n</LocomotorTemplate>\n</AssetDeclaration>`;
|
||||
const line1 = text.split("\n")[1];
|
||||
const line1Start = text.indexOf("\n") + 1;
|
||||
const valueStart = text.indexOf('Surfaces="') + 'Surfaces="'.length;
|
||||
const pos = new Position(1, line1.indexOf("W") + 1);
|
||||
|
||||
const items = await provider.provideCompletionItems(makeDocument(text), pos, token);
|
||||
const labels = items.map((i) => i.label);
|
||||
assert.ok(labels.includes("WATER"));
|
||||
assert.ok(labels.includes("WALL_RAILING"));
|
||||
assert.ok(!labels.includes("GROUND"));
|
||||
|
||||
// Replacement range covers only the second token, not "GROUND ".
|
||||
const water = items.find((i) => i.label === "WATER");
|
||||
assert.equal(water.range.start.character, valueStart + "GROUND ".length - line1Start);
|
||||
});
|
||||
|
||||
test("empty unterminated value offers all enum values", async () => {
|
||||
const text =
|
||||
`<AssetDeclaration>\n <LocomotorTemplate id="x" Surfaces=">\n <Other/>\n</LocomotorTemplate>\n</AssetDeclaration>`;
|
||||
const line1 = text.split("\n")[1];
|
||||
const pos = new Position(1, line1.indexOf('Surfaces="') + 'Surfaces="'.length);
|
||||
|
||||
const items = await provider.provideCompletionItems(makeDocument(text), pos, token);
|
||||
const labels = items.map((i) => i.label);
|
||||
assert.equal(items.length, 11);
|
||||
assert.ok(labels.includes("GROUND"));
|
||||
assert.ok(labels.includes("WATER"));
|
||||
assert.ok(labels.includes("CRUSHABLE_WALL"));
|
||||
});
|
||||
@@ -0,0 +1,42 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { parseXml } from "../out/language/xmlParser.js";
|
||||
import { analyzeContext, splitListValuePrefix } from "../out/language/context.js";
|
||||
|
||||
test("unterminated quote is still an attribute-value context", () => {
|
||||
const text = `<Locomotor id="x" Surfaces="GROUND>\n <Other/>\n</Locomotor>`;
|
||||
const cursor = text.indexOf("GROUND") + 6;
|
||||
const doc = parseXml(text);
|
||||
const ctx = analyzeContext(doc, text, cursor);
|
||||
assert.equal(ctx.kind, "attribute-value");
|
||||
assert.equal(ctx.attr?.name, "Surfaces");
|
||||
assert.equal(ctx.valuePrefix, "GROUND");
|
||||
assert.equal(ctx.element?.name, "Locomotor");
|
||||
});
|
||||
|
||||
test("empty unterminated value keeps attribute-value context", () => {
|
||||
const text = `<Locomotor id="x" Surfaces="\n</Locomotor>`;
|
||||
const cursor = text.indexOf('Surfaces="') + 'Surfaces="'.length;
|
||||
const doc = parseXml(text);
|
||||
const ctx = analyzeContext(doc, text, cursor);
|
||||
assert.equal(ctx.kind, "attribute-value");
|
||||
assert.equal(ctx.attr?.name, "Surfaces");
|
||||
assert.equal(ctx.valuePrefix, "");
|
||||
});
|
||||
|
||||
test("closed quote still resolves value context", () => {
|
||||
const text = `<Locomotor id="x" Surfaces="GROUND">\n</Locomotor>`;
|
||||
const cursor = text.indexOf("GROUND") + 6;
|
||||
const doc = parseXml(text);
|
||||
const ctx = analyzeContext(doc, text, cursor);
|
||||
assert.equal(ctx.kind, "attribute-value");
|
||||
assert.equal(ctx.attr?.name, "Surfaces");
|
||||
assert.equal(ctx.valuePrefix, "GROUND");
|
||||
});
|
||||
|
||||
test("splitListValuePrefix isolates the token being edited", () => {
|
||||
assert.deepEqual(splitListValuePrefix("GROUND WA"), { token: "WA", start: 7 });
|
||||
assert.deepEqual(splitListValuePrefix("GROUND "), { token: "", start: 7 });
|
||||
assert.deepEqual(splitListValuePrefix("WA"), { token: "WA", start: 0 });
|
||||
assert.deepEqual(splitListValuePrefix(""), { token: "", start: 0 });
|
||||
});
|
||||
@@ -0,0 +1,2 @@
|
||||
This is a fake DDS payload used to verify that binary assets included from
|
||||
an XML hub are registered as files but never parsed as XML.
|
||||
@@ -0,0 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<AssetDeclaration xmlns="uri:ea.com:eala:asset">
|
||||
<W3DMesh id="Tank_FP" />
|
||||
</AssetDeclaration>
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<AssetDeclaration xmlns="uri:ea.com:eala:asset">
|
||||
<W3DContainer id="Tank_SKN" Hierarchy="Tank_SKL" />
|
||||
<W3DHierarchy id="Tank_SKL" />
|
||||
</AssetDeclaration>
|
||||
@@ -0,0 +1,8 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<AssetDeclaration xmlns="uri:ea.com:eala:asset">
|
||||
<Includes>
|
||||
<Include type="all" source="Models/Tank_SKN.w3x" />
|
||||
<Include type="all" source="Models/Tank_FP.w3d" />
|
||||
<Include type="all" source="Models/Tank_Damaged.dds" />
|
||||
</Includes>
|
||||
</AssetDeclaration>
|
||||
Vendored
+1
@@ -9,5 +9,6 @@
|
||||
<Include type="all" source="Includes/Weapons.xml" />
|
||||
<Include type="all" source="Includes/Shared.xml" />
|
||||
<Include type="all" source="Includes/Refs.xml" />
|
||||
<Include type="all" source="Includes/VehicleArt.xml" />
|
||||
</Includes>
|
||||
</AssetDeclaration>
|
||||
|
||||
@@ -2,8 +2,16 @@ import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { dirname, join } from "node:path";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import { ModIndexer } from "../out/indexer/indexer.js";
|
||||
import { CachedDirectoryWalker } from "../out/indexer/fileScanner.js";
|
||||
import {
|
||||
DocumentCache,
|
||||
IncludeResolveCache,
|
||||
IndexRecordsCache,
|
||||
} from "../out/indexer/caches.js";
|
||||
import { resolveReferenceTargetsForType } from "../out/indexer/refs.js";
|
||||
|
||||
const root = dirname(dirname(fileURLToPath(import.meta.url)));
|
||||
const project = join(root, "test", "fixtures", "minimod");
|
||||
@@ -70,3 +78,182 @@ test("provides include source candidates", async () => {
|
||||
assert.ok(xml.some((c) => c.source === "Includes/Units.xml"));
|
||||
assert.ok(xml.some((c) => c.source === "DATA:static.xml"));
|
||||
});
|
||||
|
||||
test("indexes art-asset XML (.w3x / sniffed .w3d) via shallow scan and skips binary", async () => {
|
||||
const idx = await buildIndex();
|
||||
|
||||
// The .w3x hub chain: Mod.xml -> VehicleArt.xml -> Models/Tank_SKN.w3x.
|
||||
const skn = idx.assetsById.get("tank_skn");
|
||||
assert.ok(skn?.some((d) => d.type === "W3DContainer"), "W3DContainer from .w3x indexed");
|
||||
assert.ok(
|
||||
idx.assetsById.get("tank_skl")?.some((d) => d.type === "W3DHierarchy"),
|
||||
"W3DHierarchy from .w3x indexed",
|
||||
);
|
||||
const container = skn.find((d) => d.type === "W3DContainer");
|
||||
assert.match(container.file, /Tank_SKN\.w3x$/);
|
||||
assert.ok(container.line > 0, "definition line recorded from shallow scan");
|
||||
|
||||
// Unknown extension with XML content is sniffed and indexed.
|
||||
assert.ok(
|
||||
idx.assetsById.get("tank_fp")?.some((d) => d.type === "W3DMesh"),
|
||||
"unknown-extension XML (.w3d) sniffed and indexed",
|
||||
);
|
||||
|
||||
// Binary content is registered as a file but never parsed.
|
||||
assert.equal(idx.assetsById.get("tank_damaged"), undefined, "binary asset never indexed");
|
||||
assert.ok(
|
||||
idx.files.has(
|
||||
join(project, "Data", "Includes", "Models", "Tank_Damaged.dds").toLowerCase(),
|
||||
),
|
||||
"binary include target registered as a file",
|
||||
);
|
||||
|
||||
// The reported Harbinger scenario: <Model Name="Tank_SKN"/> (refType
|
||||
// BaseRenderAssetType) must resolve to the W3DContainer defined in the w3x.
|
||||
const targets = resolveReferenceTargetsForType(
|
||||
idx,
|
||||
"ScriptedModelDrawModel",
|
||||
"Name",
|
||||
"Tank_SKN",
|
||||
);
|
||||
assert.equal(targets.length, 1);
|
||||
assert.equal(targets[0].def.type, "W3DContainer");
|
||||
assert.match(targets[0].def.file, /Tank_SKN\.w3x$/);
|
||||
});
|
||||
|
||||
test("w3x files appear in Include source completion candidates", async () => {
|
||||
const idx = await buildIndex();
|
||||
assert.ok(
|
||||
idx.sourceCandidates.some((c) => c.source === "Includes/Models/Tank_SKN.w3x"),
|
||||
"project-relative w3x candidate",
|
||||
);
|
||||
assert.ok(
|
||||
idx.sourceCandidates.some((c) => c.source === "DATA:Includes/Models/Tank_SKN.w3x"),
|
||||
"DATA: w3x candidate",
|
||||
);
|
||||
});
|
||||
|
||||
test("shallow scans and full parses are cached across rebuilds", async () => {
|
||||
const documentCache = new DocumentCache();
|
||||
const recordsCache = new IndexRecordsCache();
|
||||
const resolveCache = new IncludeResolveCache();
|
||||
const makeIndexer = () =>
|
||||
new ModIndexer({
|
||||
projectDir: project,
|
||||
sdkDir: sdk,
|
||||
builtmodsDirs: [join(sdk, "builtmods")],
|
||||
indexSageXml: true,
|
||||
additionalDataSearchPaths: [],
|
||||
walker: new CachedDirectoryWalker(),
|
||||
documentCache,
|
||||
recordsCache,
|
||||
resolveCache,
|
||||
});
|
||||
|
||||
const first = await makeIndexer().build();
|
||||
const second = await makeIndexer().build();
|
||||
|
||||
assert.equal(first.stats.shallowScannedFiles, 2, "w3x + sniffed w3d scanned on first build");
|
||||
assert.equal(second.stats.shallowScannedFiles, 0, "unchanged art assets are not re-scanned");
|
||||
assert.equal(second.stats.shallowCacheHits, 2);
|
||||
assert.ok(second.stats.recordsCacheHits > 0, "XML records served from cache");
|
||||
assert.ok(second.stats.resolveCacheHits > 0, "include resolutions served from cache");
|
||||
assert.equal(second.stats.resolveCalls, 0, "no include re-resolved on a trusted rebuild");
|
||||
assert.equal(second.stats.assetCount, first.stats.assetCount);
|
||||
assert.equal(second.stats.indexedFiles, first.stats.indexedFiles);
|
||||
});
|
||||
|
||||
test("trusted rebuilds skip unchanged files; invalidation forces re-reads", async () => {
|
||||
const documentCache = new DocumentCache();
|
||||
const recordsCache = new IndexRecordsCache();
|
||||
const resolveCache = new IncludeResolveCache();
|
||||
const opts = () => ({
|
||||
projectDir: project,
|
||||
sdkDir: sdk,
|
||||
builtmodsDirs: [join(sdk, "builtmods")],
|
||||
indexSageXml: true,
|
||||
additionalDataSearchPaths: [],
|
||||
walker: new CachedDirectoryWalker(),
|
||||
documentCache,
|
||||
recordsCache,
|
||||
resolveCache,
|
||||
});
|
||||
|
||||
// Trusted rebuild: cached files are reused without any per-file stat/read.
|
||||
const first = await new ModIndexer({ ...opts(), trustUnchanged: true }).build();
|
||||
assert.equal(first.stats.shallowScannedFiles, 2);
|
||||
const second = await new ModIndexer({ ...opts(), trustUnchanged: true }).build();
|
||||
assert.equal(second.stats.shallowScannedFiles, 0);
|
||||
assert.equal(second.stats.shallowCacheHits, 2);
|
||||
assert.ok(second.stats.recordsCacheHits > 0);
|
||||
|
||||
// Simulate the file watcher / save handler: invalidate one art asset.
|
||||
// The next trusted rebuild must re-scan exactly that file.
|
||||
recordsCache.invalidate(join(project, "Data", "Includes", "Models", "Tank_SKN.w3x"));
|
||||
const third = await new ModIndexer({ ...opts(), trustUnchanged: true }).build();
|
||||
assert.equal(third.stats.shallowScannedFiles, 1);
|
||||
assert.equal(third.stats.shallowCacheHits, 1);
|
||||
assert.ok(
|
||||
third.assetsById.get("tank_skn")?.some((d) => d.type === "W3DContainer"),
|
||||
"re-scanned w3x asset present",
|
||||
);
|
||||
|
||||
// Invalidate one XML file: exactly its records are re-extracted.
|
||||
recordsCache.invalidate(join(project, "Data", "Includes", "Units.xml"));
|
||||
const fourth = await new ModIndexer({ ...opts(), trustUnchanged: true }).build();
|
||||
assert.equal(fourth.stats.recordsCacheHits, third.stats.recordsCacheHits - 1);
|
||||
assert.ok(
|
||||
fourth.assetsById.get("testtank")?.some((d) => d.type === "GameObject"),
|
||||
"re-parsed XML asset present",
|
||||
);
|
||||
|
||||
// A forced rebuild (ra3modxml.reindex) verifies stats but still reuses
|
||||
// content caches for unchanged files.
|
||||
const forced = await new ModIndexer({ ...opts(), trustUnchanged: false }).build();
|
||||
assert.equal(forced.stats.shallowScannedFiles, 0);
|
||||
assert.equal(forced.stats.shallowCacheHits, 2);
|
||||
});
|
||||
|
||||
test("index stats include candidate/walk phase timings", async () => {
|
||||
const idx = await buildIndex();
|
||||
assert.equal(typeof idx.stats.candidatesMs, "number");
|
||||
assert.equal(typeof idx.stats.walkMs, "number");
|
||||
});
|
||||
|
||||
test("w3x with a UTF-8 BOM is indexed with correct offsets", async (t) => {
|
||||
const tmp = fs.mkdtempSync(join(os.tmpdir(), "ra3modxml-bom-"));
|
||||
t.after(() => fs.rmSync(tmp, { recursive: true, force: true }));
|
||||
const projectDir = join(tmp, "project");
|
||||
fs.mkdirSync(join(projectDir, "Data", "Models"), { recursive: true });
|
||||
fs.writeFileSync(
|
||||
join(projectDir, "Data", "Mod.xml"),
|
||||
`<?xml version="1.0" encoding="utf-8"?>
|
||||
<AssetDeclaration>
|
||||
<Includes>
|
||||
<Include type="all" source="Models/Tank_BOM.w3x"/>
|
||||
</Includes>
|
||||
</AssetDeclaration>`,
|
||||
"utf8",
|
||||
);
|
||||
fs.writeFileSync(
|
||||
join(projectDir, "Data", "Models", "Tank_BOM.w3x"),
|
||||
"\uFEFF<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" +
|
||||
"<AssetDeclaration>\n" +
|
||||
" <W3DContainer id=\"Tank_BOM\" />\n" +
|
||||
"</AssetDeclaration>\n",
|
||||
"utf8",
|
||||
);
|
||||
|
||||
const idx = await new ModIndexer({
|
||||
projectDir,
|
||||
sdkDir: sdk,
|
||||
builtmodsDirs: [join(sdk, "builtmods")],
|
||||
indexSageXml: false,
|
||||
additionalDataSearchPaths: [],
|
||||
walker: new CachedDirectoryWalker(),
|
||||
}).build();
|
||||
|
||||
const def = idx.assetsById.get("tank_bom")?.find((d) => d.type === "W3DContainer");
|
||||
assert.ok(def, "BOM-prefixed w3x asset indexed");
|
||||
assert.equal(def.line, 3, "id line is correct despite the BOM");
|
||||
});
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { parseXml, LineMap } from "../out/language/xmlParser.js";
|
||||
import { extractIndexRecords, recordsFromShallow } from "../out/indexer/records.js";
|
||||
import { scanXmlShallow } from "../out/indexer/shallowScan.js";
|
||||
|
||||
test("extractIndexRecords mirrors the walk semantics", () => {
|
||||
const text = `<?xml version="1.0"?>
|
||||
<AssetDeclaration>
|
||||
<Defines>
|
||||
<Define name="HP" value="100"/>
|
||||
</Defines>
|
||||
<Includes>
|
||||
<Include type="all" source="Units.xml"/>
|
||||
<Include type="instance" source="Base.xml"/>
|
||||
</Includes>
|
||||
<GameObject id="Tank" CommandSet="TankCommandSet"/>
|
||||
<WeaponTemplate id="TankGun"/>
|
||||
<xi:include href="DATA:Extra.xml" xpointer="xmlns(n=uri:ea.com:eala:asset) xpointer(/n:Extra/child::*)"/>
|
||||
<GameObject id="Tank2">
|
||||
<Draws>
|
||||
<xi:include href="DATA:Nested.xml"/>
|
||||
</Draws>
|
||||
</GameObject>
|
||||
</AssetDeclaration>`;
|
||||
const lineMap = new LineMap(text);
|
||||
const records = extractIndexRecords(parseXml(text), lineMap);
|
||||
assert.deepEqual(
|
||||
records.assets.map((a) => [a.type, a.id, a.line]),
|
||||
[
|
||||
["GameObject", "Tank", 10],
|
||||
["WeaponTemplate", "TankGun", 11],
|
||||
["GameObject", "Tank2", 13],
|
||||
],
|
||||
);
|
||||
assert.deepEqual(
|
||||
records.defines.map((d) => [d.name, d.value]),
|
||||
[["HP", "100"]],
|
||||
);
|
||||
assert.deepEqual(
|
||||
records.includes.map((i) => [i.type, i.source]),
|
||||
[
|
||||
["all", "Units.xml"],
|
||||
["instance", "Base.xml"],
|
||||
],
|
||||
);
|
||||
assert.deepEqual(
|
||||
records.rootXiIncludes.map((x) => [x.href, x.line]),
|
||||
[["DATA:Extra.xml", 12]],
|
||||
);
|
||||
assert.deepEqual(
|
||||
records.nestedXiIncludes.map((x) => [x.href, x.line]),
|
||||
[["DATA:Nested.xml", 15]],
|
||||
);
|
||||
});
|
||||
|
||||
test("recordsFromShallow converts offsets to 1-based lines", () => {
|
||||
const text = `<AssetDeclaration>\n <W3DContainer id="A"/>\n</AssetDeclaration>`;
|
||||
const lineMap = new LineMap(text);
|
||||
const records = recordsFromShallow(scanXmlShallow(text), lineMap);
|
||||
assert.equal(records.assets.length, 1);
|
||||
assert.equal(records.assets[0].type, "W3DContainer");
|
||||
assert.equal(records.assets[0].id, "A");
|
||||
assert.equal(records.assets[0].line, 2);
|
||||
});
|
||||
+147
-5
@@ -4,7 +4,15 @@ import { fileURLToPath } from "node:url";
|
||||
import { dirname, join } from "node:path";
|
||||
import { ModIndexer } from "../out/indexer/indexer.js";
|
||||
import { CachedDirectoryWalker } from "../out/indexer/fileScanner.js";
|
||||
import { isReferenceAttribute, resolveReferenceTargets } from "../out/indexer/refs.js";
|
||||
import {
|
||||
isLocalReferenceAttribute,
|
||||
isReferenceAttribute,
|
||||
isReferenceAttributeOfType,
|
||||
resolveReferenceTargets,
|
||||
resolveReferenceTargetsForType,
|
||||
} from "../out/indexer/refs.js";
|
||||
import { parseXml } from "../out/language/xmlParser.js";
|
||||
import { resolveElementType } from "../out/language/typeContext.js";
|
||||
import * as model from "../out/model/schemaModel.js";
|
||||
|
||||
const root = dirname(dirname(fileURLToPath(import.meta.url)));
|
||||
@@ -63,10 +71,21 @@ test("isReferenceAttribute distinguishes references from enums/paths", () => {
|
||||
assert.equal(isReferenceAttribute("FireWeaponNugget", "WeaponName"), true);
|
||||
});
|
||||
|
||||
test("untyped references (isRef without refType) resolve to any declared id", () => {
|
||||
// LocomotorSet/@Locomotor has type AssetReference: xas:isRef="true" with
|
||||
// no refType. It must be treated as a reference and match the id regardless
|
||||
// of the target asset type.
|
||||
test("attribute-level xas:refType is preserved in the model", () => {
|
||||
// <xs:attribute name="Locomotor" type="AssetReference"
|
||||
// xas:refType="LocomotorTemplate" /> — the refType lives on
|
||||
// the attribute node, not on the AssetReference simple type.
|
||||
assert.equal(
|
||||
model.attributesOfElement("LocomotorSet").find((a) => a.name === "Locomotor")?.refType,
|
||||
"LocomotorTemplate",
|
||||
);
|
||||
assert.equal(
|
||||
model.attributesOfElement("ArmorSet").find((a) => a.name === "Armor")?.refType,
|
||||
"ArmorTemplate",
|
||||
);
|
||||
});
|
||||
|
||||
test("typed Locomotor references resolve only to LocomotorTemplate defs", () => {
|
||||
const idx = {
|
||||
assetsById: new Map([
|
||||
[
|
||||
@@ -79,6 +98,14 @@ test("untyped references (isRef without refType) resolve to any declared id", ()
|
||||
line: 5,
|
||||
origin: "project",
|
||||
},
|
||||
{
|
||||
// Same id under a different type must NOT satisfy the reference.
|
||||
type: "GameObject",
|
||||
id: "AlliedAntiVehicleVehicleTech1Locomotor",
|
||||
file: "GameObject.xml",
|
||||
line: 1,
|
||||
origin: "project",
|
||||
},
|
||||
],
|
||||
],
|
||||
]),
|
||||
@@ -100,3 +127,118 @@ test("untyped references (isRef without refType) resolve to any declared id", ()
|
||||
// A non-reference attribute still returns nothing.
|
||||
assert.equal(resolveReferenceTargets(idx, "LocomotorSet", "EditorName", "X").length, 0);
|
||||
});
|
||||
|
||||
test("module ids declare the element itself and are never global references", () => {
|
||||
// The exact reported scenario: <TruckDraw id="ModuleTag_Draw"/> inside a
|
||||
// GameObject. ModuleData@id carries xas:refType="ModuleData", which is the
|
||||
// element's own base type -> a definition site, not a reference.
|
||||
const xml = `
|
||||
<AssetDeclaration>
|
||||
<GameObject id="GuardianTank">
|
||||
<Draws>
|
||||
<TruckDraw id="ModuleTag_Draw" />
|
||||
</Draws>
|
||||
</GameObject>
|
||||
</AssetDeclaration>`;
|
||||
const doc = parseXml(xml);
|
||||
const truckDraw = doc.elements.find((e) => e.name === "TruckDraw");
|
||||
assert.ok(truckDraw);
|
||||
const elType = resolveElementType(truckDraw);
|
||||
assert.equal(elType, "W3DTruckDrawModuleData");
|
||||
assert.equal(isLocalReferenceAttribute(elType, "id"), true);
|
||||
assert.equal(isReferenceAttributeOfType(elType, "id"), false);
|
||||
const idx = { assetsById: new Map(), assets: new Map(), defines: new Map() };
|
||||
assert.equal(
|
||||
resolveReferenceTargetsForType(idx, elType, "id", "ModuleTag_Draw").length,
|
||||
0,
|
||||
);
|
||||
});
|
||||
|
||||
test("Poid attributes reference pipeline-local objects, not global assets", () => {
|
||||
// AttachModuleId / ModuleId / AutoResolveBody are Poid-typed references to
|
||||
// modules inside the same GameObject; the global index cannot judge them.
|
||||
assert.equal(isReferenceAttributeOfType("AttachNugget", "AttachModuleId"), false);
|
||||
assert.equal(isLocalReferenceAttribute("AttachNugget", "AttachModuleId"), true);
|
||||
const idx = { assetsById: new Map(), assets: new Map(), defines: new Map() };
|
||||
assert.equal(
|
||||
resolveReferenceTargetsForType(idx, "AttachNugget", "AttachModuleId", "ModuleTag_X").length,
|
||||
0,
|
||||
);
|
||||
});
|
||||
|
||||
test("cross-type id references (RoadObject@id -> Road) stay real references", () => {
|
||||
// RoadObject declares its id with xas:refType="Road" (an AssetReference,
|
||||
// NOT the element's own type), so it must keep reference semantics.
|
||||
assert.equal(
|
||||
model.attributesOfType("RoadObject").find((a) => a.name === "id")?.refType,
|
||||
"Road",
|
||||
);
|
||||
assert.equal(isReferenceAttributeOfType("RoadObject", "id"), true);
|
||||
const idx = {
|
||||
assetsById: new Map([
|
||||
[
|
||||
"road01",
|
||||
[
|
||||
{
|
||||
type: "Road",
|
||||
id: "Road01",
|
||||
file: "Roads.xml",
|
||||
line: 3,
|
||||
origin: "project",
|
||||
},
|
||||
],
|
||||
],
|
||||
]),
|
||||
assets: new Map(),
|
||||
projectDir: ".",
|
||||
sdkDir: ".",
|
||||
defines: new Map(),
|
||||
files: new Map(),
|
||||
streams: [],
|
||||
manifests: new Map(),
|
||||
sourceCandidates: [],
|
||||
diagnostics: [],
|
||||
stats: {},
|
||||
};
|
||||
const targets = resolveReferenceTargetsForType(idx, "RoadObject", "id", "Road01");
|
||||
assert.equal(targets.length, 1);
|
||||
assert.equal(targets[0].def.type, "Road");
|
||||
});
|
||||
|
||||
test("top-level asset ids with plain pipeline ids are not references", () => {
|
||||
// LocomotorSet@id / ArmorSet@id are Poid pipeline ids without a refType:
|
||||
// they identify the element and never need a global definition.
|
||||
assert.equal(isReferenceAttributeOfType("LocomotorSet", "id"), false);
|
||||
assert.equal(isReferenceAttributeOfType("ArmorTemplateSet", "id"), false);
|
||||
const idx = { assetsById: new Map(), assets: new Map(), defines: new Map() };
|
||||
assert.equal(
|
||||
resolveReferenceTargetsForType(idx, "LocomotorSet", "id", "AnyLocalId").length,
|
||||
0,
|
||||
);
|
||||
});
|
||||
|
||||
test("xi:include elements are outside the XSD model and unvalidated", () => {
|
||||
// The nested XInclude inside a GameObject draw list (HeadlightDraw2.xml)
|
||||
// must resolve to no XSD type, so href/xpointer can never be flagged as
|
||||
// unknown attributes.
|
||||
const xml = `
|
||||
<AssetDeclaration>
|
||||
<GameObject id="GuardianTank">
|
||||
<Draws>
|
||||
<xi:include
|
||||
href="DATA:Includes/HeadlightDraw2.xml"
|
||||
xpointer="xmlns(n=uri:ea.com:eala:asset) xpointer(/n:HeadlightDraw2/child::*)" />
|
||||
</Draws>
|
||||
</GameObject>
|
||||
</AssetDeclaration>`;
|
||||
const doc = parseXml(xml);
|
||||
const inc = doc.elements.find((e) => e.name === "xi:include");
|
||||
assert.ok(inc);
|
||||
assert.equal(model.isXsdElementName(inc.name), false);
|
||||
assert.equal(resolveElementType(inc), null);
|
||||
// The XInclude attributes themselves are unprefixed (model-valid name
|
||||
// shape); the element-level foreign check is what keeps them quiet.
|
||||
for (const a of inc.attrs) {
|
||||
assert.equal(model.isXsdAttributeName(a.name), true, a.name);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -17,12 +17,84 @@ test("GameObject has expected attributes", () => {
|
||||
assert.ok(attrs.some((a) => a.name === "inheritFrom"));
|
||||
});
|
||||
|
||||
test("attribute-level xas:refType is captured (module ids, map objects)", () => {
|
||||
// ModuleData@id is declared as <xs:attribute name="id" type="Poid"
|
||||
// xas:refType="ModuleData" />; the refType must reach every module subtype.
|
||||
const moduleId = model.attributesOfType("ModuleData").find((a) => a.name === "id");
|
||||
assert.equal(moduleId?.refType, "ModuleData");
|
||||
assert.equal(
|
||||
model.attributesOfType("W3DTruckDrawModuleData").find((a) => a.name === "id")?.refType,
|
||||
"ModuleData",
|
||||
);
|
||||
// MapObject@id declares itself; ThingTemplate references a GameObject.
|
||||
assert.equal(
|
||||
model.attributesOfType("MapObject").find((a) => a.name === "id")?.refType,
|
||||
"MapObject",
|
||||
);
|
||||
assert.equal(
|
||||
model.attributesOfType("MapObject").find((a) => a.name === "ThingTemplate")?.refType,
|
||||
"GameObject",
|
||||
);
|
||||
// RoadObject@id references a different global asset type (Road).
|
||||
assert.equal(
|
||||
model.attributesOfType("RoadObject").find((a) => a.name === "id")?.refType,
|
||||
"Road",
|
||||
);
|
||||
// Poid pipeline-local references keep their typed targets.
|
||||
assert.equal(
|
||||
model.attributesOfType("AttachNugget").find((a) => a.name === "AttachModuleId")?.refType,
|
||||
"ModuleData",
|
||||
);
|
||||
});
|
||||
|
||||
test("Include has reference/instance/all enum", () => {
|
||||
const attrs = model.attributesOfElement("Include");
|
||||
const type = attrs.find((a) => a.name === "type");
|
||||
assert.deepEqual(type?.enumValues, ["reference", "instance", "all"]);
|
||||
});
|
||||
|
||||
test("xs:list types expose their item enum and isList flag", () => {
|
||||
const expected = [
|
||||
"GROUND", "WATER", "CLIFF", "AIR", "RUBBLE", "OBSTACLE", "IMPASSABLE",
|
||||
"DEEP_WATER", "WALL_RAILING", "CRUSHABLE_OBSTACLE", "CRUSHABLE_WALL",
|
||||
];
|
||||
const flags = model.typeInfo("LocomotorSurfaceBitFlags");
|
||||
assert.equal(flags.isList, true);
|
||||
assert.deepEqual(flags.enumValues, expected);
|
||||
const surfaces = model
|
||||
.attributesOfType("LocomotorTemplate")
|
||||
.find((a) => a.name === "Surfaces");
|
||||
assert.equal(surfaces.isList, true);
|
||||
assert.deepEqual(surfaces.enumValues, expected);
|
||||
});
|
||||
|
||||
test("numeric list types do not invent enums", () => {
|
||||
const list = model.typeInfo("SageUnsignedIntList");
|
||||
assert.equal(list.isList, true);
|
||||
assert.deepEqual(list.enumValues, []);
|
||||
// Plain restriction enums keep isList=false.
|
||||
const plain = model.typeInfo("Surface");
|
||||
assert.equal(plain.isList, false);
|
||||
assert.ok(plain.enumValues.length > 0);
|
||||
});
|
||||
|
||||
test("foreign namespaces are excluded from XSD validation", () => {
|
||||
// XInclude elements (xi:include / xi:fallback) come from the W3C XInclude
|
||||
// namespace, not from the EA asset XSD.
|
||||
assert.equal(model.isXsdElementName("xi:include"), false);
|
||||
assert.equal(model.isXsdElementName("xi:fallback"), false);
|
||||
assert.equal(model.isXsdElementName("GameObject"), true);
|
||||
assert.equal(model.isXsdElementName("AssetDeclaration"), true);
|
||||
// EA schema attributes are unprefixed; prefixed attributes (xai:, xi:,
|
||||
// xlink:, xml:, xsi:, xmlns:*) are namespace machinery.
|
||||
assert.equal(model.isXsdAttributeName("href"), true);
|
||||
assert.equal(model.isXsdAttributeName("xpointer"), true);
|
||||
assert.equal(model.isXsdAttributeName("xai:joinAction"), false);
|
||||
assert.equal(model.isXsdAttributeName("xlink:href"), false);
|
||||
assert.equal(model.isXsdAttributeName("xml:space"), false);
|
||||
assert.equal(model.isXsdAttributeName("xmlns:xi"), false);
|
||||
});
|
||||
|
||||
test("type assignability follows inheritance", () => {
|
||||
assert.ok(model.isAssignableTo("GameObject", "BaseInheritableAsset"));
|
||||
assert.ok(model.isAssignableTo("GameObject", "GameObject"));
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { createRequire } from "node:module";
|
||||
|
||||
// Minimal vscode shim so the compiled semantic tokens provider can run under
|
||||
// plain node. Only the APIs used by the provider call path are implemented.
|
||||
class Range {
|
||||
constructor(startLine, startCharacter, endLine, endCharacter) {
|
||||
this.start = { line: startLine, character: startCharacter };
|
||||
this.end = { line: endLine, character: endCharacter };
|
||||
}
|
||||
}
|
||||
|
||||
class SemanticTokensLegend {
|
||||
constructor(tokenTypes) {
|
||||
this.tokenTypes = tokenTypes;
|
||||
}
|
||||
}
|
||||
|
||||
class SemanticTokens {
|
||||
constructor(data) {
|
||||
this.data = data;
|
||||
}
|
||||
}
|
||||
|
||||
class SemanticTokensBuilder {
|
||||
constructor(legend) {
|
||||
this.legend = legend;
|
||||
this.pushed = [];
|
||||
}
|
||||
push(range, tokenType) {
|
||||
this.pushed.push({ range, tokenType });
|
||||
}
|
||||
build() {
|
||||
return new SemanticTokens(new Uint32Array(this.pushed.length * 5));
|
||||
}
|
||||
}
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const Module = require("module");
|
||||
const origResolve = Module._resolveFilename;
|
||||
Module._resolveFilename = function (request, ...args) {
|
||||
if (request === "vscode") return "vscode-stub";
|
||||
return origResolve.call(this, request, ...args);
|
||||
};
|
||||
require.cache["vscode-stub"] = {
|
||||
id: "vscode-stub",
|
||||
filename: "vscode-stub",
|
||||
loaded: true,
|
||||
exports: {
|
||||
Range,
|
||||
SemanticTokens,
|
||||
SemanticTokensBuilder,
|
||||
SemanticTokensLegend,
|
||||
},
|
||||
};
|
||||
|
||||
const { parseXml } = require("../out/language/xmlParser.js");
|
||||
const { buildSemanticTokenRanges } = require("../out/language/semanticTokens.js");
|
||||
const { Ra3SemanticTokensProvider } = require("../out/features/semanticTokens.js");
|
||||
|
||||
function lineStartsOf(text) {
|
||||
const starts = [0];
|
||||
for (let i = 0; i < text.length; i++) {
|
||||
if (text.charCodeAt(i) === 10) starts.push(i + 1);
|
||||
}
|
||||
return starts;
|
||||
}
|
||||
|
||||
function sliceAt(text, token) {
|
||||
const starts = lineStartsOf(text);
|
||||
const offset = starts[token.line] + token.startChar;
|
||||
return text.slice(offset, offset + token.length);
|
||||
}
|
||||
|
||||
function makeDocument(text) {
|
||||
const lineStarts = lineStartsOf(text);
|
||||
return {
|
||||
getText: () => text,
|
||||
offsetAt: (pos) => lineStarts[pos.line] + pos.character,
|
||||
positionAt: (offset) => {
|
||||
let lo = 0;
|
||||
let hi = lineStarts.length - 1;
|
||||
while (lo < hi) {
|
||||
const mid = (lo + hi + 1) >> 1;
|
||||
if (lineStarts[mid] <= offset) lo = mid;
|
||||
else hi = mid - 1;
|
||||
}
|
||||
return { line: lo, character: offset - lineStarts[lo] };
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const MALFORMED =
|
||||
`<AssetDeclaration>\n <LocomotorTemplate id="x" Surfaces="GROUND>\n <Other/>\n</LocomotorTemplate>\n</AssetDeclaration>`;
|
||||
|
||||
test("fallback tokens cover names, attributes and values", () => {
|
||||
const doc = parseXml(MALFORMED);
|
||||
assert.ok(doc.errors.length > 0);
|
||||
const tokens = buildSemanticTokenRanges(doc, MALFORMED);
|
||||
|
||||
// Element names (opening and closing tags).
|
||||
assert.ok(tokens.some((t) => t.tokenType === "type" && sliceAt(MALFORMED, t) === "AssetDeclaration"));
|
||||
assert.ok(tokens.some((t) => t.tokenType === "type" && sliceAt(MALFORMED, t) === "LocomotorTemplate"));
|
||||
assert.ok(tokens.some((t) => t.tokenType === "type" && sliceAt(MALFORMED, t) === "Other"));
|
||||
assert.ok(
|
||||
tokens.some(
|
||||
(t) => t.tokenType === "type" && sliceAt(MALFORMED, t) === "LocomotorTemplate" && t.line === 3,
|
||||
),
|
||||
);
|
||||
|
||||
// Attribute names.
|
||||
assert.ok(tokens.some((t) => t.tokenType === "property" && sliceAt(MALFORMED, t) === "Surfaces"));
|
||||
assert.ok(tokens.some((t) => t.tokenType === "property" && sliceAt(MALFORMED, t) === "id"));
|
||||
|
||||
// Attribute values: closed quotes include both quotes; the unterminated
|
||||
// Surfaces value keeps the opening quote and everything up to the recovered
|
||||
// line end (the `>` is still inside the unclosed string).
|
||||
const strings = tokens.filter((t) => t.tokenType === "string").map((t) => sliceAt(MALFORMED, t));
|
||||
assert.ok(strings.includes('"x"'));
|
||||
assert.ok(strings.includes('"GROUND>'));
|
||||
|
||||
// Tokens are sorted by position for the vscode encoder.
|
||||
for (let i = 1; i < tokens.length; i++) {
|
||||
const a = tokens[i - 1];
|
||||
const b = tokens[i];
|
||||
assert.ok(a.line < b.line || (a.line === b.line && a.startChar <= b.startChar));
|
||||
}
|
||||
});
|
||||
|
||||
test("well-formed XML gets no semantic fallback tokens", async () => {
|
||||
const text = `<AssetDeclaration>\n <LocomotorTemplate id="x" Surfaces="GROUND"/>\n</AssetDeclaration>`;
|
||||
const provider = new Ra3SemanticTokensProvider();
|
||||
const result = await provider.provideDocumentSemanticTokens(makeDocument(text), {});
|
||||
assert.equal(result.data.length, 0);
|
||||
});
|
||||
|
||||
test("malformed XML gets semantic fallback tokens", async () => {
|
||||
const provider = new Ra3SemanticTokensProvider();
|
||||
const result = await provider.provideDocumentSemanticTokens(
|
||||
makeDocument(MALFORMED),
|
||||
{},
|
||||
);
|
||||
assert.ok(result.data.length > 0);
|
||||
});
|
||||
@@ -0,0 +1,122 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { scanXmlShallow } from "../out/indexer/shallowScan.js";
|
||||
|
||||
test("extracts top-level assets, includes, defines and xi:include", () => {
|
||||
const text = `<?xml version="1.0"?>
|
||||
<AssetDeclaration xmlns="uri:ea.com:eala:asset">
|
||||
<Defines>
|
||||
<Define name="MODEL_SCALE" value="1.0"/>
|
||||
</Defines>
|
||||
<Includes>
|
||||
<Include type="all" source="Art/Model_SKN.w3x"/>
|
||||
<Include type="reference" source="DATA:static.xml"/>
|
||||
</Includes>
|
||||
<W3DContainer id="Model_SKN" Hierarchy="Model_SKL">
|
||||
<SubObject SubObjectID="GUN">
|
||||
<RenderObject><Mesh>Model_SKN.GUN</Mesh></RenderObject>
|
||||
</SubObject>
|
||||
</W3DContainer>
|
||||
<W3DMesh id="Model_SKN.GUN"/>
|
||||
<xi:include href="DATA:Includes/Extra.xml" xpointer="xmlns(n=uri:ea.com:eala:asset) xpointer(/n:Extra/child::*)"/>
|
||||
</AssetDeclaration>`;
|
||||
const doc = scanXmlShallow(text);
|
||||
assert.equal(doc.errors.length, 0);
|
||||
assert.deepEqual(
|
||||
doc.assets.map((a) => `${a.name}:${a.id}`),
|
||||
["W3DContainer:Model_SKN", "W3DMesh:Model_SKN.GUN"],
|
||||
);
|
||||
assert.equal(doc.includes.length, 2);
|
||||
assert.equal(doc.includes[0].type, "all");
|
||||
assert.equal(doc.includes[0].source, "Art/Model_SKN.w3x");
|
||||
assert.equal(doc.includes[1].type, "reference");
|
||||
assert.equal(doc.defines.length, 1);
|
||||
assert.equal(doc.defines[0].name, "MODEL_SCALE");
|
||||
assert.equal(doc.defines[0].value, "1.0");
|
||||
assert.equal(doc.rootXiIncludes.length, 1);
|
||||
assert.equal(doc.rootXiIncludes[0].href, "DATA:Includes/Extra.xml");
|
||||
assert.match(doc.rootXiIncludes[0].xpointer, /xpointer\(/);
|
||||
assert.equal(doc.nestedXiIncludes.length, 0);
|
||||
|
||||
// Recorded offsets must slice the original text back out.
|
||||
const container = doc.assets[0];
|
||||
assert.equal(text.slice(container.idValueStart, container.idValueEnd), "Model_SKN");
|
||||
assert.equal(text.slice(container.start, container.startTagEnd).startsWith("<W3DContainer"), true);
|
||||
const mesh = doc.assets[1];
|
||||
assert.equal(text.slice(mesh.idValueStart, mesh.idValueEnd), "Model_SKN.GUN");
|
||||
});
|
||||
|
||||
test("handles > and / inside quoted values, self-closing tags and comments", () => {
|
||||
const text = `<?xml version="1.0"?>
|
||||
<!-- a > comment with < inside -->
|
||||
<AssetDeclaration xmlns="uri:ea.com:eala:asset">
|
||||
<W3DContainer id="Weird" Description="a > b < c" />
|
||||
<W3DMesh id="X" VertexData="a / b"/>
|
||||
</AssetDeclaration>`;
|
||||
const doc = scanXmlShallow(text);
|
||||
assert.equal(doc.errors.length, 0);
|
||||
assert.deepEqual(
|
||||
doc.assets.map((a) => a.id),
|
||||
["Weird", "X"],
|
||||
);
|
||||
});
|
||||
|
||||
test("skips CDATA and processing instructions", () => {
|
||||
const text = `<AssetDeclaration>
|
||||
<!-- <W3DMesh id="FAKE"/> -->
|
||||
<![CDATA[ <W3DMesh id="FAKE2"/> ]]>
|
||||
<?xml-stylesheet href="x"?>
|
||||
<W3DContainer id="REAL"/>
|
||||
</AssetDeclaration>`;
|
||||
const doc = scanXmlShallow(text);
|
||||
assert.equal(doc.errors.length, 0);
|
||||
assert.deepEqual(
|
||||
doc.assets.map((a) => a.id),
|
||||
["REAL"],
|
||||
);
|
||||
});
|
||||
|
||||
test("reports unterminated tags and keeps partial results", () => {
|
||||
const doc = scanXmlShallow('<AssetDeclaration><W3DMesh id="A"/><W3DMesh id="B"');
|
||||
assert.ok(doc.errors.length >= 1);
|
||||
assert.deepEqual(
|
||||
doc.assets.map((a) => a.id),
|
||||
["A"],
|
||||
);
|
||||
});
|
||||
|
||||
test("nested module payload does not create asset records", () => {
|
||||
// Only top-level elements with an id are assets; the hundreds of thousands
|
||||
// of numeric V/T elements inside a mesh are not.
|
||||
const doc = scanXmlShallow(`<AssetDeclaration>
|
||||
<W3DMesh id="MESH">
|
||||
<Vertices><V X="1" Y="2" Z="3"/><V X="4" Y="5" Z="6"/></Vertices>
|
||||
<Triangles><T A="0" B="1" C="2"/></Triangles>
|
||||
</W3DMesh>
|
||||
</AssetDeclaration>`);
|
||||
assert.deepEqual(
|
||||
doc.assets.map((a) => a.id),
|
||||
["MESH"],
|
||||
);
|
||||
});
|
||||
|
||||
test("distinguishes root-level and nested xi:include", () => {
|
||||
const doc = scanXmlShallow(`<AssetDeclaration>
|
||||
<xi:include href="DATA:Top.xml"/>
|
||||
<W3DMesh id="MESH">
|
||||
<SubObject>
|
||||
<RenderObject>
|
||||
<xi:include href="DATA:Nested.xml" xpointer="xmlns(n=uri:ea.com:eala:asset) xpointer(/n:X/child::*)"/>
|
||||
</RenderObject>
|
||||
</SubObject>
|
||||
</W3DMesh>
|
||||
</AssetDeclaration>`);
|
||||
assert.deepEqual(
|
||||
doc.rootXiIncludes.map((x) => x.href),
|
||||
["DATA:Top.xml"],
|
||||
);
|
||||
assert.deepEqual(
|
||||
doc.nestedXiIncludes.map((x) => x.href),
|
||||
["DATA:Nested.xml"],
|
||||
);
|
||||
});
|
||||
+32
-1
@@ -1,6 +1,11 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { parseXml, findElementAt } from "../out/language/xmlParser.js";
|
||||
import { parseXml, findElementAt, stripBom } from "../out/language/xmlParser.js";
|
||||
|
||||
test("stripBom removes a leading UTF-8 byte-order mark", () => {
|
||||
assert.equal(stripBom("\uFEFF<A/>"), "<A/>");
|
||||
assert.equal(stripBom("<A/>"), "<A/>");
|
||||
});
|
||||
|
||||
test("parses elements, attributes and positions", () => {
|
||||
const text = `<AssetDeclaration xmlns="uri:ea.com:eala:asset">\n\t<GameObject id="X" KindOf="A B"/>\n</AssetDeclaration>`;
|
||||
@@ -45,3 +50,29 @@ test("tolerates partial input while typing", () => {
|
||||
assert.ok(doc.errors.length >= 1); // unclosed
|
||||
assert.equal(doc.elements.length, 2);
|
||||
});
|
||||
|
||||
test("recovers from an unterminated attribute value at end of line", () => {
|
||||
// Typing an attribute value quote without its closing quote makes the tag
|
||||
// malformed; the parser must stop the broken start tag at the line break so
|
||||
// the rest of the document (and completion for it) keeps working.
|
||||
const text = `<AssetDeclaration>\n <A x="1" y="abc\n <B/>\n </A>\n</AssetDeclaration>`;
|
||||
const doc = parseXml(text);
|
||||
assert.ok(doc.errors.some((e) => e.message === "Unterminated start tag"));
|
||||
assert.deepEqual(doc.elements.map((e) => e.name), ["AssetDeclaration", "A", "B"]);
|
||||
const a = doc.elements.find((e) => e.name === "A");
|
||||
const y = a.attrs.find((at) => at.name === "y");
|
||||
assert.equal(y.quoteEnd, -1);
|
||||
assert.equal(text.slice(y.valueStart, y.valueEnd), "abc");
|
||||
const b = doc.elements.find((e) => e.name === "B");
|
||||
assert.ok(b.start > a.start);
|
||||
});
|
||||
|
||||
test("reports an unterminated attribute value at EOF", () => {
|
||||
const text = `<A x="abc`;
|
||||
const doc = parseXml(text);
|
||||
assert.ok(doc.errors.some((e) => /Unterminated start tag/.test(e.message)));
|
||||
assert.equal(doc.elements.length, 1);
|
||||
const a = doc.elements[0];
|
||||
assert.equal(a.attrs[0].value, "abc");
|
||||
assert.equal(a.attrs[0].quoteEnd, -1);
|
||||
});
|
||||
|
||||
+45
-1
@@ -168,6 +168,7 @@ function resolveTypeDescriptor(typeName, memo = new Map()) {
|
||||
name,
|
||||
refType: null,
|
||||
enumValues: [],
|
||||
isList: false,
|
||||
allowsDefine: false,
|
||||
isBoolean: name === "boolean",
|
||||
base: null,
|
||||
@@ -180,6 +181,37 @@ function resolveTypeDescriptor(typeName, memo = new Map()) {
|
||||
const st = simpleTypes.get(name);
|
||||
if (st) {
|
||||
memo.set(name, null); // guard against cycles
|
||||
const listNode = st?.list;
|
||||
if (listNode) {
|
||||
// xs:list types (bit flags such as LocomotorSurfaceBitFlags, AssetIdList,
|
||||
// PercentageList, ...) serialize as whitespace-separated items. The
|
||||
// usable enum/ref semantics come from the item type, so inherit them.
|
||||
let itemDesc = null;
|
||||
let inlineEnum = [];
|
||||
let inlineAllowsDefine = false;
|
||||
const itemTypeName = normalizeTypeName(listNode?.["@_itemType"]);
|
||||
if (itemTypeName) {
|
||||
itemDesc = resolveTypeDescriptor(itemTypeName, memo);
|
||||
} else if (listNode.simpleType) {
|
||||
inlineEnum = enumOfSimpleType(listNode.simpleType);
|
||||
const pats = patternOfSimpleType(listNode.simpleType);
|
||||
inlineAllowsDefine = pats ? pats.some(patternAllowsDefine) : false;
|
||||
}
|
||||
const desc = {
|
||||
kind: "simple",
|
||||
name,
|
||||
refType: itemDesc?.refType ?? null,
|
||||
enumValues: itemDesc?.enumValues?.length ? itemDesc.enumValues : inlineEnum,
|
||||
isList: true,
|
||||
allowsDefine: (itemDesc?.allowsDefine ?? false) || inlineAllowsDefine,
|
||||
isRef: itemDesc?.isRef ?? false,
|
||||
isBoolean: false,
|
||||
base: itemDesc?.base ?? null,
|
||||
doc: docOf(st),
|
||||
};
|
||||
memo.set(name, desc);
|
||||
return desc;
|
||||
}
|
||||
const base = normalizeTypeName(st?.restriction?.["@_base"]);
|
||||
const baseDesc = base ? resolveTypeDescriptor(base, memo) : null;
|
||||
const enumValues = enumOfSimpleType(st);
|
||||
@@ -192,6 +224,7 @@ function resolveTypeDescriptor(typeName, memo = new Map()) {
|
||||
name,
|
||||
refType: normalizeTypeName(refType) || baseDesc?.refType || null,
|
||||
enumValues: enumValues.length ? enumValues : baseDesc?.enumValues ?? [],
|
||||
isList: false,
|
||||
allowsDefine:
|
||||
(patterns ? patterns.some(patternAllowsDefine) : false) ||
|
||||
baseDesc?.allowsDefine ||
|
||||
@@ -215,6 +248,7 @@ function resolveTypeDescriptor(typeName, memo = new Map()) {
|
||||
name,
|
||||
refType: null,
|
||||
enumValues: [],
|
||||
isList: false,
|
||||
allowsDefine: false,
|
||||
isRef: false,
|
||||
isBoolean: false,
|
||||
@@ -230,6 +264,7 @@ function resolveTypeDescriptor(typeName, memo = new Map()) {
|
||||
name,
|
||||
refType: null,
|
||||
enumValues: [],
|
||||
isList: false,
|
||||
allowsDefine: false,
|
||||
isRef: false,
|
||||
isBoolean: false,
|
||||
@@ -287,6 +322,7 @@ function collectAttributes(node, out = []) {
|
||||
name: `@attr:${name}`,
|
||||
refType: attr.simpleType?.["@_refType"] ?? null,
|
||||
enumValues: inlineEnum,
|
||||
isList: false,
|
||||
allowsDefine: pats ? pats.some(patternAllowsDefine) : false,
|
||||
isBoolean: false,
|
||||
base: normalizeTypeName(attr.simpleType?.restriction?.["@_base"]),
|
||||
@@ -300,8 +336,15 @@ function collectAttributes(node, out = []) {
|
||||
doc: docOf(attr),
|
||||
kind: desc?.kind ?? "unknown",
|
||||
type: desc?.name ?? null,
|
||||
refType: normalizeTypeName(desc?.refType) ?? null,
|
||||
// xas:refType may be declared on the attribute itself (e.g.
|
||||
// <xs:attribute name="id" type="Poid" xas:refType="ModuleData" />) or
|
||||
// on its simple type. The attribute-level declaration wins.
|
||||
refType:
|
||||
normalizeTypeName(attr["@_refType"]) ??
|
||||
normalizeTypeName(desc?.refType) ??
|
||||
null,
|
||||
enumValues: desc?.enumValues ?? inlineEnum ?? [],
|
||||
isList: desc?.isList ?? false,
|
||||
allowsDefine: desc?.allowsDefine ?? false,
|
||||
isRef: desc?.isRef ?? false,
|
||||
isBoolean: desc?.isBoolean ?? false,
|
||||
@@ -390,6 +433,7 @@ for (const [name, node] of simpleTypes) {
|
||||
refType: desc?.refType ?? node?.["@_refType"] ?? null,
|
||||
isRef: node?.["@_isRef"] === "true" || desc?.refType != null,
|
||||
enumValues: desc?.enumValues ?? [],
|
||||
isList: desc?.isList ?? false,
|
||||
allowsDefine: desc?.allowsDefine ?? false,
|
||||
doc: docOf(node),
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user