diff --git a/README.md b/README.md index 02d21fb..d71a59a 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,15 @@ 400 条时标记为不完整，继续输入会重新请求，因此 `CrateDebris_01` 这类排在 列表后部的 id 不会因首屏截断而消失。 - **悬停提示**：元素/属性显示 XSD 文档、类型、必填/默认值；引用值显示定义位置；`$DEFINE` 显示值与定义位置；`Include source` / `xi:include href` 显示解析后的目标文件；`xi:include` 元素与属性给出 XInclude 说明。 -- **引用导航**：从引用值（`CommandSet="..."`、`Weapon="..."`、`inheritFrom`、`ID` 等元素文本）跳转到定义（严格按引用类型过滤，候选由 `ra3modxml.definitionMode` 控制：`all` 列出 mod + 原版、`project-only` 优先项目内定义）；`Ctrl+点击` Include / `xi:include href` 打开目标文件；Find All References 同时搜索属性值与元素文本内容；文档大纲列出顶层资产与 `$DEFINE`。 +- **引用导航**：从引用值（`CommandSet="..."`、`Weapon="..."`、`inheritFrom`、`ID` 等元素文本）跳转到定义（严格按引用类型过滤，候选由 `ra3modxml.definitionMode` 控制：`all` 列出 mod + 原版、`project-only` 优先项目内定义）；`Ctrl+点击` Include / `xi:include href` 打开目标文件；Find All References 基于**语义引用索引**（属性引用 + simple-content 文本 + `inheritFrom`，排除 id 定义点 / Poid / `$DEFINE`，不再全文搜索）；文档大纲列出顶层资产与 `$DEFINE`。 +- **引用计数（CodeLens）**：在“设计上应被引用”的顶部资产类型上显示 + `0 references` / `1 reference` / `N references`（0 也显示），点击直接打开 + references peek；设置类、地图元数据、w3x 子结构等自动注册类型不显示， + 避免满屏 0；manifest 资产有对应 SageXml 源码时，引用按源码定义归并计数 + （打开 SageXml 源码同样能看到引用数）。 +- **未引用资产**：`RA3 Mod XML: Find unreferenced assets…` 命令按类型列出 + 所有零引用的项目资产并跳转；编辑器右键菜单 + `Find unreferenced assets of this type` 可直接使用光标所在资产类型。 - **当前文档局部作用域（T1）**：即使一个文件不在任何全局流里（没有从 `Data/Mod.xml` / `additionalmaps` 可达），插件也会按当前文件自身的资产、 `$DEFINE` 及其 include 链建立局部索引。`xi:include` 会在逻辑树中展开， @@ -38,13 +46,16 @@ Include 跳转/悬停照常工作；引用类诊断会“显示但标注” （`unresolved-reference-indexing` + `(index incomplete)` 说明），不会把 未完成的索引误当成最终结论。 -- **大项目性能**：索引记录（资产 / Define / Include / 行号）与 include 解析结果 - 跨重建缓存，保存触发的重建零 stat、零重读（Corona 实测约 2 秒）；DOM 树只按需 - 保留并设元素预算，避免内存膨胀。编辑器外的文件改动（git pull、导出工具）会 - 触发防抖重建；构建期间文件再次被修改时，已发布索引会标记 `(stale)` 并自动重跑。 - include 路径解析使用目录枚举建立的文件集快照（无 statSync 风暴）；records - 缓存会持久化到磁盘（gzip + 多信号 stat 校验 + 原子写），重启 VS Code 后冷启动 - 只需秒级校验，Corona 实测约 11 秒（首次全量约 2 分钟）。 +- **大项目性能**：索引记录（资产 / Define / Include / 引用 / 行号）与 include + 解析结果跨重建缓存，保存触发的重建零 stat、零重读（Corona 实测约 2 秒）； + DOM 树只按需保留并设元素预算，避免内存膨胀。编辑器外的文件改动（git pull、 + 导出工具）会触发防抖重建；构建期间文件再次被修改时，已发布索引会标记 + `(stale)` 并自动重跑。include 路径解析使用目录枚举建立的文件集快照（无 + statSync 风暴）；records 缓存会持久化到磁盘（gzip + 多信号 stat 校验 + + 内容哈希 + 原子写），重启 VS Code 后冷启动只需秒级校验，Corona 实测约 11 秒 + （首次全量约 2 分钟）。引用索引只从构建期实际消费的 records 构建；打开文档 + 时若发现当前文本的 records 与快照不一致（如外置盘重连后缓存过时），会自动 + 定向重建自愈；`Re-index workspace` 会对 stat 匹配的 XML 也做内容校验。 ## 使用 @@ -70,6 +81,9 @@ - `RA3 Mod XML: Show index report`：查看索引统计。 - `RA3 Mod XML: Clear caches and rebuild`：清空内存/磁盘缓存并强制全量重建。 - `RA3 Mod XML: Show cache report`：查看磁盘缓存路径、大小、校验统计与命中数。 +- `RA3 Mod XML: Find unreferenced assets…`：按类型查找零引用的项目资产。 +- `RA3 Mod XML: Find unreferenced assets of this type`：右键菜单入口， + 直接查找光标所在顶部资产类型的未引用资产。 ## 开发 @@ -102,17 +116,19 @@ src/ existence.ts 文件集存在性快照（目录枚举 Set，替代逐路径 statSync） manifestParser.ts .manifest 二进制解析（移植 OpenSAGE ManifestFile.cs） fileScanner.ts 目录扫描与 Include source 候选 - refs.ts 引用目标解析（按引用类型过滤） + refs.ts 引用目标解析（按引用类型过滤）+ “设计上可被引用类型”判定 + referenceIndex.ts 引用记录 → 反向引用索引（定义 → 引用位置）+ 未引用报告 xpointer.ts xi:include xpointer 子集解析（纯 TS） logicalTree.ts 当前文档逻辑树（xi:include 拼接、局部作用域） localScope.ts 文档局部索引 overlay（自身链 + include 链） shallowScan.ts .w3x 等大体积美术资产顶层浅扫描（纯 TS，不建 DOM） - records.ts 每文件紧凑索引记录（资产/Define/Include/xi + 行号） + records.ts 每文件紧凑索引记录（资产/Define/Include/xi/引用 + 行号偏移） caches.ts 跨重建持久缓存（DocumentCache / IndexRecordsCache / IncludeResolveCache）+ 失效纪元 InvalidationsEpoch diskCache.ts 跨会话磁盘缓存（gzip JSON、原子写、多信号 stat 校验） indexer.ts 工作区索引器（后台、缓存、记录驱动重建、分阶段发布） - features/ completion / hover / navigation / diagnostics / semanticTokens + features/ completion / hover / navigation / references / codeLens / + unreferenced / diagnostics / semanticTokens syntaxes/ TextMate 注入语法 tools/ XSD → 模型、AssetType 枚举提取 ``` @@ -124,4 +140,5 @@ tools/ XSD → 模型、AssetType 枚举提取 - 领域说明与需求：`docs/requirements.md` - 调研与设计决策：`docs/plan.md` - 问题分析与修复记录：`docs/analysis-issues.md` +- 引用计数 / 语义 FAR / 未引用资产功能设计：`docs/features-reference-counts.md` - Manifest 格式参考：OpenSAGE `src/OpenSage.Game/Data/StreamFS/ManifestFile.cs`（本仓库 `OpenSAGE/` 子目录，commit `d45d361`） diff --git a/docs/analysis-issues.md b/docs/analysis-issues.md index 6dcf3c7..9330dcf 100644 --- a/docs/analysis-issues.md +++ b/docs/analysis-issues.md @@ -974,7 +974,8 @@ stale=true 并触发 follow-up。处理（按用户建议的扩展名白名单 - w3x 文件名启发式定向扫描（按约定后续再做）。 - `AssetIdList` 等“任意资产 ID 列表”的引用语义建模。 -- Find All References 目前仍是全文搜索，未走索引（对应需求 P1 高效搜索）。 +- Find All References 已改为语义引用索引（第十八轮，2026-08-05，见 + `docs/features-reference-counts.md`）；通用内容搜索与索引复用仍属 P1 远期。 --- @@ -1509,3 +1510,117 @@ XSD 模型中共 **371 处“子元素是 simple type”的声明（149 个不 若 Corona 上“输入 C 后菜单出现慢”仍然明显，下一个瓶颈大概率是 `getScope()` 每次文档版本变化都重建完整局部 include 链 / 逻辑树；本轮先把 候选截断与类型过滤造成的“列表错误”修掉，局部 scope 缓存优化留作独立一轮。 + +--- + +## 二十三、问题分析（2026-08-05）：FAR 把定义行算作“自引用”；CodeLens 漏掉 manifest 源引用 + +### 问题 1：Find All References 把 id 定义行也算进结果 + +**现象**：对任意资产执行 FAR，结果里包含它自己的 `id="..."` 定义行，看起来 +像“自己引用自己”；CodeLens 计数没有这个问题。 + +**根因**：VS Code 的 FAR 默认带 `context.includeDeclaration = true`，旧实现 +把定义位置附加进返回结果。语义反向索引本身不含 id 定义点（records 提取时 +已排除），CodeLens 只读反向索引，所以两者不一致。 + +**修复**：`findReferenceLocations` 不再附加定义位置，无论 `includeDeclaration` +取值；FAR 结果与 CodeLens 计数严格一致。 + +### 问题 2：CodeLens 不显示 manifest 定义的引用，FAR 却显示 + +**现象**：打开 SageXml 原版源码时，CodeLens 显示 0 引用，但从该资产执行 +FAR 能看到引用。 + +**根因**：反向索引的站点挂在“实际解析到的定义”上。当 SageXml 源码不在 +include 遍历里时，引用只解析到 manifest 定义（如 `static.manifest` 条目）， +站点挂在 manifest key；CodeLens 用“当前文档定义 key”精确查表所以是 0， +FAR 把同 id/同类型的 manifest 定义也合并进来所以能看到。 + +**修复**：新增 `referenceSitesForDefinition`：除了精确 key，还把 +`manifestSource` 可解析到当前打开文件的 manifest 定义的站点并入计数—— +语义上把“manifest 引用”视作“SageXml 源码对该 asset 的引用” +（Go to Definition 本来就会把 manifest 定义映射到 SageXml 源码）。 + +**验证**：AttachTest 中 326 个带引用的 manifest 定义此前全部没有对应 +`origin: "sdk"` 定义（源码未遍历），现在打开对应 SageXml 源码即可看到计数 +（如 `PlayerTemplate Allies` → 8 引用）。 + +### 举一反三的测试（146 → 147 全绿） + +- `referenceProvider.test.mjs`：FAR 即使 `includeDeclaration = true` 也不返回 + 定义行；从引用位置发起 FAR 结果一致； +- `referenceIndex.test.mjs`：`referenceSitesForDefinition` 把 manifest 源 + 站点并入 SageXml 源码定义，其他文件不串； +- `codeLens.test.mjs`：打开 manifestSource 对应的源码文件时，CodeLens 显示 + manifest 定义上挂着的引用数。 + +版本 **0.1.16 → 0.1.17**。 + +--- + +## 二十四、问题分析（2026-08-06）：引用索引与 records 缓存不同步 / 竞争 + +### 现象 + +Corona `Data/GlobalData/Weapon/Weapon_Allied.xml` 中的 +`AlliedCommandoDesertEaglesWarhead`（一个 WeaponTemplate）偶尔没有 CodeLens， +FAR 显示无引用；它实际被同文件 WeaponTemplate 的 `ProjectileNugget +WarheadTemplate="..."` 引用。该问题在移动硬盘重连 + 重新打开工作区 + 校验 +磁盘缓存 + 重新 indexing 之后出现；在 ProjectileNugget 里 Ctrl+点击一次后恢复； +清缓存重载不复发。 + +### 调查 + +1. **模型核对**：`AlliedCommandoDesertEaglesWarhead` 的元素是 + ``，`ProjectileNuggetType@WarheadTemplate` 的 refType 是 + `WeaponTemplate`，语义匹配成立。 +2. **真实文件验证**：直接对 Weapon_Allied.xml 做 records 提取 + 反向索引， + 引用记录（line 1332）能正确挂到定义（line 1341）——**新鲜构建没有问题**。 +3. 因此问题不在提取/过滤逻辑，而在“引用索引与索引状态不同步”的缓存/竞争 + 路径。 + +### 找到的不同步 / 竞争点 + +- **`buildReferences()` 读共享 recordsCache**（workspace 持有、跨重建复用）， + 而不是 walk 实际消费的 records： + - 外部盘重连 / 旧磁盘缓存条目“stat 全匹配但内容过时”（FAT32/exFAT 时间戳 + 粒度 2s、同步工具保留 mtime、size 不变）→ 资产用旧 records 入库，而新 + 引用缺失；watcher 事件在重连期间丢失时快照不会标 stale → 持久化问题； + - 构建中途 watcher 失效、或 feature `readDom` 重写 recordsCache，都可能让 + 快照的 references 与 assets 来自不同版本的 records。 +- **feature 在构建中调用 `readDom`**（定义跳转的精确定位）会改写同一个 + indexer 的 `files` / `recordsCache`，污染进行中的构建。 +- **force 重建（Re-index）之前只比 stat 信号**：stat 相同但内容不同的陈旧 + 条目会被直接复用，只有清缓存才能修复。 + +### 修复 + +1. **构建期本地 records**：`ModIndexer.buildRecords` 记录本次 walk 实际消费的 + `{ file, records, recordsHash }`，`buildReferences()` 只从这里构建反向索引。 + 中途失效 / feature 重读不再造成“资产在但引用缺失”。 +2. **构建期 readDom 闸门**：`assetDefLocation` 在 `ws.isBuilding` 时退化为 + 行级位置，避免定义跳转改写进行中的构建。 +3. **force 重建内容校验**：full XML 的 records 缓存条目带 `contentHash`； + `Re-index workspace` 对 stat 匹配的条目也读文件比对哈希，不一致才重解析 + （w3x 浅扫描不读，保持 2.6 GB 免读）。 +4. **打开文档自愈**：快照发布每文件 `recordsHashes`；CodeLens / FAR 对**已保存** + 文档比较当前文本的 records 哈希，不一致则定向 `invalidate` + + `scheduleRebuild("records-desync")`。records 哈希忽略行尾/空白差异， + 未保存编辑不触发，避免误报和循环。 +5. **磁盘缓存 v2 → v3**：旧缓存没有哈希、无法校验/自愈，一次性重建后每文件 + 都带哈希。 + +### 验证（147 → 151 全绿） + +- 构建中途 invalidate 某文件的 recordsCache，最终快照仍保留该文件的引用； +- force 重建：stat 全匹配但 contentHash 不同的条目被重新解析（trusted 路径 + 仍复用缓存不读盘）； +- 自愈：records 哈希不一致时只对干净文件触发 invalidate + records-desync； +- CodeLens 集成：打开文档与快照不同步时调度定向重建。 + +“Ctrl+点击后恢复”的精确时序无法在代码里复现；最可能是跳转前后恰好发生了一 +次重建（watcher 事件 / dirty-followup）。自愈检查让这类问题不再依赖巧合： +只要文件被打开并触发 CodeLens / FAR，不一致就会被检测并定向修复。 + +版本 **0.1.17 → 0.1.18**。 diff --git a/docs/features-reference-counts.md b/docs/features-reference-counts.md new file mode 100644 index 0000000..56a5d8d --- /dev/null +++ b/docs/features-reference-counts.md @@ -0,0 +1,122 @@ +# 功能设计：引用计数、语义 Find All References 与未引用资产 + +> 状态：已实现（v0.1.16 起）。本文档记录需求分析结论、语义定义、数据流与 +> 已知边界；问题排查记录仍放在 `docs/analysis-issues.md`，功能设计单独成文。 + +## 一、需求与结论 + +原始需求： + +1. 在每个顶部 asset 上显示它被引用了多少次，点击后直接打开“查找所有引用”， + 而不是必须通过右键菜单； +2. 能查找“某一种类型中所有没被引用的顶部 asset”。 + +调研结论（2026-08-05）： + +- XSD 模型层面，292 个有模型类型的顶层资产中：73 个有类型化引用指向它们， + 28 个只能被 `inheritFrom` 指向，**191 个在模型里根本没有任何引用指向** + （设置类、地图元数据、w3x 子结构等）。对这些类型，“0 引用”是唯一正常状态。 +- AttachTest 实测：625 个项目定义中 29% 是 0 引用、48% 是 1 引用。 + 因此“所有资产一律显示计数”会制造大量噪音，也会让用户误以为资产是孤儿。 +- 结论：**计数按类型过滤**——只在“设计上应该被引用”的类型上显示 + （0 也显示，因为对 GameObject 这类类型 0 是有效信号）；自动注册/结构类 + 类型不显示。 +- 旧版 Find All References 是全文搜索（正则匹配 `"id"` / `>id<`），会把 + `id="X"` 定义本身、`EditorName="X"` 等非引用属性也算进去。计数如果和它 + 同源，必然误导；如果不同源，点击后的结果又对不上。因此本轮把 FAR 一起 + 升级为语义引用索引，计数与点击结果共用同一数据源。 + +## 二、语义定义：什么算一次“引用” + +与补全 / hover / 跳转 / 诊断完全一致（`refs.ts` 是唯一判定来源）： + +- 带 `xas:refType` 的属性值（如 `CommandSet` → `LogicCommandSet`）； +- 带 `xas:refType` 的 simple-content 文本（如 `ID`）； +- `inheritFrom`（按元素自身类型过滤）； +- 无 `refType` 的 `isRef` 属性（按同名 ID 匹配任意声明类型）。 + +不算引用： + +- 元素自己的 `id` 定义点（除非是 `RoadObject@id→Road` 这类跨类型 id 引用）； +- Poid 管线局部引用（`ModuleId`、`AttachModuleId`、`SoundRef` 等）； +- `$DEFINE` / `=` 常量值； +- 枚举、文件路径、`Include@source` / `xi:include href`； +- w3x 内部父子结构关系（`W3DMesh` 等靠结构归属，不走全局引用）。 + +引用索引按“类型 + id + 定义位置”精确归属，同名 ID 的不同类型定义互不串扰 +（`WeaponTemplate:X` 的引用不会计到 `GameObject:X` 头上）。 + +## 三、数据流 + +``` +parse DOM + │ records.ts extractIndexRecords(parse, lineMap, text) + ▼ +IndexRecords.references[] 每文件紧凑记录（refType / selfType / value / + │ 行号 / 起止偏移），随 records 缓存跨重建复用， + │ 并持久化到磁盘缓存（缓存版本 v3，full XML + │ 附带内容哈希；快照另发布每文件 records 哈希） + ▼ +indexer.ts buildReferences() 只解算本次 build 触及的文件，防止陈旧缓存泄漏 + │ referenceIndex.ts buildReferenceIndex() + ▼ +ModIndex.references Map<定义 key, ReferenceSite[]> + ├── CodeLens 计数（O(1) 查表） + ├── 语义 Find All References（返回精确位置） + └── 未引用资产报告（0 引用 = 不在反向索引中） +``` + +引用记录在解析期就把 `refType` / `selfType` 固化下来，反向索引构建时不需要 +再解析 DOM、也不需要上下文类型解析；只需对 `assetsById` 做一次查找并按类型 +过滤。快照发布时（XML 阶段 + art 阶段）各构建一次反向索引，Corona 规模下 +开销远小于 include 遍历。 + +反向索引只从**本次构建的 walk 实际消费的 records**（`ModIndexer.buildRecords`） +构建，不读共享 recordsCache：中途 watcher 失效、或 feature 通过 `readDom` +重读文件，都不会让“资产在但引用缺失”的快照出现。打开文档时 CodeLens / FAR +还会比较当前文本的 records 哈希与快照，不一致就定向 invalidate 并触发 +`records-desync` 重建自愈（仅对已保存文档，未保存编辑不触发）。 + +## 四、CodeLens 规则 + +- 只对根级（`AssetDeclaration` 直接子元素）带 `id` 的资产显示； +- 只对 `isReferenceTargetType()` 为真的类型显示（类型化引用目标 + + `inheritFrom` 可继承类型；见 `refs.ts` 的 `referenceTargetTypes()`）； +- 0 也显示：`0 references` / `1 reference` / `N references`； +- 点击执行 `ra3modxml.showReferences` → `editor.action.showReferences` + 打开 references peek，结果与计数完全一致（不含定义本身）； +- 计数除了当前定义自己的反向索引桶，还并入“manifestSource 可解析到当前 + 文件”的 manifest 定义桶：manifest 资产有对应 SageXml 源码时，引用直接 + 视作 SageXml 源码对该 asset 的引用（Go to Definition 同样把 manifest + 定义映射到 SageXml 源码）； +- 索引重建完成后自动 `editor.action.codeLens.refresh`，计数不会停留在旧值。 + +## 五、未引用资产 + +- 主入口：命令面板 `RA3 Mod XML: Find unreferenced assets…` + 第一步 QuickPick 选类型（只列“设计上应被引用”的类型，显示每种未引用数量）， + 第二步列出资产（`id — 相对路径:行号`），点击跳转到定义； +- 加速入口：编辑器右键菜单 + `RA3 Mod XML: Find unreferenced assets of this type` + （仅 `editorLangId == xml && ra3modxml.active` 时显示），光标在顶部资产上 + 时直接预选该类型，否则回退到类型选择； +- 语义：只统计 `origin === "project"` 且非 `viaInstance` 的定义； + manifest / SDK 定义永远不参与；如果某个 id 覆盖了原版资产，来自原版/其他 + 流的引用同样计入（否则会把覆盖件误报成未使用）。 + +## 六、已知边界与后续 + +- 当前文档局部作用域（不在任何全局流里的文件）的引用不在全局反向索引中： + 这类文件打开后 CodeLens / FAR 只反映全局流；局部链内的引用暂不计数。 +- Find All References 不返回定义行本身（即使 VS Code 传入 + `includeDeclaration`），因此结果数量与 CodeLens 计数严格一致。 +- `AssetIdList` 等“任意资产 ID 列表”的引用语义仍未建模：整个属性值按一条 + 引用记录处理，与现有补全/诊断保持一致（已知缺口，后续可在 records 层扩展）。 +- w3x 文件本身不提取引用记录（浅扫描无 DOM）；对 w3x 资产的引用从引用它的 + XML 文件捕获。 +- manifest 二进制里包含 BAB 编译后的完整引用图（`AssetReferenceOffset/Count` + + 8 字节 `TypeId+InstanceId` 条目），目前仍跳过。将来可解析它给原版/manifest + 资产提供“编译器权威”计数；mod 源码资产在编译前没有 manifest，仍需源码级 + 引用索引。 +- 未引用资产目前是命令 + QuickPick 的“查询”形态；如果之后想要常驻浏览， + 可以再加 Tree View（更重，暂不计划）。 diff --git a/docs/plan.md b/docs/plan.md index ed438a5..857aad0 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -1,6 +1,6 @@ # 调研结论与实施计划（已按最新代码同步更新） -> 说明：本文档随实现演进持续同步。最近一次同步（2026-08-04）对齐了实现过程中新增的模块与设计变更：BAB 精确搜索路径、manifest 类型/ID 推导、上下文感知元素类型、属性级 refType / Poid 局部引用（`id` 定义点）、精确跳转范围、嵌套 `xi:include`、注入式语法高亮、bit-flag 列表补全（空格触发 / 排除已用 / 追加模式）、simple-content 元素文本引用（补全 / hover / 跳转 / 诊断 / Find All References）等。 +> 说明：本文档随实现演进持续同步。最近一次同步（2026-08-05）对齐了实现过程中新增的模块与设计变更：BAB 精确搜索路径、manifest 类型/ID 推导、上下文感知元素类型、属性级 refType / Poid 局部引用（`id` 定义点）、精确跳转范围、嵌套 `xi:include`、注入式语法高亮、bit-flag 列表补全（空格触发 / 排除已用 / 追加模式）、simple-content 元素文本引用（补全 / hover / 跳转 / 诊断 / Find All References）、语义引用索引 / CodeLens 引用计数 / 未引用资产命令等。 ## 一、调研结论（带证据） @@ -86,9 +86,10 @@ src/ manifestParser.ts .manifest 二进制解析 + 类型/ID 推导（纯 TS） fileScanner.ts 目录遍历缓存 + Include source 候选收集 refs.ts 引用目标解析（属性 + 元素文本内容，按 refType / isRef / - inheritFrom 过滤，纯 TS） + inheritFrom 过滤，纯 TS）+ “设计上可被引用类型”判定 + referenceIndex.ts 引用记录 → 反向引用索引（定义 → 引用位置）+ 未引用报告 shallowScan.ts 大体积美术资产（.w3x 等）顶层浅扫描（纯 TS，不建 DOM） - records.ts 每文件紧凑索引记录（资产/Define/Include/xi + 行号） + records.ts 每文件紧凑索引记录（资产/Define/Include/xi/引用 + 行号偏移） caches.ts 跨重建持久缓存（DocumentCache / IndexRecordsCache / IncludeResolveCache）+ 失效纪元 InvalidationsEpoch diskCache.ts 跨会话磁盘缓存（gzip JSON、原子写、多信号 stat 校验） @@ -99,6 +100,9 @@ src/ completion.ts 补全 provider（元素/属性/值，上下文感知；xs:list 多值按当前段过滤） hover.ts hover provider navigation.ts 定义/引用/文档链接/大纲 + references.ts 语义 FAR / 引用上下文 / CodeLens 命令共享逻辑 + codeLens.ts CodeLens 引用计数（类型过滤，0 也显示，点击开 references peek） + unreferenced.ts 未引用资产 QuickPick 命令 + 右键菜单入口 diagnostics.ts 实时诊断 semanticTokens.ts 语义 token provider（文档有解析错误时接管着色） syntaxes/ @@ -108,9 +112,10 @@ tools/ extract-asset-types.mjs OpenSAGE AssetType.cs → asset-types.json test/ fixtures/minimod 样例 Mod（include 各种情形、同名 ID、嵌套 xi:include、manifest 回退） - *.test.mjs 11 个测试文件（xmlParser / context / completion / semanticTokens / + *.test.mjs 14 个测试文件（xmlParser / context / completion / semanticTokens / includeResolver / manifestParser / indexer / schemaModel / refs / - typeContext / manifestTypes） + typeContext / manifestTypes / referenceIndex / codeLens / + referenceProvider） ``` ### 关键设计决策 @@ -251,6 +256,27 @@ test/ 用 `<` 做过滤前缀导致菜单为空——改为保留已输入的 `<`、range 从 `<` 之后 开始、插入文本不带开括号；`textContentTokenAt` 对未闭合元素用 `el.end` 作内容边界。 +28. **语义引用索引 + CodeLens 计数 + 未引用资产（第十八轮，2026-08-05）**： + - 动机与结论：292 个有模型类型的顶层资产中 191 个在 XSD 中没有任何 + 类型化引用指向（设置/地图元数据/w3x 子结构等自动注册类型）， + AttachTest 实测 29% 项目定义 0 引用、48% 恰好 1 引用。因此 CodeLens + 计数只显示在 `isReferenceTargetType()` 为真的类型上（类型化引用目标 + + `inheritFrom` 可继承类型），0 也显示；未引用报告默认也只列这些类型。 + - 数据：`IndexRecords` 增加 `references[]` + （`kind / refType / selfType / value / line / start / end`），解析期固化 + 引用上下文，反向索引构建零 DOM、零上下文重解析；`ModIndex.references` + 为“定义 key → 引用位置”的反向表，快照发布时构建；磁盘缓存版本 + v1 → v2（`index-records-v2.json.gz`），v3 起附加内容/records 哈希。 + - FAR 从全文搜索改为语义索引：不再把 `id="X"` 定义行、`EditorName="X"` + 等非引用属性计为引用；CodeLens 显示的计数与点击打开的 references peek + 严格一致；`includeDeclaration` 时才附加定义位置。 + - 未引用资产：命令面板 `Find unreferenced assets…` + 编辑器右键菜单 + `Find unreferenced assets of this type`；只统计 `origin === "project"` + 且非 `viaInstance` 的定义；覆盖原版 id 时来自原版/其他流的引用计入。 + - 边界：局部 scope（不在全局流里的文件）的引用不入全局反向索引； + `AssetIdList` 等列表引用按整值记录；w3x 不提取引用记录（对 w3x 资产的 + 引用从引用它的 XML 捕获）；manifest 编译期引用图（AssetReference 缓冲） + 留作后续。设计文档：`docs/features-reference-counts.md`。 ## 三、实施步骤 @@ -308,6 +334,24 @@ test/ id/define/local/include 候选返回 `isIncomplete` 让 VS Code 随输入重请求； 当前文档 local 资产优先、候选 top-N 用堆避免全量排序、Include source 先排序再截断；测试 128 → 136。 +21. [x] 语义引用索引 + CodeLens + 未引用资产（第十八轮，2026-08-05）： + records 引用记录提取、`referenceIndex.ts` 反向索引、语义 FAR（替换全文 + 搜索）、CodeLens 引用计数（类型过滤、0 也显示、点击开 references peek）、 + `findUnreferencedAssets` 命令 + 右键菜单、磁盘缓存 v2； + 测试 136 → 146；设计文档 `docs/features-reference-counts.md`。 +22. [x] FAR 定义行排除 + manifest 源引用归并（2026-08-05 修复轮）： + FAR 不再因 `includeDeclaration` 附加定义行（消除“自引用”观感）； + `referenceSitesForDefinition` 把 manifestSource 可解析到当前文件的 + manifest 定义站点并入 CodeLens 计数（AttachTest 326 个 manifest 定义 + 受益）；测试 146 → 147；版本 0.1.17；分析见 + `docs/analysis-issues.md` 二十三。 +23. [x] 引用索引与缓存同步加固（2026-08-06）：反向索引改为从**构建期本地 + records**（`buildRecords`）构建，不再读共享 recordsCache——中途失效、 + feature `readDom` 重读都不再造成“资产在但引用缺失”；构建中 `readDom` + 不再被定义跳转调用（避免污染进行中的构建）；force 重建（Re-index） + 对 stat 匹配的 full XML 做内容哈希校验；打开文档时比较 records 哈希， + 不一致则定向 invalidate + `records-desync` 重建自愈；磁盘缓存 v2 → v3； + 测试 147 → 151；分析见 `docs/analysis-issues.md` 二十四。 ## 四、验证结果（实测） @@ -320,19 +364,23 @@ test/ 单元测试覆盖：XML 解析（自闭合/容错/偏移/未闭合引号行尾恢复 + recovered 标记）、补全上下文（未闭合引号仍为 attribute-value、多行未闭合引号、list 多值 分段、闭合引号后为 attribute-name）、补全集成（vscode stub 下 -`LocomotorTemplate@Surfaces` 未闭合引号枚举补全、空格后第二段过滤与替换范围、 -空格后排除已用 flag、中间插入范围止于光标、闭合值末尾追加模式、`CAN_ATTACK` -前缀保护、多行未闭合 `Disposition` 完整链路、闭合引号后补空格、一行一个属性 -换行缩进、新行缩进对齐、标量类型化默认值）、语义 token（标签/属性/值范围、 -合法文档返回空、malformed 返回兜底 token）、include 解析（BAB 顺序、SDK 根 -优先于 SageXml）、manifest 二进制解析（合成 v5 样本、类型/ID 推导）、索引器 -（资产/Define/流/缺失 include/嵌套 xi:include）、XSD 模型（上下文类型、 -`childTypeOf`、大小写规范化、属性级 refType、外来命名空间判定、`xs:list` -枚举继承与 `isList` 标记）、引用过滤（`Weapon="X"` 只跳 `WeaponTemplate`、 -模块 `id` 定义点、Poid 局部引用、`xi:include` 不校验、`Side="Allies"` 命中 -manifest 的 `PlayerTemplate`）、simple-content 文本引用（`<` 后补全不产生 -`<<`、`$1` 片段、内容值按 refType 过滤、 -内容 hover / Ctrl 跳转 / 诊断）。 + `LocomotorTemplate@Surfaces` 未闭合引号枚举补全、空格后第二段过滤与替换范围、 + 空格后排除已用 flag、中间插入范围止于光标、闭合值末尾追加模式、`CAN_ATTACK` + 前缀保护、多行未闭合 `Disposition` 完整链路、闭合引号后补空格、一行一个属性 + 换行缩进、新行缩进对齐、标量类型化默认值）、语义 token（标签/属性/值范围、 + 合法文档返回空、malformed 返回兜底 token）、include 解析（BAB 顺序、SDK 根 + 优先于 SageXml）、manifest 二进制解析（合成 v5 样本、类型/ID 推导）、索引器 + （资产/Define/流/缺失 include/嵌套 xi:include）、XSD 模型（上下文类型、 + `childTypeOf`、大小写规范化、属性级 refType、外来命名空间判定、`xs:list` + 枚举继承与 `isList` 标记）、引用过滤（`Weapon="X"` 只跳 `WeaponTemplate`、 + 模块 `id` 定义点、Poid 局部引用、`xi:include` 不校验、`Side="Allies"` 命中 + manifest 的 `PlayerTemplate`）、simple-content 文本引用（`<` 后补全不产生 + `<<`、`$1` 片段、内容值按 refType 过滤、 + 内容 hover / Ctrl 跳转 / 诊断）、引用索引（records 引用提取：attr/content/ + inheritFrom、排除 id/Poid/`$DEFINE`/枚举；`buildReferenceIndex` 严格类型过滤 + 不串同名 ID；minimod 集成：`CommandSet` / `inheritFrom` 反向命中；未引用过滤 + 只含 project 非 viaInstance；CodeLens 类型过滤与 0 显示；语义 FAR 排除定义 + 行与非引用属性）。 > 注：D: 盘移动硬盘已恢复连接；Corona 已在第八 / 九轮按上述新数据回归。 diff --git a/docs/requirements.md b/docs/requirements.md index 371de03..81a1831 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -86,9 +86,21 @@ XML 之间的组织靠 `` 标签，共有三种语义： 美术资产目录（Corona 约 2.6 GB）；索引记录与 include 解析结果同样跨重建缓存， 保存触发的重建零 stat、零重读（Corona 实测约 2 秒）。 +**补充（引用计数、语义 Find All References 与未引用资产，2026-08-05）**： +- 每个顶部 asset 显示被引用次数（CodeLens，0 也显示），点击直接打开 + references peek；只在“设计上应被引用”的类型上显示，避免设置类/地图元数据/ + w3x 子结构等自动注册类型制造满屏 0； +- Find All References 改为基于语义引用索引（属性引用 + simple-content 文本 + + `inheritFrom`，排除 id 定义点 / Poid / `$DEFINE`），不再全文搜索； +- 命令 `Find unreferenced assets…`：按类型列出所有零引用的项目定义并跳转； + 编辑器右键菜单 `Find unreferenced assets of this type` 预选光标所在类型； +- 设计文档见 `docs/features-reference-counts.md`。 + ### P1：非近期目标（本期不做，但预留扩展点） 6. **高效搜索**：Mod 项目巨大（Corona 约 7500 个 XML、38MB）时直接全文搜索很慢，需要一个高效的 XML 内容索引机制。 + （2026-08-05：语义引用索引已落地，FAR / 引用计数 / 未引用报告不再全文搜索； + 通用内容搜索与索引复用仍属远期。） 7. **索引机制的复用性**：希望索引不仅能服务 VS Code 插件，也能被其他工具（如搜索、静态分析）复用，因此索引/解析核心应设计成与编辑器无关的纯模块。 ## 三、环境与参考资源 diff --git a/package-lock.json b/package-lock.json index 2ca98a2..a48fb7d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "ra3-mod-xml", - "version": "0.1.9", + "version": "0.1.18", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "ra3-mod-xml", - "version": "0.1.9", + "version": "0.1.18", "license": "MIT", "dependencies": { "fast-xml-parser": "^4.5.0" diff --git a/package.json b/package.json index f044f21..dbac54d 100644 --- a/package.json +++ b/package.json @@ -2,7 +2,7 @@ "name": "ra3-mod-xml", "displayName": "RA3 Mod XML", "description": "Red Alert 3 Mod XML tooling: syntax highlighting, completions, reference navigation and diagnostics for SAGE/BinaryAssetBuilder XML.", - "version": "0.1.15", + "version": "0.1.18", "publisher": "ra3-mod-xml", "license": "MIT", "engines": { @@ -100,6 +100,23 @@ { "command": "ra3modxml.showCacheReport", "title": "RA3 Mod XML: Show cache report" + }, + { + "command": "ra3modxml.findUnreferencedAssets", + "title": "RA3 Mod XML: Find unreferenced assets…" + }, + { + "command": "ra3modxml.findUnreferencedAssetsOfType", + "title": "RA3 Mod XML: Find unreferenced assets of this type" + } + ] + }, + "menus": { + "editor/context": [ + { + "command": "ra3modxml.findUnreferencedAssetsOfType", + "when": "editorLangId == xml && ra3modxml.active", + "group": "navigation@50" } ] }, diff --git a/src/extension.ts b/src/extension.ts index 722e508..7752938 100644 --- a/src/extension.ts +++ b/src/extension.ts @@ -8,6 +8,12 @@ import { Ra3DocumentSymbolProvider, Ra3ReferenceProvider, } from "./features/navigation"; +import { Ra3CodeLensProvider } from "./features/codeLens"; +import { showReferencesForDef } from "./features/references"; +import { + findUnreferencedAssets, + findUnreferencedAssetsOfType, +} from "./features/unreferenced"; import { Ra3Diagnostics } from "./features/diagnostics"; import { Ra3SemanticTokensProvider, @@ -60,6 +66,12 @@ export function activate(context: vscode.ExtensionContext): void { new Ra3DocumentSymbolProvider(ws), ), ); + context.subscriptions.push( + vscode.languages.registerCodeLensProvider( + XML_SELECTOR, + new Ra3CodeLensProvider(ws), + ), + ); context.subscriptions.push( vscode.languages.registerDocumentSemanticTokensProvider( XML_SELECTOR, @@ -73,6 +85,7 @@ export function activate(context: vscode.ExtensionContext): void { // Refresh diagnostics for every open XML document whenever a new index // snapshot is published (XML phase, art phase, stale/final rebuild). ws.onIndexUpdate = () => { + void vscode.commands.executeCommand("editor.action.codeLens.refresh"); for (const doc of vscode.workspace.textDocuments) { if (doc.languageId === "xml") void diagnostics.update(doc); } @@ -177,6 +190,7 @@ export function activate(context: vscode.ExtensionContext): void { `Project: ${s.projectDir}\n` + `Files: ${s.indexedFiles} (${s.parsedFiles} parsed, ${s.shallowScannedFiles} shallow-scanned, ${s.shallowCacheHits + s.recordsCacheHits} cache hits)\n` + `Assets: ${s.assetCount} (${s.manifestAssetCount} from ${s.manifestFiles} manifests)\n` + + `References: ${s.referenceCount}\n` + `Defines: ${s.defineCount} · Streams: ${s.streams} · Candidates: ${s.sourceCandidates}\n` + `Phase: ${s.phase} · Complete: ${s.complete}${stale}\n` + `Build #${ws.buildCount} (trigger: ${ws.lastTrigger})\n` + @@ -186,6 +200,25 @@ export function activate(context: vscode.ExtensionContext): void { ); }), ); + context.subscriptions.push( + vscode.commands.registerCommand( + "ra3modxml.showReferences", + (args: Parameters[1]) => + void showReferencesForDef(ws, args), + ), + ); + context.subscriptions.push( + vscode.commands.registerCommand( + "ra3modxml.findUnreferencedAssets", + () => void findUnreferencedAssets(ws), + ), + ); + context.subscriptions.push( + vscode.commands.registerCommand( + "ra3modxml.findUnreferencedAssetsOfType", + () => void findUnreferencedAssetsOfType(ws), + ), + ); void ws.initialize(); } diff --git a/src/features/codeLens.ts b/src/features/codeLens.ts new file mode 100644 index 0000000..3cc4728 --- /dev/null +++ b/src/features/codeLens.ts @@ -0,0 +1,90 @@ +import * as vscode from "vscode"; +import { LineMap, parseXml } from "../language/xmlParser"; +import { resolveElementType } from "../language/typeContext"; +import { isReferenceTargetType } from "../indexer/refs"; +import { + referenceSitesForDefinition, + scheduleRebuildIfRecordsDesync, +} from "../indexer/referenceIndex"; +import type { ShowReferencesArgs } from "./references"; +import type { ModWorkspace } from "../workspace"; + +/** Never build a DOM for huge files just to show counts (w3x safety). */ +const MAX_CODELENS_TEXT = 4 * 1024 * 1024; + +/** + * CodeLens reference counts on top-level assets. + * + * Only types that are reference targets by design get a lens (settings, map + * metadata, w3x sub-assets etc. would otherwise show a permanent, misleading + * "0 references"). Zero is still shown for the meaningful types: that is the + * signal users can click to inspect an unused asset. + */ +export class Ra3CodeLensProvider implements vscode.CodeLensProvider { + constructor(private ws: ModWorkspace) {} + + provideCodeLenses( + document: vscode.TextDocument, + _token: vscode.CancellationToken, + ): vscode.CodeLens[] { + if (!this.ws.isRa3Workspace()) return []; + const idx = this.ws.index; + if (!idx) return []; + const text = document.getText(); + if (text.length > MAX_CODELENS_TEXT) return []; + scheduleRebuildIfRecordsDesync(this.ws, document); + const doc = parseXml(text); + const root = doc.root; + if (!root) return []; + const lineMap = new LineMap(text); + const lenses: vscode.CodeLens[] = []; + + for (const child of root.children) { + const local = localName(child.name); + if (local === "Tags" || local === "Includes" || local === "Defines") continue; + const idAttr = child.attrs.find((a) => a.name === "id"); + if (!idAttr?.hasValue) continue; + const elType = resolveElementType(child); + if (!isReferenceTargetType(elType)) continue; + + const id = idAttr.value; + const line = lineMap.positionAt(idAttr.valueStart).line + 1; + const count = referenceSitesForDefinition(idx, { + type: local, + id, + file: document.uri.fsPath, + line, + }).length; + const range = new vscode.Range( + document.positionAt(child.start), + document.positionAt(child.startTagEnd), + ); + const args: ShowReferencesArgs = { + uri: document.uri, + position: document.positionAt(idAttr.valueStart), + id, + type: local, + file: document.uri.fsPath, + line, + }; + lenses.push( + new vscode.CodeLens(range, { + title: + count === 0 + ? "0 references" + : count === 1 + ? "1 reference" + : `${count} references`, + command: "ra3modxml.showReferences", + arguments: [args], + }), + ); + } + return lenses; + } +} + +function localName(tag: string): string { + const idx = tag.lastIndexOf(":"); + return idx >= 0 ? tag.slice(idx + 1) : tag; +} diff --git a/src/features/navigation.ts b/src/features/navigation.ts index ebf0f26..9c5587f 100644 --- a/src/features/navigation.ts +++ b/src/features/navigation.ts @@ -14,6 +14,7 @@ import { resolveReferenceTargetsForType, type ReferenceTarget, } from "../indexer/refs"; +import { findReferenceLocations } from "./references"; import { findContainingGameObject, findLocalId, @@ -163,6 +164,18 @@ async function assetDefLocation( scope: DocumentScope, currentDocument: vscode.TextDocument, ): Promise { + // While a rebuild is running, avoid readDom() mutating the live indexer's + // caches mid-build; a line-based location is a fine temporary fallback. + if (ws.isBuilding) { + const line = Math.max(0, def.line - 1); + return new vscode.Location( + vscode.Uri.file(def.file), + new vscode.Range( + new vscode.Position(line, 0), + new vscode.Position(line, 1), + ), + ); + } if (def.origin === "manifest") { const src = def.manifestSource; if (src?.toUpperCase().startsWith("DATA:")) { @@ -278,56 +291,7 @@ export class Ra3ReferenceProvider implements vscode.ReferenceProvider { _context: vscode.ReferenceContext, _token: vscode.CancellationToken, ): Promise { - if (!this.ws.isRa3Workspace()) return null; - const text = document.getText(); - const offset = document.offsetAt(position); - const doc = parseXml(text); - const el = findElementAt(doc, offset); - if (!el) return null; - const attr = el.attrs.find( - (a) => - (a.hasValue && offset >= a.valueStart && offset <= a.valueEnd) || - (offset >= a.nameStart && offset <= a.nameEnd), - ); - let id: string | null = null; - if (attr?.hasValue) { - id = attr.value; - } else { - // Element text content (e.g. CrateDebris_01). - const elType = resolveElementType(el); - const token = textContentTokenAt(text, el, offset); - if (token && isReferenceContentType(elType) && !token.value.startsWith("$")) { - id = token.value; - } - } - if (!id || id.startsWith("$")) return null; - - const locations: vscode.Location[] = []; - // Matches both attribute values ("id" / 'id') and simple-content - // references (>id<); the outer delimiters are stripped from the result - // range below so the returned locations cover just the id. - const pattern = `(?:["']|>)[ \\t]*${escapeRegExp(id)}[ \\t]*(?:["']|<)`; - await findTextInWorkspace( - { pattern, isRegExp: true }, - { include: "**/*.xml", maxResults: 2000 }, - (result: { uri: vscode.Uri; matches: { range: vscode.Range }[] }) => { - if (!result.uri) return; - for (const m of result.matches) { - const start = m.range.start; - const end = m.range.end; - locations.push( - new vscode.Location( - result.uri, - new vscode.Range( - new vscode.Position(start.line, start.character + 1), - new vscode.Position(end.line, end.character - 1), - ), - ), - ); - } - }, - ); - return locations.length ? locations : null; + return findReferenceLocations(this.ws, document, position); } } @@ -435,40 +399,3 @@ function localName(tag: string): string { const idx = tag.lastIndexOf(":"); return idx >= 0 ? tag.slice(idx + 1) : tag; } - -function escapeRegExp(s: string): string { - return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); -} - -/** - * `workspace.findTextInFiles` is a stable VS Code API (since 1.66) but is - * missing from the published typings, so we declare the subset we need and - * call it via a safe cast. - */ -interface TextSearchQuery { - pattern: string; - isRegExp?: boolean; - isCaseSensitive?: boolean; - isWordMatch?: boolean; -} - -interface TextSearchOptions { - include?: string; - exclude?: string; - maxResults?: number; -} - -function findTextInWorkspace( - query: TextSearchQuery, - options: TextSearchOptions, - callback: (result: { uri: vscode.Uri; matches: { range: vscode.Range }[] }) => void, -): Promise { - const api = vscode.workspace as unknown as { - findTextInFiles( - query: TextSearchQuery, - options: TextSearchOptions, - callback: (result: { uri: vscode.Uri; matches: { range: vscode.Range }[] }) => void, - ): Promise; - }; - return api.findTextInFiles(query, options, callback).then(() => undefined); -} diff --git a/src/features/references.ts b/src/features/references.ts new file mode 100644 index 0000000..83d2347 --- /dev/null +++ b/src/features/references.ts @@ -0,0 +1,227 @@ +/** + * Shared semantic reference logic used by Find All References, the CodeLens + * "N references" command and (indirectly) the unreferenced-assets report. + * + * Unlike the old text-search implementation, every result here comes from + * the reverse reference index built during indexing, so the count shown on a + * top-level asset always matches the references peek opened by clicking it. + */ + +import * as vscode from "vscode"; +import { + findElementAt, + parseXml, + textContentTokenAt, +} from "../language/xmlParser"; +import { resolveElementType } from "../language/typeContext"; +import { attributesOfType, typeInfo } from "../model/schemaModel"; +import { + filterAndScoreDefs, + isReferenceAttributeOfType, + isReferenceContentType, + mergeLocalAndGlobalDefs, +} from "../indexer/refs"; +import { + referenceSitesForDef, + scheduleRebuildIfRecordsDesync, +} from "../indexer/referenceIndex"; +import type { AssetDef, ModIndex, ReferenceSite } from "../indexer/types"; +import type { ModWorkspace } from "../workspace"; + +export interface ReferenceContext { + id: string; + /** XSD refType when the cursor is on a typed reference; null otherwise. */ + refType: string | null; + /** Element type for inheritFrom filtering; null otherwise. */ + selfType: string | null; +} + +/** + * Extracts the referenced id and its XSD context from the cursor position. + * Works on reference attribute values/names, simple-content text and on an + * asset's own `id` definition (where every same-id definition is a target). + */ +export function referenceContextAt( + document: vscode.TextDocument, + offset: number, +): ReferenceContext | null { + const text = document.getText(); + const doc = parseXml(text); + const el = findElementAt(doc, offset); + if (!el) return null; + const elType = resolveElementType(el); + const attr = el.attrs.find( + (a) => + (a.hasValue && offset >= a.valueStart && offset <= a.valueEnd) || + (offset >= a.nameStart && offset <= a.nameEnd), + ); + + if (attr?.hasValue) { + const value = attr.value; + if (!value || value.startsWith("$") || value.startsWith("=")) return null; + const nameLower = attr.name.toLowerCase(); + if (nameLower === "id") { + return { id: value, refType: null, selfType: null }; + } + if (!isReferenceAttributeOfType(elType, attr.name)) return null; + if (nameLower === "inheritfrom") { + return { id: value, refType: null, selfType: elType }; + } + const attrInfo = elType + ? attributesOfType(elType).find((a) => a.name === attr.name) + : undefined; + return { id: value, refType: attrInfo?.refType ?? null, selfType: null }; + } + + if (elType && isReferenceContentType(elType)) { + const token = textContentTokenAt(text, el, offset); + if (token && !token.value.startsWith("$") && !token.value.startsWith("=")) { + const info = typeInfo(elType); + return { + id: token.value, + refType: info?.kind === "simple" ? info.refType : null, + selfType: null, + }; + } + } + return null; +} + +/** Definitions matching a reference context (strict type filtering). */ +export function definitionsForReference( + idx: ModIndex, + ctx: ReferenceContext, +): AssetDef[] { + const defs = mergeLocalAndGlobalDefs( + idx.local?.assetsById.get(ctx.id.toLowerCase()), + idx.assetsById.get(ctx.id.toLowerCase()), + ); + return filterAndScoreDefs(defs, ctx.refType, ctx.selfType).map((t) => t.def); +} + +/** Union of reference sites for a set of definitions, de-duplicated. */ +export function collectReferenceSites( + idx: ModIndex, + defs: readonly AssetDef[], +): ReferenceSite[] { + const sites: ReferenceSite[] = []; + const seen = new Set(); + for (const def of defs) { + for (const site of referenceSitesForDef(idx, def)) { + const key = `${site.file}\u0000${site.start}\u0000${site.end}\u0000${site.kind}`; + if (seen.has(key)) continue; + seen.add(key); + sites.push(site); + } + } + return sites; +} + +/** Converts stored offsets to precise editor locations (fallback: line). */ +export async function sitesToLocations( + ws: ModWorkspace, + sites: readonly ReferenceSite[], +): Promise { + const byFile = new Map(); + for (const site of sites) { + let list = byFile.get(site.file); + if (!list) { + list = []; + byFile.set(site.file, list); + } + list.push(site); + } + + const locations: vscode.Location[] = []; + for (const [file, fileSites] of byFile) { + const parsed = await ws.indexer?.readDom(file); + const lineMap = parsed?.lineMap ?? null; + for (const site of fileSites) { + if (lineMap) { + const start = lineMap.positionAt(site.start); + const end = lineMap.positionAt(site.end); + locations.push( + new vscode.Location( + vscode.Uri.file(file), + new vscode.Range( + new vscode.Position(start.line, start.character), + new vscode.Position(end.line, end.character), + ), + ), + ); + } else { + const line = Math.max(0, site.line - 1); + locations.push( + new vscode.Location( + vscode.Uri.file(file), + new vscode.Range( + new vscode.Position(line, 0), + new vscode.Position(line, 1), + ), + ), + ); + } + } + } + return locations; +} + +/** + * Semantic Find All References (used by the reference provider). + * + * Only real reference sites are returned — the asset's own `id` definition is + * never included, regardless of VS Code's `includeDeclaration` flag, so the + * result set matches the CodeLens reference count exactly. + */ +export async function findReferenceLocations( + ws: ModWorkspace, + document: vscode.TextDocument, + position: vscode.Position, +): Promise { + if (!ws.isRa3Workspace()) return null; + scheduleRebuildIfRecordsDesync(ws, document); + const scope = await ws.getScope(document); + const idx = scope.merged; + if (!idx) return null; + const ctx = referenceContextAt(document, document.offsetAt(position)); + if (!ctx) return null; + const defs = definitionsForReference(idx, ctx); + if (!defs.length) return null; + + const locations = await sitesToLocations(ws, collectReferenceSites(idx, defs)); + return locations.length ? locations : null; +} + +/** Arguments passed from the CodeLens to the showReferences command. */ +export interface ShowReferencesArgs { + uri: vscode.Uri; + position: vscode.Position; + id: string; + type: string; + file: string; + line: number; +} + +/** Opens the references peek for one specific asset definition. */ +export async function showReferencesForDef( + ws: ModWorkspace, + args: ShowReferencesArgs, +): Promise { + const idx = ws.index; + if (!idx) return; + const def: AssetDef = { + type: args.type, + id: args.id, + file: args.file, + line: args.line, + origin: "project", + }; + const sites = referenceSitesForDef(idx, def); + const locations = await sitesToLocations(ws, sites); + await vscode.commands.executeCommand( + "editor.action.showReferences", + args.uri, + args.position, + locations, + ); +} diff --git a/src/features/unreferenced.ts b/src/features/unreferenced.ts new file mode 100644 index 0000000..2b6a5b1 --- /dev/null +++ b/src/features/unreferenced.ts @@ -0,0 +1,124 @@ +import * as vscode from "vscode"; +import { relative } from "node:path"; +import { findElementAt, parseXml } from "../language/xmlParser"; +import { unreferencedByType } from "../indexer/referenceIndex"; +import type { ModWorkspace } from "../workspace"; + +interface TypePickItem extends vscode.QuickPickItem { + type: string; +} + +interface AssetPickItem extends vscode.QuickPickItem { + file: string; + line: number; +} + +/** + * Palette command: pick an asset type, then jump to any project asset of + * that type that has zero incoming references. Only types that are reference + * targets by design are offered, so auto-registered data (settings, map + * metadata, w3x sub-assets...) is not reported as "unused". + */ +export async function findUnreferencedAssets( + ws: ModWorkspace, + args?: { type?: string }, +): Promise { + if (!ws.isRa3Workspace() || !ws.index) { + void vscode.window.showInformationMessage( + "RA3 Mod XML: no index available yet.", + ); + return; + } + const idx = ws.index; + const byType = unreferencedByType(idx); + + let type = args?.type; + if (!type) { + if (!byType.size) { + void vscode.window.showInformationMessage( + "RA3 Mod XML: no unreferenced assets found.", + ); + return; + } + const pickedType = await vscode.window.showQuickPick( + [...byType.entries()].map(([t, defs]) => ({ + label: t, + description: `${defs.length} unreferenced`, + type: t, + })), + { + placeHolder: "Select an asset type", + matchOnDescription: true, + }, + ); + if (!pickedType) return; + type = pickedType.type; + } + + const defs = byType.get(type) ?? []; + if (!defs.length) { + void vscode.window.showInformationMessage( + `RA3 Mod XML: no unreferenced ${type} assets found.`, + ); + return; + } + const pickedAsset = await vscode.window.showQuickPick( + defs.map((d) => ({ + label: d.id, + description: `${displayPath(idx.projectDir, d.file)}:${d.line}`, + file: d.file, + line: d.line, + })), + { + placeHolder: `${type}: ${defs.length} unreferenced`, + matchOnDescription: true, + }, + ); + if (!pickedAsset) return; + + const uri = vscode.Uri.file(pickedAsset.file); + const document = await vscode.workspace.openTextDocument(uri); + const line = Math.max(0, pickedAsset.line - 1); + await vscode.window.showTextDocument(document, { + selection: new vscode.Range( + new vscode.Position(line, 0), + new vscode.Position(line, 1), + ), + preview: true, + }); +} + +/** + * Editor context-menu entry: pre-selects the asset type under the cursor. + * Falls back to the type picker when the cursor is not on a top-level asset. + */ +export async function findUnreferencedAssetsOfType( + ws: ModWorkspace, +): Promise { + const editor = vscode.window.activeTextEditor; + if (editor && ws.isRa3Workspace()) { + const document = editor.document; + const offset = document.offsetAt(editor.selection.active); + const doc = parseXml(document.getText()); + const el = findElementAt(doc, offset); + if (el && el.parent === doc.root) { + const local = localName(el.name); + const isStructural = local === "Tags" || local === "Includes" || local === "Defines"; + const hasId = el.attrs.some((a) => a.name === "id" && a.hasValue); + if (!isStructural && hasId) { + return findUnreferencedAssets(ws, { type: local }); + } + } + } + return findUnreferencedAssets(ws); +} + +function localName(tag: string): string { + const idx = tag.lastIndexOf(":"); + return idx >= 0 ? tag.slice(idx + 1) : tag; +} + +function displayPath(projectDir: string, file: string): string { + const rel = relative(projectDir, file); + return rel && !rel.startsWith("..") ? rel : file; +} diff --git a/src/indexer/caches.ts b/src/indexer/caches.ts index 25b136b..6eea377 100644 --- a/src/indexer/caches.ts +++ b/src/indexer/caches.ts @@ -10,6 +10,7 @@ */ import { resolve } from "node:path"; +import { createHash } from "node:crypto"; import type { IndexedFile, ParsedFile } from "./types"; import type { IndexRecords } from "./records"; import type { ResolveResult } from "./includeResolver"; @@ -19,6 +20,28 @@ export function normKey(path: string): string { return resolve(path).toLowerCase(); } +/** + * SHA-1 of the BOM-stripped text a file's records were extracted from. + * Lets force rebuilds verify that a stat-matching cache entry is not stale + * (FAT32/exFAT timestamps can match after a rewrite), and lets features + * detect "open document != indexed snapshot" without reading the whole index. + */ +export function contentHash(text: string): string { + return createHash("sha1").update(text, "utf8").digest("hex"); +} + +/** + * Hash of a file's compact index records. Semantic records (assets, defines, + * includes, references) are what the index actually consumes, so comparing + * records hashes ignores cosmetic text changes (line endings, whitespace) + * and only fires the self-heal when the index would really be out of date. + */ +export function recordsHash(records: IndexRecords): string { + return createHash("sha1") + .update(JSON.stringify(records), "utf8") + .digest("hex"); +} + /** * LRU cache for fully parsed XML documents. * @@ -119,6 +142,12 @@ export interface IndexRecordsCacheEntry { records: IndexRecords; /** "shallow" for art-asset scans (.w3x), "full" for parsed XML. */ kind: "shallow" | "full"; + /** + * Hash of the BOM-stripped file text (full parses only). Absent for + * shallow scans (avoid hashing multi-MB model files) and for cache entries + * produced before this field existed. + */ + contentHash?: string; } /** diff --git a/src/indexer/diskCache.ts b/src/indexer/diskCache.ts index 953c6dd..d0a34ae 100644 --- a/src/indexer/diskCache.ts +++ b/src/indexer/diskCache.ts @@ -34,7 +34,14 @@ import type { IndexedFile } from "./types"; const gzipAsync = promisify(gzip); const gunzipAsync = promisify(gunzip); -export const DISK_CACHE_VERSION = 1; +/** + * v2: per-file records now carry typed reference records (`references`), + * so caches produced by v1 (assets/defines/includes only) are stale. + * v3: full XML records carry `contentHash`, and snapshots publish per-file + * `recordsHashes` for the desync self-heal; caches without hashes cannot be + * verified, so v2 files are regenerated once. + */ +export const DISK_CACHE_VERSION = 3; /** How many stat validations run concurrently on load. */ const VALIDATE_CONCURRENCY = 32; @@ -53,6 +60,8 @@ export interface DiskCacheRecord { stat: NonNullable; records: IndexRecords; kind: "full" | "shallow"; + /** Content hash for full XML parses (see `IndexRecordsCacheEntry`). */ + contentHash?: string; } interface DiskCacheFile { @@ -173,6 +182,7 @@ export class DiskRecordsCache { stat: entry.stat, records: entry.records, kind: entry.kind, + contentHash: entry.contentHash, }); } const payload: DiskCacheFile = { diff --git a/src/indexer/indexer.ts b/src/indexer/indexer.ts index 80d81e7..7cf59b1 100644 --- a/src/indexer/indexer.ts +++ b/src/indexer/indexer.ts @@ -40,14 +40,26 @@ import { } from "./manifestParser"; import { canonicalTypeName } from "../model/schemaModel"; import { collectSourceCandidates } from "./fileScanner"; -import { DocumentCache, IncludeResolveCache, IndexRecordsCache, normKey } from "./caches"; +import { + contentHash, + DocumentCache, + IncludeResolveCache, + IndexRecordsCache, + normKey, + recordsHash, +} from "./caches"; import type { IndexRecordsCacheEntry } from "./caches"; import { scanXmlShallow } from "./shallowScan"; import { extractIndexRecords, recordsFromShallow, + type IndexRecords, type IndexRecordXi, } from "./records"; +import { + buildReferenceIndex, + type ReferenceRecordSource, +} from "./referenceIndex"; import type { AssetDef, DefineDef, @@ -55,6 +67,7 @@ import type { IndexedFile, ModIndex, ParsedFile, + ReferenceSite, SourceCandidate, StreamInfo, } from "./types"; @@ -110,6 +123,16 @@ export class ModIndexer { private assetsById = new Map(); private defines = new Map(); private files = new Map(); + /** + * Records exactly as the current build's walk saw them (path -> records + + * content hash). The reverse reference index is built from this map, never + * from the shared records cache, so a watcher invalidation or a feature + * re-read (readDom) mid-build cannot desync references from assets. + */ + private buildRecords = new Map< + string, + { file: string; records: IndexRecords; recordsHash: string } + >(); private streams: StreamInfo[] = []; private manifests = new Map(); private sourceCandidates: SourceCandidate[] = []; @@ -171,6 +194,20 @@ export class ModIndexer { rec.stat.birthtimeMs === st.birthtimeMs && rec.stat.ctimeMs === st.ctimeMs ) { + // Force rebuilds (Re-index workspace) verify full-XML content even + // when every stat signal matches: external drives (FAT32/exFAT) can + // rewrite a file with the same size and coarse timestamps. + if ( + this.opts.trustUnchanged === false && + rec.kind === "full" && + rec.contentHash + ) { + const text = stripBom(await readFile(path, "utf8")); + if (contentHash(text) === rec.contentHash) { + return this.recordsParsed(path, rec); + } + return this.parseFullXml(path, st, text); + } return this.recordsParsed(path, rec); } const hit = this.docs.get(key); @@ -235,27 +272,7 @@ export class ModIndexer { return parsed; } const text = stripBom(await readFile(path, "utf8")); - const lineMap = new LineMap(text); - const parse = parseXml(text); - const records = extractIndexRecords(parse, lineMap); - const parsed: ParsedFile = { - file: { - path: resolve(path), - stat: { - mtimeMs: st.mtimeMs, - size: st.size, - birthtimeMs: st.birthtimeMs, - ctimeMs: st.ctimeMs, - }, - }, - parse, - records, - lineMap, - }; - this.docs.set(parsed); - this.recordsCache.set(key, { stat: parsed.file.stat, records, kind: "full" }); - this.files.set(key, parsed.file); - return parsed; + return this.parseFullXml(path, st, text); } catch { const parsed: ParsedFile = { file: { path: resolve(path), stat: null }, @@ -313,6 +330,40 @@ export class ModIndexer { return { file, parse: null, records: entry.records, lineMap: null }; } + /** + * Parses a full XML document from its text, caches records (with a content + * hash) and the DOM, and registers the file in this build. + */ + private parseFullXml(path: string, st: Stats, text: string): ParsedFile { + const key = normKey(path); + const lineMap = new LineMap(text); + const parse = parseXml(text); + const records = extractIndexRecords(parse, lineMap, text); + const parsed: ParsedFile = { + file: { + path: resolve(path), + stat: { + mtimeMs: st.mtimeMs, + size: st.size, + birthtimeMs: st.birthtimeMs, + ctimeMs: st.ctimeMs, + }, + }, + parse, + records, + lineMap, + }; + this.docs.set(parsed); + this.recordsCache.set(key, { + stat: parsed.file.stat, + records, + kind: "full", + contentHash: contentHash(text), + }); + this.files.set(key, parsed.file); + return parsed; + } + /** * Reads a document and guarantees a DOM parse tree. Used for root-level * xpointer selection (rare) and by the document-local scope @@ -341,27 +392,7 @@ export class ModIndexer { } if (st.size > MAX_PARSE_BYTES) return null; const text = stripBom(await readFile(path, "utf8")); - const lineMap = new LineMap(text); - const parse = parseXml(text); - const records = extractIndexRecords(parse, lineMap); - const parsed: ParsedFile = { - file: { - path: resolve(path), - stat: { - mtimeMs: st.mtimeMs, - size: st.size, - birthtimeMs: st.birthtimeMs, - ctimeMs: st.ctimeMs, - }, - }, - parse, - records, - lineMap, - }; - this.docs.set(parsed); - this.recordsCache.set(key, { stat: parsed.file.stat, records, kind: "full" }); - this.files.set(key, parsed.file); - return parsed; + return this.parseFullXml(path, st, text); } catch { return null; } @@ -424,6 +455,7 @@ export class ModIndexer { async build(onPhase?: (index: ModIndex) => void | Promise): Promise { const start = Date.now(); + this.buildRecords.clear(); // Root list only; directories are listed lazily on first query, so the // XML phase does not pay an upfront recursive enumeration of the SDK. this.existence = buildExistenceSnapshot(this.searchPaths); @@ -561,6 +593,15 @@ export class ModIndexer { for (const [id, defs] of this.assetsById) assetsById.set(id, defs.slice()); const defines = new Map(); for (const [name, defs] of this.defines) defines.set(name, defs.slice()); + const references = this.buildReferences(); + const referenceCount = [...references.values()].reduce( + (sum, sites) => sum + sites.length, + 0, + ); + const recordsHashes = new Map(); + for (const [key, entry] of this.buildRecords) { + recordsHashes.set(key, entry.recordsHash); + } return { projectDir: resolve(this.opts.projectDir), @@ -575,6 +616,8 @@ export class ModIndexer { manifests: new Map(this.manifests), sourceCandidates: this.sourceCandidates.slice(), diagnostics: this.diagnostics.slice(), + references, + recordsHashes, stats: { projectDir: resolve(this.opts.projectDir), sdkDir: resolve(this.opts.sdkDir), @@ -596,6 +639,7 @@ export class ModIndexer { walkMs: this.timings.walkMs, artScanMs: this.timings.artScanMs, assetCount: [...this.assets.values()].reduce((sum, byId) => sum + byId.size, 0), + referenceCount, defineCount: this.defines.size, manifestFiles: this.manifests.size, manifestAssetCount, @@ -606,6 +650,38 @@ export class ModIndexer { }; } + /** + * Resolves the per-file reference records collected during this build + * against the current asset maps. Only files touched by this build are + * included, so stale cache entries for files that left the include graph + * never leak into the reverse index. + */ + private buildReferences(): Map { + const sources: ReferenceRecordSource[] = []; + for (const { file, records } of this.buildRecords.values()) { + sources.push({ file, records }); + } + return buildReferenceIndex(sources, { + assets: this.assets, + assetsById: this.assetsById, + }); + } + + /** + * Remembers the records exactly as this build saw them (plus the content + * hash when the file was fully parsed / cached with one), so snapshots can + * build references from the same source of truth as the asset maps. + */ + private noteBuildRecords(parsed: ParsedFile): void { + if (!parsed.records) return; + const key = normKey(parsed.file.path); + this.buildRecords.set(key, { + file: parsed.file.path, + records: parsed.records, + recordsHash: recordsHash(parsed.records), + }); + } + // ── Include walk ────────────────────────────────────────────────── private async walk( @@ -670,6 +746,7 @@ export class ModIndexer { ): Promise { const records = parsed.records; if (!records) return; + this.noteBuildRecords(parsed); const file = parsed.file.path; const origin = this.originOf(file); diff --git a/src/indexer/localScope.ts b/src/indexer/localScope.ts index 0822642..7d6da9f 100644 --- a/src/indexer/localScope.ts +++ b/src/indexer/localScope.ts @@ -59,7 +59,7 @@ export async function buildDocumentScope( const lineMap = new LineMap(text); const parse = parseXml(text); const builder = new OverlayBuilder(ctx); - await builder.addEntry(uri, parse, lineMap); + await builder.addEntry(uri, parse, lineMap, text); const expanded = await expandDocument(uri, parse, { resolve: (source, currentDir) => @@ -115,6 +115,8 @@ export function withLocalOverlay( manifests: new Map(), sourceCandidates: [], diagnostics: [], + references: new Map(), + recordsHashes: new Map(), stats: { projectDir, sdkDir, @@ -134,6 +136,7 @@ export function withLocalOverlay( walkMs: 0, artScanMs: 0, assetCount: 0, + referenceCount: 0, defineCount: 0, manifestFiles: 0, manifestAssetCount: 0, @@ -164,12 +167,13 @@ class OverlayBuilder { path: string, parse: XmlDocument, lineMap: LineMap, + text: string, ): Promise { this.lineMaps.set(scopePathKey(path), lineMap); await this.addParsed({ file: { path: resolve(path), stat: null }, parse, - records: extractIndexRecords(parse, lineMap), + records: extractIndexRecords(parse, lineMap, text), lineMap, }, 0); } diff --git a/src/indexer/records.ts b/src/indexer/records.ts index 822565d..f842d1a 100644 --- a/src/indexer/records.ts +++ b/src/indexer/records.ts @@ -12,6 +12,12 @@ import type { LineMap, XmlDocument } from "../language/xmlParser"; import type { ShallowDocument } from "./shallowScan"; +import { attributesOfType, typeInfo } from "../model/schemaModel"; +import { resolveElementType } from "../language/typeContext"; +import { + isReferenceAttributeOfType, + isReferenceContentType, +} from "./refs"; export interface IndexRecordAsset { /** Top-level element name, e.g. "W3DContainer". */ @@ -42,6 +48,27 @@ export interface IndexRecordXi { line: number; } +export interface IndexRecordReference { + /** "attr" for attribute values, "content" for simple-content text. */ + kind: "attr" | "content"; + /** + * XSD reference target type (from `xas:refType`), or null for untyped + * `isRef` references and `inheritFrom` (which filters by the element's own + * type via `selfType`). + */ + refType: string | null; + /** Element type used by `inheritFrom` filtering; null otherwise. */ + selfType: string | null; + /** The referenced id text (whole attribute value / trimmed content). */ + value: string; + /** 1-based line of the value. */ + line: number; + /** Character offset of the value start (relative to the file text). */ + start: number; + /** Character offset one past the value end. */ + end: number; +} + export interface IndexRecords { assets: IndexRecordAsset[]; defines: IndexRecordDefine[]; @@ -50,6 +77,8 @@ export interface IndexRecords { rootXiIncludes: IndexRecordXi[]; /** elements nested anywhere else in the document. */ nestedXiIncludes: IndexRecordXi[]; + /** Typed global-asset references (attribute values + simple content). */ + references: IndexRecordReference[]; } const INCLUDE_TYPES = new Set(["all", "instance", "reference"]); @@ -69,14 +98,21 @@ function localName(tag: string): string { * Tags/Includes/Defines), $DEFINE constants, the top-level block * and root/nested elements. */ -export function extractIndexRecords(parse: XmlDocument, lineMap: LineMap): IndexRecords { +export function extractIndexRecords( + parse: XmlDocument, + lineMap: LineMap, + text: string, +): IndexRecords { const assets: IndexRecordAsset[] = []; const defines: IndexRecordDefine[] = []; const includes: IndexRecordInclude[] = []; const rootXiIncludes: IndexRecordXi[] = []; const nestedXiIncludes: IndexRecordXi[] = []; + const references: IndexRecordReference[] = []; const root = parse.root; - if (!root) return { assets, defines, includes, rootXiIncludes, nestedXiIncludes }; + if (!root) { + return { assets, defines, includes, rootXiIncludes, nestedXiIncludes, references }; + } for (const child of root.children) { const local = localName(child.name); @@ -142,7 +178,86 @@ export function extractIndexRecords(parse: XmlDocument, lineMap: LineMap): Index }); } - return { assets, defines, includes, rootXiIncludes, nestedXiIncludes }; + collectReferenceRecords(parse, lineMap, text, references); + + return { assets, defines, includes, rootXiIncludes, nestedXiIncludes, references }; +} + +/** + * Walks every element of a fully parsed document and records typed + * global-asset references: reference attributes, `inheritFrom` and + * simple-content reference text. Local `id` definitions, Poid pipeline-local + * references and `$DEFINE`/`=` values are intentionally skipped (the same + * semantics as diagnostics / hover / navigation). + * + * The stored `refType` / `selfType` pair is exactly what + * `resolveReferenceTargetsForType` derives from the element context, so the + * reverse reference index can resolve these records after the whole index is + * built without re-walking the document or re-resolving element types. + */ +function collectReferenceRecords( + parse: XmlDocument, + lineMap: LineMap, + text: string, + out: IndexRecordReference[], +): void { + for (const el of parse.elements) { + const elType = resolveElementType(el); + + for (const attr of el.attrs) { + if (!attr.hasValue) continue; + if (!isReferenceAttributeOfType(elType, attr.name)) continue; + const value = attr.value; + if (!value || value.startsWith("$") || value.startsWith("=")) continue; + let refType: string | null = null; + let selfType: string | null = null; + if (attr.name.toLowerCase() === "inheritfrom") { + selfType = elType; + } else if (elType) { + refType = + attributesOfType(elType).find((a) => a.name === attr.name)?.refType ?? + null; + } + out.push({ + kind: "attr", + refType, + selfType, + value, + line: lineOf(lineMap, attr.valueStart), + start: attr.valueStart, + end: attr.valueEnd, + }); + } + + if ( + elType && + isReferenceContentType(elType) && + !el.selfClosing && + el.closeTagStart >= 0 + ) { + const raw = text.slice(el.startTagEnd, el.closeTagStart); + const value = raw.trim(); + if ( + !value || + value.startsWith("$") || + value.startsWith("=") || + value.includes("<") + ) { + continue; + } + const start = el.startTagEnd + raw.indexOf(value); + const info = typeInfo(elType); + out.push({ + kind: "content", + refType: info?.kind === "simple" ? info.refType : null, + selfType: null, + value, + line: lineOf(lineMap, start), + start, + end: start + value.length, + }); + } + } } /** Converts a shallow scan (offsets) into index records (1-based lines). */ @@ -173,5 +288,6 @@ export function recordsFromShallow(scan: ShallowDocument, lineMap: LineMap): Ind xpointer: x.xpointer, line: lineOf(lineMap, x.start), })), + references: [], }; } diff --git a/src/indexer/referenceIndex.ts b/src/indexer/referenceIndex.ts new file mode 100644 index 0000000..e7be024 --- /dev/null +++ b/src/indexer/referenceIndex.ts @@ -0,0 +1,219 @@ +/** + * Reverse reference index built from per-file reference records. + * + * The indexer stores compact reference records per file (attribute values, + * simple-content text and inheritFrom, with XSD `refType`/`selfType` context + * captured at parse time). After the include walk, this module resolves every + * record against the final asset maps and produces: + * + * definition key -> reference sites + * + * which powers CodeLens reference counts, semantic Find All References and + * the "unreferenced assets" report. Pure TypeScript: no vscode dependency. + */ + +import { extractIndexRecords, type IndexRecords } from "./records"; +import { + filterAndScoreDefs, + isReferenceTargetType, + type ReferenceLookup, +} from "./refs"; +import { buildSearchPaths, resolveSource } from "./includeResolver"; +import { normKey, recordsHash } from "./caches"; +import { LineMap, parseXml } from "../language/xmlParser"; +import type { AssetDef, ModIndex, ReferenceSite } from "./types"; + +/** A file whose reference records should be resolved. */ +export interface ReferenceRecordSource { + /** Absolute path of the referencing file. */ + file: string; + records: IndexRecords; +} + +/** Stable key identifying one specific asset definition. */ +export function assetDefKey( + def: Pick, +): string { + return `${def.type}\u0000${def.id.toLowerCase()}\u0000${def.file.toLowerCase()}\u0000${def.line}`; +} + +/** + * Resolves per-file reference records against the asset lookup and returns + * the reverse map. A record resolves to every definition that satisfies its + * `refType` / `selfType` context (same strict filtering as go-to-definition), + * so same-name ids of different types never share reference counts. + */ +export function buildReferenceIndex( + sources: Iterable, + lookup: ReferenceLookup, +): Map { + const map = new Map(); + for (const { file, records } of sources) { + for (const ref of records.references) { + const defs = lookup.assetsById.get(ref.value.toLowerCase()); + if (!defs?.length) continue; + const targets = filterAndScoreDefs(defs, ref.refType, ref.selfType); + for (const target of targets) { + const key = assetDefKey(target.def); + let sites = map.get(key); + if (!sites) { + sites = []; + map.set(key, sites); + } + sites.push({ + file, + line: ref.line, + start: ref.start, + end: ref.end, + kind: ref.kind, + }); + } + } + } + return map; +} + +/** Reference sites for one definition (empty when the index has none). */ +export function referenceSitesForDef( + idx: Pick | null | undefined, + def: Pick, +): ReferenceSite[] { + return idx?.references?.get(assetDefKey(def)) ?? []; +} + +function normFileKey(path: string): string { + return path.replace(/\\/g, "/").toLowerCase(); +} + +/** + * Reference sites that belong to a definition opened in the editor. + * + * Besides the definition's own reverse-index bucket, this unions the sites + * of manifest definitions that map back to the same XML source file via + * `manifestSource`. A manifest asset with a resolvable SageXml source is + * semantically the same asset as that XML definition, so references to it + * should show up on the source file's CodeLens too (Find All References + * already sees them because it unions every same-id/type definition). + */ +export function referenceSitesForDefinition( + idx: ModIndex, + def: Pick, +): ReferenceSite[] { + const sites = referenceSitesForDef(idx, def); + const byId = idx.assets.get(def.type)?.get(def.id.toLowerCase()); + if (!byId?.length) return sites; + + const defFile = normFileKey(def.file); + const searchPaths = buildSearchPaths(idx.sdkDir, idx.projectDir); + const seen = new Set( + sites.map((s) => `${s.file}\u0000${s.start}\u0000${s.end}\u0000${s.kind}`), + ); + for (const other of byId) { + if (other.origin !== "manifest" || !other.manifestSource) continue; + const resolved = resolveSource(other.manifestSource, null, searchPaths).path; + if (!resolved || normFileKey(resolved) !== defFile) continue; + for (const site of referenceSitesForDef(idx, other)) { + const key = `${site.file}\u0000${site.start}\u0000${site.end}\u0000${site.kind}`; + if (seen.has(key)) continue; + seen.add(key); + sites.push(site); + } + } + return sites; +} + +export interface UnreferencedOptions { + /** + * When true (default), only report types that are reference targets by + * design. Auto-registered / structural types (settings, map metadata, + * w3x sub-assets...) are excluded because zero references is their normal + * state. + */ + onlyReferenceTargetTypes?: boolean; +} + +/** + * Project asset definitions with zero incoming references, grouped by type. + * Manifest / SDK definitions and `instance`-only assets are never reported + * (they are not part of the compiled stream in the same way). + */ +export function unreferencedByType( + idx: ModIndex, + options: UnreferencedOptions = {}, +): Map { + const onlyReferenceTargets = options.onlyReferenceTargetTypes ?? true; + const out = new Map(); + for (const [type, byId] of idx.assets) { + if (onlyReferenceTargets && !isReferenceTargetType(type)) continue; + const defs: AssetDef[] = []; + for (const arr of byId.values()) { + for (const def of arr) { + if (def.origin !== "project" || def.viaInstance) continue; + if (referenceSitesForDef(idx, def).length > 0) continue; + defs.push(def); + } + } + if (defs.length) { + defs.sort((a, b) => a.id.localeCompare(b.id)); + out.set(type, defs); + } + } + return out; +} + +/** Minimal workspace surface needed by the records-desync self-heal. */ +export interface RecordsSyncWorkspace { + index: ModIndex | null; + invalidate(path: string): void; + scheduleRebuild(reason: string): void; +} + +/** Minimal document surface (vscode.TextDocument subset, no vscode dep). */ +export interface RecordsSyncDocument { + uri: { fsPath: string; scheme?: string }; + isDirty?: boolean; + getText(): string; +} + +/** + * True when a clean (saved) document's text no longer matches the records the + * published snapshot was built from. This catches cache entries that slipped + * through stat validation (e.g. a rewrite with preserved timestamps on an + * external drive) or watcher events lost during a drive reconnect. + */ +export function documentRecordsDesynced( + idx: ModIndex, + fsPath: string, + text: string, +): boolean { + const expected = idx.recordsHashes?.get(normKey(fsPath)); + if (expected == null) return false; + const lineMap = new LineMap(text); + const records = extractIndexRecords(parseXml(text), lineMap, text); + return recordsHash(records) !== expected; +} + +/** + * Self-heal: when the open, saved document's content differs from the + * snapshot's records hash, invalidate exactly that file and schedule a + * rebuild. Returns true when a rebuild was scheduled. Unsaved (dirty) + * documents are skipped — the editor text is intentionally ahead of disk. + */ +export function scheduleRebuildIfRecordsDesync( + ws: RecordsSyncWorkspace, + document: RecordsSyncDocument, +): boolean { + if ( + document.isDirty || + (document.uri.scheme != null && document.uri.scheme !== "file") + ) { + return false; + } + const idx = ws.index; + if (!idx) return false; + const fsPath = document.uri.fsPath; + if (!documentRecordsDesynced(idx, fsPath, document.getText())) return false; + ws.invalidate(fsPath); + ws.scheduleRebuild("records-desync"); + return true; +} diff --git a/src/indexer/refs.ts b/src/indexer/refs.ts index ed384aa..50d205e 100644 --- a/src/indexer/refs.ts +++ b/src/indexer/refs.ts @@ -1,17 +1,33 @@ import { + allTypeNames, attributesOfType, + canonicalTypeName, elementTypeName, isAssignableTo, typeChain, typeInfo, } from "../model/schemaModel"; -import type { AssetDef, ModIndex } from "./types"; +import type { AssetDef, LocalOverlay } from "./types"; export interface ReferenceTarget { def: AssetDef; score: number; } +/** + * The subset of `ModIndex` that reference resolution needs. Kept narrow so + * the reverse reference index can resolve records against the indexer's live + * maps without constructing a full index snapshot. + */ +export interface ReferenceLookup { + /** type -> id -> definitions. */ + assets: Map>; + /** id -> definitions across all types. */ + assetsById: Map; + /** Optional document-local overlay (consulted first). */ + local?: LocalOverlay; +} + /** * True when an attribute is a "pipeline-local" reference that the global * asset index cannot judge: @@ -78,7 +94,7 @@ export function isReferenceAttributeOfType( * Returns [] when the attribute is not a typed reference or nothing matches. */ export function resolveReferenceTargets( - idx: ModIndex, + idx: ReferenceLookup, elementType: string, attrName: string, id: string, @@ -93,7 +109,7 @@ export function resolveReferenceTargets( /** Same resolution, driven by a resolved XSD type name. */ export function resolveReferenceTargetsForType( - idx: ModIndex, + idx: ReferenceLookup, typeName: string | null, attrName: string, id: string, @@ -148,7 +164,7 @@ export function isReferenceContentType(typeName: string | null): boolean { * (e.g. `GameObjectWeakRef` -> `GameObject`). */ export function resolveContentReferenceTargets( - idx: ModIndex, + idx: ReferenceLookup, typeName: string | null, id: string, ): ReferenceTarget[] { @@ -164,7 +180,7 @@ export function resolveContentReferenceTargets( return filterAndScoreDefs(defs, refType, null); } -function filterAndScoreDefs( +export function filterAndScoreDefs( defs: readonly AssetDef[], refType: string | null, selfType: string | null, @@ -204,3 +220,49 @@ export function mergeLocalAndGlobalDefs( } return out; } + +let referenceTargetTypeSet: Set | null = null; + +/** + * The set of XSD types that are "reference targets by design": at least one + * typed reference attribute / simple-content reference points at them, or + * they are inheritable (`inheritFrom`). Types outside this set are + * auto-registered / structural (settings, map metadata, w3x sub-assets...), + * so a zero reference count is their normal state and counts would only be + * noise. + */ +export function referenceTargetTypes(): ReadonlySet { + if (referenceTargetTypeSet) return referenceTargetTypeSet; + const set = new Set(); + const add = (t: string | null) => { + if (!t) return; + set.add(canonicalTypeName(t) ?? t); + }; + for (const typeName of allTypeNames()) { + const info = typeInfo(typeName); + if (!info) continue; + if (info.kind === "complex") { + for (const attr of info.attributes) { + if (isLocalReferenceAttribute(typeName, attr.name)) continue; + if (attr.refType) add(attr.refType); + } + if (info.attributes.some((a) => a.name.toLowerCase() === "inheritfrom")) { + add(typeName); + } + } else if ( + info.kind === "simple" && + info.refType && + !typeChain(typeName).includes("Poid") + ) { + add(info.refType); + } + } + referenceTargetTypeSet = set; + return set; +} + +/** True when the type is a designed reference target (see above). */ +export function isReferenceTargetType(typeName: string | null): boolean { + if (!typeName) return false; + return referenceTargetTypes().has(canonicalTypeName(typeName) ?? typeName); +} diff --git a/src/indexer/types.ts b/src/indexer/types.ts index 443a5fa..7c1fb15 100644 --- a/src/indexer/types.ts +++ b/src/indexer/types.ts @@ -24,6 +24,22 @@ export interface AssetDef { manifestSource?: string; } +/** + * One resolved reference occurrence pointing at an asset definition. + * Produced by `buildReferenceIndex` from per-file reference records. + */ +export interface ReferenceSite { + /** Absolute path of the referencing file. */ + file: string; + /** 1-based line of the reference value. */ + line: number; + /** Character offset of the value start (relative to the file text). */ + start: number; + /** Character offset one past the value end. */ + end: number; + kind: "attr" | "content"; +} + export interface DefineDef { name: string; value: string; @@ -121,6 +137,8 @@ export interface IndexStats { /** Time spent shallow-scanning deferred art assets (ms). */ artScanMs: number; assetCount: number; + /** Total resolved reference sites in the reverse index. */ + referenceCount: number; defineCount: number; manifestFiles: number; manifestAssetCount: number; @@ -159,6 +177,20 @@ export interface ModIndex { sourceCandidates: SourceCandidate[]; /** Problems found while indexing (unresolved includes, cycles, ...). */ diagnostics: IndexerDiagnostic[]; + /** + * Reverse reference index: asset definition key (see + * `referenceIndex.assetDefKey`) -> reference sites. Built from the compact + * per-file reference records when a snapshot is published, so counts and + * Find All References share one semantic source of truth. + */ + references: Map; + /** + * normKey(file) -> SHA-1 of the file's compact index records as this + * snapshot consumed them. Lets open documents detect "my file changed on + * disk but the index still uses older records" and trigger a targeted + * rebuild, while ignoring cosmetic text changes (line endings, whitespace). + */ + recordsHashes: Map; stats: IndexStats; /** * Document-local overlay (when the index was obtained through the diff --git a/src/model/schemaModel.ts b/src/model/schemaModel.ts index 07a16a2..9325e6c 100644 --- a/src/model/schemaModel.ts +++ b/src/model/schemaModel.ts @@ -92,6 +92,11 @@ export const modelMeta = { typeCount: Object.keys(model.types).length, }; +/** All type names in the XSD model (complex + simple), in model order. */ +export function allTypeNames(): string[] { + return Object.keys(model.types); +} + export function topLevelElements(): string[] { return model.topLevelElements; } diff --git a/src/workspace.ts b/src/workspace.ts index 2bdafc0..7600659 100644 --- a/src/workspace.ts +++ b/src/workspace.ts @@ -92,7 +92,7 @@ export class ModWorkspace { this.settings = readSettings(); const storageUri = context.storageUri ?? context.globalStorageUri; if (storageUri) { - this.diskCachePath = join(storageUri.fsPath, "index-records-v1.json.gz"); + this.diskCachePath = join(storageUri.fsPath, "index-records-v3.json.gz"); } this.output = vscode.window.createOutputChannel("RA3 Mod XML"); this.statusBar = vscode.window.createStatusBarItem( @@ -145,6 +145,11 @@ export class ModWorkspace { async initialize(): Promise { this.projectRoot = this.detectProjectRoot(); + void vscode.commands.executeCommand( + "setContext", + "ra3modxml.active", + this.projectRoot != null, + ); if (!this.projectRoot) { this.statusBar.hide(); return; @@ -346,6 +351,7 @@ export class ModWorkspace { stat: rec.stat, records: rec.records, kind: rec.kind, + contentHash: rec.contentHash, }); } } @@ -451,6 +457,7 @@ export class ModWorkspace { `${s.projectDir}\n` + `${s.indexedFiles} files indexed (${s.parsedFiles} parsed, ${s.shallowScannedFiles} art assets shallow-scanned, ${(s.elapsedMs / 1000).toFixed(1)}s)\n` + `${s.assetCount} assets (${s.manifestAssetCount} from ${s.manifestFiles} manifests)\n` + + `${s.referenceCount} reference sites\n` + `${s.defineCount} defines, ${s.streams} streams, ${s.sourceCandidates} include candidates\n` + `Phase: ${s.phase} · Complete: ${s.complete}${stale}`; } @@ -589,7 +596,7 @@ export class ModWorkspace { return { file: { path: resolve(path), stat: null }, parse, - records: extractIndexRecords(parse, lineMap), + records: extractIndexRecords(parse, lineMap, text), lineMap, }; } catch { diff --git a/test/caches.test.mjs b/test/caches.test.mjs index 20f9f57..a6590ed 100644 --- a/test/caches.test.mjs +++ b/test/caches.test.mjs @@ -52,7 +52,7 @@ test("IndexRecordsCache stores and invalidates entries", () => { const cache = new IndexRecordsCache(); const entry = { stat, - records: { assets: [], defines: [], includes: [], rootXiIncludes: [], nestedXiIncludes: [] }, + records: { assets: [], defines: [], includes: [], rootXiIncludes: [], nestedXiIncludes: [], references: [] }, kind: "full", }; cache.set("a.xml", entry); @@ -65,7 +65,7 @@ test("IndexRecordsCache exposes entries for disk persistence", () => { const cache = new IndexRecordsCache(); const entry = { stat, - records: { assets: [], defines: [], includes: [], rootXiIncludes: [], nestedXiIncludes: [] }, + records: { assets: [], defines: [], includes: [], rootXiIncludes: [], nestedXiIncludes: [], references: [] }, kind: "full", }; cache.set("a.xml", entry); diff --git a/test/codeLens.test.mjs b/test/codeLens.test.mjs new file mode 100644 index 0000000..bc64fe9 --- /dev/null +++ b/test/codeLens.test.mjs @@ -0,0 +1,210 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { createRequire } from "node:module"; +import { fileURLToPath } from "node:url"; +import { dirname, join, resolve } from "node:path"; + +// Minimal vscode shim: the CodeLens provider only constructs CodeLens/Range. +class Range { + constructor(start, end) { + this.start = start; + this.end = end; + } +} +class CodeLens { + constructor(range, command) { + this.range = range; + this.command = command; + } +} + +const require = createRequire(import.meta.url); +const Module = require("module"); +const origResolve = Module._resolveFilename; +Module._resolveFilename = function (request, ...args) { + if (request === "vscode") return "vscode-stub"; + return origResolve.call(this, request, ...args); +}; +require.cache["vscode-stub"] = { + id: "vscode-stub", + filename: "vscode-stub", + loaded: true, + exports: { + Range, + CodeLens, + }, +}; + +const { Ra3CodeLensProvider } = require("../out/features/codeLens.js"); +const { assetDefKey } = require("../out/indexer/referenceIndex.js"); +const { normKey } = require("../out/indexer/caches.js"); + +const root = dirname(dirname(fileURLToPath(import.meta.url))); +const PROJECT = join(root, "test", "fixtures", "minimod"); +const SDK = join(root, "test", "fixtures", "fakesdk"); +// A real file whose path matches what DATA:Includes/Units.xml resolves to. +const FILE = resolve(join(PROJECT, "Data", "Includes", "Units.xml")); +const TEXT = ` + + + +`; + +function makeDocument(text = TEXT) { + const lineStarts = [0]; + for (let i = 0; i < text.length; i++) { + if (text.charCodeAt(i) === 10) lineStarts.push(i + 1); + } + return { + uri: { fsPath: FILE }, + getText: () => text, + positionAt: (offset) => { + let lo = 0; + let hi = lineStarts.length - 1; + while (lo < hi) { + const mid = (lo + hi + 1) >> 1; + if (lineStarts[mid] <= offset) lo = mid; + else hi = mid - 1; + } + return { line: lo, character: offset - lineStarts[lo] }; + }, + }; +} + +function makeIndex() { + const tankSite = { + file: "C:/mod/Data/Other.xml", + line: 7, + start: 40, + end: 48, + kind: "content", + }; + const secondSite = { + file: "C:/mod/Data/Third.xml", + line: 2, + start: 12, + end: 20, + kind: "attr", + }; + const references = new Map(); + references.set( + assetDefKey({ + type: "GameObject", + id: "TestTank", + file: FILE, + line: 2, + }), + [tankSite, secondSite], + ); + references.set( + assetDefKey({ + type: "GameObject", + id: "BaseVehicle", + file: FILE, + line: 3, + }), + [], + ); + return { + references, + assets: new Map(), + sdkDir: SDK, + projectDir: PROJECT, + }; +} + +test("CodeLens shows counts on reference-target types only, including zero", () => { + const provider = new Ra3CodeLensProvider({ + isRa3Workspace: () => true, + index: makeIndex(), + }); + const lenses = provider.provideCodeLenses(makeDocument(), {}); + + assert.equal(lenses.length, 2, "no lens for auto-registered CameraSettings"); + const tank = lenses.find((l) => l.command.arguments[0].id === "TestTank"); + const base = lenses.find((l) => l.command.arguments[0].id === "BaseVehicle"); + + assert.ok(tank, "GameObject TestTank gets a lens"); + assert.equal(tank.command.title, "2 references"); + assert.equal(tank.command.command, "ra3modxml.showReferences"); + assert.equal(tank.command.arguments[0].type, "GameObject"); + assert.equal(tank.command.arguments[0].line, 2); + + assert.ok(base, "zero is still displayed for reference-target types"); + assert.equal(base.command.title, "0 references"); + + // Lenses anchor on the element start tag. + assert.equal(tank.range.start.line, 1); + assert.ok(tank.range.start.character < tank.range.end.character); +}); + +test("CodeLens returns nothing without a workspace or index", () => { + const noWorkspace = new Ra3CodeLensProvider({ + isRa3Workspace: () => false, + index: makeIndex(), + }); + assert.deepEqual(noWorkspace.provideCodeLenses(makeDocument(), {}), []); + + const noIndex = new Ra3CodeLensProvider({ + isRa3Workspace: () => true, + index: null, + }); + assert.deepEqual(noIndex.provideCodeLenses(makeDocument(), {}), []); +}); + +test("CodeLens counts references attached to a manifest definition with the same SageXml source", () => { + const manifestDef = { + type: "GameObject", + id: "TestTank", + file: resolve(join(SDK, "builtmods", "static.manifest")), + line: 0, + origin: "manifest", + manifestSource: "DATA:Includes/Units.xml", + }; + const site = { + file: "C:/mod/Data/Other.xml", + line: 7, + start: 40, + end: 48, + kind: "content", + }; + const references = new Map(); + references.set(assetDefKey(manifestDef), [site]); + const idx = { + references, + assets: new Map([ + ["GameObject", new Map([["testtank", [manifestDef]]])], + ]), + sdkDir: SDK, + projectDir: PROJECT, + }; + const provider = new Ra3CodeLensProvider({ + isRa3Workspace: () => true, + index: idx, + }); + const lenses = provider.provideCodeLenses(makeDocument(), {}); + const tank = lenses.find((l) => l.command.arguments[0].id === "TestTank"); + assert.ok(tank, "lens is shown for the SageXml-backed definition"); + assert.equal(tank.command.title, "1 reference"); +}); + +test("CodeLens schedules a targeted rebuild when the open document desyncs from the snapshot", () => { + const idx = makeIndex(); + idx.recordsHashes = new Map([[normKey(FILE), "stale-hash"]]); + const calls = []; + const provider = new Ra3CodeLensProvider({ + isRa3Workspace: () => true, + index: idx, + invalidate: (p) => calls.push(["invalidate", p]), + scheduleRebuild: (r) => calls.push(["schedule", r]), + }); + provider.provideCodeLenses(makeDocument(), {}); + assert.ok( + calls.some(([kind]) => kind === "invalidate"), + "the stale file is invalidated", + ); + assert.ok( + calls.some(([kind, reason]) => kind === "schedule" && reason === "records-desync"), + "a targeted rebuild is scheduled", + ); +}); diff --git a/test/diskCache.test.mjs b/test/diskCache.test.mjs index 84c5a64..f0bea03 100644 --- a/test/diskCache.test.mjs +++ b/test/diskCache.test.mjs @@ -19,6 +19,7 @@ const sampleRecords = { includes: [{ type: "all", source: "Units.xml", line: 4 }], rootXiIncludes: [], nestedXiIncludes: [], + references: [], }; function stampOf(file) { @@ -47,7 +48,12 @@ test("disk cache roundtrip keeps records and leaves no temp file", async (t) => await cache.save([ [ file.toLowerCase(), - { stat: stampOf(file), records: sampleRecords, kind: "full" }, + { + stat: stampOf(file), + records: sampleRecords, + kind: "full", + contentHash: "abc123", + }, ], ]); assert.equal(fs.existsSync(`${filePath}.tmp`), false, "atomic write leaves no temp"); @@ -60,6 +66,7 @@ test("disk cache roundtrip keeps records and leaves no temp file", async (t) => assert.equal(stats.dropped, 0); assert.equal(records.length, 1); assert.deepEqual(records[0].records, sampleRecords); + assert.equal(records[0].contentHash, "abc123"); }); test("stat mismatch drops the cached entry", async (t) => { diff --git a/test/localScope.test.mjs b/test/localScope.test.mjs index 70dc478..55b1ca3 100644 --- a/test/localScope.test.mjs +++ b/test/localScope.test.mjs @@ -31,7 +31,7 @@ async function readParsed(path) { return { file: { path, stat: null }, parse, - records: extractIndexRecords(parse, lineMap), + records: extractIndexRecords(parse, lineMap, text), lineMap, }; } diff --git a/test/records.test.mjs b/test/records.test.mjs index 3ece3d5..619b0db 100644 --- a/test/records.test.mjs +++ b/test/records.test.mjs @@ -24,7 +24,7 @@ test("extractIndexRecords mirrors the walk semantics", () => { `; const lineMap = new LineMap(text); - const records = extractIndexRecords(parseXml(text), lineMap); + const records = extractIndexRecords(parseXml(text), lineMap, text); assert.deepEqual( records.assets.map((a) => [a.type, a.id, a.line]), [ @@ -62,4 +62,54 @@ test("recordsFromShallow converts offsets to 1-based lines", () => { assert.equal(records.assets[0].type, "W3DContainer"); assert.equal(records.assets[0].id, "A"); assert.equal(records.assets[0].line, 2); + assert.deepEqual(records.references, []); +}); + +test("extractIndexRecords records typed references and skips non-references", () => { + const text = ` + + + + Tank + + + +`; + const lineMap = new LineMap(text); + const records = extractIndexRecords(parseXml(text), lineMap, text); + + const attrs = records.references.filter((r) => r.kind === "attr"); + const content = records.references.filter((r) => r.kind === "content"); + + // Typed attribute reference keeps the XSD refType. + const cs = attrs.find((r) => r.value === "CS"); + assert.ok(cs, "CommandSet reference is recorded"); + assert.equal(cs.refType, "LogicCommandSet"); + assert.equal(cs.selfType, null); + assert.equal(cs.line, 2); + assert.equal(records.references.some((r) => r.start === cs.start && r.end === cs.end), true); + + // inheritFrom records the element type as selfType instead of refType. + const base = attrs.find((r) => r.value === "Base"); + assert.ok(base, "inheritFrom reference is recorded"); + assert.equal(base.refType, null); + assert.equal(base.selfType, "GameObject"); + + // Enums and non-reference values are not references. + assert.equal(attrs.some((r) => r.value === "SELECTABLE"), false); + + // Simple-content text is recorded with its content refType and offsets. + const tank = content.find((r) => r.value === "Tank"); + assert.ok(tank, "content reference is recorded"); + assert.equal(tank.refType, "GameObject"); + const line = text.split("\n")[4]; + assert.equal(line.slice(tank.start - text.indexOf(line), tank.end - text.indexOf(line)), "Tank"); + + // The id definition itself is never recorded as a reference. + assert.equal( + records.references.some( + (r) => r.value === "Tank" && r.kind === "attr", + ), + false, + ); }); diff --git a/test/referenceIndex.test.mjs b/test/referenceIndex.test.mjs new file mode 100644 index 0000000..0dab5e2 --- /dev/null +++ b/test/referenceIndex.test.mjs @@ -0,0 +1,378 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { fileURLToPath } from "node:url"; +import { dirname, join } from "node:path"; +import { mkdtempSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { ModIndexer } from "../out/indexer/indexer.js"; +import { CachedDirectoryWalker } from "../out/indexer/fileScanner.js"; +import { + IndexRecordsCache, + contentHash, + normKey, + recordsHash, +} from "../out/indexer/caches.js"; +import { + assetDefKey, + buildReferenceIndex, + documentRecordsDesynced, + referenceSitesForDef, + referenceSitesForDefinition, + scheduleRebuildIfRecordsDesync, + unreferencedByType, +} from "../out/indexer/referenceIndex.js"; +import { LineMap, parseXml } from "../out/language/xmlParser.js"; +import { extractIndexRecords } from "../out/indexer/records.js"; + +const root = dirname(dirname(fileURLToPath(import.meta.url))); +const project = join(root, "test", "fixtures", "minimod"); +const sdk = join(root, "test", "fixtures", "fakesdk"); + +async function buildIndex() { + const indexer = new ModIndexer({ + projectDir: project, + sdkDir: sdk, + builtmodsDirs: [join(sdk, "builtmods")], + indexSageXml: true, + additionalDataSearchPaths: [], + walker: new CachedDirectoryWalker(), + }); + return indexer.build(); +} + +function makeDef(type, id, file, line, extra = {}) { + return { + type, + id, + file, + line, + origin: "project", + ...extra, + }; +} + +test("buildReferenceIndex resolves records with strict type filtering", () => { + const tank = makeDef("GameObject", "Tank", "C:/mod/A.xml", 2); + const gun = makeDef("WeaponTemplate", "Tank", "C:/mod/B.xml", 1); + const cs = makeDef("LogicCommandSet", "CS", "C:/mod/C.xml", 4); + const lookup = { + assets: new Map(), + assetsById: new Map([ + ["tank", [tank, gun]], + ["cs", [cs]], + ]), + }; + const recordsA = { + assets: [], + defines: [], + includes: [], + rootXiIncludes: [], + nestedXiIncludes: [], + references: [ + { + kind: "content", + refType: "GameObject", + selfType: null, + value: "Tank", + line: 5, + start: 40, + end: 44, + }, + ], + }; + const recordsB = { + assets: [], + defines: [], + includes: [], + rootXiIncludes: [], + nestedXiIncludes: [], + references: [ + { + kind: "attr", + refType: "LogicCommandSet", + selfType: null, + value: "CS", + line: 3, + start: 10, + end: 12, + }, + ], + }; + + const map = buildReferenceIndex( + [ + { file: "C:/mod/A.xml", records: recordsA }, + { file: "C:/mod/B.xml", records: recordsB }, + ], + lookup, + ); + + // The content reference resolves only to the GameObject definition, never + // to the same-name WeaponTemplate. + const tankSites = map.get(assetDefKey(tank)); + assert.equal(tankSites.length, 1); + assert.equal(tankSites[0].file, "C:/mod/A.xml"); + assert.equal(tankSites[0].kind, "content"); + assert.equal(map.get(assetDefKey(gun)), undefined); + assert.equal(map.get(assetDefKey(cs)).length, 1); +}); + +test("records extracted from XML resolve through the reference index", () => { + const text = ` + + + + + Tank + + +`; + const lineMap = new LineMap(text); + const records = extractIndexRecords(parseXml(text), lineMap, text); + const file = "C:/mod/D.xml"; + const tank = makeDef("GameObject", "Tank", file, 2); + const cs = makeDef("LogicCommandSet", "CS", file, 3); + const lookup = { + assets: new Map(), + assetsById: new Map([ + ["tank", [tank]], + ["cs", [cs]], + ]), + }; + const map = buildReferenceIndex([{ file, records }], lookup); + + const tankSites = map.get(assetDefKey(tank)); + assert.equal(tankSites.length, 1); + assert.equal(tankSites[0].kind, "content"); + assert.equal(records.references.find((r) => r.value === "Tank").start, tankSites[0].start); + + const csSites = map.get(assetDefKey(cs)); + assert.equal(csSites.length, 1); + assert.equal(csSites[0].kind, "attr"); +}); + +test("referenceSitesForDefinition unions manifest-source sites onto the SageXml source file", () => { + const sourceFile = join(project, "Data", "Includes", "Units.xml"); + const manifestDef = { + type: "GameObject", + id: "Tank", + file: join(sdk, "builtmods", "static.manifest"), + line: 0, + origin: "manifest", + manifestSource: "DATA:Includes/Units.xml", + }; + const site = { + file: "C:/mod/ref.xml", + line: 3, + start: 10, + end: 14, + kind: "attr", + }; + const idx = { + assets: new Map([["GameObject", new Map([["tank", [manifestDef]]])]]), + assetsById: new Map([["tank", [manifestDef]]]), + references: new Map([[assetDefKey(manifestDef), [site]]]), + projectDir: project, + sdkDir: sdk, + }; + + const sites = referenceSitesForDefinition(idx, { + type: "GameObject", + id: "Tank", + file: sourceFile, + line: 4, + }); + assert.equal(sites.length, 1); + assert.equal(sites[0].file, "C:/mod/ref.xml"); + + // A different file does not inherit the manifest definition's sites. + const other = referenceSitesForDefinition(idx, { + type: "GameObject", + id: "Tank", + file: "C:/mod/elsewhere.xml", + line: 4, + }); + assert.equal(other.length, 0); +}); + +test("the minimod indexer publishes a semantic reverse reference index", async () => { + const idx = await buildIndex(); + assert.ok(idx.stats.referenceCount > 0, "reverse index is populated"); + + // Units.xml has CommandSet="TestTankCommandSet" and inheritFrom="BaseVehicle". + const lcs = idx.assets.get("LogicCommandSet").get("testtankcommandset")[0]; + const lcsSites = idx.references.get(assetDefKey(lcs)); + assert.ok(lcsSites && lcsSites.length >= 1); + assert.ok(lcsSites.some((s) => s.kind === "attr")); + + const baseVehicle = idx.assets.get("GameObject").get("basevehicle")[0]; + const bvSites = idx.references.get(assetDefKey(baseVehicle)); + assert.ok(bvSites && bvSites.length >= 1); +}); + +test("references survive a records-cache invalidation during the build", async () => { + const recordsCache = new IndexRecordsCache(); + const indexer = new ModIndexer({ + projectDir: project, + sdkDir: sdk, + builtmodsDirs: [join(sdk, "builtmods")], + indexSageXml: true, + additionalDataSearchPaths: [], + walker: new CachedDirectoryWalker(), + recordsCache, + }); + const unitsPath = join(project, "Data", "Includes", "Units.xml"); + const idx = await indexer.build((phaseIndex) => { + if (!phaseIndex.complete) recordsCache.invalidate(unitsPath); + }); + const lcs = idx.assets.get("LogicCommandSet").get("testtankcommandset")[0]; + const sites = idx.references.get(assetDefKey(lcs)); + assert.ok( + sites && sites.length >= 1, + "final snapshot keeps the walk-time reference records", + ); +}); + +test("force rebuild verifies content even when every stat signal matches", async (t) => { + const tmp = mkdtempSync(join(tmpdir(), "ra3-refidx-")); + t.after(() => rmSync(tmp, { recursive: true, force: true })); + const file = join(tmp, "units.xml"); + const diskText = ``; + const cachedText = ``; + writeFileSync(file, diskText); + const st = statSync(file); + const stamp = { + mtimeMs: st.mtimeMs, + size: st.size, + birthtimeMs: st.birthtimeMs, + ctimeMs: st.ctimeMs, + }; + const cache = new IndexRecordsCache(); + cache.set(file, { + stat: stamp, + records: extractIndexRecords( + parseXml(cachedText), + new LineMap(cachedText), + cachedText, + ), + kind: "full", + contentHash: contentHash(cachedText), + }); + const opts = { + projectDir: tmp, + sdkDir: tmp, + builtmodsDirs: [], + indexSageXml: false, + additionalDataSearchPaths: [], + walker: new CachedDirectoryWalker(), + recordsCache: cache, + }; + + const trusted = new ModIndexer({ ...opts, trustUnchanged: true }); + const trustedParsed = await trusted.readDocument(file); + assert.equal( + trustedParsed.records.assets[0].id, + "Cached", + "trusted rebuilds reuse the cached records without reading", + ); + + const forced = new ModIndexer({ ...opts, trustUnchanged: false }); + const forcedParsed = await forced.readDocument(file); + assert.equal( + forcedParsed.records.assets[0].id, + "Tank", + "force rebuild re-reads a stat-matching but content-stale entry", + ); + assert.equal(cache.get(file).contentHash, contentHash(diskText)); +}); + +test("records-desync self-heal schedules a targeted rebuild only for clean files", () => { + const file = join(project, "Data", "Includes", "Units.xml"); + const emptyRecords = extractIndexRecords( + parseXml(""), + new LineMap(""), + "", + ); + const idx = { + recordsHashes: new Map([[normKey(file), recordsHash(emptyRecords)]]), + references: new Map(), + assets: new Map(), + sdkDir: sdk, + projectDir: project, + }; + assert.equal( + documentRecordsDesynced(idx, file, ""), + false, + ); + assert.equal( + documentRecordsDesynced( + idx, + file, + "", + ), + true, + ); + + let invalidated = null; + let scheduled = null; + const ws = { + index: idx, + invalidate: (p) => { + invalidated = p; + }, + scheduleRebuild: (r) => { + scheduled = r; + }, + }; + assert.equal( + scheduleRebuildIfRecordsDesync(ws, { + uri: { fsPath: file, scheme: "file" }, + isDirty: false, + getText: () => "", + }), + true, + ); + assert.equal(invalidated, file); + assert.equal(scheduled, "records-desync"); + + invalidated = null; + scheduled = null; + assert.equal( + scheduleRebuildIfRecordsDesync(ws, { + uri: { fsPath: file, scheme: "file" }, + isDirty: true, + getText: () => "", + }), + false, + ); + assert.equal(scheduled, null); +}); + +test("unreferencedByType reports only meaningful project assets", async () => { + const idx = await buildIndex(); + const map = unreferencedByType(idx); + const all = [...map.values()].flat(); + + assert.ok(all.length > 0, "some project assets are unreferenced"); + assert.ok( + all.every((d) => d.origin === "project" && !d.viaInstance), + "only compiled-stream project definitions are reported", + ); + assert.ok( + all.every((d) => !d.file.toLowerCase().includes("fakesdk")), + "SDK/manifest definitions are never reported", + ); + + // WeaponTemplate TestTankCannon is defined but never referenced. + const weapons = map.get("WeaponTemplate"); + assert.ok(weapons.some((d) => d.id === "TestTankCannon")); + + // LogicCommandSet TestTankCommandSet is referenced, so it must not appear. + const commandSets = map.get("LogicCommandSet"); + assert.ok(!commandSets?.some((d) => d.id === "TestTankCommandSet")); + + // referenceSitesForDef is stable across lookups and safe without a map. + const tankCannon = idx.assets.get("WeaponTemplate").get("testtankcannon")[0]; + assert.deepEqual(referenceSitesForDef(idx, tankCannon), []); + assert.deepEqual(referenceSitesForDef(null, tankCannon), []); +}); diff --git a/test/referenceProvider.test.mjs b/test/referenceProvider.test.mjs new file mode 100644 index 0000000..c3c1dca --- /dev/null +++ b/test/referenceProvider.test.mjs @@ -0,0 +1,169 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { createRequire } from "node:module"; + +// Minimal vscode shim for the semantic reference provider. +class Position { + constructor(line, character) { + this.line = line; + this.character = character; + } +} +class Range { + constructor(start, end) { + this.start = start; + this.end = end; + } +} +class Location { + constructor(uri, range) { + this.uri = uri; + this.range = range; + } +} +const Uri = { + file: (p) => ({ fsPath: p }), +}; + +const require = createRequire(import.meta.url); +const Module = require("module"); +const origResolve = Module._resolveFilename; +Module._resolveFilename = function (request, ...args) { + if (request === "vscode") return "vscode-stub"; + return origResolve.call(this, request, ...args); +}; +require.cache["vscode-stub"] = { + id: "vscode-stub", + filename: "vscode-stub", + loaded: true, + exports: { + Position, + Range, + Location, + Uri, + SymbolKind: {}, + DocumentSymbol: class {}, + }, +}; + +const { Ra3ReferenceProvider } = require("../out/features/navigation.js"); +const { LineMap, parseXml } = require("../out/language/xmlParser.js"); +const { extractIndexRecords } = require("../out/indexer/records.js"); +const { buildReferenceIndex } = require("../out/indexer/referenceIndex.js"); + +const FILE = "C:/mod/Data/Units.xml"; +const TEXT = ` + + + + Tank + + +`; + +function makeDocument(text = TEXT) { + const lineStarts = [0]; + for (let i = 0; i < text.length; i++) { + if (text.charCodeAt(i) === 10) lineStarts.push(i + 1); + } + return { + uri: { fsPath: FILE }, + getText: () => text, + offsetAt: (pos) => lineStarts[pos.line] + pos.character, + positionAt: (offset) => { + let lo = 0; + let hi = lineStarts.length - 1; + while (lo < hi) { + const mid = (lo + hi + 1) >> 1; + if (lineStarts[mid] <= offset) lo = mid; + else hi = mid - 1; + } + return new Position(lo, offset - lineStarts[lo]); + }, + }; +} + +function makeScope() { + const parse = parseXml(TEXT); + const lineMap = new LineMap(TEXT); + const records = extractIndexRecords(parse, lineMap, TEXT); + const def = { + type: "GameObject", + id: "Tank", + file: FILE, + line: 2, + origin: "project", + }; + const lookup = { + assets: new Map([["GameObject", new Map([["tank", [def]]])]]), + assetsById: new Map([["tank", [def]]]), + }; + const references = buildReferenceIndex([{ file: FILE, records }], lookup); + const idx = { + ...lookup, + references, + complete: true, + phase: "art", + projectDir: "C:/mod", + sdkDir: "C:/sdk", + defines: new Map(), + files: new Map(), + streams: [], + manifests: new Map(), + sourceCandidates: [], + diagnostics: [], + stats: {}, + }; + return { + merged: idx, + }; +} + +function makeWs(scope) { + const parse = parseXml(TEXT); + const lineMap = new LineMap(TEXT); + return { + isRa3Workspace: () => true, + getScope: async () => scope, + indexer: { + readDom: async (path) => + path === FILE + ? { file: { path: FILE }, parse, lineMap, records: null } + : null, + }, + }; +} + +test("semantic Find All References excludes the definition even when includeDeclaration is set", async () => { + const scope = makeScope(); + const provider = new Ra3ReferenceProvider(makeWs(scope)); + const document = makeDocument(); + const defLine = TEXT.split("\n")[1]; + const defPos = new Position(1, defLine.indexOf('id="') + 4); + + const refs = await provider.provideReferences(document, defPos, { + includeDeclaration: true, + }, {}); + assert.ok(refs, "references are returned"); + assert.equal(refs.length, 1, "only the typed content reference is returned"); + assert.equal(refs[0].uri.fsPath, FILE); + assert.equal(refs[0].range.start.line, 4); + assert.equal( + TEXT.split("\n")[4].slice(refs[0].range.start.character, refs[0].range.end.character), + "Tank", + ); +}); + +test("FAR from the reference site itself returns the same result", async () => { + const scope = makeScope(); + const provider = new Ra3ReferenceProvider(makeWs(scope)); + const document = makeDocument(); + const contentLine = TEXT.split("\n")[4]; + const contentPos = new Position(4, contentLine.indexOf("Tank") + 1); + + const refs = await provider.provideReferences(document, contentPos, { + includeDeclaration: false, + }, {}); + assert.equal(refs.length, 1); + assert.equal(refs[0].range.start.line, 4); +});