diff --git a/README.md b/README.md index 1ad3b07..7854e94 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,12 @@ - `` 补全可解析的 `DATA:` / `ART:` / `AUDIO:` 与项目相对路径。 - **悬停提示**:元素/属性显示 XSD 文档、类型、必填/默认值;引用值显示定义位置;`$DEFINE` 显示值与定义位置;`Include source` / `xi:include href` 显示解析后的目标文件;`xi:include` 元素与属性给出 XInclude 说明。 - **引用导航**:从引用值(`CommandSet="..."`、`Weapon="..."`、`inheritFrom`)跳转到定义(严格按引用类型过滤,候选由 `ra3modxml.definitionMode` 控制:`all` 列出 mod + 原版、`project-only` 优先项目内定义);`Ctrl+点击` Include / `xi:include href` 打开目标文件;Find All References 搜索整个工作区;文档大纲列出顶层资产与 `$DEFINE`。 +- **当前文档局部作用域(T1)**:即使一个文件不在任何全局流里(没有从 + `Data/Mod.xml` / `additionalmaps` 可达),插件也会按当前文件自身的资产、 + `$DEFINE` 及其 include 链建立局部索引。`xi:include` 会在逻辑树中展开, + 使 include 进来的内容获得正确的父上下文;`AttachModuleId` / `ModuleId` / + `AutoResolveBody` 等管线局部(Poid)引用可以补全、悬停与跳转到同一 + GameObject 内的模块(含通过 `xi:include` 拼入的兄弟模块)。 - **错误检查**:XML 格式错误、未知元素/属性(`xi:` 等外来命名空间不误报)、顶层资产缺 `id`、重复 ID、未解析引用(含类型不匹配)、Include / 嵌套 `xi:include` 目标找不到、`$DEFINE` 未定义。 - **manifest 支持**:`` 指向的 `static/global/audio.manifest`(SDK `builtmods`)会被解析,manifest 中的原版资产 ID 可用于补全/悬停/导航/诊断。 - **美术资产(`.w3x`)**:`W3X.xml` / `ART:` include 链中的 `.w3x` 模型文件会被 @@ -23,14 +29,24 @@ `Model@Name`、`Hierarchy`、`Mesh` 等引用可以解析、悬停与跳转。超大模型 (几十 MB 的顶点/三角形数据)采用浅扫描——只提取顶层资产记录、不建 DOM 树, 结果在 workspace 级缓存并跨重建复用,保存文件触发的重建不会重读未变化的模型文件。 +- **索引分阶段与部分可用性**:先建立 XML + manifest 索引(首建早期即可用), + w3x 美术资产随后台扫描补齐。索引完成前,语法/模型诊断、枚举与子元素补全、 + Include 跳转/悬停照常工作;引用类诊断会“显示但标注” + (`unresolved-reference-indexing` + `(index incomplete)` 说明),不会把 + 未完成的索引误当成最终结论。 - **大项目性能**:索引记录(资产 / Define / Include / 行号)与 include 解析结果 跨重建缓存,保存触发的重建零 stat、零重读(Corona 实测约 2 秒);DOM 树只按需 - 保留并设元素预算,避免内存膨胀。 + 保留并设元素预算,避免内存膨胀。编辑器外的文件改动(git pull、导出工具)会 + 触发防抖重建;构建期间文件再次被修改时,已发布索引会标记 `(stale)` 并自动重跑。 + include 路径解析使用目录枚举建立的文件集快照(无 statSync 风暴);records + 缓存会持久化到磁盘(gzip + 多信号 stat 校验 + 原子写),重启 VS Code 后冷启动 + 只需秒级校验,Corona 实测约 11 秒(首次全量约 2 分钟)。 ## 使用 1. 用 VS Code 打开 RA3 Mod 项目文件夹(含 `Data/Mod.xml` 或 `mod.babproj`)。 -2. 插件自动激活并开始后台索引(状态栏显示资产数量)。 +2. 插件自动激活并开始后台索引(状态栏显示阶段与资产数量;扩展也会在打开 + 任意 XML 文件时激活,非 RA3 工作区不会显示 RA3 专属功能)。 3. 编辑任意 `*.xml` 即可获得补全、跳转与诊断。 ### 设置(`settings.json`) @@ -48,6 +64,8 @@ - `RA3 Mod XML: Re-index workspace`:手动重建索引。 - `RA3 Mod XML: Show index report`:查看索引统计。 +- `RA3 Mod XML: Clear caches and rebuild`:清空内存/磁盘缓存并强制全量重建。 +- `RA3 Mod XML: Show cache report`:查看磁盘缓存路径、大小、校验统计与命中数。 ## 开发 @@ -77,14 +95,19 @@ src/ schemaModel.ts XSD 模型运行时(schema-model.json / asset-types.json 由 tools 生成) indexer/ includeResolver.ts Include 路径解析(纯 TS,移植 check_duplicate_ids.py) + existence.ts 文件集存在性快照(目录枚举 Set,替代逐路径 statSync) manifestParser.ts .manifest 二进制解析(移植 OpenSAGE ManifestFile.cs) fileScanner.ts 目录扫描与 Include source 候选 refs.ts 引用目标解析(按引用类型过滤) + xpointer.ts xi:include xpointer 子集解析(纯 TS) + logicalTree.ts 当前文档逻辑树(xi:include 拼接、局部作用域) + localScope.ts 文档局部索引 overlay(自身链 + include 链) shallowScan.ts .w3x 等大体积美术资产顶层浅扫描(纯 TS,不建 DOM) records.ts 每文件紧凑索引记录(资产/Define/Include/xi + 行号) caches.ts 跨重建持久缓存(DocumentCache / IndexRecordsCache / - IncludeResolveCache) - indexer.ts 工作区索引器(后台、缓存、记录驱动重建) + IncludeResolveCache)+ 失效纪元 InvalidationsEpoch + diskCache.ts 跨会话磁盘缓存(gzip JSON、原子写、多信号 stat 校验) + indexer.ts 工作区索引器(后台、缓存、记录驱动重建、分阶段发布) features/ completion / hover / navigation / diagnostics / semanticTokens syntaxes/ TextMate 注入语法 tools/ XSD → 模型、AssetType 枚举提取 diff --git a/docs/analysis-issues.md b/docs/analysis-issues.md index cce1246..2ad3264 100644 --- a/docs/analysis-issues.md +++ b/docs/analysis-issues.md @@ -607,7 +607,8 @@ AttachTest `Harbinger Gunship\GameObject.xml` 中 ` 两条是同一缺失定义的两种呈现(诊断 + hover),不是两个独立 bug。 `W3DContainer:AUGunship_SKN` 确实由 mod 定义——但定义在 -`Harbinger Gunship\W3X\AUGUNSHIP_SKN.w3x` 里,而索引器只解析 `.xml` / `.manifestxml`。 +`Harbinger Gunship\W3X\AUGUNSHIP_SKN.w3x` 里,而索引器只解析 `.xml` +(manifest 是 `*.manifest` 二进制,不存在 `.manifestxml` 源码格式)。 ### 关键事实(实测) @@ -644,7 +645,7 @@ AttachTest `Harbinger Gunship\GameObject.xml` 中 ` 单次线性扫描,只提取顶层元素 `name + id`(含精确 offset)、顶层 `` 的 `Include@type/source`、任意层级 `` 的 `href/xpointer`、`` 常量;注释 / CDATA / DOCTYPE / PI 整体跳过;属性解析兼容引号内 `>` 与 `/`。 -2. **索引模式三分**:`.xml`/`.manifestxml` 全量解析(4 MB 上限不变); +2. **索引模式三分**:`.xml` 全量解析(4 MB 上限不变); `.w3x` 一律浅扫描;未知扩展名先嗅探文件头(512 字节,BOM/空白后以 `<` 开头且无 NUL 字节 → 按 XML 浅扫描,否则按二进制仅登记)。w3x 自身的 `` 与 嵌套 `xi:include` 会继续被 walk(BAB 语义)。 @@ -661,7 +662,8 @@ AttachTest `Harbinger Gunship\GameObject.xml` 中 ` - 单元测试 **53 → 63 全绿**:新增 `test/shallowScan.test.mjs`(顶层资产 / Includes / xi:include / Defines / 引号内 `>` / CDATA / 未闭合标签 / 数值载荷不产生记录); - indexer 新增 w3x 链、`.w3d` 嗅探、二进制 `.dds` 跳过、跨重建缓存命中、 + indexer 新增 w3x 链、未知扩展名 XML 嗅探(fixture 用 `.dat`)、二进制 `.dds` + 跳过、跨重建缓存命中、 BOM 偏移断言;xmlParser 新增 `stripBom` 用例。 - AttachTest 实机: - 首次构建 1.2 s,浅扫 62 个文件;`Model@Name=AUGunship_SKN` → @@ -762,3 +764,303 @@ resolve 缓存命中与"信任重建 0 次重解析"断言。 并行浅扫描(worker)或把 w3x 顶层记录持久化到磁盘缓存。 - `ra3modxml.reindex` 出于正确性会清空解析缓存(可能 ~15-25s);如接受 watcher 可靠性可改为保留。 + +--- + +## 十五、问题分析(第十轮,2026-08-03):索引未完成时的部分可用性与分阶段索引 + +### 目标 + +第九轮遗留:Corona 首次全量建索引约 4 分钟(2.6GB w3x 读取 + 冷 statSync), +期间插件所有功能被整体关闭(`ws.index == null`)。本轮让插件在索引完成前 +分层可用,并把最耗时的 w3x 扫描放到后台阶段。 + +### 现状证据 + +- `diagnostics.update` 无索引时直接清空诊断:连 XML 语法错误、未知元素、缺 id + 这类不依赖索引的检查也被关闭; +- 补全的 `attribute-value` / `content` 分支无索引直接返回空,但枚举 / 布尔 / + Include type / 子元素补全并不需要索引(`contentItems` 甚至没用 idx); +- hover / 文档链接的 Include 解析只依赖搜索路径(settings),不需要索引; +- Find All References 完全不使用索引。 + +### 设计 + +1. **索引状态模型**:`ModIndex` 增加 `complete` / `phase`(`"xml" | "art"`)/ + `stale`;`stats` 增加 `phase` / `complete` / `deferredArtFiles` / + `artScanMs`。 +2. **分阶段索引**: + - 阶段 A(xml):walk include 链时只**登记** w3x / 嗅探 XML 文件 + (`readDocument` 的 `deferArt` 模式),不读内容;manifest、项目 XML、 + mapmetadata 全部正常索引;结束后发布不可变快照; + - 阶段 B(art):按队列浅扫描 w3x、应用资产记录,并继续走 w3x 内的 + Include / xi:include(此时新遇到的美术文件立即扫描,不再入队); + - 快照不可变性的关键:`addAsset` 在阶段 B 仍会向数组 push,所以阶段 A + 发布时必须复制 `assets` / `assetsById` / `defines` / `streams.files` / + `diagnostics`(`snapshotIndex` 深拷贝嵌套 Map/数组)。 +3. **部分可用性(T0 解耦)**: + - 诊断:语法错误、未知元素/属性、缺 id、同文件重复 ID、Include 目标存在性 + 不再依赖索引;引用 / `$DEFINE` / 跨文件重复在索引未完成或 stale 时 + “显示但标注”:code 变为 `unresolved-reference-indexing` / + `undefined-define-indexing`,消息追加 `(index incomplete — may be a + false positive)`;跨文件重复追加 `(based on a partial index)`; + - 补全:枚举 / 布尔 / Include type / `xai:joinAction` / 子元素(content) + 在无索引时可用;资产 ID / define / include source 仍需索引; + - hover / 文档链接:Include source / `xi:include href` 用 settings 搜索路径 + 即可解析;无索引时引用值 hover 提示 “Index is still building”。 +4. **构建中文件被修改**: + - watcher 的 change / create / delete 现在都会 `scheduleRebuild()`(此前 + 只 `invalidate`,外部修改后没有任何东西触发重建); + - 新增 `InvalidationsEpoch`:`invalidate` / `invalidateExistence` 递增; + 构建开始时记录 epoch,发布每个快照(含最终)时若 epoch 变化则标记 + `stale`(状态栏显示 `(stale)`);dirty 机制保证构建结束后立即再重建收敛; + - 构建失败时保留上一个可用快照并标记 stale,不再清空索引。 +5. **stat 结构扩展**:`IndexedFile.stat` 从 `{ mtimeMs, size }` 变为 + `{ mtimeMs, size, birthtimeMs, ctimeMs }`,为磁盘持久化缓存铺路 + (FAT32 的 mtime 只有 2 秒粒度,多信号可捕捉“保留 mtime 的整体替换写入”)。 + +### 验证 + +- 单元测试 **73 → 79 全绿**: + - indexer:phase-A 快照发布(XML/manifest 资产可用、美术资产缺席)、快照 + 不可变性、`deferredArtFiles` 统计、`artScanMs`、mtime 变更强制重读; + - caches:`InvalidationsEpoch` 递增 / 快照语义; + - completion:无索引时枚举 / 元素名 / 属性名 / 子元素补全仍工作。 +- `tsc` + esbuild 构建通过。 + +### 实机复现与补充修复(2026-08-03) + +在 Corona 上实测新版索引器: + +- 阶段 A(xml)**27.0s** 发布:54,283 资产(含 manifest 35,322)、8,399 文件、 + 4,797 个 w3x 待扫;最终 **118.0s**:64,868 资产、8,976 文件、4,829 浅扫 + (artScanMs ≈ 91s,walkMs ≈ 26s)。 +- `OnSeaUnitCrate.xml` 中的引用在阶段 A 即可解析(`Locomotor=JapanEggLocomotor` + 、`CommandSet=EmptyCommandSet` 均有候选);`LocomotorSet@Condition` 是 + 19 个枚举值的模型属性,无索引时即可补全。 + +由此确认两个真实体验问题并修复: + +1. **状态栏初始不显示 indexing**:`workspaceContains` 激活事件在大目录上扫描 + 较慢,打开工作区后扩展尚未激活,看起来“没有任何功能”。`activationEvents` + 增加 `onLanguage:xml`,打开任意 XML 文件即激活;同时各 provider 增加 + `isRa3Workspace()` 守卫,避免在非 RA3 工作区误补全/误诊断。 +2. 索引构建中执行 “Show index report” 提示 “no index available” 有误导 → + 新增 `ws.isBuilding`,构建中改为提示 “index is still building”。 + +版本 **0.1.1 → 0.1.2**(重新打包 `ra3-mod-xml-0.1.2.vsix`)。 + +--- + +## 十六、问题分析(第十一轮,2026-08-03):冷启动磁盘缓存与首建 statSync 消除 + +### 目标 + +第十轮后 Corona 首次建索引仍需 ~2 分钟,且每次新会话(重启 VS Code)都要 +重来一遍。本轮做两件事: + +1. **文件集快照替代 statSync**:`resolveSource` 的逐 base `statSync` 存在性 + 检查(Corona 首建约 11 万次)改为目录枚举建立的 `Set` 查询; +2. **磁盘持久化缓存**:records 缓存(含 w3x 浅扫记录)跨会话落盘,冷启动 + 只做 stat 校验,不再重读 2.6GB 美术资产。 + +### 设计 + +1. **ExistenceSnapshot**(`src/indexer/existence.ts`): + - **惰性按目录 readdir**:只在实际查询某个候选路径时读取它的父目录 + (`readdirSync` + `withFileTypes`,不 stat 单个文件),结果按目录缓存; + 不做首建前的全量递归枚举(该方案曾让 XML 阶段从 27s 涨到 45s); + - 覆盖根判定:候选父目录落在根内 → 目录条目 Set 查询(`hits`);落在 + 根外 → `statSync` 回退(`fallbacks`); + - 盘符根(`C:\` 等)与不存在的根不枚举,避免遍历整个磁盘; + - 子根被更宽的根覆盖时跳过(如 `sdkDir` 覆盖 `sdkDir/SageXml`)。 +2. **DiskRecordsCache**(`src/indexer/diskCache.ts`): + - gzip JSON,原子写(tmp + rename),版本号 + identity key + (项目/SDK/设置哈希,配置变化自动忽略旧缓存); + - 每条记录保存多信号 stamp `{ size, mtimeMs, birthtimeMs, ctimeMs }`; + - 加载时并发(32)stat 校验,不匹配/缺失丢弃,构建时重读; + - 缺失/损坏/身份不符 → 空结果,不报错。 +3. **workspace 集成**: + - 构建前若内存 records 缓存为空,从磁盘加载并校验(状态栏显示 + “validating cache…”);构建成功后异步回写(entries 先快照,避免与 + 下一次构建竞争); + - 新命令 `ra3modxml.clearCache`(清内存 + 删磁盘文件 + 强制重建)与 + `ra3modxml.showCacheReport`(缓存路径/大小/加载校验统计/命中数)。 + +### 实测(Corona) + +| 场景 | 结果 | +|---|---| +| 首次构建 | 118.7s(phaseA **24.0s**);snapshotHits **75,926**、fallbacks **0**、resolveCalls 11,061 | +| 保存缓存 | 0.1s;gzip 后 **651 KB** | +| 加载 + stat 校验 | 0.2s(8,976 条全部校验通过) | +| 新会话二次构建 | **10.9s**(w3x 重扫 0、shallowCacheHits 4,829、recordsCacheHits 4,166) | + +冷启动(加载 + 校验 + 构建)约 **11s**,对比之前每次 ~2.5 分钟。 + +### 测试与验证 + +- 单元测试 **79 → 90 全绿**: + - `existence.test.mjs`:覆盖/未覆盖判定、hits/fallbacks、盘符根识别、 + 搜索 base 枚举、`resolveSource` 快照命中与 statSync 回退; + - `diskCache.test.mjs`:roundtrip、原子写无残留 tmp、stat 变更丢弃、 + identity 不符忽略、损坏文件空结果、clear、key 稳定性; + - `caches.test.mjs` 增加 `entries()`;indexer 统计断言 snapshotHits。 +- `tsc` + esbuild + `vsce package` 通过。 + +### 遗留 + +- 首次构建 phase A 现在包含快照目录枚举成本(Corona 实测约 +10-17s,后续 + 构建由 walker 缓存吸收);如 SDK 根目录特别大可再优化根覆盖策略。 +- w3x 并行/流水线浅扫描仍未做(HDD 收益存疑,SSD 再做)。 + +### 补充(用户实测反馈,2026-08-03) + +用户在 0.1.3 上删除 workspace storage 缓存后实测: + +- XML 阶段约 45s(比之前 27s 多)→ 根因是磁盘缓存轮实现的**全量目录枚举 + 快照**在首建前递归枚举 SDK 根等搜索根。已改为惰性按目录 readdir,复测 + phase A **24.0s**,statSync 消除效果不变(75,926 次快照命中、0 回退)。 +- “show index report 显示 Indexed in 1.1s、0 浅扫、8995 缓存命中”与观察 + 不一致 → 首建完成后又发生了一次信任重建(follow-up)。为定位触发源, + 新增**重建插桩**:`buildCount` / `lastBuildTrigger`(initial、save、 + watcher-create/change/delete、config、reindex-command、clear-cache、 + dirty-followup)+ “RA3 Mod XML” 输出通道(每次构建记录 trigger、phase A + 发布时间、完成耗时);索引报告与缓存报告均显示构建序号与触发原因。 + +### 补充 2(0.1.4 日志复现,2026-08-03) + +0.1.4 输出通道日志: + +- build #1:phase A 31.4s、done 120.1s、**stale=true**; +- build #2:`dirty-followup (initial)`,0.6s(报告计时不一致的直接来源); +- build #3/#4/#5:每 ~35s 一次 `watcher-change`,每次 0.5s 重建。 + +build #1 的 stale=true 与周期性 watcher-change 均不符合预期:说明有后台进程 +在周期性触碰被监视目录(最可疑是 `.git` 内部文件,如后台 fetch/maintenance)。 +处理: + +1. watcher 事件现在把**触发 URI 写入输出通道**(`[watcher-change] `), + 可直接定位是哪个文件/目录在变化; +2. 新增 `isWatcherNoisePath`:路径含 `.git` 段的事件直接忽略(不 invalidate、 + 不标记 stale、不触发重建);单元测试 90 → 91。 + +版本 **0.1.4 → 0.1.5**。 + +### 补充 3(0.1.5 日志复现,2026-08-03) + +0.1.5 日志中周期性重建已消失,但首建期间仍有一次: + +``` +[watcher-change] d:\...\corona\Data\Neutral\Crate\UnitCrate.xml.git +``` + +该文件**并不存在**(疑似其他 VSCode 扩展产生的瞬时临时文件),且它是 +`*.xml.git` 文件名,不是 `.git` 目录段,绕过了上一轮过滤;它也导致 build #1 +stale=true 并触发 follow-up。处理(按用户建议的扩展名白名单思路): + +1. `isWatcherNoisePath` 增加临时文件命名模式:`.git` / `.tmp` / `.lock` / + `~` / `.swp` / `.bak` / `.orig` 后缀,以及 `.#` / `.~` 前缀; +2. `onDidChange` 只响应**内容相关**文件:扩展名白名单(`.xml` / `.w3x`; + 领域修正:RA3 合理文本格式为 xml/w3x/lua,lua 暂未索引,manifest 为 + `*.manifest` 二进制,不存在 `.manifestxml`)或已在当前索引中的文件 + (`ModIndexer.isIndexedFile`,覆盖被嗅探为 XML 的未知扩展名);纹理等 + 二进制内容变更不触发重建; +3. 创建/删除仍对所有真实文件响应(影响 include 存在性),临时文件模式除外。 + +测试 91 → 92;版本 **0.1.5 → 0.1.6**。 + +### 遗留(此处的磁盘缓存与文件集快照已在第十一轮完成,T1 已在第十二轮完成) + +- w3x 文件名启发式定向扫描(按约定后续再做)。 +- `AssetIdList` 等“任意资产 ID 列表”的引用语义建模。 +- Find All References 目前仍是全文搜索,未走索引(对应需求 P1 高效搜索)。 + +--- + +## 十七、问题分析(第十二轮,2026-08-03):当前文档局部链与 include 逻辑树展开(T1) + +### 目标 + +上一轮遗留的 T1:让**不在任何全局流里的文件**也能解析自身引用,并为 GameObject +内模块 `id` 的局部作用域(`AttachModuleId` / `ModuleId` / `AutoResolveBody` 等) +铺路。两件事一起做: + +1. **T1a 文档局部 overlay**:当前打开的文档(含未保存文本)自身资产 / `$DEFINE` + 及其 include 链进入一个轻量局部索引,与全局索引叠加使用; +2. **T1b 逻辑树展开**:`xi:include` 按现有 `xpointer` 子集拼入当前文档的逻辑树, + 使 include 进来的内容获得正确的父上下文,并让 Poid 引用能在同一 GameObject + 子树内解析。 + +### 现状证据 + +- 全局索引只从 `Data/Mod.xml` 与 `additionalmaps/mapmetadata_*.xml` 出发; + `Data/Standalone.xml` 这类未进流的文件,其自身 `GameObject` / `$DEFINE` 不会 + 出现在 `ws.index`,`inheritFrom`、`CommandSet` 全部无法解析; +- `xi:include` 此前只做到“目标文件可索引 / 缺失可诊断”,没有拼入当前文档树; + include 进来的 `TruckDraw` 等模块拿不到 `Draws` 子元素的上下文类型; +- `isLocalReferenceAttribute` 对所有 Poid 属性一律“不检查、不解析”,导致 + 同一 GameObject 内完全可静态判断的模块引用也没有 hover / 跳转 / 补全。 + +### 设计 + +1. **共享 `xpointer.ts`**:`localName` / `findXPointerContainer` 从 indexer 迁出, + 全局索引与局部展开共用同一份 xpointer 子集实现。 +2. **`logicalTree.ts`**: + - 按解析器扁平元素表预建逻辑节点壳(保留原始 `sourceFile` 与偏移),再按 + 真实根 + 容错孤儿根遍历,重建 parent/child 链; + - `xi:include` 解析 href → `readDom` 目标 → 按 `xpointer` 选择容器子节点 → + 拼入逻辑父元素;目标缺失 / 环 / 超深时跳过(环与深度用 visited + 64); + - 保留 xi 节点本身在 `elements` 中,hover 仍可解释 XInclude。 +3. **`localScope.ts`**: + - `buildDocumentScope` 返回原始 parse、逻辑树、per-source LineMap、局部 + overlay 与 overlay-aware merged index; + - overlay 沿 `` 与 `xi:include` 递归收集资产 / + Define;`reference` 指向 manifest,资产由全局索引提供,不重复解析; + - `withLocalOverlay` 不复制全局 Map(Corona ~65k 资产),只把 overlay 挂到 + `ModIndex.local`,查询函数“局部优先、全局兜底”。 +4. **workspace 集成**:`getScope(document)` 按 URI + 文档 version + 全局索引 + 代次缓存;全局索引发布 / 文件关闭时失效;首个 indexer 创建前的兜底读取走 + `fallbackRead`(小 XML 直接解析)。 +5. **features 接入**: + - diagnostics / hover / navigation / completion 改从 `getScope` 取逻辑树与 + merged index;诊断只上报 `sourceFile === 当前文件` 的节点; + - 引用解析 / 补全 / Define 查询支持 `idx.local` 优先; + - Poid 属性:`AttachModuleId` / `ModuleId` / `UpdateModuleId` 等可在最近 + GameObject 子树内补全、hover、跳转;未命中**不新增诊断**(保守,避免 + “武器引用另一 GameObject 模块”这类跨文件语义误报); + - 导航对当前未保存文件优先用编辑器文本定位,再回退磁盘 DOM。 + +### 验证 + +- 单元测试 **92 → 98 全绿**: + - `localScope.test.mjs`:未进流文件 overlay(自身资产 / Define / instance + include 链)、inheritFrom 局部解析、xi:include 展开后模块上下文类型、 + Poid 引用找到 include 进来的兄弟模块、局部定义优先于全局同名定义、 + xi:include 环终止; + - `completion.test.mjs`:Poid 属性只补全所在 GameObject 子树内的 id。 +- `tsc` / `esbuild` 构建通过(打包验证见版本发布步骤)。 + +### 边界与后续 + +- 顶层 `` 仍**不**并入逻辑树:它通常是独立整文件或大体积 + w3x,展开收益与风险不成比例;如需要“当前文档视角的全量合并诊断”再单独做。 +- `AttachModuleId` 若出现在独立 `WeaponTemplate`(不在 GameObject 子树内), + 本轮仍不解析;等真实样本确认跨文件语义后再扩展。 +- 未命中 Poid 不报诊断是刻意保守,后续可加配置项开启。 + +版本 **0.1.7 → 0.1.8**。 + +### 补充:构建期局部作用域闸门(2026-08-04) + +用户清空缓存后在 Corona 上测得 phase A **101.2s** / 完成 **254.6s**,明显高于 +第十一轮冷建基线(phase A 约 24s / 总约 118s)。代码审查确认 T1 没有改动 indexer +的构建路径(`localScope` / `logicalTree` 不参与 `build()`),但自动诊断在构建中 +会触发 `getScope()`,可能和 indexer 抢磁盘 / CPU。 + +修复:`getScope()` 在 `building === true` 时返回 **parse-only 轻量 scope** +(只解析当前文件,不沿 include 链读盘、不展开逻辑树);构建完全结束后再刷新 +全量局部 scope。这样首建期间诊断 / 补全仍可用,但不会拖慢索引。索引报告与 +输出通道同步增加 `walk / candidates / art` 耗时分解,便于下次直接定位慢在哪一段。 + +版本 **0.1.8 → 0.1.9**。 diff --git a/docs/plan.md b/docs/plan.md index adb8c58..e70288a 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -82,14 +82,17 @@ src/ asset-types.json 由 tools/extract-asset-types.mjs 生成(TypeId 哈希→类型名) indexer/ includeResolver.ts Include 路径解析(纯 TS,BAB /data /art /audio 顺序) + existence.ts 文件集存在性快照(目录枚举 Set,替代逐路径 statSync) manifestParser.ts .manifest 二进制解析 + 类型/ID 推导(纯 TS) fileScanner.ts 目录遍历缓存 + Include source 候选收集 refs.ts 引用目标解析(按 refType / isRef / inheritFrom 过滤,纯 TS) shallowScan.ts 大体积美术资产(.w3x 等)顶层浅扫描(纯 TS,不建 DOM) records.ts 每文件紧凑索引记录(资产/Define/Include/xi + 行号) caches.ts 跨重建持久缓存(DocumentCache / IndexRecordsCache / - IncludeResolveCache) - indexer.ts 工作区索引器(资产/Define/流/manifest/w3x 合并) + IncludeResolveCache)+ 失效纪元 InvalidationsEpoch + diskCache.ts 跨会话磁盘缓存(gzip JSON、原子写、多信号 stat 校验) + indexer.ts 工作区索引器(资产/Define/流/manifest/w3x 合并, + 分阶段:XML → art,中间快照可发布) types.ts 共享类型 features/ completion.ts 补全 provider(元素/属性/值,上下文感知;xs:list 多值按当前段过滤) @@ -144,8 +147,9 @@ test/ - 全量解析内存放大约 17 倍(6.3 MB 文本 → +109 MB DOM),不可接受;`scanXmlShallow` 单次线性扫描只提取顶层 `name+id`、``、``、``, 22 MB 文件 ~600 ms、保留内存≈0。 - - 索引按扩展名三分:`.xml`/`.manifestxml` 全量解析(4 MB 上限不变);`.w3x` 浅扫描; - 未知扩展名嗅探文件头(`<` 开头、无 NUL)决定按 XML 浅扫描或二进制登记。 + - 索引按扩展名三分:`.xml` 全量解析(4 MB 上限不变);`.w3x` 浅扫描; + 未知扩展名嗅探文件头(`<` 开头、无 NUL)决定按 XML 浅扫描或二进制登记 + (manifest 是 `*.manifest` 二进制,不存在 `.manifestxml` 源码格式)。 - `DocumentCache` / `ShallowScanCache` 由 `ModWorkspace` 持有,每次重建传入新的 `ModIndexer`;按 `mtimeMs + size` 校验,未变化不重读。Corona 第二次构建 w3x 重扫数为 0(4,829 次缓存命中)。 @@ -164,6 +168,52 @@ test/ 堆保留确认为构建期可回收垃圾,常驻 ~100MB;强制 reindex ~5-25s。 - 候选目录扫描并行化;`stats` 新增 `candidatesMs` / `walkMs` / `resolveCalls` / `resolveCacheHits` 供索引报告定位耗时。 +16. **分阶段索引与部分可用性**(第十轮,2026-08-03):索引分两阶段发布—— + 阶段 A(xml)只走 XML + manifest include 链,w3x 只登记进待扫队列; + 阶段 B(art)浅扫描队列并继续走 w3x 内的 include。阶段 A 结束即发布 + 不可变快照(`snapshotIndex` 深拷贝嵌套 Map/数组),XML/枚举/语法类功能 + 在首建早期即可用;引用类诊断在 `!complete || stale` 时“显示但标注” + (code 为 `*-indexing`,消息注明 index incomplete)。快照携带 + `complete` / `phase` / `stale`,状态栏显示阶段。 +17. **构建中失效与 stale 标记**(第十轮):watcher 的 change / create / + delete 均触发防抖重建;`InvalidationsEpoch` 记录失效次数,快照发布时若 + 期间有失效则标记 stale,由 dirty 机制随后重建收敛;构建失败保留上一个 + 快照而非清空索引。 +18. **多信号文件 stamp**(第十轮):`IndexedFile.stat` 扩展为 + `{ size, mtimeMs, birthtimeMs, ctimeMs }`,任一不匹配即重读,为磁盘 + 持久化缓存铺路(FAT32 mtime 2s 粒度、工具保留 mtime 等场景)。 +19. **文件集快照替代 statSync**(第十一轮,2026-08-03):`ExistenceSnapshot` + 用**惰性按目录 readdir**(只读查询到的父目录,按目录缓存)回答 include + 存在性,覆盖根之外才 statSync 回退;盘符根不枚举。首版全量递归枚举使 + XML 阶段从 27s 涨到 45s,已改为惰性模式(复测 24.0s)。 + `stats.snapshotHits / snapshotFallbacks` 入报告;Corona 首建 75,926 次 + 查询全部由快照回答、0 回退。 +20. **磁盘持久化缓存**(第十一轮):`DiskRecordsCache` 持久化 records 缓存 + (gzip JSON、原子写、identity key);启动时并发 stat 多信号校验,不匹配 + 丢弃重读;构建后异步回写。Corona 缓存仅 651 KB,冷启动约 11s(原 + ~2.5 分钟)。 +21. **缓存命令**(第十一轮):`ra3modxml.clearCache`(清内存 + 磁盘 + + 强制重建)、`ra3modxml.showCacheReport`(路径/大小/校验统计/命中数)。 +22. **重建插桩**(第十一轮补充):`buildCount` / `lastBuildTrigger` + (initial、save、watcher-*、config、reindex、clear-cache、 + dirty-followup)进入索引/缓存报告;“RA3 Mod XML” 输出通道记录每次构建 + 的触发原因、phase A 发布时间与完成耗时,用于定位“首建后又重建一次” + 之类的现象。 +23. **watcher 噪声过滤与 URI 日志**(第十一轮补充):watcher 事件把触发 + URI 写入输出通道;路径含 `.git` 段的事件直接忽略(后台 fetch / + maintenance 会周期性触碰 `.git`,不应触发重建或 stale 标记)。 +24. **watcher 内容白名单**(第十一轮补充):临时文件命名模式 + (`.git`/`.tmp`/`.lock`/`~`/`.swp`/`.bak`/`.orig` 后缀、`.#`/`.~` + 前缀)全部忽略;`onDidChange` 只响应扩展名白名单(`.xml` / `.w3x`, + RA3 合理文本格式为 xml/w3x/lua,lua 暂未索引、manifest 为二进制)或已在 + 索引中的文件;创建/删除仍响应所有真实文件(影响 include 存在性)。 +25. **当前文档局部链 + 逻辑树展开(第十二轮,T1)**:新增 `xpointer.ts` / + `logicalTree.ts` / `localScope.ts`。打开文件时按当前文本建立局部 overlay + (自身资产 / `$DEFINE` / include 链),并生成展开 `xi:include` 的逻辑树; + features 经 `ws.getScope(document)` 拿到 overlay-aware 索引。Poid 引用 + (`AttachModuleId` / `ModuleId` / `AutoResolveBody` 等)在最近 GameObject + 子树内解析;未命中不新增诊断(保守策略,避免跨文件误报)。 + 顶层 `` 暂不并入逻辑树(保留为后续扩展)。 ## 三、实施步骤 @@ -182,6 +232,24 @@ test/ 9. [x] Corona 性能与内存优化(第九轮,v0.1.1):`records.ts` 记录驱动索引、 `IncludeResolveCache` 零 stat 重建、DOM 元素预算淘汰、w3x LineMap 移除、 候选并行扫描、阶段计时;Corona 信任重建 2.0s;确认首建 2.5GB 为可回收垃圾。 +10. [x] 部分可用性 + 分阶段索引(第十轮,2026-08-03):T0 解耦(语法/模型 + 诊断、枚举/子元素补全、Include 链接无索引可用)、A/B 分阶段发布、watcher + 触发重建 + 失效纪元 stale 标记、stat 多信号扩展;测试 73 → 79。 +11. [x] 冷启动提速(第十一轮,2026-08-03):文件集快照替代 statSync、 + 磁盘持久化缓存(多信号校验、原子写)、clearCache/showCacheReport 命令; + 测试 79 → 90;Corona 冷启动 ~11s。 +12. [x] 惰性存在性快照 + 重建插桩(第十一轮补充,2026-08-03):全量目录枚举 + 改为惰性按目录 readdir(phase A 45s → 24.0s);新增 buildCount / + lastBuildTrigger 与输出通道日志;索引/缓存报告显示构建序号与触发原因。 +13. [x] watcher 噪声过滤 + URI 日志(第十一轮补充,2026-08-03):输出通道 + 记录 `[watcher-*] `;`.git` 段路径忽略;测试 90 → 91。 +14. [x] watcher 内容白名单(第十一轮补充,2026-08-03):临时文件命名模式 + 过滤 + 内容变更扩展名白名单 + `isIndexedFile` 兜底;测试 91 → 92。 +15. [x] T1 当前文档局部链 + include 展开(第十二轮,2026-08-03):局部 + overlay(不在任何流里的文件也能解析自身引用)+ `xi:include` 逻辑树展开 + + Poid 局部作用域补全/悬停/跳转;测试 92 → 98。2026-08-04 补充构建期 + 闸门:`getScope` 在重建进行中只返回 parse-only scope,避免与 indexer + 抢盘(版本 0.1.9)。 ## 四、验证结果(实测) @@ -200,16 +268,11 @@ test/ - 假设 SDK 路径默认 `C:\Apps\RA3-MODSDK-X`(与 prompts 一致),可在设置中修改。 - 假设补全/导航以“文本语义分析”为主,不做完整 XSD 校验(BAB 才是权威校验器)。 - 开放:是否发布到 VS Code Marketplace(需要 publisher)——本期先保证本地 `vsce package` 可安装。 -- 开放:**“宏展开”式虚拟合并**(用户提议,方向已确认):解析前先把 `xi:include` - (以及顶层 ``、`inheritFrom` 继承合并)展开成不含 include 的 - 文档树,再对展开后的树做 XSD 校验、补全与诊断——与 BAB 编译时把整个 Mod 合并成 - 一份大 XML 的行为一致。展开树需携带**来源追溯**(错误/跳转仍定位到原始文件), - 并处理 xpointer 子集解析与 include 循环。当前仅做到:目标文件可索引、可导航、 - 缺失可诊断;`xi:include` 本身不参与 XSD 校验(第五轮)。 - 该功能也是“GameObject 内模块 id 局部作用域解析”(第四轮遗留)的地基。 - 详细设计备忘见下一节。 +- 开放:**“宏展开”式虚拟合并**(用户提议,方向已确认):`xi:include` 已按逻辑树 + 展开(第十二轮,见第六节);顶层 `` 与 `inheritFrom` + + `xai:joinAction` 的深合并仍未实现,后续如需要“当前文档视角的全量合并诊断”再继续。 -## 六、include 展开设计备忘(2026-08-01,待实施) +## 六、include 展开设计备忘(2026-08-01;xi:include 部分已实施于第十二轮) > 目的:集中记录 include 处理相关的现状、结论与设计,下次遇到 include 问题时从这里继续, > 并在实施后把结果回写本节。 @@ -222,7 +285,7 @@ test/ | `reference` → builtmods manifest 解析 / 缺失回退占位 XML | 已实现 | | 嵌套 `xi:include`(任意层级):目标可索引、缺失报 `include-not-found`、Ctrl+点击跳转、`href` hover 解析目标 | 已实现(第二轮 + 第五轮) | | `xi:include` 及其属性不参与 XSD 校验(外来命名空间守卫 `isXsdElementName` / `isXsdAttributeName`) | 已实现(第五轮) | -| include 目标内容“虚拟合并”进父文档的逻辑树 | **未实现**(本文档主题) | +| include 目标内容“虚拟合并”进父文档的逻辑树 | 已实现 `xi:include`(第十二轮);顶层 `` 仍不展开 | ### 2. 已确认的方向 diff --git a/docs/requirements.md b/docs/requirements.md index cc5ca5b..307df50 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -90,7 +90,9 @@ XML 之间的组织靠 `` 标签,共有三种语义: - 中小项目:`C:\Apps\RA3-MODSDK-X\Mods\AttachTest`、`D:\Mods\CoronaMod\mods\mods\GenEvoTest`。 - 大型项目:`D:\Mods\CoronaMod\mods\mods\corona`(自带 `xsd/`)。 - 现有工具:工作区 `check_duplicate_ids.py`(include 解析与重复 ID 检测的参考实现)。 -- manifest 格式参考:OpenSAGE `src/OpenSage.Game/Data/StreamFS/ManifestFile.cs`(commit `d45d361`,最新分支已移除该文件)。本机网络受限未能拉取,且 manifest 为压缩/哈希的二进制,本期不实现其解析。 +- manifest 格式参考:OpenSAGE `src/OpenSage.Game/Data/StreamFS/ManifestFile.cs` + (commit `d45d361`,本仓库 `OpenSAGE/` 子目录)。插件已实现 v5/v6/v7 二进制 + 解析(`src/indexer/manifestParser.ts`),未知 TypeId 哈希时按资产名前缀推导类型。 ## 四、验收标准 diff --git a/package-lock.json b/package-lock.json index 4ef5a32..2ca98a2 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "ra3-mod-xml", - "version": "0.1.1", + "version": "0.1.9", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "ra3-mod-xml", - "version": "0.1.1", + "version": "0.1.9", "license": "MIT", "dependencies": { "fast-xml-parser": "^4.5.0" diff --git a/package.json b/package.json index 7e8d3d8..2a5b070 100644 --- a/package.json +++ b/package.json @@ -2,7 +2,7 @@ "name": "ra3-mod-xml", "displayName": "RA3 Mod XML", "description": "Red Alert 3 Mod XML tooling: syntax highlighting, completions, reference navigation and diagnostics for SAGE/BinaryAssetBuilder XML.", - "version": "0.1.1", + "version": "0.1.9", "publisher": "ra3-mod-xml", "license": "MIT", "engines": { @@ -22,6 +22,7 @@ ], "main": "./dist/extension.js", "activationEvents": [ + "onLanguage:xml", "workspaceContains:**/Data/Mod.xml", "workspaceContains:**/mod.babproj", "workspaceContains:**/*.babproj" @@ -91,6 +92,14 @@ { "command": "ra3modxml.openIndexReport", "title": "RA3 Mod XML: Show index report" + }, + { + "command": "ra3modxml.clearCache", + "title": "RA3 Mod XML: Clear caches and rebuild" + }, + { + "command": "ra3modxml.showCacheReport", + "title": "RA3 Mod XML: Show cache report" } ] }, diff --git a/src/extension.ts b/src/extension.ts index b73c088..b1dcdff 100644 --- a/src/extension.ts +++ b/src/extension.ts @@ -44,7 +44,7 @@ export function activate(context: vscode.ExtensionContext): void { context.subscriptions.push( vscode.languages.registerReferenceProvider( XML_SELECTOR, - new Ra3ReferenceProvider(), + new Ra3ReferenceProvider(ws), ), ); context.subscriptions.push( @@ -56,19 +56,26 @@ export function activate(context: vscode.ExtensionContext): void { context.subscriptions.push( vscode.languages.registerDocumentSymbolProvider( XML_SELECTOR, - new Ra3DocumentSymbolProvider(), + new Ra3DocumentSymbolProvider(ws), ), ); context.subscriptions.push( vscode.languages.registerDocumentSemanticTokensProvider( XML_SELECTOR, - new Ra3SemanticTokensProvider(), + new Ra3SemanticTokensProvider(ws), RA3_SEMANTIC_TOKENS_LEGEND, ), ); const diagnostics = new Ra3Diagnostics(ws); context.subscriptions.push(diagnostics); + // Refresh diagnostics for every open XML document whenever a new index + // snapshot is published (XML phase, art phase, stale/final rebuild). + ws.onIndexUpdate = () => { + for (const doc of vscode.workspace.textDocuments) { + if (doc.languageId === "xml") void diagnostics.update(doc); + } + }; const diagnosticTimers = new Map>(); const scheduleDiagnostics = (doc: vscode.TextDocument) => { @@ -110,7 +117,7 @@ export function activate(context: vscode.ExtensionContext): void { vscode.workspace.onDidSaveTextDocument((doc) => { if (doc.languageId !== "xml") return; ws.invalidate(doc.uri.fsPath); - ws.scheduleRebuild(); + ws.scheduleRebuild("save"); void diagnostics.update(doc); }), ); @@ -120,31 +127,60 @@ export function activate(context: vscode.ExtensionContext): void { // Search paths / builtmods locations may have changed: cached include // resolutions and manifest lookups are no longer valid. ws.invalidateExistence(); - ws.scheduleRebuild(); + ws.scheduleRebuild("config"); } }), ); context.subscriptions.push( - vscode.commands.registerCommand("ra3modxml.reindex", () => ws.rebuild(true)), + vscode.commands.registerCommand( + "ra3modxml.reindex", + () => void ws.rebuild(true, "reindex-command"), + ), + ); + context.subscriptions.push( + vscode.commands.registerCommand("ra3modxml.clearCache", () => { + ws.clearCaches(); + void vscode.window.showInformationMessage( + "RA3 Mod XML: caches cleared; rebuilding from scratch…", + ); + }), + ); + context.subscriptions.push( + vscode.commands.registerCommand("ra3modxml.showCacheReport", async () => { + void vscode.window.showInformationMessage(await ws.cacheReport(), { + modal: false, + }); + }), ); context.subscriptions.push( vscode.commands.registerCommand("ra3modxml.openIndexReport", () => { const idx = ws.index; if (!idx) { + if (ws.isBuilding) { + void vscode.window.showInformationMessage( + "RA3 Mod XML: index is still building — check the status bar. " + + "Most features become available after the XML phase.", + ); + return; + } void vscode.window.showInformationMessage( "RA3 Mod XML: no index available. Open a workspace that contains Data/Mod.xml.", ); return; } const s = idx.stats; + const stale = idx.stale ? " (stale)" : ""; void vscode.window.showInformationMessage( `RA3 Mod XML index\n` + `Project: ${s.projectDir}\n` + `Files: ${s.indexedFiles} (${s.parsedFiles} parsed, ${s.shallowScannedFiles} shallow-scanned, ${s.shallowCacheHits + s.recordsCacheHits} cache hits)\n` + `Assets: ${s.assetCount} (${s.manifestAssetCount} from ${s.manifestFiles} manifests)\n` + `Defines: ${s.defineCount} · Streams: ${s.streams} · Candidates: ${s.sourceCandidates}\n` + - `Indexed in ${(s.elapsedMs / 1000).toFixed(1)}s`, + `Phase: ${s.phase} · Complete: ${s.complete}${stale}\n` + + `Build #${ws.buildCount} (trigger: ${ws.lastTrigger})\n` + + `Indexed in ${(s.elapsedMs / 1000).toFixed(1)}s\n` + + `XML walk: ${(s.walkMs / 1000).toFixed(1)}s · Candidates: ${(s.candidatesMs / 1000).toFixed(1)}s · Art scan: ${(s.artScanMs / 1000).toFixed(1)}s`, { modal: false }, ); }), diff --git a/src/features/completion.ts b/src/features/completion.ts index 56acfdc..ff74cc6 100644 --- a/src/features/completion.ts +++ b/src/features/completion.ts @@ -1,5 +1,5 @@ import * as vscode from "vscode"; -import { parseXml, type XmlElement } from "../language/xmlParser"; +import type { XmlElement } from "../language/xmlParser"; import { analyzeContext, splitListValuePrefix, @@ -8,6 +8,11 @@ import { import { resolveElementType } from "../language/typeContext"; import * as model from "../model/schemaModel"; import { isLocalReferenceAttribute } from "../indexer/refs"; +import { + findContainingGameObject, + collectLocalIds, + type LogicalElement, +} from "../indexer/logicalTree"; import type { ModWorkspace } from "../workspace"; import type { ModIndex, AssetDef } from "../indexer/types"; @@ -21,11 +26,13 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider { position: vscode.Position, _token: vscode.CancellationToken, ): Promise { + if (!this.ws.isRa3Workspace()) return []; const text = document.getText(); const offset = document.offsetAt(position); - const doc = parseXml(text); + const scope = await this.ws.getScope(document); + const doc = scope.expanded; const ctx = analyzeContext(doc, text, offset); - const idx = this.ws.index; + const idx = scope.merged; switch (ctx.kind) { case "element-name": @@ -33,9 +40,9 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider { case "attribute-name": return this.attributeNameItems(ctx, document, position); case "attribute-value": - return idx ? this.valueItems(ctx, document, position, idx) : []; + return this.valueItems(ctx, document, position, idx); case "content": - return idx ? this.contentItems(ctx, document, position, idx) : []; + return this.contentItems(ctx, document, position, idx); default: return []; } @@ -177,7 +184,7 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider { ctx: CompletionContext, document: vscode.TextDocument, position: vscode.Position, - idx: ModIndex, + idx: ModIndex | null, ): vscode.CompletionItem[] { const el = ctx.element; const attr = ctx.attr; @@ -232,6 +239,7 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider { ); } if (isInclude && attrName === "source") { + if (!idx) return []; return this.includeSourceItems(idx, prefix, make); } if (attrName === "xai:joinaction" || attrName === "joinaction") { @@ -242,16 +250,19 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider { // inheritFrom: same element type first, then everything. if (attrName === "inheritfrom") { + if (!idx) return []; return this.assetIdItems(idx, el.name, null, prefix, make); } // `id` attributes are definitions and Poid attributes are pipeline-local // references; offering global asset ids for them would be wrong. if (attrInfo && isLocalReferenceAttribute(elType, attr.name)) { - return []; + if (attrName === "id") return []; + return this.localIdItems(el as LogicalElement, prefix, make); } if (attrInfo?.refType) { + if (!idx) return []; return this.assetIdItems(idx, null, attrInfo.refType, prefix, make); } if (attrInfo?.enumValues?.length) { @@ -264,7 +275,7 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider { .filter((v) => v.startsWith(prefix.toLowerCase())) .map((v) => make(v, vscode.CompletionItemKind.Value, "boolean")); } - if (attrInfo?.allowsDefine) { + if (idx && attrInfo?.allowsDefine) { return this.defineItems(idx, prefix, make); } return []; @@ -304,8 +315,12 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider { ): vscode.CompletionItem[] { const lower = prefix.toLowerCase(); const scored: { def: AssetDef; score: number }[] = []; + const seen = new Set(); const consider = (def: AssetDef) => { + const key = `${def.type}:${def.id.toLowerCase()}:${def.file}:${def.line}`; + if (seen.has(key)) return; + seen.add(key); if (!def.id.toLowerCase().startsWith(lower)) return; let score = 3; if (refType && model.isAssignableTo(def.type, refType)) score = 1; @@ -315,6 +330,18 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider { }; const targetType = selfType ?? refType; + const localAssets = idx.local?.assets; + const localById = idx.local?.assetsById; + if (localAssets || localById) { + if (!targetType) { + for (const list of localById!.values()) for (const d of list) consider(d); + } else { + for (const [typeName, byId] of localAssets!) { + if (!model.isAssignableTo(typeName, targetType)) continue; + for (const list of byId.values()) for (const d of list) consider(d); + } + } + } if (!targetType) { for (const list of idx.assetsById.values()) for (const d of list) consider(d); } else { @@ -343,24 +370,54 @@ export class Ra3CompletionProvider implements vscode.CompletionItemProvider { ): vscode.CompletionItem[] { const lower = prefix.replace(/^[=$]*/, "").toLowerCase(); const items: vscode.CompletionItem[] = []; - for (const [key, defs] of idx.defines) { - if (!key.includes(lower)) continue; - const def = defs[0]; - const label = `$${def.name}`; - const item = make(label, vscode.CompletionItemKind.Constant, "Define", def.value); - item.insertText = label; - items.push(item); + const seen = new Set(); + for (const defines of [idx.local?.defines, idx.defines]) { + if (!defines) continue; + for (const [key, defs] of defines) { + if (!key.includes(lower)) continue; + const def = defs[0]; + const dedupe = `${def.name.toLowerCase()}:${def.file}:${def.line}`; + if (seen.has(dedupe)) continue; + seen.add(dedupe); + const label = `$${def.name}`; + const item = make(label, vscode.CompletionItemKind.Constant, "Define", def.value); + item.insertText = label; + items.push(item); + } } return items.slice(0, MAX_VALUE_ITEMS); } + private localIdItems( + el: LogicalElement, + prefix: string, + make: (label: string, kind: vscode.CompletionItemKind, detail: string, doc?: string) => vscode.CompletionItem, + ): vscode.CompletionItem[] { + const root = findContainingGameObject(el); + if (!root) return []; + const lower = prefix.toLowerCase(); + const items: vscode.CompletionItem[] = []; + for (const { id } of collectLocalIds(root)) { + if (!id.toLowerCase().startsWith(lower)) continue; + items.push( + make( + id, + vscode.CompletionItemKind.Value, + "local module", + "Pipeline-local id in the enclosing GameObject (includes xi:include targets).", + ), + ); + } + return items; + } + // ── Element content ─────────────────────────────────────────────── private contentItems( ctx: CompletionContext, _document: vscode.TextDocument, _position: vscode.Position, - _idx: ModIndex, + _idx: ModIndex | null, ): vscode.CompletionItem[] { const el = ctx.element; if (!el) return []; diff --git a/src/features/diagnostics.ts b/src/features/diagnostics.ts index 7563040..84cab1e 100644 --- a/src/features/diagnostics.ts +++ b/src/features/diagnostics.ts @@ -1,6 +1,6 @@ import * as vscode from "vscode"; import { dirname } from "node:path"; -import { LineMap, parseXml, type XmlElement } from "../language/xmlParser"; +import { LineMap, type XmlElement } from "../language/xmlParser"; import { resolveElementType } from "../language/typeContext"; import { resolveSource, buildSearchPaths } from "../indexer/includeResolver"; import * as model from "../model/schemaModel"; @@ -8,8 +8,11 @@ import type { ModWorkspace } from "../workspace"; import type { ModIndex } from "../indexer/types"; import { isReferenceAttributeOfType, + mergeLocalAndGlobalDefs, resolveReferenceTargetsForType, } from "../indexer/refs"; +import type { LogicalElement } from "../indexer/logicalTree"; +import { scopePathKey } from "../indexer/localScope"; export class Ra3Diagnostics { private collection: vscode.DiagnosticCollection; @@ -19,15 +22,19 @@ export class Ra3Diagnostics { } async update(document: vscode.TextDocument): Promise { - const idx = this.ws.index; - if (!idx) { + if (!this.ws.isRa3Workspace()) { this.collection.set(document.uri, []); return; } + const scope = await this.ws.getScope(document); + const idx = scope.merged; const text = document.getText(); const lineMap = new LineMap(text); - const doc = parseXml(text); + const doc = scope.parse; const diags: vscode.Diagnostic[] = []; + // Reference/duplicate checks are provisional while the index is + // incomplete or stale: "not found" may be a false positive. + const provisional = idx ? !idx.complete || idx.stale === true : false; for (const err of doc.errors) { diags.push( @@ -44,7 +51,15 @@ export class Ra3Diagnostics { } if (doc.root) { - this.checkElements(doc.root, doc, lineMap, idx, document, diags); + this.checkElements( + scope.expanded.root, + scope.expanded, + lineMap, + idx, + document, + diags, + provisional, + ); } this.collection.set(document.uri, diags); @@ -59,19 +74,29 @@ export class Ra3Diagnostics { } private checkElements( - root: XmlElement, - doc: { elements: XmlElement[] }, + root: LogicalElement | null, + doc: { elements: LogicalElement[] }, lineMap: LineMap, - idx: ModIndex, + idx: ModIndex | null, document: vscode.TextDocument, diags: vscode.Diagnostic[], + provisional: boolean, ): void { const settings = this.ws.settings; const fileDuplicates = new Map(); for (const el of doc.elements) { + // Only report diagnostics for nodes that belong to the document being + // edited. Nodes spliced in through xi:include keep their own source + // file and are diagnosed when that file is opened. + if (scopePathKey(el.sourceFile) !== scopePathKey(document.uri.fsPath)) { + continue; + } const local = localName(el.name); - const isTopLevel = el.parent === root && !["Tags", "Includes", "Defines"].includes(local); + const isTopLevel = + root !== null && + el.parent === root && + !["Tags", "Includes", "Defines"].includes(local); const range = tagRange(document, el); // Top-level assets must have an id. @@ -111,6 +136,7 @@ export class Ra3Diagnostics { document, idx, diags, + provisional, ); } } @@ -169,6 +195,7 @@ export class Ra3Diagnostics { document, idx, diags, + provisional, ); } } @@ -184,12 +211,17 @@ export class Ra3Diagnostics { type: string, id: string, document: vscode.TextDocument, - idx: ModIndex, + idx: ModIndex | null, diags: vscode.Diagnostic[], + provisional: boolean, ): void { + if (!idx) return; const byType = idx.assets.get(type); - const defs = byType?.get(id.toLowerCase()); - if (!defs || defs.length < 2) return; + const defs = mergeLocalAndGlobalDefs( + idx.local?.assets.get(type)?.get(id.toLowerCase()), + byType?.get(id.toLowerCase()), + ); + if (defs.length < 2) return; const self = defs.filter( (d) => d.origin === "project" && @@ -211,7 +243,8 @@ export class Ra3Diagnostics { diags.push( this.diag( range, - `Duplicate id "${id}" for <${type}> (also defined in ${other.file})`, + `Duplicate id "${id}" for <${type}> (also defined in ${other.file})` + + (provisional ? " (based on a partial index)" : ""), vscode.DiagnosticSeverity.Error, "duplicate-id", ), @@ -225,8 +258,9 @@ export class Ra3Diagnostics { value: string, attr: { valueStart: number; valueEnd: number }, document: vscode.TextDocument, - idx: ModIndex, + idx: ModIndex | null, diags: vscode.Diagnostic[], + provisional: boolean, ): void { if (!value) return; const range = new vscode.Range( @@ -238,13 +272,19 @@ export class Ra3Diagnostics { const defineRe = /\$([A-Za-z_][A-Za-z0-9_]*)/g; let m: RegExpExecArray | null; while ((m = defineRe.exec(value)) !== null) { - if (!idx.defines.has(m[1].toLowerCase())) { + if ( + idx && + !(idx.local?.defines.has(m[1].toLowerCase()) ?? + idx.defines.has(m[1].toLowerCase())) + ) { + const code = provisional ? "undefined-define-indexing" : "undefined-define"; diags.push( this.diag( range, - `Undefined define "$${m[1]}"`, + `Undefined define "$${m[1]}"` + + (provisional ? " (index incomplete — may be a false positive)" : ""), vscode.DiagnosticSeverity.Warning, - "undefined-define", + code, ), ); } @@ -253,10 +293,13 @@ export class Ra3Diagnostics { if (value.startsWith("$") || value.startsWith("=")) return; const severity = this.ws.settings.reportUnresolvedReferences; if (severity === "none") return; + if (!idx) return; if (!isReferenceAttributeOfType(elType, attrName)) return; const targets = resolveReferenceTargetsForType(idx, elType, attrName, value); if (targets.length) return; - const anyDef = idx.assetsById.has(value.toLowerCase()); + const anyDef = + (idx.local?.assetsById.has(value.toLowerCase()) ?? false) || + idx.assetsById.has(value.toLowerCase()); const attrRef = model .attributesOfType(elType) .find((a) => a.name === attrName); @@ -265,16 +308,20 @@ export class Ra3Diagnostics { : attrRef?.isRef ? "of the expected declared type" : "matching"; + const code = provisional ? "unresolved-reference-indexing" : "unresolved-reference"; + const baseMessage = anyDef + ? `Reference "${value}" has no definition ${expected} (ids with the same name exist for other types)` + : `Unresolved reference "${value}" (not found in the current index)`; diags.push( this.diag( range, - anyDef - ? `Reference "${value}" has no definition ${expected} (ids with the same name exist for other types)` - : `Unresolved reference "${value}" (not found in the current index)`, + provisional + ? `${baseMessage} (index incomplete — may be a false positive)` + : baseMessage, severity === "warning" ? vscode.DiagnosticSeverity.Warning : vscode.DiagnosticSeverity.Information, - "unresolved-reference", + code, ), ); } @@ -282,7 +329,7 @@ export class Ra3Diagnostics { private checkInclude( el: XmlElement, document: vscode.TextDocument, - idx: ModIndex, + idx: ModIndex | null, diags: vscode.Diagnostic[], ): void { const typeAttr = el.attrs.find((a) => a.name === "type"); @@ -301,12 +348,18 @@ export class Ra3Diagnostics { ); } if (!sourceAttr?.hasValue) return; + const searchPaths = idx + ? buildSearchPaths(idx.sdkDir, idx.projectDir) + : this.ws.searchPaths(); + if (!searchPaths) return; const resolved = resolveSource( sourceAttr.value, dirname(document.uri.fsPath), - buildSearchPaths(idx.sdkDir, idx.projectDir), + searchPaths, ); - if (!resolved.path && !idx.sourceCandidates.some((c) => c.source === sourceAttr.value)) { + const candidateHit = + idx?.sourceCandidates.some((c) => c.source === sourceAttr.value) ?? false; + if (!resolved.path && !candidateHit) { diags.push( this.diag( new vscode.Range( diff --git a/src/features/hover.ts b/src/features/hover.ts index 31ddb1b..91b853a 100644 --- a/src/features/hover.ts +++ b/src/features/hover.ts @@ -1,13 +1,19 @@ import * as vscode from "vscode"; -import { findElementAt, parseXml } from "../language/xmlParser"; +import { findElementAt } from "../language/xmlParser"; import { resolveElementType } from "../language/typeContext"; import * as model from "../model/schemaModel"; import type { ModWorkspace } from "../workspace"; -import type { ModIndex } from "../indexer/types"; import { + isLocalReferenceAttribute, isReferenceAttributeOfType, resolveReferenceTargetsForType, } from "../indexer/refs"; +import { + findContainingGameObject, + findLocalId, + type LogicalElement, +} from "../indexer/logicalTree"; +import { scopePathKey, type DocumentScope } from "../indexer/localScope"; import { dirname } from "node:path"; import { buildSearchPaths, resolveSource } from "../indexer/includeResolver"; @@ -19,9 +25,10 @@ export class Ra3HoverProvider implements vscode.HoverProvider { position: vscode.Position, _token: vscode.CancellationToken, ): Promise { - const text = document.getText(); + if (!this.ws.isRa3Workspace()) return null; const offset = document.offsetAt(position); - const doc = parseXml(text); + const scope = await this.ws.getScope(document); + const doc = scope.expanded; const el = findElementAt(doc, offset); if (!el) return null; const elType = resolveElementType(el); @@ -35,7 +42,7 @@ export class Ra3HoverProvider implements vscode.HoverProvider { // Attribute value. for (const attr of el.attrs) { if (attr.hasValue && offset >= attr.valueStart && offset <= attr.valueEnd) { - return this.valueHover(el, elType, attr.name, attr.value, document, this.ws.index); + return this.valueHover(el, elType, attr.name, attr.value, document, scope); } } // Element name. @@ -116,14 +123,17 @@ export class Ra3HoverProvider implements vscode.HoverProvider { attrName: string, value: string, document: vscode.TextDocument, - idx: ModIndex | null, + scope: DocumentScope, ): vscode.Hover | null { + const idx = scope.merged; const md = new vscode.MarkdownString(); // $DEFINE reference. const defineMatch = /\$([A-Za-z_][A-Za-z0-9_]*)/.exec(value); if (defineMatch && idx) { - const defs = idx.defines.get(defineMatch[1].toLowerCase()); + const defs = + idx.local?.defines.get(defineMatch[1].toLowerCase()) ?? + idx.defines.get(defineMatch[1].toLowerCase()); if (defs?.length) { const d = defs[0]; md.appendMarkdown(`**Define** \`$${d.name}\` \n`); @@ -139,11 +149,14 @@ export class Ra3HoverProvider implements vscode.HoverProvider { (el.name === "Include" && attrName === "source") || (el.name === "xi:include" && attrName === "href") ) { - const resolved = idx + const searchPaths = idx + ? buildSearchPaths(idx.sdkDir, idx.projectDir) + : this.ws.searchPaths(); + const resolved = searchPaths ? resolveSource( value, dirname(document.uri.fsPath), - buildSearchPaths(idx.sdkDir, idx.projectDir), + searchPaths, ).path : null; if (resolved) { @@ -161,8 +174,26 @@ export class Ra3HoverProvider implements vscode.HoverProvider { return new vscode.Hover(md); } + // Pipeline-local (Poid) references: resolve inside the enclosing + // GameObject's logical subtree (including xi:include targets). + if ( + isLocalReferenceAttribute(elType, attrName) && + attrName.toLowerCase() !== "id" + ) { + return this.localIdHover(scope, el as LogicalElement, value, document); + } + // Asset reference / inheritFrom. - if (idx) { + if (!idx) { + if (isReferenceAttributeOfType(elType, attrName)) { + md.appendMarkdown( + "Index is still building — references cannot be resolved yet.", + ); + return new vscode.Hover(md); + } + return null; + } + { if (!isReferenceAttributeOfType(elType, attrName)) return null; const targets = resolveReferenceTargetsForType(idx, elType, attrName, value); if (targets.length) { @@ -191,8 +222,28 @@ export class Ra3HoverProvider implements vscode.HoverProvider { ); return new vscode.Hover(md); } + } - return null; + private localIdHover( + scope: DocumentScope, + el: LogicalElement, + value: string, + document: vscode.TextDocument, + ): vscode.Hover | null { + const root = findContainingGameObject(el); + if (!root) return null; + const target = findLocalId(root, value); + if (!target) return null; + const idAttr = target.attrs.find((a) => a.name === "id"); + if (!idAttr?.hasValue) return null; + const lineMap = scope.lineMaps.get(scopePathKey(target.sourceFile)); + const line = lineMap ? lineMap.positionAt(idAttr.valueStart).line + 1 : 0; + const md = new vscode.MarkdownString(); + md.appendMarkdown(`**Local pipeline id** \`${value}\` \n`); + md.appendCodeblock(`<${target.name}>`); + const rel = relativePath(document, target.sourceFile); + md.appendMarkdown(`Defined in \`${rel}:${line}\``); + return new vscode.Hover(md); } } diff --git a/src/features/navigation.ts b/src/features/navigation.ts index 287cfbe..a321236 100644 --- a/src/features/navigation.ts +++ b/src/features/navigation.ts @@ -7,7 +7,16 @@ import { resolveSource, type SearchPaths, } from "../indexer/includeResolver"; -import { resolveReferenceTargetsForType } from "../indexer/refs"; +import { + isLocalReferenceAttribute, + resolveReferenceTargetsForType, +} from "../indexer/refs"; +import { + findContainingGameObject, + findLocalId, + type LogicalElement, +} from "../indexer/logicalTree"; +import { scopePathKey, type DocumentScope } from "../indexer/localScope"; import type { ModWorkspace } from "../workspace"; import type { AssetDef, ModIndex } from "../indexer/types"; @@ -25,11 +34,11 @@ export class Ra3DefinitionProvider implements vscode.DefinitionProvider { position: vscode.Position, _token: vscode.CancellationToken, ): Promise { - const idx = this.ws.index; - if (!idx) return null; - const text = document.getText(); + if (!this.ws.isRa3Workspace()) return null; + const scope = await this.ws.getScope(document); + const idx = scope.merged; const offset = document.offsetAt(position); - const doc = parseXml(text); + const doc = scope.expanded; const el = findElementAt(doc, offset); if (!el) return null; const elType = resolveElementType(el); @@ -46,17 +55,31 @@ export class Ra3DefinitionProvider implements vscode.DefinitionProvider { (el.name === "Include" && nameLower === "source") || (el.name === "include" && nameLower === "href") ) { - const resolved = - resolveSource(value, dirname(document.uri.fsPath), searchPathsFor(idx)).path ?? - idx.sourceCandidates.find((c) => c.source === value)?.path ?? - null; - return resolved - ? new vscode.Location(vscode.Uri.file(resolved), new vscode.Position(0, 0)) + const searchPaths = idx ? searchPathsFor(idx) : this.ws.searchPaths(); + const resolved = searchPaths + ? resolveSource(value, dirname(document.uri.fsPath), searchPaths).path + : null; + const fallback = idx?.sourceCandidates.find((c) => c.source === value)?.path; + const target = resolved ?? fallback ?? null; + return target + ? new vscode.Location(vscode.Uri.file(target), new vscode.Position(0, 0)) : null; } // Asset reference / inheritFrom (filtered by the attribute's ref type). + if (!idx) return null; if (value && !value.startsWith("$")) { + if ( + isLocalReferenceAttribute(elType, attr.name) && + nameLower !== "id" + ) { + const local = this.localIdLocation( + scope, + el as LogicalElement, + value, + ); + if (local) return local; + } let targets = resolveReferenceTargetsForType(idx, elType, attr.name, value); if (!targets.length) return null; if ( @@ -67,13 +90,40 @@ export class Ra3DefinitionProvider implements vscode.DefinitionProvider { } const locations: vscode.Location[] = []; for (const { def } of targets.slice(0, 8)) { - const loc = await assetDefLocation(this.ws, def, idx); + const loc = await assetDefLocation(this.ws, def, idx, scope, document); if (loc) locations.push(loc); } return locations.length ? locations : null; } return null; } + + private localIdLocation( + scope: DocumentScope, + el: LogicalElement, + value: string, + ): vscode.Location | null { + const root = findContainingGameObject(el); + if (!root) return null; + const target = findLocalId(root, value); + if (!target) return null; + const idAttr = target.attrs.find((a) => a.name === "id"); + if (!idAttr?.hasValue) return null; + const lineMap = scope.lineMaps.get(scopePathKey(target.sourceFile)); + if (!lineMap) { + return new vscode.Location( + vscode.Uri.file(target.sourceFile), + new vscode.Position(0, 0), + ); + } + return new vscode.Location( + vscode.Uri.file(target.sourceFile), + new vscode.Range( + toVscodePosition(lineMap.positionAt(idAttr.valueStart)), + toVscodePosition(lineMap.positionAt(idAttr.valueEnd)), + ), + ); + } } /** @@ -85,6 +135,8 @@ async function assetDefLocation( ws: ModWorkspace, def: AssetDef, idx: ModIndex, + scope: DocumentScope, + currentDocument: vscode.TextDocument, ): Promise { if (def.origin === "manifest") { const src = def.manifestSource; @@ -100,6 +152,11 @@ async function assetDefLocation( return null; } + if (scopePathKey(def.file) === scopePathKey(currentDocument.uri.fsPath)) { + const precise = locationInCurrentDocument(scope, def.id, currentDocument); + if (precise) return precise; + } + return ( (await locationInDocument(ws, def.file, def.id)) ?? new vscode.Location( @@ -109,6 +166,36 @@ async function assetDefLocation( ); } +function locationInCurrentDocument( + scope: DocumentScope, + id: string, + document: vscode.TextDocument, +): vscode.Location | null { + const el = scope.parse.elements.find((e) => + e.attrs.some( + (a) => a.name === "id" && a.value.toLowerCase() === id.toLowerCase(), + ), + ); + if (!el) return null; + const idAttr = el.attrs.find((a) => a.name === "id"); + if (idAttr?.hasValue) { + return new vscode.Location( + document.uri, + new vscode.Range( + document.positionAt(idAttr.valueStart), + document.positionAt(idAttr.valueEnd), + ), + ); + } + return new vscode.Location( + document.uri, + new vscode.Range( + document.positionAt(el.start), + document.positionAt(el.startTagEnd), + ), + ); +} + /** * Finds the precise range of an asset definition inside an XML file: the id * attribute value when present, otherwise the element start tag. @@ -118,7 +205,9 @@ async function locationInDocument( file: string, id: string, ): Promise { - const parsed = await ws.indexer?.readDocument(file); + // readDom (not readDocument) guarantees a DOM even when the compact + // records cache already has an entry for the file. + const parsed = await ws.indexer?.readDom(file); if (parsed?.parse && parsed.lineMap) { const el = parsed.parse.elements.find( (e) => @@ -156,12 +245,15 @@ function toVscodePosition(p: { line: number; character: number }): vscode.Positi // ── Find all references ───────────────────────────────────────────── export class Ra3ReferenceProvider implements vscode.ReferenceProvider { + constructor(private ws: ModWorkspace) {} + async provideReferences( document: vscode.TextDocument, position: vscode.Position, _context: vscode.ReferenceContext, _token: vscode.CancellationToken, ): Promise { + if (!this.ws.isRa3Workspace()) return null; const text = document.getText(); const offset = document.offsetAt(position); const doc = parseXml(text); @@ -201,8 +293,10 @@ export class Ra3DocumentLinkProvider implements vscode.DocumentLinkProvider { document: vscode.TextDocument, _token: vscode.CancellationToken, ): Promise { + if (!this.ws.isRa3Workspace()) return []; const idx = this.ws.index; - if (!idx) return []; + const searchPaths = idx ? searchPathsFor(idx) : this.ws.searchPaths(); + if (!searchPaths) return []; const text = document.getText(); const doc = parseXml(text); const links: vscode.DocumentLink[] = []; @@ -214,9 +308,9 @@ export class Ra3DocumentLinkProvider implements vscode.DocumentLinkProvider { resolveSource( srcAttr.value, dirname(document.uri.fsPath), - searchPathsFor(idx), + searchPaths, ).path ?? - idx.sourceCandidates.find((c) => c.source === srcAttr.value)?.path ?? + idx?.sourceCandidates.find((c) => c.source === srcAttr.value)?.path ?? null; if (!target) continue; links.push( @@ -236,10 +330,13 @@ export class Ra3DocumentLinkProvider implements vscode.DocumentLinkProvider { // ── Document symbols (outline) ────────────────────────────────────── export class Ra3DocumentSymbolProvider implements vscode.DocumentSymbolProvider { + constructor(private ws: ModWorkspace) {} + async provideDocumentSymbols( document: vscode.TextDocument, _token: vscode.CancellationToken, ): Promise { + if (!this.ws.isRa3Workspace()) return []; const text = document.getText(); const doc = parseXml(text); const root = doc.root; diff --git a/src/features/semanticTokens.ts b/src/features/semanticTokens.ts index 33c0994..b62e885 100644 --- a/src/features/semanticTokens.ts +++ b/src/features/semanticTokens.ts @@ -1,6 +1,7 @@ import * as vscode from "vscode"; import { parseXml } from "../language/xmlParser"; import { buildSemanticTokenRanges } from "../language/semanticTokens"; +import type { ModWorkspace } from "../workspace"; const TOKEN_TYPES = ["type", "property", "string"] as const; @@ -20,10 +21,15 @@ export const RA3_SEMANTIC_TOKENS_LEGEND = new vscode.SemanticTokensLegend([ export class Ra3SemanticTokensProvider implements vscode.DocumentSemanticTokensProvider { + constructor(private ws: ModWorkspace) {} + async provideDocumentSemanticTokens( document: vscode.TextDocument, _token: vscode.CancellationToken, ): Promise { + if (!this.ws.isRa3Workspace()) { + return new vscode.SemanticTokens(new Uint32Array(0)); + } const text = document.getText(); const doc = parseXml(text); if (doc.errors.length === 0) { diff --git a/src/indexer/caches.ts b/src/indexer/caches.ts index d798fc7..25b136b 100644 --- a/src/indexer/caches.ts +++ b/src/indexer/caches.ts @@ -146,6 +146,11 @@ export class IndexRecordsCache { return this.map.size; } + /** Iterates [normalized key, entry] pairs (used by disk persistence). */ + entries(): IterableIterator<[string, IndexRecordsCacheEntry]> { + return this.map.entries(); + } + set(path: string, entry: IndexRecordsCacheEntry): void { const key = normKey(path); this.map.delete(key); @@ -224,3 +229,35 @@ export class IncludeResolveCache { return this.map.size; } } + +/** + * Monotonic counter for workspace-level invalidations. + * + * A build captures `snapshot()` when it starts; if `changedSince()` is true + * when a phase snapshot is about to be published, files may have changed + * mid-build, so the published index is marked stale (and the workspace's + * dirty/rebuild mechanism converges to fresh data shortly after). + */ +export class InvalidationsEpoch { + private value = 0; + + /** Records a new invalidation (content, creation or deletion). */ + mark(): void { + this.value++; + } + + /** Returns the current epoch value. */ + snapshot(): number { + return this.value; + } + + /** True when at least one invalidation happened since `epoch`. */ + changedSince(epoch: number): boolean { + return this.value !== epoch; + } + + /** Current epoch value (read-only accessor). */ + get current(): number { + return this.value; + } +} diff --git a/src/indexer/diskCache.ts b/src/indexer/diskCache.ts new file mode 100644 index 0000000..953c6dd --- /dev/null +++ b/src/indexer/diskCache.ts @@ -0,0 +1,208 @@ +/** + * On-disk persistence for per-file index records. + * + * The records cache (top-level assets / defines / includes / xi:include with + * line numbers) is tiny compared to the source corpus (Corona: ~9k files, + * ~10 MB in memory), but rebuilding it from scratch means reading ~2.6 GB of + * art assets again. Persisting it makes a cold start cost a stat validation + * pass (~seconds on SSD, a few to tens of seconds on a mechanical drive) + * instead of a full rebuild. + * + * Correctness model (layered): + * - every cached record stores a multi-signal stamp + * `{ size, mtimeMs, birthtimeMs, ctimeMs }`; + * - on load, each file is stat-validated (no content reads); mismatches and + * missing files are dropped and re-read during the build; + * - during a session the file watcher invalidates entries precisely; + * - `ra3modxml.reindex` / `ra3modxml.clearCache` remain the final authority. + * + * The file is gzip-compressed JSON written atomically (temp + rename), keyed + * by project identity + settings so stale caches are ignored automatically. + * + * Pure TypeScript: no vscode dependency. + */ + +import { createHash } from "node:crypto"; +import { mkdir, readFile, rename, rm, stat, writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; +import { gzip, gunzip } from "node:zlib"; +import { promisify } from "node:util"; +import type { IndexRecordsCacheEntry } from "./caches"; +import type { IndexRecords } from "./records"; +import type { IndexedFile } from "./types"; + +const gzipAsync = promisify(gzip); +const gunzipAsync = promisify(gunzip); + +export const DISK_CACHE_VERSION = 1; +/** How many stat validations run concurrently on load. */ +const VALIDATE_CONCURRENCY = 32; + +/** Settings that change what the index contains; a mismatch ignores the cache. */ +export interface DiskCacheIdentity { + projectDir: string; + sdkDir: string; + indexSageXml: boolean; + additionalDataSearchPaths: string[]; + builtmodsDirs: string[]; +} + +export interface DiskCacheRecord { + /** Normalized cache key (see `normKey`). */ + key: string; + stat: NonNullable; + records: IndexRecords; + kind: "full" | "shallow"; +} + +interface DiskCacheFile { + version: number; + key: string; + savedAt: string; + records: DiskCacheRecord[]; +} + +export interface DiskCacheLoadStats { + fileExists: boolean; + keyMatched: boolean; + /** Records stored in the file. */ + loaded: number; + /** Records whose stat still matches (kept). */ + validated: number; + /** Records dropped because the file changed, moved or was deleted. */ + dropped: number; +} + +export function diskCacheKey(identity: DiskCacheIdentity): string { + return createHash("sha256") + .update(JSON.stringify(identity)) + .digest("hex") + .slice(0, 16); +} + +export class DiskRecordsCache { + constructor( + private readonly filePath: string, + private readonly identity: DiskCacheIdentity, + ) {} + + get path(): string { + return this.filePath; + } + + /** + * Loads and stat-validates the cache. Returns the kept records plus load + * statistics; missing/corrupt/key-mismatched caches yield an empty result + * instead of an error. + */ + async loadValidated(): Promise<{ + records: DiskCacheRecord[]; + stats: DiskCacheLoadStats; + }> { + const stats: DiskCacheLoadStats = { + fileExists: false, + keyMatched: false, + loaded: 0, + validated: 0, + dropped: 0, + }; + let raw: DiskCacheFile | null = null; + try { + const buf = await readFile(this.filePath); + stats.fileExists = true; + const text = (await gunzipAsync(buf)).toString("utf8"); + const parsed = JSON.parse(text); + if ( + parsed && + parsed.version === DISK_CACHE_VERSION && + parsed.key === diskCacheKey(this.identity) && + Array.isArray(parsed.records) + ) { + raw = parsed as DiskCacheFile; + } + } catch { + // Missing or corrupt cache: fall through with an empty result. + } + if (!raw) return { records: [], stats }; + + stats.keyMatched = true; + stats.loaded = raw.records.length; + const kept: DiskCacheRecord[] = []; + for (let i = 0; i < raw.records.length; i += VALIDATE_CONCURRENCY) { + const chunk = raw.records.slice(i, i + VALIDATE_CONCURRENCY); + const results = await Promise.all( + chunk.map(async (rec): Promise => { + try { + const s = await stat(rec.key); + if ( + s.isFile() && + s.size === rec.stat.size && + s.mtimeMs === rec.stat.mtimeMs && + s.birthtimeMs === rec.stat.birthtimeMs && + s.ctimeMs === rec.stat.ctimeMs + ) { + return rec; + } + } catch { + // File missing or inaccessible. + } + return null; + }), + ); + for (const r of results) { + if (r) { + kept.push(r); + stats.validated++; + } else { + stats.dropped++; + } + } + } + return { records: kept, stats }; + } + + /** Writes the current records cache atomically (temp file + rename). */ + async save( + entries: Iterable<[string, IndexRecordsCacheEntry]>, + ): Promise { + const records: DiskCacheRecord[] = []; + for (const [key, entry] of entries) { + if (!entry.stat) continue; + records.push({ + key, + stat: entry.stat, + records: entry.records, + kind: entry.kind, + }); + } + const payload: DiskCacheFile = { + version: DISK_CACHE_VERSION, + key: diskCacheKey(this.identity), + savedAt: new Date().toISOString(), + records, + }; + const buf = await gzipAsync(Buffer.from(JSON.stringify(payload), "utf8")); + await mkdir(dirname(this.filePath), { recursive: true }); + const tmp = `${this.filePath}.tmp`; + await writeFile(tmp, buf); + await rename(tmp, this.filePath); + } + + /** Deletes the cache file (used by the clear-cache command). */ + async clear(): Promise { + try { + await rm(this.filePath, { force: true }); + } catch { + // Best effort. + } + } + + async status(): Promise<{ exists: boolean; sizeBytes: number }> { + try { + const s = await stat(this.filePath); + return { exists: s.isFile(), sizeBytes: s.size }; + } catch { + return { exists: false, sizeBytes: 0 }; + } + } +} diff --git a/src/indexer/existence.ts b/src/indexer/existence.ts new file mode 100644 index 0000000..b8268f9 --- /dev/null +++ b/src/indexer/existence.ts @@ -0,0 +1,117 @@ +/** + * Lazy file-existence snapshot for include resolution. + * + * `resolveSource` performs synchronous `statSync` existence checks against + * every search base; a cold Corona build does ~110k of them (tens of seconds + * on a mechanical drive). Instead, existence is answered by reading the + * candidate's **parent directory** once (`readdir`, no per-file stat) and + * caching the entry set for the rest of the build. Only directories that are + * actually queried are ever listed, so a cold build pays a handful of + * readdir calls instead of an upfront recursive enumeration of every search + * root (which measurably slowed the XML phase). + * + * Correctness: the workspace clears `IncludeResolveCache` on file + * create/delete; each rebuild creates a fresh snapshot, and the debounced + * rebuild triggered by the watcher converges if anything changed mid-build. + * + * Pure TypeScript: no vscode dependency. + */ + +import { existsSync, readdirSync } from "node:fs"; +import { basename, dirname, parse, resolve, sep } from "node:path"; +import { normKey } from "./caches"; +import type { SearchPaths } from "./includeResolver"; + +/** + * Answers file-existence questions from lazily read directory listings. + * `has()` is authoritative for paths inside the roots and returns null for + * paths the snapshot does not cover (the caller falls back to `statSync`). + */ +export class ExistenceSnapshot { + private roots: string[] = []; + /** parent dir (normalized) -> lowercased file names, or null (no dir). */ + private dirCache = new Map | null>(); + /** Existence answers served from cached directory listings. */ + hits = 0; + /** Paths outside the snapshot that required a statSync fallback. */ + fallbacks = 0; + + constructor(roots: string[]) { + // Roots and lookups must use the same normalization (case-insensitive on + // Windows), otherwise `startsWith` misses due to case differences. + this.roots = roots.map((r) => { + const n = normKey(r); + return n.endsWith(sep) ? n : n + sep; + }); + } + + /** true/false when covered by the snapshot, null when not covered. */ + has(absPath: string): boolean | null { + const parentKey = normKey(dirname(absPath)); + if (!this.isCovered(parentKey)) { + this.fallbacks++; + return null; + } + let entries = this.dirCache.get(parentKey); + if (entries === undefined) { + entries = listDirEntries(parentKey); + this.dirCache.set(parentKey, entries); + } + this.hits++; + return entries ? entries.has(basename(absPath).toLowerCase()) : false; + } + + private isCovered(dirKey: string): boolean { + const key = dirKey.endsWith(sep) ? dirKey : dirKey + sep; + for (const root of this.roots) { + if (key.startsWith(root)) return true; + } + return false; + } +} + +function listDirEntries(dir: string): Set | null { + try { + const out = new Set(); + for (const entry of readdirSync(dir, { withFileTypes: true })) { + if (entry.isFile()) out.add(entry.name.toLowerCase()); + } + return out; + } catch { + return null; + } +} + +/** + * True when `dir` is a filesystem root (e.g. "C:\" or "/"). Such roots are + * never treated as search bases (they may contain the whole disk). + */ +export function isDriveRoot(dir: string): boolean { + const resolved = resolve(dir); + return parse(resolved).root === resolved; +} + +/** + * Builds the snapshot root list from the search bases: drive roots and + * missing directories are skipped, and a root covered by a broader root is + * dropped (e.g. `sdkDir` covers `sdkDir/SageXml`). No directory is listed + * here; listings happen lazily per queried parent directory. + */ +export function buildExistenceSnapshot(searchPaths: SearchPaths): ExistenceSnapshot { + const candidates = [ + ...searchPaths.DATA, + ...searchPaths.ART, + ...searchPaths.AUDIO, + ].map((r) => resolve(r)); + candidates.sort((a, b) => a.length - b.length); + + const roots: string[] = []; + for (const root of candidates) { + if (isDriveRoot(root)) continue; + if (!existsSync(root)) continue; + const normalized = normKey(root); + if (roots.some((r) => normalized.startsWith(r + sep))) continue; + roots.push(normalized); + } + return new ExistenceSnapshot(roots); +} diff --git a/src/indexer/fileScanner.ts b/src/indexer/fileScanner.ts index e74a3c2..2bb3b82 100644 --- a/src/indexer/fileScanner.ts +++ b/src/indexer/fileScanner.ts @@ -61,6 +61,38 @@ export class CachedDirectoryWalker implements FileWalker { } } +/** + * True for paths whose changes can never affect the index: `.git` internals + * touched by background fetch/maintenance, and transient temp/backup files + * created by editors or other extensions (`UnitCrate.xml.git`, `*.tmp`, + * `*.lock`, `file~`, `.#file`, ...). Such events are ignored by the file + * watcher instead of triggering rebuilds. + */ +export function isWatcherNoisePath(fsPath: string): boolean { + const segments = fsPath.split(/[\\/]/); + if (segments.some((seg) => seg.toLowerCase() === ".git")) return true; + const base = segments[segments.length - 1] ?? ""; + const lower = base.toLowerCase(); + const noiseSuffixes = [".git", ".tmp", ".lock", "~", ".swp", ".bak", ".orig"]; + if (noiseSuffixes.some((suffix) => lower.endsWith(suffix))) return true; + return lower.startsWith(".#") || lower.startsWith(".~"); +} + +/** + * True for files whose *content* participates in the index: XML documents + * and art-asset XML (.w3x). Reasonable text formats in RA3 mods are `.xml`, + * `.w3x` and `.lua`; lua is not indexed yet, and compiled manifests are + * binary `*.manifest` (there is no `.manifestxml` source format). Files + * already in the index with other extensions (e.g. sniffed XML) are handled + * separately via `ModIndexer.isIndexedFile`. Content changes to binary art + * (e.g. textures, `.w3d`) cannot change index records, so they do not need + * to trigger a rebuild. + */ +export function isContentRelevantPath(fsPath: string): boolean { + const ext = extname(fsPath).toLowerCase(); + return ext === ".xml" || ext === ".w3x"; +} + /** * Builds the candidate list for Include/@source completion from a set of * search directories. For DATA directories only *.xml files are listed; for diff --git a/src/indexer/includeResolver.ts b/src/indexer/includeResolver.ts index e7f6d67..792984b 100644 --- a/src/indexer/includeResolver.ts +++ b/src/indexer/includeResolver.ts @@ -8,6 +8,7 @@ import { join, resolve, normalize, isAbsolute } from "node:path"; import { statSync } from "node:fs"; +import type { ExistenceSnapshot } from "./existence"; export type IncludeKind = "all" | "instance" | "reference"; export type SourcePrefix = "DATA" | "ART" | "AUDIO" | null; @@ -99,41 +100,58 @@ export function resolveSource( source: string, currentDir: string | null, searchPaths: SearchPaths, + existence?: ExistenceSnapshot, ): ResolveResult { const raw = source.trim().replace(/\\/g, "/"); const { prefix, rest } = splitPrefix(raw); if (prefix) { const bases = searchPaths[prefix] ?? []; - const direct = findInBases(rest, bases); + const direct = findInBases(rest, bases, existence); if (direct) return { path: direct, prefix, raw }; if (prefix === "ART" && !rest.includes("/")) { const two = rest.slice(0, 2).toLowerCase(); - const prefixed = findInBases(`${two}/${rest}`, bases); + const prefixed = findInBases(`${two}/${rest}`, bases, existence); if (prefixed) return { path: prefixed, prefix, raw }; } return { path: null, prefix, raw }; } if (currentDir && isAbsolute(rest)) { - return { path: fileExists(rest) ? rest : null, prefix: null, raw }; + return { + path: fileExists(rest, existence) ? rest : null, + prefix: null, + raw, + }; } if (currentDir) { const candidate = resolve(currentDir, rest); - return { path: fileExists(candidate) ? candidate : null, prefix: null, raw }; + return { + path: fileExists(candidate, existence) ? candidate : null, + prefix: null, + raw, + }; } return { path: null, prefix: null, raw }; } -function findInBases(relPath: string, bases: string[]): string | null { +function findInBases( + relPath: string, + bases: string[], + existence?: ExistenceSnapshot, +): string | null { for (const base of bases) { const candidate = normalize(resolve(base, relPath)); - if (fileExists(candidate)) return candidate; + if (fileExists(candidate, existence)) return candidate; } return null; } -function fileExists(path: string): boolean { +function fileExists(path: string, existence?: ExistenceSnapshot): boolean { + if (existence) { + const known = existence.has(path); + if (known !== null) return known; + } try { return statSync(path).isFile(); } catch { diff --git a/src/indexer/indexer.ts b/src/indexer/indexer.ts index c6d681f..80d81e7 100644 --- a/src/indexer/indexer.ts +++ b/src/indexer/indexer.ts @@ -18,7 +18,6 @@ import { LineMap, parseXml, stripBom, - type XmlDocument, type XmlElement, } from "../language/xmlParser"; import { @@ -28,6 +27,11 @@ import { type ResolveResult, type SearchPaths, } from "./includeResolver"; +import { + buildExistenceSnapshot, + type ExistenceSnapshot, +} from "./existence"; +import { findXPointerContainer, localName } from "./xpointer"; import { deriveAssetId, deriveAssetType, @@ -59,10 +63,11 @@ const MAX_DEPTH = 300; /** Files above this size are never parsed (safety against binary blobs). */ const MAX_PARSE_BYTES = 4 * 1024 * 1024; /** - * Fully parsed XML documents. `.xml` / `.manifestxml` files are small enough - * that a full DOM is affordable. + * Fully parsed XML documents. `.xml` files are small enough that a full DOM + * is affordable. (Compiled manifests are binary `*.manifest` files parsed by + * `manifestParser`; there is no `.manifestxml` source format.) */ -const FULL_XML_EXTENSIONS = new Set([".xml", ".manifestxml"]); +const FULL_XML_EXTENSIONS = new Set([".xml"]); /** * XML documents whose top-level structure is all the index needs (art-asset * files exported by modeling tools, e.g. .w3x). They are shallow-scanned so @@ -79,6 +84,8 @@ export class ModIndexer { private docs: DocumentCache; private recordsCache: IndexRecordsCache; private resolveCache: IncludeResolveCache; + /** Directory-based file existence snapshot (avoids cold statSync storms). */ + private existence: ExistenceSnapshot | null = null; private scanCounters = { shallowScannedFiles: 0, shallowCacheHits: 0, @@ -86,7 +93,19 @@ export class ModIndexer { resolveCacheHits: 0, resolveCalls: 0, }; - private phase = { candidatesMs: 0, walkMs: 0 }; + private timings = { candidatesMs: 0, walkMs: 0, artScanMs: 0 }; + /** + * Phase A ("xml") registers art-asset XML files (`.w3x` and sniffed XML) + * without reading their content; the queue is drained by phase B ("art"), + * which shallow-scans them and walks any includes they contain. + */ + private deferArtScan = false; + private artQueue: { + path: string; + stream: StreamInfo; + depth: number; + viaInstance: boolean; + }[] = []; private assets = new Map>(); private assetsById = new Map(); private defines = new Map(); @@ -111,8 +130,7 @@ export class ModIndexer { /** * Returns a document for indexing/navigation: - * - `.xml` / `.manifestxml` files are fully parsed (bounded by - * MAX_PARSE_BYTES); + * - `.xml` files are fully parsed (bounded by MAX_PARSE_BYTES); * - `.w3x` (and unknown-extension files whose content looks like XML) are * shallow-scanned, so huge model files never become a DOM; * - everything else is registered as a file but never parsed. @@ -120,7 +138,10 @@ export class ModIndexer { * Cached entries are reused when the file stat is unchanged, which lets a * workspace-owned cache survive rebuilds. */ - async readDocument(path: string): Promise { + async readDocument( + path: string, + opts?: { deferArt?: boolean }, + ): Promise { const key = normKey(path); const trust = this.opts.trustUnchanged === true && !this.opts.changedFiles?.has(key); @@ -143,29 +164,71 @@ export class ModIndexer { try { const st = await stat(path); const rec = this.recordsCache.get(key); - if (rec?.stat && rec.stat.mtimeMs === st.mtimeMs && rec.stat.size === st.size) { + if ( + rec?.stat && + rec.stat.mtimeMs === st.mtimeMs && + rec.stat.size === st.size && + rec.stat.birthtimeMs === st.birthtimeMs && + rec.stat.ctimeMs === st.ctimeMs + ) { return this.recordsParsed(path, rec); } const hit = this.docs.get(key); if ( hit?.file.stat && hit.file.stat.mtimeMs === st.mtimeMs && - hit.file.stat.size === st.size + hit.file.stat.size === st.size && + hit.file.stat.birthtimeMs === st.birthtimeMs && + hit.file.stat.ctimeMs === st.ctimeMs ) { this.files.set(key, hit.file); return hit; } const mode = await this.detectXmlMode(path); - if (mode === "shallow") return this.scanShallow(path, st); + if (mode === "shallow") { + // Phase A: register the art file, defer the shallow scan to phase B. + if (opts?.deferArt) { + const file: IndexedFile = { + path: resolve(path), + stat: { + mtimeMs: st.mtimeMs, + size: st.size, + birthtimeMs: st.birthtimeMs, + ctimeMs: st.ctimeMs, + }, + }; + // Deliberately NOT stored in the DocumentCache: a deferred entry + // has no records, and a later phase-B read must re-scan it. + this.files.set(key, file); + return { file, parse: null, records: null, lineMap: null, deferredArt: true }; + } + return this.scanShallow(path, st); + } if (mode === "binary") { - const file: IndexedFile = { path: resolve(path), stat: { mtimeMs: st.mtimeMs, size: st.size } }; + const file: IndexedFile = { + path: resolve(path), + stat: { + mtimeMs: st.mtimeMs, + size: st.size, + birthtimeMs: st.birthtimeMs, + ctimeMs: st.ctimeMs, + }, + }; const parsed: ParsedFile = { file, parse: null, records: null, lineMap: null }; this.docs.set(parsed); this.files.set(key, file); return parsed; } if (st.size > MAX_PARSE_BYTES) { - const file: IndexedFile = { path: resolve(path), stat: { mtimeMs: st.mtimeMs, size: st.size } }; + const file: IndexedFile = { + path: resolve(path), + stat: { + mtimeMs: st.mtimeMs, + size: st.size, + birthtimeMs: st.birthtimeMs, + ctimeMs: st.ctimeMs, + }, + }; const parsed: ParsedFile = { file, parse: null, records: null, lineMap: null }; this.docs.set(parsed); this.files.set(key, file); @@ -176,7 +239,15 @@ export class ModIndexer { const parse = parseXml(text); const records = extractIndexRecords(parse, lineMap); const parsed: ParsedFile = { - file: { path: resolve(path), stat: { mtimeMs: st.mtimeMs, size: st.size } }, + file: { + path: resolve(path), + stat: { + mtimeMs: st.mtimeMs, + size: st.size, + birthtimeMs: st.birthtimeMs, + ctimeMs: st.ctimeMs, + }, + }, parse, records, lineMap, @@ -211,7 +282,15 @@ export class ModIndexer { const lineMap = new LineMap(text); const records = recordsFromShallow(scanXmlShallow(text), lineMap); const parsed: ParsedFile = { - file: { path: resolve(path), stat: { mtimeMs: st.mtimeMs, size: st.size } }, + file: { + path: resolve(path), + stat: { + mtimeMs: st.mtimeMs, + size: st.size, + birthtimeMs: st.birthtimeMs, + ctimeMs: st.ctimeMs, + }, + }, parse: null, records, lineMap: null, @@ -235,11 +314,11 @@ export class ModIndexer { } /** - * Reads a document and guarantees a DOM parse tree. Used only for - * root-level xpointer selection (rare), where the target's - * container children are needed. + * Reads a document and guarantees a DOM parse tree. Used for root-level + * xpointer selection (rare) and by the document-local scope + * (logical include expansion + precise definition locations). */ - private async readDom(path: string): Promise { + async readDom(path: string): Promise { const key = normKey(path); const cached = this.docs.get(key); if (cached?.parse?.root) { @@ -253,7 +332,9 @@ export class ModIndexer { hit?.parse?.root && hit.file.stat && hit.file.stat.mtimeMs === st.mtimeMs && - hit.file.stat.size === st.size + hit.file.stat.size === st.size && + hit.file.stat.birthtimeMs === st.birthtimeMs && + hit.file.stat.ctimeMs === st.ctimeMs ) { this.files.set(key, hit.file); return hit; @@ -264,7 +345,15 @@ export class ModIndexer { const parse = parseXml(text); const records = extractIndexRecords(parse, lineMap); const parsed: ParsedFile = { - file: { path: resolve(path), stat: { mtimeMs: st.mtimeMs, size: st.size } }, + file: { + path: resolve(path), + stat: { + mtimeMs: st.mtimeMs, + size: st.size, + birthtimeMs: st.birthtimeMs, + ctimeMs: st.ctimeMs, + }, + }, parse, records, lineMap, @@ -299,7 +388,12 @@ export class ModIndexer { return hit; } this.scanCounters.resolveCalls++; - const result = resolveSource(source, currentDir, this.searchPaths); + const result = resolveSource( + source, + currentDir, + this.searchPaths, + this.existence ?? undefined, + ); this.resolveCache.set(key, result); return result; } @@ -323,14 +417,26 @@ export class ModIndexer { return this.docs.get(path); } - async build(): Promise { + /** True when the file is part of the current build's index. */ + isIndexedFile(path: string): boolean { + return this.files.has(normKey(path)); + } + + async build(onPhase?: (index: ModIndex) => void | Promise): Promise { const start = Date.now(); + // Root list only; directories are listed lazily on first query, so the + // XML phase does not pay an upfront recursive enumeration of the SDK. + this.existence = buildExistenceSnapshot(this.searchPaths); const projectData = await findCaseInsensitiveDir(join(this.opts.projectDir, "Data")); const additionalMaps = projectData ? await findCaseInsensitiveDir(join(projectData, "additionalmaps")) : null; // ── Streams ── + // Phase A: walk the include graph without reading art-asset content. + // `.w3x` (and sniffed XML) files are registered and queued; their + // top-level assets and nested includes are processed in phase B. + this.deferArtScan = true; const walkStart = Date.now(); const staticEntry = projectData ? join(projectData, "Mod.xml") : null; if (staticEntry) { @@ -360,7 +466,7 @@ export class ModIndexer { await this.walk(entry, "all", stream, 0); } } - this.phase.walkMs = Date.now() - walkStart; + this.timings.walkMs = Date.now() - walkStart; // ── Source completion candidates ── const candidatesStart = Date.now(); @@ -408,45 +514,94 @@ export class ModIndexer { ...sdkRootCandidates, ...this.sourceCandidates, ]); - this.phase.candidatesMs = Date.now() - candidatesStart; + this.timings.candidatesMs = Date.now() - candidatesStart; + // Publish the XML phase as an immutable snapshot: features get usable + // completions/navigation/diagnostics for XML + manifest data immediately, + // while the slow art scan continues in the background. + const xmlPhase = this.snapshotIndex("xml", false, start); + if (onPhase) await onPhase(xmlPhase); + + // ── Phase B: art assets ── + this.deferArtScan = false; + const artStart = Date.now(); + while (this.artQueue.length) { + const entry = this.artQueue.shift()!; + const parsed = await this.readDocument(entry.path); + if (parsed?.records) { + await this.applyRecords(parsed, entry.stream, entry.depth, entry.viaInstance); + } + } + this.timings.artScanMs = Date.now() - artStart; + + return this.snapshotIndex("art", true, start); + } + + /** + * Produces a copy of the current index state. Phase-A snapshots must be + * immutable: phase B keeps mutating the live maps after the snapshot has + * been handed to features, so every nested map/array is cloned here. + */ + private snapshotIndex( + phase: "xml" | "art", + complete: boolean, + startedAt: number, + ): ModIndex { const manifestAssetCount = [...this.manifests.values()].reduce( (sum, m) => sum + m.assets.length, 0, ); + const assets = new Map>(); + for (const [type, byId] of this.assets) { + const copied = new Map(); + for (const [id, defs] of byId) copied.set(id, defs.slice()); + assets.set(type, copied); + } + const assetsById = new Map(); + for (const [id, defs] of this.assetsById) assetsById.set(id, defs.slice()); + const defines = new Map(); + for (const [name, defs] of this.defines) defines.set(name, defs.slice()); return { projectDir: resolve(this.opts.projectDir), sdkDir: resolve(this.opts.sdkDir), - assets: this.assets, - assetsById: this.assetsById, - defines: this.defines, - files: this.files, - streams: this.streams, - manifests: this.manifests, - sourceCandidates: this.sourceCandidates, - diagnostics: this.diagnostics, + complete, + phase, + assets, + assetsById, + defines, + files: new Map(this.files), + streams: this.streams.map((s) => ({ ...s, files: new Set(s.files) })), + manifests: new Map(this.manifests), + sourceCandidates: this.sourceCandidates.slice(), + diagnostics: this.diagnostics.slice(), stats: { projectDir: resolve(this.opts.projectDir), sdkDir: resolve(this.opts.sdkDir), + phase, + complete, indexedFiles: this.files.size, parsedFiles: [...this.files.values()].filter( (f) => f.stat != null && f.stat.size <= MAX_PARSE_BYTES, ).length, shallowScannedFiles: this.scanCounters.shallowScannedFiles, + deferredArtFiles: this.artQueue.length, shallowCacheHits: this.scanCounters.shallowCacheHits, recordsCacheHits: this.scanCounters.recordsCacheHits, resolveCacheHits: this.scanCounters.resolveCacheHits, resolveCalls: this.scanCounters.resolveCalls, - candidatesMs: this.phase.candidatesMs, - walkMs: this.phase.walkMs, + snapshotHits: this.existence?.hits ?? 0, + snapshotFallbacks: this.existence?.fallbacks ?? 0, + candidatesMs: this.timings.candidatesMs, + walkMs: this.timings.walkMs, + artScanMs: this.timings.artScanMs, assetCount: [...this.assets.values()].reduce((sum, byId) => sum + byId.size, 0), defineCount: this.defines.size, manifestFiles: this.manifests.size, manifestAssetCount, streams: this.streams.length, sourceCandidates: this.sourceCandidates.length, - elapsedMs: Date.now() - start, + elapsedMs: Date.now() - startedAt, }, }; } @@ -483,9 +638,19 @@ export class ModIndexer { // readDocument returns compact index records for every indexable XML // document (full parse or shallow scan), or a bare file registration - // for binary / unparseable targets. - const parsed = await this.readDocument(path); + // for binary / unparseable targets. During phase A, art-asset XML files + // are registered and queued instead of scanned (deferredArt). + const parsed = await this.readDocument(path, this.deferArtScan ? { deferArt: true } : undefined); if (!parsed) return; + if (parsed.deferredArt) { + this.artQueue.push({ + path: parsed.file.path, + stream, + depth, + viaInstance: mode === "instance", + }); + return; + } if (parsed.records) { await this.applyRecords(parsed, stream, depth, mode === "instance"); return; @@ -722,11 +887,6 @@ export class ModIndexer { // ── Module-level helpers ───────────────────────────────────────────── -function localName(tag: string): string { - const idx = tag.lastIndexOf(":"); - return idx >= 0 ? tag.slice(idx + 1) : tag; -} - function lineOf(parsed: ParsedFile, offset: number): number { if (!parsed.lineMap) return 0; return parsed.lineMap.positionAt(offset).line + 1; @@ -774,15 +934,6 @@ async function findCaseInsensitiveDir(dir: string): Promise { } } -function findXPointerContainer(doc: XmlDocument, xpointer: string): XmlElement | null { - // Supports the form used by the mods: - // xmlns(n=uri:ea.com:eala:asset) xpointer(/n:ElementName/child::*) - const m = /xpointer\(\/\w+:(\w+)\/child::\*\)/.exec(xpointer); - if (!m) return null; - const name = m[1]; - return doc.elements.find((el) => localName(el.name) === name) ?? null; -} - /** Keeps the first candidate for each case-insensitive source string. */ function dedupeSourceCandidates(candidates: SourceCandidate[]): SourceCandidate[] { const seen = new Set(); diff --git a/src/indexer/localScope.ts b/src/indexer/localScope.ts new file mode 100644 index 0000000..0822642 --- /dev/null +++ b/src/indexer/localScope.ts @@ -0,0 +1,265 @@ +import { dirname, resolve } from "node:path"; +import { LineMap, parseXml, type XmlDocument } from "../language/xmlParser"; +import { extractIndexRecords } from "./records"; +import { resolveSource, type SearchPaths } from "./includeResolver"; +import { expandDocument, type LogicalDocument } from "./logicalTree"; +import type { + AssetDef, + DefineDef, + LocalOverlay, + ModIndex, + ParsedFile, +} from "./types"; + +export interface LocalScopeContext { + projectDir: string; + sdkDir: string; + searchPaths: SearchPaths; + /** Reads a file's compact index records (full parse or shallow scan). */ + readRecords(path: string): Promise; + /** Reads a file and guarantees a DOM parse tree. */ + readDom(path: string): Promise; +} + +/** + * Everything the features need for the currently open document: the original + * parse, the expanded logical tree, per-source line maps, the local overlay + * and the overlay-aware index for lookups. + */ +export interface DocumentScope { + uri: string; + version: number; + parse: XmlDocument; + lineMap: LineMap; + expanded: LogicalDocument; + /** scopePathKey(file) -> line map (current file + expanded include targets). */ + lineMaps: Map; + /** Local assets/defines from the document itself and its include chain. */ + overlay: LocalOverlay; + /** Global index with `local` attached (or a minimal standalone index). */ + merged: ModIndex | null; +} + +/** Normalized key used for source-file identity / line-map lookup. */ +export function scopePathKey(path: string): string { + return path.replace(/\\/g, "/").toLowerCase(); +} + +/** + * Builds the document scope for the current (possibly unsaved) text: + * - a local overlay of assets/defines reachable from this file; + * - a logical tree with supported xi:include targets spliced in place. + */ +export async function buildDocumentScope( + uri: string, + text: string, + version: number, + ctx: LocalScopeContext, +): Promise { + const lineMap = new LineMap(text); + const parse = parseXml(text); + const builder = new OverlayBuilder(ctx); + await builder.addEntry(uri, parse, lineMap); + + const expanded = await expandDocument(uri, parse, { + resolve: (source, currentDir) => + resolveSource(source, currentDir, ctx.searchPaths).path, + readDom: async (path) => { + const parsed = await ctx.readDom(path); + return parsed?.parse && parsed.lineMap + ? { parse: parsed.parse, lineMap: parsed.lineMap } + : null; + }, + }); + + const lineMaps = new Map(); + lineMaps.set(scopePathKey(uri), lineMap); + for (const [path, lm] of builder.lineMaps) { + if (!lineMaps.has(path)) lineMaps.set(path, lm); + } + + return { + uri, + version, + parse, + lineMap, + expanded, + lineMaps, + overlay: builder.overlay, + merged: null, + }; +} + +/** + * Attaches a document-local overlay to a global index without copying the + * global maps. When there is no global index yet, returns a minimal standalone + * index so the local chain alone can serve completions / references. + */ +export function withLocalOverlay( + global: ModIndex | null, + overlay: LocalOverlay, + projectDir: string, + sdkDir: string, +): ModIndex { + if (!global) { + return { + projectDir, + sdkDir, + complete: false, + phase: "xml", + assets: new Map(), + assetsById: new Map(), + defines: new Map(), + files: new Map(), + streams: [], + manifests: new Map(), + sourceCandidates: [], + diagnostics: [], + stats: { + projectDir, + sdkDir, + phase: "xml", + complete: false, + indexedFiles: 0, + parsedFiles: 0, + shallowScannedFiles: 0, + deferredArtFiles: 0, + shallowCacheHits: 0, + recordsCacheHits: 0, + resolveCacheHits: 0, + resolveCalls: 0, + snapshotHits: 0, + snapshotFallbacks: 0, + candidatesMs: 0, + walkMs: 0, + artScanMs: 0, + assetCount: 0, + defineCount: 0, + manifestFiles: 0, + manifestAssetCount: 0, + streams: 0, + sourceCandidates: 0, + elapsedMs: 0, + }, + local: overlay, + }; + } + return { ...global, local: overlay }; +} + +const MAX_LOCAL_DEPTH = 64; + +class OverlayBuilder { + readonly overlay: LocalOverlay = { + assets: new Map(), + assetsById: new Map(), + defines: new Map(), + }; + readonly lineMaps = new Map(); + private visited = new Set(); + + constructor(private ctx: LocalScopeContext) {} + + async addEntry( + path: string, + parse: XmlDocument, + lineMap: LineMap, + ): Promise { + this.lineMaps.set(scopePathKey(path), lineMap); + await this.addParsed({ + file: { path: resolve(path), stat: null }, + parse, + records: extractIndexRecords(parse, lineMap), + lineMap, + }, 0); + } + + async addFile(path: string, depth: number): Promise { + if (depth > MAX_LOCAL_DEPTH) return; + const parsed = await this.ctx.readRecords(path); + if (parsed) await this.addParsed(parsed, depth); + } + + private async addParsed(parsed: ParsedFile, depth: number): Promise { + const path = parsed.file.path; + const key = scopePathKey(path); + if (!parsed.records || this.visited.has(key)) return; + this.visited.add(key); + if (parsed.lineMap) this.lineMaps.set(key, parsed.lineMap); + + const origin = this.originOf(path); + for (const asset of parsed.records.assets) { + this.addAsset({ + type: asset.type, + id: asset.id, + file: path, + line: asset.line, + origin, + stream: "local", + }); + } + for (const define of parsed.records.defines) { + const entry: DefineDef = { + name: define.name, + value: define.value, + file: path, + line: define.line, + origin, + }; + const arr = this.overlay.defines.get(define.name.toLowerCase()); + if (arr) arr.push(entry); + else this.overlay.defines.set(define.name.toLowerCase(), [entry]); + } + + for (const inc of parsed.records.includes) { + const resolved = resolveSource(inc.source, dirname(path), this.ctx.searchPaths); + if (!resolved.path) continue; + if (inc.type === "all" || inc.type === "instance") { + await this.addFile(resolved.path, depth + 1); + } + // type="reference" points at compiled manifests; their assets are + // provided by the global index, so the local text overlay skips them. + } + + for (const xi of [ + ...parsed.records.nestedXiIncludes, + ...parsed.records.rootXiIncludes, + ]) { + const resolved = resolveSource(xi.href, dirname(path), this.ctx.searchPaths); + if (resolved.path) await this.addFile(resolved.path, depth + 1); + } + } + + private addAsset(def: AssetDef): void { + const typeKey = def.type; + const idKey = def.id.toLowerCase(); + let byId = this.overlay.assets.get(typeKey); + if (!byId) { + byId = new Map(); + this.overlay.assets.set(typeKey, byId); + } + const arr = byId.get(idKey); + if (arr) { + if (arr.some((a) => a.file === def.file && a.line === def.line)) return; + arr.push(def); + } else { + byId.set(idKey, [def]); + } + const all = this.overlay.assetsById.get(idKey); + if (all) { + if (all.some((a) => a.file === def.file && a.line === def.line)) return; + all.push(def); + } else { + this.overlay.assetsById.set(idKey, [def]); + } + } + + private originOf(path: string): "project" | "sdk" | "manifest" { + const p = resolve(path).toLowerCase(); + const project = resolve(this.ctx.projectDir).toLowerCase(); + const sdk = resolve(this.ctx.sdkDir).toLowerCase(); + if (p.startsWith(project + "\\")) return "project"; + if (sdk && p.startsWith(sdk + "\\")) return "sdk"; + return "project"; + } +} diff --git a/src/indexer/logicalTree.ts b/src/indexer/logicalTree.ts new file mode 100644 index 0000000..bcc13df --- /dev/null +++ b/src/indexer/logicalTree.ts @@ -0,0 +1,249 @@ +import { dirname } from "node:path"; +import type { + LineMap, + XmlDocument, + XmlElement, + XmlParseError, +} from "../language/xmlParser"; +import { resolveElementType } from "../language/typeContext"; +import { isAssignableTo } from "../model/schemaModel"; +import { findXPointerContainer, localName } from "./xpointer"; + +/** + * A logical document is the parsed tree of the currently open file with + * supported `xi:include` targets spliced in place. Every node keeps its + * original source file and offsets so diagnostics / hover / navigation can + * map back to the real file. + */ +export interface LogicalElement extends Omit { + parent: LogicalElement | null; + children: LogicalElement[]; + sourceFile: string; +} + +export interface LogicalDocument { + root: LogicalElement | null; + elements: LogicalElement[]; + /** Mirrors XmlDocument so existing helpers (findElementAt etc.) work. */ + errors: XmlParseError[]; + declarationEnd: number; +} + +export interface ExpandContext { + /** Resolves an xi:include href (BAB search order). */ + resolve(source: string, currentDir: string): string | null; + /** Reads a target and guarantees a DOM parse tree. */ + readDom(path: string): Promise<{ parse: XmlDocument; lineMap: LineMap } | null>; + /** Include-depth guard. Defaults to 64. */ + maxDepth?: number; +} + +const DEFAULT_MAX_DEPTH = 64; + +/** + * Builds the logical document for `entryPath` by replacing supported + * `xi:include` elements with their selected target children. + * + * The original xi:include node stays in `elements` (so hover keeps working) + * but is removed from the logical child list; its selected content is spliced + * in as siblings. Nodes are shallow-cloned shells with a rebuilt parent/child + * chain, so cached parse trees in the indexer are never mutated. + */ +export async function expandDocument( + entryPath: string, + parse: XmlDocument, + ctx: ExpandContext, +): Promise { + const elements: LogicalElement[] = []; + const map = new Map(); + // Pre-create a logical shell for every original element (including orphan + // nodes produced by the parser's unterminated-quote recovery). Parent + // pointers follow the original tree so type context is preserved. + for (const orig of parse.elements) { + const parent = orig.parent ? map.get(orig.parent) ?? null : null; + const clone = cloneNode(orig, parent, entryPath); + map.set(orig, clone); + elements.push(clone); + } + // Cycle guard is a recursion stack, not a global visited set: the same + // fragment may legitimately be included under several parents. + const stack = new Set(); + const root = parse.root ? map.get(parse.root) ?? null : null; + + // Traverse from the real root plus any parser-recovery orphans (elements + // whose parent is null but are not the root). + const roots = new Set(); + if (parse.root) roots.add(parse.root); + for (const orig of parse.elements) { + if (orig !== parse.root && orig.parent === null) roots.add(orig); + } + for (const origRoot of roots) { + const logicalRoot = map.get(origRoot)!; + await expandChildren( + origRoot, + logicalRoot, + entryPath, + 0, + ctx, + elements, + stack, + map, + ); + } + return { root, elements, errors: parse.errors, declarationEnd: parse.declarationEnd }; +} + +function cloneNode( + el: XmlElement, + parent: LogicalElement | null, + sourceFile: string, +): LogicalElement { + return { ...el, parent, children: [], sourceFile }; +} + +async function expandChildren( + origParent: XmlElement, + logicalParent: LogicalElement, + file: string, + depth: number, + ctx: ExpandContext, + elements: LogicalElement[], + stack: Set, + map?: Map, +): Promise { + for (const child of origParent.children) { + await handleChild(child, logicalParent, file, depth, ctx, elements, stack, map); + } +} + +async function handleChild( + orig: XmlElement, + logicalParent: LogicalElement, + file: string, + depth: number, + ctx: ExpandContext, + elements: LogicalElement[], + stack: Set, + map?: Map, +): Promise { + const isXi = orig.name.toLowerCase().startsWith("xi:") && + localName(orig.name).toLowerCase() === "include"; + if (isXi) { + // Keep the xi:include itself discoverable (hover), but let its selected + // content replace it in the logical child list. + if (!map?.has(orig)) elements.push(cloneNode(orig, logicalParent, file)); + await expandXi(orig, logicalParent, file, depth, ctx, elements, stack); + return; + } + + let clone = map?.get(orig); + if (!clone) { + clone = cloneNode(orig, logicalParent, file); + elements.push(clone); + } + logicalParent.children.push(clone); + await expandChildren(orig, clone, file, depth + 1, ctx, elements, stack, map); +} + +async function expandXi( + xi: XmlElement, + logicalParent: LogicalElement, + parentFile: string, + depth: number, + ctx: ExpandContext, + elements: LogicalElement[], + stack: Set, +): Promise { + const href = xi.attrs.find((a) => a.name === "href")?.value; + if (!href) return; + const resolved = ctx.resolve(href, dirname(parentFile)); + if (!resolved) return; + + const key = normPath(resolved); + if (stack.has(key)) return; // include cycle + if (depth > (ctx.maxDepth ?? DEFAULT_MAX_DEPTH)) return; + stack.add(key); + + try { + const target = await ctx.readDom(resolved); + if (!target?.parse?.root) return; + + const xpointer = xi.attrs.find((a) => a.name === "xpointer")?.value ?? ""; + const selected = xpointer + ? findXPointerContainer(target.parse, xpointer)?.children ?? [] + : target.parse.root.children; + + for (const sel of selected) { + await handleChild(sel, logicalParent, resolved, depth + 1, ctx, elements, stack); + } + } finally { + stack.delete(key); + } +} + +/** True when a logical element's resolved XSD type is a GameObject. */ +export function isGameObjectElement(el: LogicalElement): boolean { + const type = resolveElementType(el); + return type != null && isAssignableTo(type, "GameObject"); +} + +/** Nearest ancestor whose resolved type is a GameObject (or subclass). */ +export function findContainingGameObject( + el: LogicalElement, +): LogicalElement | null { + let cur = el.parent; + while (cur) { + if (isGameObjectElement(cur)) return cur; + cur = cur.parent; + } + return null; +} + +export interface LocalIdInfo { + id: string; + el: LogicalElement; +} + +/** + * Collects every `id` defined inside a GameObject subtree (including modules + * spliced in through xi:include). These are the candidates for Poid-typed + * pipeline-local references such as AttachModuleId / ModuleId. + */ +export function collectLocalIds(root: LogicalElement): LocalIdInfo[] { + const out: LocalIdInfo[] = []; + const seen = new Set(); + const stack: LogicalElement[] = [root]; + while (stack.length) { + const el = stack.pop()!; + const idAttr = el.attrs.find((a) => a.name === "id"); + if (idAttr?.hasValue) { + const key = idAttr.value.toLowerCase(); + if (!seen.has(key)) { + seen.add(key); + out.push({ id: idAttr.value, el }); + } + } + for (const child of el.children) stack.push(child); + } + return out; +} + +/** Finds an id inside a GameObject subtree (case-insensitive). */ +export function findLocalId( + root: LogicalElement, + id: string, +): LogicalElement | null { + const wanted = id.toLowerCase(); + const stack: LogicalElement[] = [root]; + while (stack.length) { + const el = stack.pop()!; + const idAttr = el.attrs.find((a) => a.name === "id"); + if (idAttr?.hasValue && idAttr.value.toLowerCase() === wanted) return el; + for (const child of el.children) stack.push(child); + } + return null; +} + +function normPath(path: string): string { + return path.replace(/\\/g, "/").toLowerCase(); +} diff --git a/src/indexer/refs.ts b/src/indexer/refs.ts index 5e0648a..d010e0f 100644 --- a/src/indexer/refs.ts +++ b/src/indexer/refs.ts @@ -96,8 +96,11 @@ export function resolveReferenceTargetsForType( attrName: string, id: string, ): ReferenceTarget[] { - const defs = idx.assetsById.get(id.toLowerCase()); - if (!defs?.length) return []; + const defs = mergeLocalAndGlobalDefs( + idx.local?.assetsById.get(id.toLowerCase()), + idx.assetsById.get(id.toLowerCase()), + ); + if (!defs.length) return []; const nameLower = attrName.toLowerCase(); let refType: string | null = null; @@ -127,3 +130,25 @@ export function resolveReferenceTargetsForType( targets.sort((a, b) => a.score - b.score || a.def.id.localeCompare(b.def.id)); return targets; } + +/** + * Merges document-local definitions with the global index, keeping local + * entries first and de-duplicating definitions that exist in both. + */ +export function mergeLocalAndGlobalDefs( + local: readonly AssetDef[] | undefined, + global: readonly AssetDef[] | undefined, +): AssetDef[] { + const seen = new Set(); + const out: AssetDef[] = []; + for (const list of [local, global]) { + if (!list) continue; + for (const def of list) { + const key = `${def.type}\u0000${def.id.toLowerCase()}\u0000${def.file}\u0000${def.line}`; + if (seen.has(key)) continue; + seen.add(key); + out.push(def); + } + } + return out; +} diff --git a/src/indexer/types.ts b/src/indexer/types.ts index cdd4fa2..443a5fa 100644 --- a/src/indexer/types.ts +++ b/src/indexer/types.ts @@ -32,9 +32,36 @@ export interface DefineDef { origin: AssetOrigin; } +/** + * Document-local overlay produced by `localScope.ts`. It contains assets / + * defines reachable from the currently open document (its own text plus its + * include chain), even when that file is not part of any global stream. + * + * The overlay is attached to a `ModIndex` as `local` rather than merged into + * the global maps, so large indexes (Corona: ~65k assets) are never copied + * on every keystroke. Lookup helpers consult `local` first. + */ +export interface LocalOverlay { + /** type -> id -> definitions. */ + assets: Map>; + /** id -> definitions across all types. */ + assetsById: Map; + /** `$NAME` -> definitions. */ + defines: Map; +} + export interface IndexedFile { path: string; - stat: { mtimeMs: number; size: number } | null; + /** + * Multi-signal file stamp used to validate cached entries without + * re-reading content: size, last-write time, creation time and change + * time. Creation/change time catch tools that rewrite a file while + * preserving its mtime (e.g. temp-file + rename exporters) — important on + * removable drives where mtime resolution can be coarse (FAT32: 2 s). + */ + stat: + | { mtimeMs: number; size: number; birthtimeMs: number; ctimeMs: number } + | null; } export interface StreamInfo { @@ -65,10 +92,16 @@ export interface IndexerDiagnostic { export interface IndexStats { projectDir: string; sdkDir: string; + /** Last finished phase ("xml" or "art"). */ + phase: "xml" | "art"; + /** Whether the index is fully complete (art assets included). */ + complete: boolean; indexedFiles: number; parsedFiles: number; /** Art-asset documents indexed via shallow scan (no DOM tree). */ shallowScannedFiles: number; + /** Art files registered during the XML phase, scanned later in phase B. */ + deferredArtFiles: number; /** Shallow scans served from the persistent cache (unchanged files). */ shallowCacheHits: number; /** Parsed XML files served from the persistent records cache. */ @@ -77,10 +110,16 @@ export interface IndexStats { resolveCacheHits: number; /** Include/xi:include resolutions performed during this build. */ resolveCalls: number; + /** Existence checks answered by the directory snapshot (no statSync). */ + snapshotHits: number; + /** Existence checks outside the snapshot that fell back to statSync. */ + snapshotFallbacks: number; /** Time spent enumerating Include source candidates (ms). */ candidatesMs: number; /** Time spent walking the include graph (ms). */ walkMs: number; + /** Time spent shallow-scanning deferred art assets (ms). */ + artScanMs: number; assetCount: number; defineCount: number; manifestFiles: number; @@ -93,6 +132,19 @@ export interface IndexStats { export interface ModIndex { projectDir: string; sdkDir: string; + /** + * True when every indexing phase (including the art-asset shallow scan) + * has finished. Features use this to decide whether unresolved references + * are final errors or provisional "may be a false positive" diagnostics. + */ + complete: boolean; + /** Last finished phase: "xml" (XML + manifests) or "art" (final). */ + phase: "xml" | "art"; + /** + * True when files changed while this snapshot was being built, so some + * entries may be stale. A follow-up rebuild is scheduled by the workspace. + */ + stale?: boolean; /** type -> id -> definitions (project + sdk + manifest, deduplicated). */ assets: Map>; /** id -> definitions across all types. */ @@ -108,6 +160,12 @@ export interface ModIndex { /** Problems found while indexing (unresolved includes, cycles, ...). */ diagnostics: IndexerDiagnostic[]; stats: IndexStats; + /** + * Document-local overlay (when the index was obtained through the + * workspace's `getScope` / `getIndex` path). Optional so plain indexer + * snapshots remain overlay-free. + */ + local?: LocalOverlay; } export interface IndexOptions { @@ -160,4 +218,9 @@ export interface ParsedFile { */ records: IndexRecords | null; lineMap: LineMap | null; + /** + * True when the file is an art-asset XML that was only registered during + * the XML phase (its shallow scan is deferred to the art phase). + */ + deferredArt?: boolean; } diff --git a/src/indexer/xpointer.ts b/src/indexer/xpointer.ts new file mode 100644 index 0000000..9aff5fb --- /dev/null +++ b/src/indexer/xpointer.ts @@ -0,0 +1,24 @@ +import type { XmlDocument, XmlElement } from "../language/xmlParser"; + +/** Lowercases nothing; returns the part after the last ":" in a tag name. */ +export function localName(tag: string): string { + const idx = tag.lastIndexOf(":"); + return idx >= 0 ? tag.slice(idx + 1) : tag; +} + +/** + * Resolves the xpointer subset used by real RA3 mods: + * xmlns(n=uri:ea.com:eala:asset) xpointer(/n:ElementName/child::*) + * + * Returns the container element whose children are selected by the xpointer, + * or null when the form is unsupported / the container is missing. + */ +export function findXPointerContainer( + doc: XmlDocument, + xpointer: string, +): XmlElement | null { + const m = /xpointer\(\/\w+:(\w+)\/child::\*\)/.exec(xpointer); + if (!m) return null; + const name = m[1]; + return doc.elements.find((el) => localName(el.name) === name) ?? null; +} diff --git a/src/workspace.ts b/src/workspace.ts index 7930791..6e0e675 100644 --- a/src/workspace.ts +++ b/src/workspace.ts @@ -1,10 +1,35 @@ import * as vscode from "vscode"; import { existsSync } from "node:fs"; -import { join, dirname } from "node:path"; -import { CachedDirectoryWalker } from "./indexer/fileScanner"; +import { readFile, stat } from "node:fs/promises"; +import { join, dirname, resolve } from "node:path"; +import { + CachedDirectoryWalker, + isContentRelevantPath, + isWatcherNoisePath, +} from "./indexer/fileScanner"; import { ModIndexer } from "./indexer/indexer"; -import { DocumentCache, IncludeResolveCache, IndexRecordsCache } from "./indexer/caches"; -import type { ModIndex } from "./indexer/types"; +import { + DocumentCache, + IncludeResolveCache, + IndexRecordsCache, + InvalidationsEpoch, + normKey, +} from "./indexer/caches"; +import { + DiskRecordsCache, + diskCacheKey, + type DiskCacheIdentity, + type DiskCacheLoadStats, +} from "./indexer/diskCache"; +import { buildSearchPaths, type SearchPaths } from "./indexer/includeResolver"; +import { extractIndexRecords } from "./indexer/records"; +import { LineMap, parseXml, stripBom } from "./language/xmlParser"; +import { + buildDocumentScope, + withLocalOverlay, + type DocumentScope, +} from "./indexer/localScope"; +import type { ModIndex, ParsedFile } from "./indexer/types"; import { readSettings, type ExtensionSettings } from "./settings"; const REBUILD_DEBOUNCE_MS = 1500; @@ -22,6 +47,39 @@ export class ModWorkspace { private documentCache = new DocumentCache(); private recordsCache = new IndexRecordsCache(); private resolveCache = new IncludeResolveCache(); + /** + * Monotonic invalidation counter. A build captures the epoch when it + * starts; snapshots published after any invalidation are marked stale so + * features can tell users "this index may be slightly out of date" while + * the follow-up rebuild converges. + */ + private epoch = new InvalidationsEpoch(); + /** On-disk records cache (cold-start acceleration). */ + private diskCachePath: string | null = null; + private diskCache: DiskRecordsCache | null = null; + private diskCacheStats: DiskCacheLoadStats = { + fileExists: false, + keyMatched: false, + loaded: 0, + validated: 0, + dropped: 0, + }; + private diskSaved = false; + private saving: Promise | null = null; + /** Document-local scopes (parse + expanded tree + overlay), per open doc. */ + private localScopes = new Map< + string, + { version: number; indexEpoch: number; scope: DocumentScope } + >(); + private localScopeBuilds = new Map>(); + private indexEpochValue = 0; + /** Diagnostics: how many builds ran and what triggered the last one. */ + private buildCountValue = 0; + private lastBuildTrigger = "initial"; + private pendingTrigger: string | null = null; + private output: vscode.OutputChannel; + /** Called whenever a new index snapshot is published (phase or final). */ + onIndexUpdate?: () => void; private context: vscode.ExtensionContext; private watchers: vscode.FileSystemWatcher[] = []; private statusBar: vscode.StatusBarItem; @@ -32,6 +90,11 @@ export class ModWorkspace { constructor(context: vscode.ExtensionContext) { this.context = context; this.settings = readSettings(); + const storageUri = context.storageUri ?? context.globalStorageUri; + if (storageUri) { + this.diskCachePath = join(storageUri.fsPath, "index-records-v1.json.gz"); + } + this.output = vscode.window.createOutputChannel("RA3 Mod XML"); this.statusBar = vscode.window.createStatusBarItem( vscode.StatusBarAlignment.Left, 100, @@ -39,12 +102,32 @@ export class ModWorkspace { this.statusBar.name = "RA3 Mod XML"; this.statusBar.command = "ra3modxml.openIndexReport"; context.subscriptions.push(this.statusBar); + context.subscriptions.push( + vscode.workspace.onDidCloseTextDocument((document) => { + this.localScopes.delete(document.uri.toString()); + }), + ); } isRa3Workspace(): boolean { return this.projectRoot != null; } + /** True while a rebuild is running (before any snapshot is published). */ + get isBuilding(): boolean { + return this.building; + } + + /** Number of index builds performed in this session. */ + get buildCount(): number { + return this.buildCountValue; + } + + /** Why the last build started ("initial", "save", "watcher-*", ...). */ + get lastTrigger(): string { + return this.lastBuildTrigger; + } + detectProjectRoot(): string | null { const folders = vscode.workspace.workspaceFolders; if (!folders?.length) return null; @@ -64,7 +147,7 @@ export class ModWorkspace { this.startWatching(); this.statusBar.text = "$(sync~spin) RA3 XML: indexing…"; this.statusBar.show(); - await this.rebuild(); + await this.rebuild(false, "initial"); } /** @@ -74,6 +157,7 @@ export class ModWorkspace { */ invalidate(path: string): void { if (!path) return; + this.epoch.mark(); this.documentCache.invalidate(path); this.recordsCache.invalidate(path); } @@ -83,6 +167,7 @@ export class ModWorkspace { * (which encode file existence) are no longer trustworthy. */ invalidateExistence(): void { + this.epoch.mark(); this.resolveCache.clear(); } @@ -107,13 +192,32 @@ export class ModWorkspace { new vscode.RelativePattern(root, "**/*"), ); watcher.onDidCreate((uri) => { + if (isWatcherNoisePath(uri.fsPath)) return; + this.output.appendLine(`[watcher-create] ${uri.fsPath}`); this.invalidate(uri.fsPath); this.invalidateExistence(); + this.scheduleRebuild("watcher-create"); + }); + watcher.onDidChange((uri) => { + if (isWatcherNoisePath(uri.fsPath)) return; + // Content changes only matter for files that can change index + // records (XML-ish documents); textures/binary art changes do not. + if ( + !isContentRelevantPath(uri.fsPath) && + !this.isIndexedPath(uri.fsPath) + ) { + return; + } + this.output.appendLine(`[watcher-change] ${uri.fsPath}`); + this.invalidate(uri.fsPath); + this.scheduleRebuild("watcher-change"); }); - watcher.onDidChange((uri) => this.invalidate(uri.fsPath)); watcher.onDidDelete((uri) => { + if (isWatcherNoisePath(uri.fsPath)) return; + this.output.appendLine(`[watcher-delete] ${uri.fsPath}`); this.invalidate(uri.fsPath); this.invalidateExistence(); + this.scheduleRebuild("watcher-delete"); }); this.watchers.push(watcher); this.context.subscriptions.push(watcher); @@ -124,15 +228,22 @@ export class ModWorkspace { } } - scheduleRebuild(): void { + /** True when the path is part of the current index (any build state). */ + private isIndexedPath(fsPath: string): boolean { + if (this.indexer?.isIndexedFile(fsPath)) return true; + return this.index?.files.has(normKey(fsPath)) ?? false; + } + + scheduleRebuild(reason = "unknown"): void { if (!this.projectRoot) return; + this.pendingTrigger = reason; if (this.rebuildTimer) clearTimeout(this.rebuildTimer); this.rebuildTimer = setTimeout(() => { - void this.rebuild(); + void this.rebuild(false, this.pendingTrigger ?? reason); }, REBUILD_DEBOUNCE_MS); } - async rebuild(force = false): Promise { + async rebuild(force = false, trigger = "unknown"): Promise { if (!this.projectRoot) return; if (this.building) { this.dirty = true; @@ -140,8 +251,18 @@ export class ModWorkspace { } if (force) this.resolveCache.clear(); this.building = true; + this.buildCountValue++; + this.lastBuildTrigger = trigger; + this.output.appendLine( + `[build #${this.buildCountValue}] trigger=${trigger} force=${force} start=${new Date().toISOString()}`, + ); this.settings = readSettings(); + const epochAtStart = this.epoch.snapshot(); try { + // Cold start: seed the records cache from disk (stat-validated) so a + // fresh session does not re-read unchanged files (Corona: 2.6 GB of + // art assets) just because the in-memory caches are empty. + await this.seedRecordsFromDisk(); this.statusBar.text = "$(sync~spin) RA3 XML: indexing…"; const indexer = new ModIndexer({ projectDir: this.projectRoot, @@ -157,30 +278,320 @@ export class ModWorkspace { // verification (ra3modxml.reindex). trustUnchanged: !force, }); - const started = Date.now(); - this.index = await indexer.build(); this.indexer = indexer; - const secs = ((Date.now() - started) / 1000).toFixed(1); - const s = this.index.stats; - this.statusBar.text = `$(symbol-misc) RA3 XML: ${formatCount(s.assetCount)} assets`; - this.statusBar.tooltip = - `${s.projectDir}\n` + - `${s.indexedFiles} files indexed (${s.parsedFiles} parsed, ${s.shallowScannedFiles} art assets shallow-scanned, ${secs}s)\n` + - `${s.assetCount} assets (${s.manifestAssetCount} from ${s.manifestFiles} manifests)\n` + - `${s.defineCount} defines, ${s.streams} streams, ${s.sourceCandidates} include candidates`; + // The XML phase is published as soon as it is ready, so completion / + // navigation / diagnostics work while the art scan continues. + const finalIndex = await indexer.build((phaseIndex) => { + this.publishIndex(phaseIndex, epochAtStart); + }); + this.publishIndex(finalIndex, epochAtStart); + this.output.appendLine( + `[build #${this.buildCountValue}] done in ${(finalIndex.stats.elapsedMs / 1000).toFixed(1)}s (phase=${finalIndex.phase}, assets=${finalIndex.stats.assetCount}, stale=${finalIndex.stale === true}, walk=${(finalIndex.stats.walkMs / 1000).toFixed(1)}s, candidates=${(finalIndex.stats.candidatesMs / 1000).toFixed(1)}s, art=${(finalIndex.stats.artScanMs / 1000).toFixed(1)}s)`, + ); + this.saveRecordsToDisk(); } catch (err) { - this.index = null; - this.statusBar.text = "$(error) RA3 XML: indexing failed"; - this.statusBar.tooltip = err instanceof Error ? err.message : String(err); + if (this.index) { + // Keep the last good snapshot (marked stale) instead of disabling the + // extension entirely; a later rebuild can recover. + this.index.stale = true; + this.statusBar.text = "$(error) RA3 XML: indexing failed (stale index kept)"; + this.statusBar.tooltip = err instanceof Error ? err.message : String(err); + this.onIndexUpdate?.(); + } else { + this.statusBar.text = "$(error) RA3 XML: indexing failed"; + this.statusBar.tooltip = err instanceof Error ? err.message : String(err); + } } finally { this.building = false; if (this.dirty) { this.dirty = false; - void this.rebuild(); + void this.rebuild(false, `dirty-followup (${this.lastBuildTrigger})`); + } else if (this.index) { + // Build is fully over: refresh diagnostics with full local scopes + // (the snapshot published while `building` was still true only got + // cheap parse-only scopes). + this.onIndexUpdate?.(); } } } + private diskCacheIdentity(): DiskCacheIdentity | null { + if (!this.projectRoot) return null; + return { + projectDir: this.projectRoot, + sdkDir: this.settings.sdkPath, + indexSageXml: this.settings.indexSageXml, + additionalDataSearchPaths: this.settings.additionalDataSearchPaths, + builtmodsDirs: this.settings.builtmodsDirs, + }; + } + + /** Loads + stat-validates the disk cache into the records cache once. */ + private async seedRecordsFromDisk(): Promise { + if (this.recordsCache.size > 0) return; + const identity = this.diskCacheIdentity(); + if (!this.diskCachePath || !identity) return; + this.diskCache = new DiskRecordsCache(this.diskCachePath, identity); + this.statusBar.text = "$(sync~spin) RA3 XML: validating cache…"; + const { records, stats } = await this.diskCache.loadValidated(); + this.diskCacheStats = stats; + this.diskSaved = false; + for (const rec of records) { + this.recordsCache.set(rec.key, { + stat: rec.stat, + records: rec.records, + kind: rec.kind, + }); + } + } + + /** Persists the records cache after a successful build (best-effort). */ + private saveRecordsToDisk(): void { + if (!this.diskCache) return; + // Snapshot the entries now: the save runs in the background while the + // next rebuild may already be mutating the live cache. + const entries = [...this.recordsCache.entries()]; + const prev = this.saving ?? Promise.resolve(); + this.saving = prev + .then(async () => { + await this.diskCache!.save(entries); + this.diskSaved = true; + }) + .catch(() => { + // Disk persistence is best-effort; the in-memory cache still works. + }); + } + + /** + * Clears every cache (in-memory + disk + directory walker) and starts a + * full forced rebuild. Used by the `ra3modxml.clearCache` command. + */ + clearCaches(): void { + this.localScopes.clear(); + this.documentCache.clear(); + this.recordsCache.clear(); + this.resolveCache.clear(); + this.walker.clear(); + this.diskCacheStats = { + fileExists: false, + keyMatched: false, + loaded: 0, + validated: 0, + dropped: 0, + }; + this.diskSaved = false; + void this.diskCache?.clear(); + void this.rebuild(true, "clear-cache"); + } + + /** Human-readable cache status for the `ra3modxml.showCacheReport` command. */ + async cacheReport(): Promise { + const lines: string[] = ["RA3 Mod XML cache report"]; + lines.push(`Disk cache: ${this.diskCachePath ?? "not available"}`); + if (this.diskCachePath) { + const status = await this.diskCache?.status(); + lines.push( + ` file: ${status?.exists ? `${(status.sizeBytes / 1024).toFixed(1)} KB` : "missing"}`, + ); + const identity = this.diskCacheIdentity(); + lines.push(` identity key: ${identity ? diskCacheKey(identity) : "-"}`); + lines.push( + ` last load: file=${this.diskCacheStats.fileExists} keyMatched=${this.diskCacheStats.keyMatched} loaded=${this.diskCacheStats.loaded} validated=${this.diskCacheStats.validated} dropped=${this.diskCacheStats.dropped}`, + ); + lines.push(` saved after last build: ${this.diskSaved}`); + } + lines.push( + `In-memory: ${this.recordsCache.size} record entries · ${this.documentCache.size} documents (${this.documentCache.elements} elements) · ${this.resolveCache.size} include resolutions`, + ); + lines.push(`Builds: #${this.buildCount} (last trigger: ${this.lastBuildTrigger})`); + if (this.index) { + const s = this.index.stats; + lines.push( + `Last build: snapshotHits=${s.snapshotHits} snapshotFallbacks=${s.snapshotFallbacks} recordsCacheHits=${s.recordsCacheHits} shallowCacheHits=${s.shallowCacheHits}`, + ); + } + return lines.join("\n"); + } + + /** + * Publishes an index snapshot (intermediate phase or final). If any file + * was invalidated while the snapshot was being built, it is marked stale; + * the dirty/rebuild mechanism converges shortly after. + */ + private publishIndex(index: ModIndex, epochAtStart: number): void { + if (this.epoch.changedSince(epochAtStart)) index.stale = true; + this.index = index; + this.indexEpochValue++; + // The merged index attached to a document scope changes with every + // published snapshot, so cached scopes are rebuilt lazily on next use. + this.localScopes.clear(); + this.updateStatusBar(index); + this.onIndexUpdate?.(); + if (!index.complete) { + this.output.appendLine( + `[build #${this.buildCountValue}] phase A published in ${(index.stats.elapsedMs / 1000).toFixed(1)}s (${index.stats.assetCount} assets, ${index.stats.deferredArtFiles} art files pending)`, + ); + } + } + + private updateStatusBar(idx: ModIndex): void { + const s = idx.stats; + const stale = idx.stale ? " (stale)" : ""; + if (!idx.complete) { + this.statusBar.text = `$(sync~spin) RA3 XML: XML indexed, scanning art…${stale}`; + } else { + this.statusBar.text = `$(symbol-misc) RA3 XML: ${formatCount(s.assetCount)} assets${stale}`; + } + this.statusBar.tooltip = + `${s.projectDir}\n` + + `${s.indexedFiles} files indexed (${s.parsedFiles} parsed, ${s.shallowScannedFiles} art assets shallow-scanned, ${(s.elapsedMs / 1000).toFixed(1)}s)\n` + + `${s.assetCount} assets (${s.manifestAssetCount} from ${s.manifestFiles} manifests)\n` + + `${s.defineCount} defines, ${s.streams} streams, ${s.sourceCandidates} include candidates\n` + + `Phase: ${s.phase} · Complete: ${s.complete}${stale}`; + } + + /** + * Search paths derived from the current settings, usable even before the + * first index snapshot exists (include links / hover / diagnostics). + */ + searchPaths(): SearchPaths | null { + if (!this.projectRoot) return null; + return buildSearchPaths(this.settings.sdkPath, this.projectRoot); + } + + /** + * Returns the document scope for the current text: original parse, expanded + * logical tree, local overlay and overlay-aware merged index. Cached by + * URI + document version + global index epoch. + */ + async getScope(document: vscode.TextDocument): Promise { + const key = document.uri.toString(); + const cached = this.localScopes.get(key); + if ( + cached && + cached.version === document.version && + cached.indexEpoch === this.indexEpochValue + ) { + return cached.scope; + } + // While a rebuild is running, avoid competing with the indexer for disk + // I/O: serve a parse-only scope (current file + XSD context, no include + // chain / logical expansion). The published snapshot clears this cache, + // so the next provider call after the build gets the full local scope. + if (this.building) { + return this.buildCheapScope(document); + } + const pending = this.localScopeBuilds.get(key); + if (pending) return pending; + const versionAtStart = document.version; + const promise = this.buildScope(document) + .then((scope) => { + this.localScopes.set(key, { + version: versionAtStart, + indexEpoch: this.indexEpochValue, + scope, + }); + return scope; + }) + .finally(() => { + this.localScopeBuilds.delete(key); + }); + this.localScopeBuilds.set(key, promise); + return promise; + } + + /** + * Returns the global index with this document's local overlay attached, or + * a minimal local-only index while the global index is still building. + */ + async getIndex(document: vscode.TextDocument): Promise { + if (!this.isRa3Workspace()) return null; + return (await this.getScope(document)).merged; + } + + private async buildScope( + document: vscode.TextDocument, + ): Promise { + const projectRoot = this.projectRoot; + if (!projectRoot) throw new Error("RA3 workspace root is not available"); + const searchPaths = + this.searchPaths() ?? buildSearchPaths(this.settings.sdkPath, projectRoot); + const readRecords = async (path: string): Promise => + this.indexer ? this.indexer.readDocument(path) : this.fallbackRead(path); + const readDom = async (path: string): Promise => + this.indexer ? this.indexer.readDom(path) : this.fallbackRead(path); + const scope = await buildDocumentScope( + document.uri.fsPath, + document.getText(), + document.version, + { + projectDir: projectRoot, + sdkDir: this.settings.sdkPath, + searchPaths, + readRecords, + readDom, + }, + ); + scope.merged = withLocalOverlay( + this.index, + scope.overlay, + projectRoot, + this.settings.sdkPath, + ); + return scope; + } + + private async buildCheapScope( + document: vscode.TextDocument, + ): Promise { + const projectRoot = this.projectRoot; + if (!projectRoot) throw new Error("RA3 workspace root is not available"); + const searchPaths = + this.searchPaths() ?? buildSearchPaths(this.settings.sdkPath, projectRoot); + const scope = await buildDocumentScope( + document.uri.fsPath, + document.getText(), + document.version, + { + projectDir: projectRoot, + sdkDir: this.settings.sdkPath, + searchPaths, + readRecords: async () => null, + readDom: async () => null, + }, + ); + scope.merged = withLocalOverlay( + this.index, + scope.overlay, + projectRoot, + this.settings.sdkPath, + ); + return scope; + } + + /** + * Fallback used before the first ModIndexer exists (e.g. during initial + * activation): parses an XML file directly so the document-local scope can + * still follow small include chains. + */ + private async fallbackRead(path: string): Promise { + try { + const st = await stat(path); + if (st.size > 4 * 1024 * 1024) return null; + const text = stripBom(await readFile(path, "utf8")); + const lineMap = new LineMap(text); + const parse = parseXml(text); + return { + file: { path: resolve(path), stat: null }, + parse, + records: extractIndexRecords(parse, lineMap), + lineMap, + }; + } catch { + return null; + } + } + /** Parses the (possibly unsaved) in-memory text of the active document. */ async parseText(path: string, text: string) { const { parseXml, LineMap } = await import("./language/xmlParser"); @@ -196,6 +607,7 @@ export class ModWorkspace { dispose(): void { if (this.rebuildTimer) clearTimeout(this.rebuildTimer); this.statusBar.dispose(); + this.output.dispose(); } } diff --git a/test/caches.test.mjs b/test/caches.test.mjs index 9f7b85a..20f9f57 100644 --- a/test/caches.test.mjs +++ b/test/caches.test.mjs @@ -4,11 +4,15 @@ import { DocumentCache, IncludeResolveCache, IndexRecordsCache, + InvalidationsEpoch, } from "../out/indexer/caches.js"; +import { resolve } from "node:path"; + +const stat = { mtimeMs: 1, size: 1, birthtimeMs: 1, ctimeMs: 1 }; function parsed(path, elements) { return { - file: { path, stat: { mtimeMs: 1, size: 1 } }, + file: { path, stat }, parse: { root: { name: "r" }, elements: new Array(elements), errors: [] }, records: null, lineMap: null, @@ -47,7 +51,7 @@ test("DocumentCache invalidate frees budget", () => { test("IndexRecordsCache stores and invalidates entries", () => { const cache = new IndexRecordsCache(); const entry = { - stat: { mtimeMs: 1, size: 1 }, + stat, records: { assets: [], defines: [], includes: [], rootXiIncludes: [], nestedXiIncludes: [] }, kind: "full", }; @@ -57,6 +61,20 @@ test("IndexRecordsCache stores and invalidates entries", () => { assert.equal(cache.get("a.xml"), undefined); }); +test("IndexRecordsCache exposes entries for disk persistence", () => { + const cache = new IndexRecordsCache(); + const entry = { + stat, + records: { assets: [], defines: [], includes: [], rootXiIncludes: [], nestedXiIncludes: [] }, + kind: "full", + }; + cache.set("a.xml", entry); + const list = [...cache.entries()]; + assert.equal(list.length, 1); + assert.equal(list[0][0], resolve("a.xml").toLowerCase()); + assert.equal(list[0][1], entry); +}); + test("IncludeResolveCache stores sources and manifest lookups", () => { const cache = new IncludeResolveCache(); const key = "dir|DATA:static.xml"; @@ -71,3 +89,15 @@ test("IncludeResolveCache stores sources and manifest lookups", () => { assert.equal(cache.get(key), undefined); assert.equal(cache.getManifest("static.xml"), undefined); }); + +test("InvalidationsEpoch tracks changes since a snapshot", () => { + const epoch = new InvalidationsEpoch(); + const before = epoch.snapshot(); + assert.equal(epoch.changedSince(before), false); + epoch.mark(); + assert.equal(epoch.changedSince(before), true); + assert.equal(epoch.changedSince(epoch.snapshot()), false); + epoch.mark(); + epoch.mark(); + assert.equal(epoch.current, 3); +}); diff --git a/test/completion.test.mjs b/test/completion.test.mjs index fffd734..c49b086 100644 --- a/test/completion.test.mjs +++ b/test/completion.test.mjs @@ -76,6 +76,8 @@ require.cache["vscode-stub"] = { }; const { Ra3CompletionProvider } = require("../out/features/completion.js"); +const { parseXml, LineMap } = require("../out/language/xmlParser.js"); +const { expandDocument } = require("../out/indexer/logicalTree.js"); function makeDocument(text) { const lineStarts = [0]; @@ -98,9 +100,26 @@ function makeDocument(text) { }; } -// Enum completions do not consult the index, but provideCompletionItems only -// routes value contexts when the workspace index is present. -const provider = new Ra3CompletionProvider({ index: {} }); +async function makeScope(text, idx) { + const lineMap = new LineMap(text); + const parse = parseXml(text); + const expanded = await expandDocument("test.xml", parse, { + resolve: () => null, + readDom: async () => null, + }); + return { expanded, merged: idx, overlay: {} }; +} + +// Enum completions do not consult the index, and value contexts now work +// even before the workspace index exists. +const makeProvider = (idx) => + new Ra3CompletionProvider({ + index: idx, + isRa3Workspace: () => true, + getScope: async (document) => makeScope(document.getText(), idx), + }); +const provider = makeProvider({}); +const providerNoIndex = makeProvider(null); const token = { isCancellationRequested: false }; test("Surfaces enum completion works with an unclosed quote", async () => { @@ -155,3 +174,73 @@ test("empty unterminated value offers all enum values", async () => { assert.ok(labels.includes("WATER")); assert.ok(labels.includes("CRUSHABLE_WALL")); }); + +test("enum completions work without an index", async () => { + const text = + `\n i.label); + assert.ok(labels.includes("GROUND"), "enum value offered without an index"); + assert.ok(!labels.includes("WATER")); +}); + +test("element and attribute name completions work without an index", async () => { + const text = `\n i.label); + assert.ok(labels.includes("id")); + assert.ok(labels.includes("Surfaces")); +}); + +test("content (child element) completions work without an index", async () => { + const text = + `\n \n \n \n`; + const pos = new Position(2, 4); + + const items = await providerNoIndex.provideCompletionItems( + makeDocument(text), + pos, + token, + ); + const labels = items.map((i) => i.label); + assert.ok(labels.length > 0, "child elements offered without an index"); +}); + +test("Poid attributes offer ids from the enclosing GameObject's local scope", async () => { + const text = + `\n` + + ` \n` + + ` \n` + + ` \n` + + ` \n` + + ` \n` + + ` \n` + + ` \n` + + ` \n` + + ``; + const line6 = text.split("\n")[6]; + const pos = new Position(6, line6.indexOf("ModuleTag_D") + "ModuleTag_D".length); + + const items = await provider.provideCompletionItems(makeDocument(text), pos, token); + const labels = items.map((i) => i.label); + assert.ok(labels.includes("ModuleTag_Draw")); + assert.ok(items.every((i) => i.kind === CompletionItemKind.Value)); + assert.ok( + items.every((i) => i.detail === "local module"), + "Poid completions are labelled as local-scope ids", + ); +}); diff --git a/test/diskCache.test.mjs b/test/diskCache.test.mjs new file mode 100644 index 0000000..84c5a64 --- /dev/null +++ b/test/diskCache.test.mjs @@ -0,0 +1,132 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { join } from "node:path"; +import fs from "node:fs"; +import os from "node:os"; +import { DiskRecordsCache, diskCacheKey } from "../out/indexer/diskCache.js"; + +const identity = { + projectDir: "C:/proj", + sdkDir: "C:/sdk", + indexSageXml: true, + additionalDataSearchPaths: [], + builtmodsDirs: ["C:/sdk/builtmods"], +}; + +const sampleRecords = { + assets: [{ type: "GameObject", id: "TankA", line: 3 }], + defines: [{ name: "HP", value: "100", line: 2 }], + includes: [{ type: "all", source: "Units.xml", line: 4 }], + rootXiIncludes: [], + nestedXiIncludes: [], +}; + +function stampOf(file) { + const s = fs.statSync(file); + return { + mtimeMs: s.mtimeMs, + size: s.size, + birthtimeMs: s.birthtimeMs, + ctimeMs: s.ctimeMs, + }; +} + +function makeTmp(t) { + const tmp = fs.mkdtempSync(join(os.tmpdir(), "ra3-diskcache-")); + t.after(() => fs.rmSync(tmp, { recursive: true, force: true })); + return tmp; +} + +test("disk cache roundtrip keeps records and leaves no temp file", async (t) => { + const tmp = makeTmp(t); + const file = join(tmp, "a.xml"); + fs.writeFileSync(file, "0123456789"); + const filePath = join(tmp, "index-records.json.gz"); + const cache = new DiskRecordsCache(filePath, identity); + + await cache.save([ + [ + file.toLowerCase(), + { stat: stampOf(file), records: sampleRecords, kind: "full" }, + ], + ]); + assert.equal(fs.existsSync(`${filePath}.tmp`), false, "atomic write leaves no temp"); + + const { records, stats } = await cache.loadValidated(); + assert.equal(stats.fileExists, true); + assert.equal(stats.keyMatched, true); + assert.equal(stats.loaded, 1); + assert.equal(stats.validated, 1); + assert.equal(stats.dropped, 0); + assert.equal(records.length, 1); + assert.deepEqual(records[0].records, sampleRecords); +}); + +test("stat mismatch drops the cached entry", async (t) => { + const tmp = makeTmp(t); + const file = join(tmp, "a.xml"); + fs.writeFileSync(file, "0123456789"); + const filePath = join(tmp, "index-records.json.gz"); + const cache = new DiskRecordsCache(filePath, identity); + await cache.save([ + [file.toLowerCase(), { stat: stampOf(file), records: sampleRecords, kind: "full" }], + ]); + + const past = new Date(Date.now() - 60000); + fs.utimesSync(file, past, past); + const { records, stats } = await cache.loadValidated(); + assert.equal(stats.validated, 0); + assert.equal(stats.dropped, 1); + assert.equal(records.length, 0); +}); + +test("identity mismatch ignores the cache", async (t) => { + const tmp = makeTmp(t); + const file = join(tmp, "a.xml"); + fs.writeFileSync(file, "0123456789"); + const filePath = join(tmp, "index-records.json.gz"); + const cache = new DiskRecordsCache(filePath, identity); + await cache.save([ + [file.toLowerCase(), { stat: stampOf(file), records: sampleRecords, kind: "full" }], + ]); + + const other = new DiskRecordsCache(filePath, { + ...identity, + sdkDir: "D:/other-sdk", + }); + const { records, stats } = await other.loadValidated(); + assert.equal(stats.fileExists, true); + assert.equal(stats.keyMatched, false); + assert.equal(records.length, 0); +}); + +test("corrupt cache file yields an empty result", async (t) => { + const tmp = makeTmp(t); + const filePath = join(tmp, "index-records.json.gz"); + fs.writeFileSync(filePath, "this is not gzip json"); + const cache = new DiskRecordsCache(filePath, identity); + const { records, stats } = await cache.loadValidated(); + assert.equal(stats.fileExists, true); + assert.equal(records.length, 0); +}); + +test("clear removes the cache file", async (t) => { + const tmp = makeTmp(t); + const file = join(tmp, "a.xml"); + fs.writeFileSync(file, "0123456789"); + const filePath = join(tmp, "index-records.json.gz"); + const cache = new DiskRecordsCache(filePath, identity); + await cache.save([ + [file.toLowerCase(), { stat: stampOf(file), records: sampleRecords, kind: "full" }], + ]); + assert.ok(fs.existsSync(filePath)); + await cache.clear(); + assert.equal(fs.existsSync(filePath), false); +}); + +test("diskCacheKey differs when the identity changes", () => { + const a = diskCacheKey(identity); + const b = diskCacheKey({ ...identity, indexSageXml: false }); + assert.notEqual(a, b); + assert.equal(a, diskCacheKey(identity)); +}); diff --git a/test/existence.test.mjs b/test/existence.test.mjs new file mode 100644 index 0000000..7e2c452 --- /dev/null +++ b/test/existence.test.mjs @@ -0,0 +1,102 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { join, parse, resolve } from "node:path"; +import fs from "node:fs"; +import os from "node:os"; +import { resolveSource } from "../out/indexer/includeResolver.js"; +import { + ExistenceSnapshot, + buildExistenceSnapshot, + isDriveRoot, +} from "../out/indexer/existence.js"; + +function makeTmp(t) { + const tmp = fs.mkdtempSync(join(os.tmpdir(), "ra3-existence-")); + t.after(() => fs.rmSync(tmp, { recursive: true, force: true })); + return tmp; +} + +test("isDriveRoot detects filesystem roots", () => { + assert.equal(isDriveRoot(parse(process.cwd()).root), true); + assert.equal(isDriveRoot(process.cwd()), false); +}); + +test("ExistenceSnapshot answers covered paths and falls back outside roots", (t) => { + const tmp = makeTmp(t); + const dataDir = join(tmp, "data"); + fs.mkdirSync(dataDir); + const existing = join(dataDir, "Units.xml"); + fs.writeFileSync(existing, ""); + + const snap = new ExistenceSnapshot([dataDir]); + assert.equal(snap.has(existing), true); + assert.equal(snap.has(join(dataDir, "Missing.xml")), false); + assert.equal(snap.has(join(tmp, "outside.xml")), null, "outside roots is unknown"); + assert.ok(snap.hits >= 2, "covered lookups counted as hits"); + assert.equal(snap.fallbacks, 1); + + if (process.platform === "win32") { + assert.equal( + snap.has(existing.toUpperCase()), + true, + "lookup is case-insensitive on Windows", + ); + } +}); + +test("buildExistenceSnapshot covers bounded search bases lazily", async (t) => { + const tmp = makeTmp(t); + const dataDir = join(tmp, "data"); + const artDir = join(tmp, "art"); + fs.mkdirSync(join(dataDir, "sub"), { recursive: true }); + fs.mkdirSync(artDir); + fs.writeFileSync(join(dataDir, "Units.xml"), ""); + fs.writeFileSync(join(dataDir, "sub", "Nested.xml"), ""); + fs.writeFileSync(join(artDir, "Tank.w3x"), ""); + + const snap = buildExistenceSnapshot({ + DATA: [dataDir], + ART: [artDir], + AUDIO: [], + }); + assert.equal(snap.has(join(dataDir, "Units.xml")), true); + assert.equal(snap.has(join(dataDir, "sub", "Nested.xml")), true); + assert.equal(snap.has(join(artDir, "Tank.w3x")), true); + assert.equal(snap.has(join(dataDir, "Missing.xml")), false); +}); + +test("resolveSource uses the snapshot and falls back to statSync outside it", async (t) => { + const tmp = makeTmp(t); + const dataDir = join(tmp, "data"); + const outsideDir = join(tmp, "outside"); + fs.mkdirSync(dataDir); + fs.mkdirSync(outsideDir); + const units = join(dataDir, "Units.xml"); + const outside = join(outsideDir, "Extra.xml"); + fs.writeFileSync(units, ""); + fs.writeFileSync(outside, ""); + + const searchPaths = { DATA: [dataDir], ART: [], AUDIO: [] }; + const snap = buildExistenceSnapshot(searchPaths); + + const found = resolveSource("DATA:Units.xml", null, searchPaths, snap); + assert.equal(found.path, units); + assert.ok(snap.hits > 0, "covered lookup served by the snapshot"); + + const missing = resolveSource("DATA:Missing.xml", null, searchPaths, snap); + assert.equal(missing.path, null); + + const emptySearchPaths = { DATA: [], ART: [], AUDIO: [] }; + const outsideResolved = resolveSource( + outside, + outsideDir, + emptySearchPaths, + snap, + ); + assert.equal( + resolve(outsideResolved.path ?? ""), + resolve(outside), + "uncovered path falls back to statSync", + ); + assert.ok(snap.fallbacks > 0, "fallback counted"); +}); diff --git a/test/fileScanner.test.mjs b/test/fileScanner.test.mjs new file mode 100644 index 0000000..53a0897 --- /dev/null +++ b/test/fileScanner.test.mjs @@ -0,0 +1,72 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { join } from "node:path"; +import { + isContentRelevantPath, + isWatcherNoisePath, +} from "../out/indexer/fileScanner.js"; + +test("isWatcherNoisePath filters .git internals", () => { + assert.equal( + isWatcherNoisePath(join("C:/proj", ".git", "index")), + true, + ".git files are noise", + ); + assert.equal( + isWatcherNoisePath(join("C:/proj", "data", ".git", "FETCH_HEAD")), + true, + "nested .git directories are noise", + ); + assert.equal( + isWatcherNoisePath(join("C:/proj", "Data", "Mod.xml")), + false, + "project XML is not noise", + ); + assert.equal( + isWatcherNoisePath(join("C:/proj", ".gitignore")), + false, + ".gitignore is a real project file", + ); + assert.equal( + isWatcherNoisePath(join("C:/proj", "Data", "UnitCrate.xml.git")), + true, + "editor temp files ending in .git are noise", + ); + assert.equal( + isWatcherNoisePath(join("C:/proj", "Data", "UnitCrate.xml.tmp")), + true, + ".tmp files are noise", + ); + assert.equal( + isWatcherNoisePath(join("C:/proj", "Data", "Mod.xml~")), + true, + "backup files ending in ~ are noise", + ); + assert.equal( + isWatcherNoisePath(join("C:/proj", "Data", ".#Mod.xml")), + true, + "lock files starting with .# are noise", + ); +}); + +test("isContentRelevantPath only reacts to XML-ish content", () => { + assert.equal(isContentRelevantPath("a.xml"), true); + assert.equal(isContentRelevantPath("a.w3x"), true); + assert.equal( + isContentRelevantPath("a.manifestxml"), + false, + "there is no .manifestxml source format", + ); + assert.equal( + isContentRelevantPath("a.w3d"), + false, + ".w3d is binary art, not text XML", + ); + assert.equal(isContentRelevantPath("a.dds"), false); + assert.equal(isContentRelevantPath("a.xml.git"), false); + assert.equal( + isContentRelevantPath("a.lua"), + false, + "lua is not indexed yet", + ); +}); diff --git a/test/fixtures/minimod/Data/Includes/HeadlightModules.xml b/test/fixtures/minimod/Data/Includes/HeadlightModules.xml new file mode 100644 index 0000000..795de37 --- /dev/null +++ b/test/fixtures/minimod/Data/Includes/HeadlightModules.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/test/fixtures/minimod/Data/Includes/Models/Tank_FP.w3d b/test/fixtures/minimod/Data/Includes/Models/Tank_FP.dat similarity index 100% rename from test/fixtures/minimod/Data/Includes/Models/Tank_FP.w3d rename to test/fixtures/minimod/Data/Includes/Models/Tank_FP.dat diff --git a/test/fixtures/minimod/Data/Includes/StandaloneBase.xml b/test/fixtures/minimod/Data/Includes/StandaloneBase.xml new file mode 100644 index 0000000..e4a8dc6 --- /dev/null +++ b/test/fixtures/minimod/Data/Includes/StandaloneBase.xml @@ -0,0 +1,4 @@ + + + + diff --git a/test/fixtures/minimod/Data/Includes/VehicleArt.xml b/test/fixtures/minimod/Data/Includes/VehicleArt.xml index f643e1a..f0dbebd 100644 --- a/test/fixtures/minimod/Data/Includes/VehicleArt.xml +++ b/test/fixtures/minimod/Data/Includes/VehicleArt.xml @@ -2,7 +2,7 @@ - + diff --git a/test/fixtures/minimod/Data/Standalone.xml b/test/fixtures/minimod/Data/Standalone.xml new file mode 100644 index 0000000..6df37ba --- /dev/null +++ b/test/fixtures/minimod/Data/Standalone.xml @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + diff --git a/test/indexer.test.mjs b/test/indexer.test.mjs index 9cd0933..d254222 100644 --- a/test/indexer.test.mjs +++ b/test/indexer.test.mjs @@ -79,7 +79,7 @@ test("provides include source candidates", async () => { assert.ok(xml.some((c) => c.source === "DATA:static.xml")); }); -test("indexes art-asset XML (.w3x / sniffed .w3d) via shallow scan and skips binary", async () => { +test("indexes art-asset XML (.w3x / sniffed unknown extension) via shallow scan and skips binary", async () => { const idx = await buildIndex(); // The .w3x hub chain: Mod.xml -> VehicleArt.xml -> Models/Tank_SKN.w3x. @@ -96,7 +96,7 @@ test("indexes art-asset XML (.w3x / sniffed .w3d) via shallow scan and skips bin // Unknown extension with XML content is sniffed and indexed. assert.ok( idx.assetsById.get("tank_fp")?.some((d) => d.type === "W3DMesh"), - "unknown-extension XML (.w3d) sniffed and indexed", + "unknown-extension XML (.dat) sniffed and indexed", ); // Binary content is registered as a file but never parsed. @@ -133,6 +133,107 @@ test("w3x files appear in Include source completion candidates", async () => { ); }); +test("build publishes an immutable XML phase before art scanning", async () => { + let phaseA; + const indexer = new ModIndexer({ + projectDir: project, + sdkDir: sdk, + builtmodsDirs: [join(sdk, "builtmods")], + indexSageXml: true, + additionalDataSearchPaths: [], + walker: new CachedDirectoryWalker(), + }); + const idx = await indexer.build((p) => { + phaseA = p; + }); + + assert.ok(phaseA, "phase-A snapshot published"); + assert.equal(phaseA.complete, false); + assert.equal(phaseA.phase, "xml"); + assert.ok(phaseA.assetsById.has("testtank"), "XML assets available in phase A"); + assert.ok( + phaseA.assetsById.has("vanillatank"), + "manifest assets available in phase A", + ); + assert.equal( + phaseA.assetsById.has("tank_skn"), + false, + "art assets deferred in phase A", + ); + assert.ok(phaseA.stats.deferredArtFiles >= 2, "deferred art queue recorded"); + assert.equal(phaseA.stats.shallowScannedFiles, 0, "no art scanned during phase A"); + + assert.equal(idx.complete, true); + assert.equal(idx.phase, "art"); + assert.ok(idx.assetsById.has("tank_skn"), "art assets present in the final index"); + assert.equal( + phaseA.assetsById.has("tank_skn"), + false, + "phase-A snapshot is immutable (phase B did not mutate it)", + ); + assert.equal(typeof idx.stats.artScanMs, "number"); + assert.equal( + indexer.isIndexedFile(join(project, "Data", "Mod.xml")), + true, + "indexed files are recognized", + ); + assert.equal( + indexer.isIndexedFile(join(project, "Data", "NotIndexed.xml")), + false, + "unrelated files are not recognized", + ); +}); + +test("stat validation re-reads a file whose mtime changed", async (t) => { + const tmp = fs.mkdtempSync(join(os.tmpdir(), "ra3modxml-mtime-")); + t.after(() => fs.rmSync(tmp, { recursive: true, force: true })); + const projectDir = join(tmp, "project"); + fs.mkdirSync(join(projectDir, "Data"), { recursive: true }); + const modPath = join(projectDir, "Data", "Mod.xml"); + fs.writeFileSync( + modPath, + `\n\n \n\n`, + "utf8", + ); + + const documentCache = new DocumentCache(); + const recordsCache = new IndexRecordsCache(); + const resolveCache = new IncludeResolveCache(); + const make = () => + new ModIndexer({ + projectDir, + sdkDir: sdk, + builtmodsDirs: [join(sdk, "builtmods")], + indexSageXml: false, + additionalDataSearchPaths: [], + walker: new CachedDirectoryWalker(), + documentCache, + recordsCache, + resolveCache, + trustUnchanged: false, + }); + + const first = await make().build(); + assert.ok(first.assetsById.has("tanka")); + assert.equal(first.stats.recordsCacheHits, 0); + + const past = new Date(Date.now() - 60000); + fs.utimesSync(modPath, past, past); + const second = await make().build(); + assert.equal( + second.stats.recordsCacheHits, + 0, + "mtime change invalidates the cached records", + ); + assert.ok(second.assetsById.has("tanka")); + + const third = await make().build(); + assert.ok( + third.stats.recordsCacheHits > 0, + "unchanged files are served from the records cache", + ); +}); + test("shallow scans and full parses are cached across rebuilds", async () => { const documentCache = new DocumentCache(); const recordsCache = new IndexRecordsCache(); @@ -218,6 +319,8 @@ test("index stats include candidate/walk phase timings", async () => { const idx = await buildIndex(); assert.equal(typeof idx.stats.candidatesMs, "number"); assert.equal(typeof idx.stats.walkMs, "number"); + assert.ok(idx.stats.snapshotHits > 0, "existence snapshot answered lookups"); + assert.equal(typeof idx.stats.snapshotFallbacks, "number"); }); test("w3x with a UTF-8 BOM is indexed with correct offsets", async (t) => { diff --git a/test/localScope.test.mjs b/test/localScope.test.mjs new file mode 100644 index 0000000..70dc478 --- /dev/null +++ b/test/localScope.test.mjs @@ -0,0 +1,210 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { fileURLToPath } from "node:url"; +import { dirname, join } from "node:path"; +import { parseXml, LineMap, stripBom } from "../out/language/xmlParser.js"; +import { extractIndexRecords } from "../out/indexer/records.js"; +import { buildSearchPaths } from "../out/indexer/includeResolver.js"; +import { + buildDocumentScope, + withLocalOverlay, +} from "../out/indexer/localScope.js"; +import { + findContainingGameObject, + findLocalId, + collectLocalIds, +} from "../out/indexer/logicalTree.js"; +import { resolveReferenceTargetsForType } from "../out/indexer/refs.js"; +import { resolveElementType } from "../out/language/typeContext.js"; + +const root = dirname(dirname(fileURLToPath(import.meta.url))); +const project = join(root, "test", "fixtures", "minimod"); +const sdk = join(root, "test", "fixtures", "fakesdk"); +const standalonePath = join(project, "Data", "Standalone.xml"); + +async function readParsed(path) { + const text = stripBom(await readFile(path, "utf8")); + const lineMap = new LineMap(text); + const parse = parseXml(text); + return { + file: { path, stat: null }, + parse, + records: extractIndexRecords(parse, lineMap), + lineMap, + }; +} + +async function makeScope() { + const searchPaths = buildSearchPaths(sdk, project); + const text = await readFile(standalonePath, "utf8"); + return buildDocumentScope(standalonePath, text, 1, { + projectDir: project, + sdkDir: sdk, + searchPaths, + readRecords: readParsed, + readDom: readParsed, + }); +} + +test("local overlay resolves refs for a file outside every global stream", async () => { + const scope = await makeScope(); + const merged = withLocalOverlay(null, scope.overlay, project, sdk); + + assert.ok(merged.local.assetsById.has("standalonetank")); + assert.ok(merged.local.assetsById.has("standalonebase")); + assert.ok(merged.local.defines.has("standalone_health")); + + const targets = resolveReferenceTargetsForType( + merged, + "GameObject", + "inheritFrom", + "StandaloneBase", + ); + assert.equal(targets.length, 1); + assert.match(targets[0].def.file, /StandaloneBase\.xml$/); + assert.equal(targets[0].def.origin, "project"); +}); + +test("logical xi:include expansion gives included modules their Draws context", async () => { + const scope = await makeScope(); + const included = scope.expanded.elements.find( + (e) => + e.name === "TruckDraw" && + e.attrs.some((a) => a.name === "id" && a.value === "ModuleTag_Headlight"), + ); + assert.ok(included, "xi:include target spliced into the logical tree"); + assert.match(included.sourceFile, /HeadlightModules\.xml$/i); + assert.equal( + resolveElementType(included), + "W3DTruckDrawModuleData", + "included module resolves through the logical Draws parent", + ); + + const update = scope.expanded.elements.find( + (e) => e.name === "ReconstituteStateSpecialAbility", + ); + assert.ok(update); + const gameObject = findContainingGameObject(update); + assert.equal(gameObject?.name, "GameObject"); + assert.equal( + findLocalId(gameObject, "ModuleTag_Headlight"), + included, + "Poid reference can reach a module spliced in through xi:include", + ); + + const localIds = collectLocalIds(gameObject).map((i) => i.id); + assert.ok(localIds.includes("ModuleTag_Draw")); + assert.ok(localIds.includes("ModuleTag_Headlight")); +}); + +test("local overlay wins over a global definition with the same id", async () => { + const scope = await makeScope(); + const global = { + assets: new Map(), + assetsById: new Map([ + [ + "standalonebase", + [ + { + type: "GameObject", + id: "StandaloneBase", + file: join(sdk, "SageXml", "VanillaBase.xml"), + line: 1, + origin: "sdk", + }, + ], + ], + ]), + defines: new Map(), + }; + const merged = withLocalOverlay(global, scope.overlay, project, sdk); + const targets = resolveReferenceTargetsForType( + merged, + "GameObject", + "inheritFrom", + "StandaloneBase", + ); + assert.equal(targets.length, 2); + assert.match(targets[0].def.file, /StandaloneBase\.xml$/); + assert.equal(targets[0].def.origin, "project"); + assert.match(targets[1].def.file, /VanillaBase\.xml$/); +}); + +test("logical expansion terminates on xi:include cycles", async (t) => { + const tmp = await mkdtemp(join(tmpdir(), "ra3-local-cycle-")); + t.after(() => rm(tmp, { recursive: true, force: true })); + const dataDir = join(tmp, "Data"); + const includesDir = join(dataDir, "Includes"); + await mkdir(includesDir, { recursive: true }); + const aPath = join(dataDir, "A.xml"); + const bPath = join(includesDir, "B.xml"); + await writeFile( + aPath, + ``, + "utf8", + ); + await writeFile( + bPath, + ``, + "utf8", + ); + const searchPaths = buildSearchPaths(sdk, tmp); + const text = await readFile(aPath, "utf8"); + const scope = await buildDocumentScope(aPath, text, 1, { + projectDir: tmp, + sdkDir: sdk, + searchPaths, + readRecords: readParsed, + readDom: readParsed, + }); + assert.ok( + scope.expanded.elements.some( + (e) => + e.name === "GameObject" && + e.attrs.some((a) => a.name === "id" && a.value === "A"), + ), + "entry document survives a cycle", + ); +}); + +test("the same xi:include target can expand under multiple parents", async (t) => { + const tmp = await mkdtemp(join(tmpdir(), "ra3-local-shared-")); + t.after(() => rm(tmp, { recursive: true, force: true })); + const dataDir = join(tmp, "Data"); + const includesDir = join(dataDir, "Includes"); + await mkdir(includesDir, { recursive: true }); + const mainPath = join(dataDir, "Main.xml"); + const fragmentPath = join(includesDir, "Fragment.xml"); + await writeFile( + fragmentPath, + ``, + "utf8", + ); + await writeFile( + mainPath, + `` + + `` + + `` + + ``, + "utf8", + ); + const searchPaths = buildSearchPaths(sdk, tmp); + const text = await readFile(mainPath, "utf8"); + const scope = await buildDocumentScope(mainPath, text, 1, { + projectDir: tmp, + sdkDir: sdk, + searchPaths, + readRecords: readParsed, + readDom: readParsed, + }); + const shared = scope.expanded.elements.filter( + (e) => + e.name === "TruckDraw" && + e.attrs.some((a) => a.name === "id" && a.value === "ModuleTag_Shared"), + ); + assert.equal(shared.length, 2, "same fragment expands once per parent"); +}); diff --git a/test/semanticTokens.test.mjs b/test/semanticTokens.test.mjs index 1dd1264..57bf1aa 100644 --- a/test/semanticTokens.test.mjs +++ b/test/semanticTokens.test.mjs @@ -130,13 +130,13 @@ test("fallback tokens cover names, attributes and values", () => { test("well-formed XML gets no semantic fallback tokens", async () => { const text = `\n \n`; - const provider = new Ra3SemanticTokensProvider(); + const provider = new Ra3SemanticTokensProvider({ isRa3Workspace: () => true }); const result = await provider.provideDocumentSemanticTokens(makeDocument(text), {}); assert.equal(result.data.length, 0); }); test("malformed XML gets semantic fallback tokens", async () => { - const provider = new Ra3SemanticTokensProvider(); + const provider = new Ra3SemanticTokensProvider({ isRa3Workspace: () => true }); const result = await provider.provideDocumentSemanticTokens( makeDocument(MALFORMED), {},