Add persistent index, incremental reparse, and .def language support #44

Merged

Christoph merged 0 commits from refs/pull/44/head into main

2026-09-24 05:54:11 +00:00

Christoph commented

2026-09-24 05:53:50 +00:00

(Migrated from git.cbsk-tech.de)

Summary of important behavior changes

Client
- The language client now watches **/*.{tcl,psc,def} (was {tcl,psc}) so .def file changes are observed by the server.
- The client passes the workspace storage folder to the server as indexCachePath (when available) so the server can persist its index cache between restarts.
Server / indexing
- Introduces a persistent IndexCache (load/save) and wires it into background indexing. The cache is loaded during initialized and saved at the end of background indexing.
- Adds an incremental reparse facility and per-document last-parse state (_last_parse) so the server can reparse only the edited parts of a document when possible instead of reparsing whole files.
- Keeps navigation definition identities cached and exposes navigation_state() to return both indexes and precomputed definitions so request handlers avoid recomputing definition sets repeatedly.
- Avoids unnecessary work in several places (for example, may_contain_classes gates class/method highlighting) so some flows skip work when the file contents or class information make it irrelevant.
.def files and completions
- Adds a tools/def_symbols parser to read BLOCK_TEMPLATE/ADDRESS/value candidates from .def files and exposes them to the server.
- The server refreshes def symbols on watched .def file changes and indexes them (server state def_symbols) for use in completions.
- Completion logic now supports offering quoted .def symbol completions and variable candidates for argument contexts where a .def name or a variable is allowed. A helper _quoted_item produces insert edits that insert quoted strings or replace already-typed quotes.
- Additional tooling support: file sourcing reads PSC DefinedEvents and integrates .def resolution; tcl command completion and tcloo completion are extended to make use of indexed symbols and class data when available.
Debug helper and VS Code configurations
- The Python debug helper (server/src/_debug_server.py) no longer loops retrying debugpy.connect. Instead it attempts a single connect in a background thread and bounds the wait with join(timeout). The helper prints a clear error and exits (os._exit(1)) when a stale adapter accepts connections but never attaches, or when no listener is present.
- Adds Linux-specific VS Code launch configurations and tasks to start the Extension Development Host with the inspector bound to 127.0.0.1:9333 and a variant that sets USE_DEBUGPY/NXPS_DEBUG_HOST/NXPS_DEBUG_PORT to make it easier to attach both the extension host and debugpy on Linux where js-debug probes localhost and ::1 in parallel.
Miscellaneous
- Commit updates package-lock.json version.
- Tests: tests were added/updated for .def completions, uplevel formatting, and for the debugpy attach behavior (tests changed/added in the diff). No test execution output is included in this change.

Testing

Tests added/changed: server tests for .def completions and uplevel formatting; debug-related tests updated. No test execution results were provided.

Recommended quick checks for reviewers

With a workspace open, start the extension and check extension storage is passed: inspect the server's initialization options to see indexCachePath (the client uses context.storageUri?.fsPath).
Add or edit a .def file and confirm the server refreshes def symbols (watcher triggered) and completion requests in affected TCL files return .def-derived candidates. Also verify quoted insertion behavior for name arguments.
Trigger background indexing and confirm the server loads the index cache (if present) and saves it after indexing (look for the log message mentioning background indexing completed and the load/save logic).
On Linux, use the provided task/launch combos to start the dev host with the inspector on 127.0.0.1:9333 and the Python debug helper on 127.0.0.1:5678 to exercise the new debug helper behavior.

Compatibility, migration, and reviewer notes

The index cache is optional and only used when a workspace storage folder is available: the client sends indexCachePath only when context.storageUri exists. Workspaces without storage will continue to operate without persistence.
The persistent cache is stored under the provided indexCachePath. Reviewers should consider whether the storage location and lifecycle are acceptable for the extension's persisted files.
The debug helper now exits with a non-zero condition when it believes a stale adapter is holding the port. This is intentional to avoid leaving background debugpy threads alive; verify any debug/test automation accounts for that behavior.
The change touches many internal parsing/indexing code paths (incremental parsing, navigation caches). The description above summarizes the end-state behavior rather than the commit sequence.

If you want, I can point to specific changed files for each area (e.g. server/src/lsp_server.py, server/src/lsp_tclserver.py, server/src/_debug_server.py, client/src/common/server.ts, .vscode/launch.json, .vscode/tasks.json, and tools/def_symbols.py) to help review particular implementations.

This change adds persistent workspace indexing, incremental reparsing of edited documents, and first-class support for .def files in the client and server. It also hardens the debugpy attach logic for the Python debug helper and adds Linux-focused VS Code launch/tasks to work around js-debug attach probing. Summary of important behavior changes - Client - The language client now watches `**/*.{tcl,psc,def}` (was `{tcl,psc}`) so .def file changes are observed by the server. - The client passes the workspace storage folder to the server as `indexCachePath` (when available) so the server can persist its index cache between restarts. - Server / indexing - Introduces a persistent `IndexCache` (load/save) and wires it into background indexing. The cache is loaded during `initialized` and saved at the end of background indexing. - Adds an incremental reparse facility and per-document last-parse state (`_last_parse`) so the server can reparse only the edited parts of a document when possible instead of reparsing whole files. - Keeps navigation definition identities cached and exposes navigation_state() to return both indexes and precomputed definitions so request handlers avoid recomputing definition sets repeatedly. - Avoids unnecessary work in several places (for example, `may_contain_classes` gates class/method highlighting) so some flows skip work when the file contents or class information make it irrelevant. - .def files and completions - Adds a `tools/def_symbols` parser to read BLOCK_TEMPLATE/ADDRESS/value candidates from .def files and exposes them to the server. - The server refreshes def symbols on watched .def file changes and indexes them (server state `def_symbols`) for use in completions. - Completion logic now supports offering quoted `.def` symbol completions and variable candidates for argument contexts where a .def name or a variable is allowed. A helper `_quoted_item` produces insert edits that insert quoted strings or replace already-typed quotes. - Additional tooling support: file sourcing reads PSC DefinedEvents and integrates .def resolution; tcl command completion and tcloo completion are extended to make use of indexed symbols and class data when available. - Debug helper and VS Code configurations - The Python debug helper (`server/src/_debug_server.py`) no longer loops retrying `debugpy.connect`. Instead it attempts a single connect in a background thread and bounds the wait with `join(timeout)`. The helper prints a clear error and exits (`os._exit(1)`) when a stale adapter accepts connections but never attaches, or when no listener is present. - Adds Linux-specific VS Code launch configurations and tasks to start the Extension Development Host with the inspector bound to `127.0.0.1:9333` and a variant that sets `USE_DEBUGPY`/`NXPS_DEBUG_HOST`/`NXPS_DEBUG_PORT` to make it easier to attach both the extension host and debugpy on Linux where js-debug probes `localhost` and `::1` in parallel. - Miscellaneous - Commit updates `package-lock.json` version. - Tests: tests were added/updated for `.def` completions, uplevel formatting, and for the debugpy attach behavior (tests changed/added in the diff). No test execution output is included in this change. Testing - Tests added/changed: server tests for `.def` completions and uplevel formatting; debug-related tests updated. No test execution results were provided. Recommended quick checks for reviewers 1. With a workspace open, start the extension and check extension storage is passed: inspect the server's initialization options to see `indexCachePath` (the client uses `context.storageUri?.fsPath`). 2. Add or edit a `.def` file and confirm the server refreshes def symbols (watcher triggered) and completion requests in affected TCL files return `.def`-derived candidates. Also verify quoted insertion behavior for name arguments. 3. Trigger background indexing and confirm the server loads the index cache (if present) and saves it after indexing (look for the log message mentioning background indexing completed and the load/save logic). 4. On Linux, use the provided task/launch combos to start the dev host with the inspector on `127.0.0.1:9333` and the Python debug helper on `127.0.0.1:5678` to exercise the new debug helper behavior. Compatibility, migration, and reviewer notes - The index cache is optional and only used when a workspace storage folder is available: the client sends `indexCachePath` only when `context.storageUri` exists. Workspaces without storage will continue to operate without persistence. - The persistent cache is stored under the provided `indexCachePath`. Reviewers should consider whether the storage location and lifecycle are acceptable for the extension's persisted files. - The debug helper now exits with a non-zero condition when it believes a stale adapter is holding the port. This is intentional to avoid leaving background debugpy threads alive; verify any debug/test automation accounts for that behavior. - The change touches many internal parsing/indexing code paths (incremental parsing, navigation caches). The description above summarizes the end-state behavior rather than the commit sequence. If you want, I can point to specific changed files for each area (e.g. `server/src/lsp_server.py`, `server/src/lsp_tclserver.py`, `server/src/_debug_server.py`, `client/src/common/server.ts`, `.vscode/launch.json`, `.vscode/tasks.json`, and `tools/def_symbols.py`) to help review particular implementations.