Merge PR #9: PathFilter, file_path scoping, and progress reporting#10
Merge PR #9: PathFilter, file_path scoping, and progress reporting#10Codeturion wants to merge 26 commits intomasterfrom
Conversation
…r into server indexing
…embers, clean up docstring
rglob() cannot skip directories mid-walk — it traverses the entire tree (including all git worktrees) before the path_filter check runs. With 16 worktrees the indexer was doing ~17x the necessary work, causing 1000+s startup times instead of ~2 minutes. Switch all six walk sites (base.py, server.py x2, typescript.py, python_parser.py, go.py, java.py) to os.walk() with in-place dirs[:] pruning. Excluded directories (.worktrees, submodules, _SKIP_DIRS) are now dropped before descent so os.walk never enters them.
…orktree-support feat: file filtering, worktree exclusions, and query-time file_path scoping
Also forward on_progress in TypeScriptParser's parse_directory override. The callback is invoked after each successful parse_file call, passing the Path of the parsed file; skipped files and parse errors do not trigger it.
…ertions - Remove `print(summary, file=sys.stderr)` from `main()` after calling `_index_full()`, since `_index_full` already prints the done line itself. - Strengthen `test_index_full_emits_progress_to_stderr` to also assert that the scanning line and 0% baseline indexing line appear in stderr.
…le level Move on_progress callback to a finally block in all five parsers so it fires whether parse_file succeeds or raises. Previously, unparseable files were counted by _count_files but never triggered on_progress, causing the startup progress display to stall below 100% on repos with problematic files. Also move `import sys` from inside except blocks to module-level imports in typescript.py, python_parser.py, go.py, and java.py.
- Add _DEFAULT_EXCLUDED_DIRS to PathFilter (node_modules, .git, dist, build, vendor, .nx, .yarn, etc.) so vendored dirs are always skipped - Rewrite detect_languages to use os.walk with PathFilter instead of rglob, which crawled into node_modules and hung indefinitely - Consolidate duplicate parse_directory from all 4 parsers into BaseParser with str ops instead of pathlib (saves ~3s on 30K files) - Rewrite _count_braces_and_parens: regex strip + str.count with fast path for lines without strings/comments (saves ~4s on 30K files) - Replace Path.read_text with open()+read(), path.relative_to with os.path.relpath, and _file_to_module with pure string ops - Remove per-parser _SKIP_DIRS (now centralized in PathFilter) - Remove __tests__/__mocks__/test/spec file exclusions from TS parser so test files are indexed for coding reference - Add skip_suffixes, skip_files, _should_skip_dir hooks to BaseParser for per-parser file filtering without duplicating the walk loop Benchmark on 34K-file TS monorepo: 22s -> 15s (32% faster)
Path('~/work/cloud').is_dir() returns False because Python's pathlib
does not expand ~ — causing the server to skip indexing entirely and
sit idle in the MCP listen loop without emitting the done: line.
These are internal planning artifacts and fork install instructions that don't belong in the upstream PR.
- Remove CppParser.parse_directory override, use BaseParser's os.walk with _should_skip_dir for C++-specific dirs (Debug, Release, x64, x86, cmake-build-*) - Remove _SKIP_DIRS from cpp.py (now centralized in PathFilter) - Apply open()+read() and os.path.relpath to cpp.py for consistency - Restore .test.ts/.spec.ts skip suffixes in TypeScript parser
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive path filtering system and significant performance optimizations for the indexing process. Key changes include the addition of a PathFilter class to handle default and user-defined exclusions, the integration of progress reporting during full indexing, and the refactoring of language parsers to use os.walk for faster directory traversal. The database layer was also updated to support scoped searches via a new file_path parameter. Feedback focuses on further performance improvements, such as consolidating multiple directory walks, reducing pathlib overhead in hot loops, and addressing code duplication in SQL construction. Additionally, the file counting logic for progress reporting needs to be synchronized with parser-specific exclusion rules to ensure accuracy.
| if file_path: | ||
| if file_path.endswith("/"): | ||
| conditions.append("r.file_path LIKE ?") | ||
| params.append(file_path + "%") | ||
| else: | ||
| conditions.append("(r.file_path = ? OR r.file_path LIKE ?)") | ||
| params.extend([file_path, file_path + "/%"]) |
There was a problem hiding this comment.
The logic for constructing the file_path SQL condition is duplicated here and in get_class_members (lines 212-218). Consider extracting this into a shared utility function to ensure consistency and reduce maintenance overhead.
| d for d in dirs | ||
| if not path_filter.is_dir_excluded_name(d) | ||
| and not self._should_skip_dir(d) | ||
| and not path_filter.is_dir_excluded(Path(os.path.join(root, d))) |
There was a problem hiding this comment.
Creating a Path object for every directory in the walk (Path(os.path.join(root, d))) introduces overhead that the class documentation explicitly aims to avoid. Furthermore, path_filter.is_dir_excluded performs I/O (checking for .git files) on every directory, which is costly. Consider using string-based checks and caching the results of the .git file detection to avoid redundant disk access.
| if not self._globs: | ||
| return False | ||
| try: | ||
| rel = str(path.relative_to(self._root)).replace("\\", "/") |
There was a problem hiding this comment.
Using path.relative_to(self._root) inside a tight loop for every file is inefficient. Since the walk already provides the current root and filename, string-based path joining and slicing would be significantly faster and more consistent with the performance goals stated in the parser base class.
- Extract _walk_files from parse_directory so _count_files can reuse the same filter logic (skip_suffixes, skip_files, _should_skip_dir) - Collect mtimes in on_progress callback instead of a separate walk - _index_full now does 2 walks (count + parse) instead of 3
2 participants
Summary
Merges #9 from @michael-howell-island with conflict resolution against #8 (C++ parser).
namespaceandfile_pathparams now coexist inget_class_members()BaseParser.parse_directory(consolidated os.walk with PathFilter).test.ts/.spec.tsskip suffixes in TypeScript parserWhat's included from #9
.codesurfaceignore,--excludeCLI globs, worktree/submodule skippingsearch,get_signature,get_classtools accept optionalfile_pathfilterscanning N files...,indexing: 50%,done:)parse_directory— all parsers now useBaseParser's os.walk loop withskip_suffixes,skip_files,_should_skip_dirhooksTest plan