[Bug]: watch mode fails when git maintenance runs

[thw0rted]

Version

29.4.0

Steps to reproduce

1. Run your tests with jest --watch and wait until they finish.
2. In another terminal, run git maintenance run.
3. The watcher will fail

Expected behavior

Outside process shouldn't kill the watcher.

Actual behavior

Watcher dies immediately with

Error: EPERM: operation not permitted, lstat 'C:\Workspace\my-project\.git\objects\dc\a-very-long-hash'
Emitted 'error' event on NodeWatcher instance at:
    at C:\Workspace\my-project\node_modules\jest-haste-map\build\watchers\NodeWatcher.js:274:14
    at FSReqCallback.oncomplete (node:fs:207:21) {
  errno: -4048,
  code: 'EPERM',
  syscall: 'lstat',
  path: 'C:\\Workspace\\my-project\\.git\\objects\\dc\\a-very-long-hash'
}

I have also seen it point to .git\objects\maintenance.lock.

Additional context

This other issue is where I found the suggestion that it was a background git maintenance command that was causing the conflict, and once I knew that it was easy to reproduce.

Environment

System:
    OS: Windows 10 10.0.22621
    CPU: (12) x64 Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz
  Binaries:
    Node: 18.12.1 - ~\AppData\Local\fnm_multishells\30672_1675721836564\node.EXE
    npm: 8.19.2 - ~\AppData\Local\fnm_multishells\30672_1675721836564\npm.CMD
  npmPackages:
    jest: ^29.3.1 => 29.4.0

[thw0rted]

Oh and in case it wasn't clear from the linked issue, this is a high priority since VSCode's built in Git integration just runs maintenance every couple of minutes in the background, making watch mode impossible to use while Code is open.

[mrazauskas]

Does not reproduce on MacOS. Might be configuration issue or something Windows specific.

Strangely it is mentioned in the docs that the watcher ignores .git folder. See the watchPathIgnorePatterns option.

[thw0rted]

I wouldn't be surprised if this is Windows-specific, I assume the watcher used varies by filesystem type. One thing that didn't come up in the Environment dump is that I'm running both commands (jest and git) from Git Bash; I don't know if it would fail under e.g. PowerShell or CMD.

The error I pasted in the report is all the stack-dump I get when this happens on the command line. I made a new commit to the repository in question, and now I'm having trouble reproducing the error. I did reproduce the error on a different project, using the same method. I got the same limited stack info there, so I edited the NodeWatcher.js file referenced in the output to add a console.error, hoping to see a full stack dump, but now I can't reproduce the issue in the second project either.

Maybe git maintenance run only makes the problematic filesystem interactions once, after a recent change/commit? I'm having some trouble nailing it down.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[rentalhost]

This bug keeps happening on Windows. It's quite intermittent. Maybe it's related to SmartGit running git maintenance in the background or something?

[rentalhost]

I've debugged the error starting point, and everything indicates that it activates within here:

https://github.com/facebook/jest/blob/285b40d5954fd81e07b02db92185fd317ba92578/packages/jest-haste-map/src/watchers/NodeWatcher.js#L281

And because it is not handled properly, it forces the process to terminate when maintenance.lock allows lstat() to fail.

[mrazauskas]

Hm.. But why the .git directory is being watched? Are you including it explicitly? Or this pattern does not work on Windows:

https://github.com/facebook/jest/blob/285b40d5954fd81e07b02db92185fd317ba92578/packages/jest-haste-map/src/index.ts#L130-L132

[rentalhost]

@mrazauskas I don't know why. Seems that .git directory is not being ignored in fact.

My jest.config.ts is:

import type { Config } from "jest";

export const jestConfig = (sourcePath = "src"): Config => ({
  moduleNameMapper: {
    "^@/(.*)$": `<rootDir>/${sourcePath}/$1`,
    "^@Tests/(.*)$": "<rootDir>/tests/$1",
  },
  setupFiles: ["dotenv/config"],
  coverageThreshold: {
    global: {
      branches: 100,
      functions: 100,
      lines: 100,
      statements: 100,
    },
  },
});

export default jestConfig() as unknown;

But I also have tried to ignore it by using watchPathIgnorePatterns but not works.

I will try debug that, but I don't know exactly how for now.

By now, I have modified the file NodeWatcher.js and the problems is gone:

273:
-     if (error && error.code !== 'ENOENT') {
+     if (error && error.code !== 'ENOENT' && error.code !== 'EPERM') {

301:
-       if (error && error.code === 'ENOENT') {
+       if (error && error.code !== 'ENOENT') {

[rentalhost]

Right, I found a glue...

The file common.js uses anymatch to check if ignore pattern match or no with directories.

So basically we have an ignore pattern like /\\\.git\\|\\\.hg\\|\\\.sl\\/ against a directory like eg. D:\\project\\.git\\refs\\tags, which at first glance seems to match. But anymatch() normalizes directories to use / in place of \\ instead of treating it literally/as-is. So the ignore pattern will always fail on Windows.

const anymatch = require('anymatch')

// Using "\" on matcher (current behavior):
console.log(anymatch(/\.git\\/, 'D:\\project\\.git\\refs\\heads')); // Will fail!
// >> false

// Using "/" on matcher:
console.log(anymatch(/\.git\//, 'D:\\project\\.git\\refs\\heads')); // Will works!
// >> true

// What anymatch sent to matcher:
anymatch(dir => console.log({dir}), 'D:\\project\\.git\\refs\\heads'); // The "\" is converted to "/"
// >> D:/project/.git/refs/heads

See: https://runkit.com/rentalhost/facebook.jest.issues.13869

So, I believe that the ignore pattern used in common.js should always use / as separator, instead of respecting the used OS separator.

In this case, the problem starts exactly where you mentioned:

https://github.com/facebook/jest/blob/e2196cab463c5a3e15668b1d7663058b09b7c0ed/packages/jest-haste-map/src/index.ts#L130-L132

The problem is whether it is safe to change path.sep to / in any case.

In the local tests I've done, it seems to work fine. The tests continue to work perfectly and the problem no longer happens.

[rentalhost]

ref: micromatch/anymatch#7

[mrazauskas]

Good catch!

I looked around. Seems like the ignorePattern is passed not only to anymatch. Perhaps the best fix would be to keep the pattern as is and to normalize it just before passing to anymatch? slash can be used (if it works in this case), because it is a dependency already.

Interesting why this test is passing in CI on Windows (might be that anymatch is simply not involved because of mocking):

https://github.com/facebook/jest/blob/285b40d5954fd81e07b02db92185fd317ba92578/packages/jest-haste-map/src/__tests__/index.test.js#L341-L347