Slow start times due to use of barrel files

[greyscalemotif]

🐛 Bug Report

It is unclear that large dependency graphs caused by the use of barrel files can slow test initialization dramatically. There are some tangential mentions of this in the documentation, but it is not outlined in a clear and direct manner. This likely leads to a lot of inefficiency in many projects.

Here are the most important points to stress:

• Jest simulates Node's require cache to allow for isolation and mocking
• The require cache is built independently for every test suite
• All dependencies of a file are included, even if unused

It is the last bullet that leads to the largest reduction in efficiency, due mainly to barrel files. Barrel files are index files that re-export the exports of other files in a directory. They make it possible to import multiple related dependencies in a single import statement. The downside of this is that Jest sees all of the re-exported contents of the barrel file as dependencies and crawls through them, used or not.

Reducing the use of barrel files can help quite a bit in reducing the amount of time it takes before a test can start. This is especially true of dependencies pulled from NPM packages. Packages are typically developed with a root index file that acts as a barrel file for the entire package. If the package is rolled up into a single file at build time, there is only one file for the Jest runtime to open and parse. If the package publishes its source files independently, without a rollup stage, the Jest runtime will need to open and parse every file in the package independently.

In an enterprise setting, there are often internally developed tools and libraries. These packages can grow to be fairly large, and given their internal use, it can be tempting to provide the source files as the output of the packages. In fact, this can improve tree shaking when building the applications that depend on them. Jest, however, can suffer greatly in this environment because the raw number of files can grow without bounds. This problem becomes exponentially worse when overly tight internal dependency semvers reduce the ability of package managers to de-duplicate their installs.

Resolving these issues can lead to tremendous decreases in Jest's test suite initialization and this should be highlighted in the documentation. Barrel files can have a huge impact on the number of files that the Jest runtime needs to parse, and that is not clear without a deep dive into the way dependencies are evaluated. Sharing this knowledge more broadly could make an already fantastic test runner that much better and improve the quality of many products that rely upon it in the process.

To Reproduce

As this is a documentation issue and not a code issue, this may not apply. However, in the spirit of completeness:

• Read the Jest documentation
• Experience slow test start times in enterprise scale development
• Be unaware of the impact that barrel files can have on running Jest

Expected behavior

The expectation is that the Jest documentation should more explicitly explain the impact that barrel files can have on test start times.

Link to repl or repo (highly encouraged)

Given that this issue is manifested in large scale applications with many inter-related dependencies, it is not feasible to provide a replication of the issue.

envinfo

N/A

[gilamran]

On our project we have ~100 test suites, ~600 tests. it take about 6 minutes to run all the tests.
The tests are very fast (less than 50ms. per test), but the startup time is very slow (And annoying)

I've created a script that changes all the imports from barrel files to explicit import, and the tests run 10x faster (Still not optimal, but shows the problem)

Anyone with a solution/workaround/idea?

[SimenB]

I'm happy to take a PR outlining this issue. 👍

---

In Jest itself we've made lots of imports lazy to avoid this issue, but doing across the entire codebase requires the use of babel (or some other code transform). Might be an OK workaround in tests, but won't help in node_modules as it's not transformed by default (you can have Jest transform node_modules if you want, but at some point there are diminishing returns since you then have to wait for the file to be transformed).

[fadi-george]

@gilamran could you please share that script for changing the barrel files import? I'm in a similar bind where changing one reusable component is running 100s of tests. I've tried using babel transform plugin imports but it was not working for me.

[gilamran]

@fadi-george sorry I don't have the script anymore (Maybe I can look it up in the git history), but I've found even faster solution.
I'm using esbuild to bundle ALL my tests + code into one huge file, then I run jest on this file only. so instead of 350sec. with jest as it is, it's taking 30sec tops.

I had to do some juggling to make esbuild and jest work together, like importing svgs etc.
you can see the script here

[fadi-george]

@gilamran ah interesting approach, thank you for sharing! I'll definitely play around with that idea.

[fadi-george]

@gilamran I assume you are doing something like this? Have you ran into any memory issues?

node build-tests && jest ./src/bundle.spec.ts

bundle.spec.ts:

import testBundle from './tests/bundle';

test('should pass', () => {
  testBundle();
});

[fadi-george]

I believe I fixed my memory issues but now I ran into this problem with jest mocks and mockImplementations failing.