build, doc: use new api doc tooling

[flakey5]

Switches over to using the new doc generation tooling. For more background on this, please see #52343

Currently a draft just to get feedback on the approach to this integration.

cc @nodejs/web-infra

[nodejs-github-bot]

Review requested:

• @nodejs/nodejs-website
• @nodejs/web-infra

[ovflowd]

@flakey5 I guess you also need to check our GitHub Action Workflows, and also other mentions of these files within the source.

Like within the Contributor Docs, there is a file that describes how the legacy API doc tooling works, and I believe there are other references also.

[flakey5]

lint-js-and-md is failing because of the linting errors since it exits with a non-zero status code if there's anything wrong with the docs. I think we should skip the REPLACEME checks for normal ci runs.

It also looks like synopsis.md fails the introduced_in check because it's not under the top-level header, should it be enforced that introduced_in goes under the top-level header or should we change it to just check that it exists somewhere in the file (like it was doing previously)?

[flakey5]

Also not sure what's going on with lint-addon-docs? cc @araujogui

[ovflowd]

lint-js-and-md is failing because of the linting errors since it exits with a non-zero status code if there's anything wrong with the docs. I think we should skip the REPLACEME checks for normal ci runs.

It also looks like synopsis.md fails the introduced_in check because it's not under the top-level header, should it be enforced that introduced_in goes under the top-level header or should we change it to just check that it exists somewhere in the file (like it was doing previously)?

REPLACEME shouldn't error, imo, just give a warning. Our linter should have warn and error levels.

And yes introduced_in must be top level!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.97%. Comparing base (049664b) to head (6b0cd66).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #57343      +/-   ##
==========================================
- Coverage   90.06%   89.97%   -0.09%     
==========================================
  Files         645      643       -2     
  Lines      189130   188695     -435     
  Branches    37094    36945     -149     
==========================================
- Hits       170339   169783     -556     
- Misses      11511    11680     +169     
+ Partials     7280     7232      -48     

see 45 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[araujogui]

lint-js-and-md is failing because of the linting errors since it exits with a non-zero status code if there's anything wrong with the docs. I think we should skip the REPLACEME checks for normal ci runs.

It also looks like synopsis.md fails the introduced_in check because it's not under the top-level header, should it be enforced that introduced_in goes under the top-level header or should we change it to just check that it exists somewhere in the file (like it was doing previously)?

Actually, the linter only returns 1 if there's an error-level issue, and I don't think that's the case here.

[araujogui]

Also not sure what's going on with lint-addon-docs? cc @araujogui

I’m not sure either, but I’ll check it out.

[ovflowd]

@flakey5:

3:1-3:9   warning Use "the Node.js" instead of "Node.js'" prohibited-strings remark-lint
4:46-4:50 warning Use "Node.js" instead of "Node"         prohibited-strings remark-lint

On the README.md file you updated (tools/doc/README.md) after updating those can you run make format-md (?)

[ovflowd]

This is the result of many months of arduous work between many awesome folks, including @flakey5 @AugustinMauroy @araujogui @ovflowd @avivkeller and others.

I'm so proud of what we are achieving here and this is a huge step towards a modern tooling and a revamped API docs within Node.js

Approving, as I believe this is ready!

[ovflowd]

cc @nodejs/collaborators can we have another approval here? 🙏

[lpinca]

RSLGTM because it is hard to review and outside of my comfort zone.

[flakey5]

Hmmm -c $(call available-node, -p 'url.pathToFileURL("CHANGELOG.md")') \ is causing this to be thrown: /bin/sh: 4: Syntax error: "then" unexpected (expecting "fi")

[aduh95]

Hmmm -c $(call available-node, -p 'url.pathToFileURL("CHANGELOG.md")') \ is causing this to be thrown: /bin/sh: 4: Syntax error: "then" unexpected (expecting "fi")

The fix would be to have the tooling take a path rather than a URL

[ovflowd]

cc @aduh95 are we good to merge?

[ovflowd]

(or is something still failing?)

[flakey5]

cc @aduh95 are we good to merge?

bump 😄

[aduh95]

I'm getting the following error when I try to generate several files:

Error: EEXIST: file already exists, mkdir '/Users/duhamean/Documents/node/out/doc/api/assets'
    at async mkdir (node:internal/fs/promises:857:10)
    at async mkDirAndCopy (node:internal/fs/cp/cp:308:3)
    at async Object.generate (file:///Users/duhamean/Documents/node/tools/doc/node_modules/@node-core/api-docs-tooling/src/generators/legacy-html/index.mjs:180:7)
Makefile:827: warning: pattern recipe did not update peer target 'out/doc/api/http2.json'.
make[1]: *** [Makefile:827: out/doc/api/embedding.html] Error 1

[ovflowd]

I'm getting the following error when I try to generate several files:

Error: EEXIST: file already exists, mkdir '/Users/duhamean/Documents/node/out/doc/api/assets'
    at async mkdir (node:internal/fs/promises:857:10)
    at async mkDirAndCopy (node:internal/fs/cp/cp:308:3)
    at async Object.generate (file:///Users/duhamean/Documents/node/tools/doc/node_modules/@node-core/api-docs-tooling/src/generators/legacy-html/index.mjs:180:7)
Makefile:827: warning: pattern recipe did not update peer target 'out/doc/api/http2.json'.
make[1]: *** [Makefile:827: out/doc/api/embedding.html] Error 1

@flakey5 any idea here? Is this trying to generate all the files at the same time? Individual file generation should still work, no?

[flakey5]

any idea here? Is this trying to generate all the files at the same time? Individual file generation should still work, no?

What's the command being used?

[ovflowd]

any idea here? Is this trying to generate all the files at the same time? Individual file generation should still work, no?

What's the command being used?

☝️ @aduh95

[aduh95]

any idea here? Is this trying to generate all the files at the same time? Individual file generation should still work, no?

What's the command being used?

make docclean out/doc/api/addons.html out/doc/api/assert.html out/doc/api/async_context.html out/doc/api/async_hooks.html out/doc/api/buffer.html out/doc/api/child_process.html out/doc/api/cli.html out/doc/api/cluster.html out/doc/api/console.html out/doc/api/corepack.html out/doc/api/crypto.html out/doc/api/debugger.html out/doc/api/deprecations.html out/doc/api/dgram.html out/doc/api/diagnostics_channel.html out/doc/api/dns.html out/doc/api/documentation.html out/doc/api/domain.html out/doc/api/embedding.html out/doc/api/errors.html out/doc/api/esm.html out/doc/api/events.html out/doc/api/fs.html out/doc/api/globals.html out/doc/api/http.html out/doc/api/http2.html out/doc/api/https.html out/doc/api/index.html out/doc/api/inspector.html out/doc/api/intl.html out/doc/api/module.html out/doc/api/modules.html out/doc/api/n-api.html out/doc/api/net.html out/doc/api/os.html out/doc/api/packages.html out/doc/api/path.html out/doc/api/perf_hooks.html out/doc/api/permissions.html out/doc/api/process.html out/doc/api/punycode.html out/doc/api/querystring.html out/doc/api/quic.html out/doc/api/readline.html out/doc/api/repl.html out/doc/api/report.html out/doc/api/single-executable-applications.html out/doc/api/sqlite.html out/doc/api/stream.html out/doc/api/string_decoder.html out/doc/api/synopsis.html out/doc/api/test.html out/doc/api/timers.html out/doc/api/tls.html out/doc/api/tracing.html out/doc/api/tty.html out/doc/api/typescript.html out/doc/api/url.html out/doc/api/util.html out/doc/api/v8.html out/doc/api/vm.html out/doc/api/wasi.html out/doc/api/webcrypto.html out/doc/api/webstreams.html out/doc/api/worker_threads.html out/doc/api/zlib.html out/doc/api/addons.json out/doc/api/assert.json out/doc/api/async_context.json out/doc/api/async_hooks.json out/doc/api/buffer.json out/doc/api/child_process.json out/doc/api/cli.json out/doc/api/cluster.json out/doc/api/console.json out/doc/api/corepack.json out/doc/api/crypto.json out/doc/api/debugger.json out/doc/api/deprecations.json out/doc/api/dgram.json out/doc/api/diagnostics_channel.json out/doc/api/dns.json out/doc/api/documentation.json out/doc/api/domain.json out/doc/api/embedding.json out/doc/api/errors.json out/doc/api/esm.json out/doc/api/events.json out/doc/api/fs.json out/doc/api/globals.json out/doc/api/http.json out/doc/api/http2.json out/doc/api/https.json out/doc/api/index.json out/doc/api/inspector.json out/doc/api/intl.json out/doc/api/module.json out/doc/api/modules.json out/doc/api/n-api.json out/doc/api/net.json out/doc/api/os.json out/doc/api/packages.json out/doc/api/path.json out/doc/api/perf_hooks.json out/doc/api/permissions.json out/doc/api/process.json out/doc/api/punycode.json out/doc/api/querystring.json out/doc/api/quic.json out/doc/api/readline.json out/doc/api/repl.json out/doc/api/report.json out/doc/api/single-executable-applications.json out/doc/api/sqlite.json out/doc/api/stream.json out/doc/api/string_decoder.json out/doc/api/synopsis.json out/doc/api/test.json out/doc/api/timers.json out/doc/api/tls.json out/doc/api/tracing.json out/doc/api/tty.json out/doc/api/typescript.json out/doc/api/url.json out/doc/api/util.json out/doc/api/v8.json out/doc/api/vm.json out/doc/api/wasi.json out/doc/api/webcrypto.json out/doc/api/webstreams.json out/doc/api/worker_threads.json out/doc/api/zlib.json -j12

[avivkeller]

Shouldn't opts be passed here, which'll include the force used in the cp call in api-docs-tooling?

[aduh95]

It looks like CI's failing on a missing fi

[avivkeller]

@aduh95 We’ve drastically improved performance, so I hope we’ve resolved all your concerns 😃

[ovflowd]

@aduh95 We’ve drastically improved performance, so I hope we’ve resolved all your concerns 😃

Shouldn't we update the version being used here on the package.json?

[aduh95]

@aduh95 We’ve drastically improved performance, so I hope we’ve resolved all your concerns 😃

Shouldn't we update the version being used here on the package.json?

I think that's indeed needed as I'm still getting Error: EEXIST: file already exists, mkdir '…/out/doc/api/assets' on 0194b0e when running make docclean docserve -j.
Also there are conflicts to solve.