Skip to content

loky: remove misleading set_start_method docs and refresh Task & results serialization section#631

Open
mvanhorn wants to merge 1 commit into
joblib:masterfrom
mvanhorn:fix/404-loky-deprecate-misleading-start-method-docs
Open

loky: remove misleading set_start_method docs and refresh Task & results serialization section#631
mvanhorn wants to merge 1 commit into
joblib:masterfrom
mvanhorn:fix/404-loky-deprecate-misleading-start-method-docs

Conversation

@mvanhorn
Copy link
Copy Markdown

Summary

Single-file edit to docs/API.rst.

  1. Remove the "Processes start methods in loky" section header and its body. This is the entire block beginning Processes start methods in :mod:\loky`and ending before "Protection against memory leaks". The section currently lives between theautofunction:: loky.wrap_non_picklable_objects` directive and the "Protection against memory leaks" heading.

  2. In the "Task & results serialization" section, rewrite the paragraph that currently reads:

    As this library is slower than the pickle module in the standard library, by default, loky uses it only to serialize objects which are detected to be in the main module.

    New wording (keep the same overall flow, just correct the two outdated claims):

    cloudpickle is used by default to serialize tasks and results, because it can handle interactively defined functions, lambdas, and locally defined classes that the standard pickle module cannot pickle. Recent versions of cloudpickle are based on the C implementation of pickle, so this is not a significant performance trade-off compared to plain pickle for most workloads.

  3. Keep the existing list of three ways to control serialization (job_reducers/result_reducers, LOKY_PICKLER, wrap_non_picklable_objects) verbatim. Do not change them in this PR; deprecation of wrap_non_picklable_objects is out of scope per the deferral above.

  4. Build the docs locally to verify no Sphinx warnings or broken cross-references are introduced (the removed section did not contain any .. _xxx: labels that other pages link to; a quick grep -r "set_start_method" docs/ after the edit confirms no leftover references).

PR title: DOC Remove misleading set_start_method section and refresh cloudpickle wording

PR body (concise, factual):

Closes #404.

  • Removes the Processes start methods in loky section from docs/API.rst. loky.set_start_method is not exposed at the top-level loky module, so documenting it as public API was misleading.
  • Rewords the cloudpickle paragraph in Task & results serialization to remove the "slower than pickle" claim (no longer true with cloudpickle's C-pickle base) and the "only objects detected to be in main" claim (loky in fact uses cloudpickle for everything cloudpickle can handle).

Per the maintainer comment on the issue, deprecating loky.wrap_non_picklable_objects and the loky_init_main start method is left for a separate PR since it requires a runtime deprecation cycle.

Why this matters

Issue #404 reports that the loky docs imply loky.set_start_method is public API, when the actual import path is loky.backend.context.set_start_method and the function is not exposed at the top-level module. Maintainer @ogrisel responded on 2025-03-06 stating he does not see value in changing the start method at the loky level, and proposed three concrete docs cleanups in the same comment:

  1. Delete the "Processes start methods in loky" section in docs/API.rst so the docs stop implying loky.set_start_method is public API.
  2. Update the "Task & results serialization" section. Two statements there are no longer accurate:
    • "As this library [cloudpickle] is slower than the pickle module in the standard library" - no longer true now that cloudpickle is built on the C pickle implementation.
    • "by default, loky uses it only to serialize objects which are detected to be in the main module" - the current behavior is that cloudpickle is used for all objects (including locally defined functions outside __main__), as demonstrated by the small reproducer in the maintainer comment.
  3. (Deferred from this plan) Deprecate loky.wrap_non_picklable_objects and the loky_init_main start method. These touch runtime code and the deprecation cycle is a larger conversation; this PR will keep the autofunction directive for wrap_non_picklable_objects so the deprecation can be staged in a follow-up.

Testing

  • cd docs && make html completes without new Sphinx warnings or errors compared to a clean master build (capture both before and after stderr to diff).
  • grep -rn "set_start_method" docs/ returns no matches after the edit. (Before the edit, it matches the section being removed.)
  • grep -rn "slower than the :mod:\pickle`" docs/` returns no matches after the edit.
  • Open the generated docs/_build/html/API.html locally and visually confirm that the API page renders, the autofunction directive for wrap_non_picklable_objects still appears, and the Protection against memory leaks section now follows the Task & results serialization section directly.
  • ReadTheDocs preview build (auto-triggered by the PR) succeeds.

Fixes #404

AI was used for assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs suggest that loky.set_start_method exists

1 participant