Architectural concerns over translations and archives

[aterrel]

As we have two GSOC projects (#315, #316) that will add switcher to the book website, it makes sense to discuss how we are going to add these two dimensions to the projects can maintain some independence.

Two suggestions from @OriolAbril:

1. one way to support the 2 dimensional version+language switcher is hosting at readthedocs
2. another options is to emulate the workflow of the cpython docs or the spyder ones, this might require some changes to the jupyter-book project itself to support multiple switches

Personally I would prefer to see us not rely on readthedocs. While I think it is a great service, I don't know that we want to require the book to be served there rather than on Github Pages as it is done now.

@speco29 and @AR21SM, let us know if you need any guidance but we should probably solve this before working too much on either independent project

[speco29]

I'm not sure about the readthedocs suggestions as I didn't understand how it will work.
To emulate CPython’s documentation workflow for language and version switchers, we’ll need to separate the switcher logic from the main repository and manage it dynamically using structured metadata.
It will also require integrating with jupyter notebooks.
Although I am not much familiar with these, but I think its a better option and I'm ready to learn along the process.

[OriolAbril]

I myself am going over the docs and repositories of the cpython project to see if and how we could emulate their setup. I don't understand your comments though.

To emulate CPython’s documentation workflow for language and version switchers, we’ll need to separate the switcher logic from the main repository and manage it dynamically using structured metadata.

When you say we'll need to separate the switcher logic from the main repository what do you mean? We'll need a 2nd cookbook github repository for the switcher javascript? The metadata? Something else? I am also not sure about what will be managed dynamically.

It will also require integrating with jupyter notebooks.

My best guess here is you mean something different than "jupyter notebooks", maybe "jupyter book". If it is indeed jupyterbook, as the python docs use sphinx and jupyterbook also uses sphinx, why does it need integrating and what exactly will need integrating?

[speco29]

When you say we'll need to separate the switcher logic from the main repository what do you mean? We'll need a 2nd cookbook github repository for the switcher javascript? The metadata? Something else? I am also not sure about what will be managed dynamically.

Regarding separating the switcher logic, my intention was to modularize the handling of language and version selection, ensuring that metadata defining available versions/languages is structured independently from the main documentation repository. Would you recommend a separate GitHub repository specifically for metadata, or should version management remain within the main repository with a dynamic mechanism for switchers?

My best guess here is you mean something different than "jupyter notebooks", maybe "jupyter book". If it is indeed jupyterbook, as the python docs use sphinx and jupyterbook also uses sphinx, why does it need integrating and what exactly will need integrating?

For Jupyter integration, I meant Jupyter Book rather than Jupyter notebooks. Since it already uses Sphinx, the integration need depends on whether we require versioned content handling or compatibility adjustments for multilingual documentation. Do you see any potential concerns with this approach?

[OriolAbril]

Given there already are working solutions to multiple versions and languages I would like to have an overview of some of them so we can then either directly follow one of these or combine them into something else that fits our use case better. I tried my best to do it for readthedocs, but there are still at least the cpython and spyder docs setups too which achieve serving multiple languages and versions without readthedocs.

It doesn't need to be today not even tomorrow, as long as it is before the start of the coding period so we can properly define the scope of each project independently it should be ok, but I would like to have the same for the cpython setup and the spyder docs setup which is why I suggested you and @AR21SM look into them.

I also expect this to be a back an forth, not a single post for either of the cases so we can all get a good idea of what concrete work would each of the approaches entail. By back and forth I mean both me and @aterrel asking for further details on the setup but also both of you asking for details about the cookbook setup in case something is not clear, about the setups being described by other people and about the setup you are exploring. As it was mentioned in the idea list, these are hard projects and we will probably have to compromise somewhere. We don't expect you to know how to tackle everything beforehand (we also don't have all the information yet either) now we expect you to fully understand the cpython/spyder setup in a quick skim of the website.

From the thread so far I am not sure what cpython is doing or if the proposals you are mentioning are related to it or not.

Would you recommend a separate GitHub repository specifically for metadata, or should version management remain within the main repository with a dynamic mechanism for switchers?

If I understood correctly, @aterrel initial preference was for using tags for versioning. I myself have never seen tags used with translatable content (but maybe cpython/spyder do and I just don't know yet!) but branches instead and sometimes even one repository per language. Translations generally lag behind the original version so completely freezing content and translations doesn't work, that is why in general what I have seen are tags for releases but translations are in a different repo or branches where the content is frozen but its translations can still be updated. I don't think I have seen a repository specific for metadata but we could do it if it solves our use case.

Since it already uses Sphinx, the integration need depends on whether we require versioned content handling or compatibility adjustments for multilingual documentation. Do you see any potential concerns with this approach?

If we have the translated content in .po files no integration or any other change should be needed to build the translated website.

[aterrel]

Perhaps a repo with mirrored tags would be best. If we did a branch per language that would lead to a lot of branches (n languages, v versions, n*v branches for all languages).

[OriolAbril]

@aterrel in general, each version is a branch, and all translations live in that branch. The main content is no longer updated but the translations are (until they catch up to the content and everything is translated)

[OriolAbril]

I'll add more details on the readthedocs option. ReadTheDocs (RTD) is a service designed to host websites with multiple versions and/or languages and simplify the process of publishing and versioning among other potential challenges.

In the cookbooks case, as it already uses jupyterbook which is based on sphinx, using readthedocs to build and host the website would be quite straightforward. At most it would mean converting the _config.yml into a conf.py but everything else can stay the same, even the _toc.yml. The maintainers of the cookbook would need to create an account on readthedocs and create a cookbook project linked to this repo. RTD will then rebuild and publish the website every time a new commit is pushed, it also has the option of rendering a preview for each PR in addition to the published version.

Once the RTD project has been created, any branch or tag within the github repository can be published as a different version from the RTD dashboard. On RTD, documentation versions can have 3 statuses: active (published and visible on the version switcher), hidden (published but not visible on the version switcher, e.g. to keep the number of version small but avoid breaking links that point to it) and inactive (not published).

Language versions are a bit more complex to set up. They require an extra readthedocs project per language which is then linked from the RTD dashboard to the main project. This can only be done by someone who is an owner of both RTD projects, so the maintainers should be added to the main RTD projects and all the translation ones. The translation RTD projects can be linked to this same GitHub repository or to a different one.

Once the different versions and languages are configures, RTD adds a flyout switcher that when clicked on shows all the active versions and languages. For example, from the sphinx docs which have 1 active version and multiple languages available:

The contents of the switcher are updated automatically whenever versions or languages are modified through the RTD dashboard.

This would hand off part of the complexity of the projects. It would still be necessary to agree on a versioning scheme and to translate and define how to build standalone versions of the translated cookbook.

[speco29]

I'll add more details on the readthedocs option. ReadTheDocs (RTD) is a service designed to host websites with multiple versions and/or languages and simplify the process of publishing and versioning among other potential challenges.

In the cookbooks case, as it already uses jupyterbook which is based on sphinx, using readthedocs to build and host the website would be quite straightforward. At most it would mean converting the _config.yml into a conf.py but everything else can stay the same, even the _toc.yml. The maintainers of the cookbook would need to create an account on readthedocs and create a cookbook project linked to this repo. RTD will then rebuild and publish the website every time a new commit is pushed, it also has the option of rendering a preview for each PR in addition to the published version.

Once the RTD project has been created, any branch or tag within the github repository can be published as a different version from the RTD dashboard. On RTD, documentation versions can have 3 statuses: active (published and visible on the version switcher), hidden (published but not visible on the version switcher, e.g. to keep the number of version small but avoid breaking links that point to it) and inactive (not published).

Language versions are a bit more complex to set up. They require an extra readthedocs project per language which is then linked from the RTD dashboard to the main project. This can only be done by someone who is an owner of both RTD projects, so the maintainers should be added to the main RTD projects and all the translation ones. The translation RTD projects can be linked to this same GitHub repository or to a different one.

Once the different versions and languages are configures, RTD adds a flyout switcher that when clicked on shows all the active versions and languages. For example, from the sphinx docs which have 1 active version and multiple languages available:

The contents of the switcher are updated automatically whenever versions or languages are modified through the RTD dashboard.

This would hand off part of the complexity of the projects. It would still be necessary to agree on a versioning scheme and to translate and define how to build standalone versions of the translated cookbook.

Yes, this gives a lot of clarity on readthedocs.
I'll explore it by tomorrow.

[aterrel]

While I am all for exploring the RTD version of archives and translations, I don't know how much I like relying on them for deliver of this content completely. If there was a way we could modify the jupyter-book template to not requires a 3rd party service, that would be preferred.

[OriolAbril]

I have gone over the cpython setup and I'll try to describe at a high level focusing on the elements I think are relevant to us to as well as I can. I am starting a new thread because their whole process is extremely complex, thankfully the extracted version/language logic not so much.

The python documentation website (everything inside docs.python.org is build using the infrastructure in https://github.com/python/docsbuild-scripts. These scripts are executed automatically from a cron for the python versions that are in active. Cadence wise it looks like once a day for translations, a couple times per day for english version based on. The metadata about the languages is in this repository config.toml whereas the version metadata is in the devguide repo. They use /language/version/ infixes in the urls with the exception of English that skips the language part.

Important

Both versions and languages are static in the python docs. The elements of the switchers are added at the time of building, and updating either of the two metadata documents doesn't update the switchers. Some examples: if you look at the 3.13 English version the versions available range from 2.6 to 3.15 whereas 3.0-3-4 have no version nor language switcher and 2.7 has versions 3.5-3.10 and is missing several languages.

To build the documentation, the main cpython repository is cloned and the requested branch is checked out, and if building a translated version the corresponding translation repo is also cloned into the locale directory and has the matching branch checked out. Tags are used for code releases but the docs are not built from the tags, not even the English version.

Then sphinx is triggered to build the docs, they have their own custom theme and some extra functionality within docsbuild-scripts to inject the switcher contents from the read metadata as html. The generated website (as for all sphinx builds, each version is a full website with all their own css/js/... in their own folder) is then copied over to the server into the corresponding /language/version/ folder.

---

If we wanted to follow a similar pattern but without multiple repositories, I believe that would mean having a branch per release (where content is frozen, maybe it could even be enforced with pre-commit in a github action to make sure only the locale folder can be updated once a branch is "released") and the multiple translations inside that branch as a locale folder (which is what the repo looks like in the cpython setup after cloning the translation repos). We would then have a GitHub action to build all the languages and in that process inject the existing versions and languages into the html.

As mentioned, this has the significant drawback that old versions are not aware of newer ones which is one of the most common navigations we'd want visitors to do.

One of the nice features of having specific repositories for translations is that most things within the translation repos are in the target language (e.g. python-docs-es is mostly in spanish) which allows people who don't speak English to contribute. They obviously can't translate but the cpython docs are huge so pages are translated, reviewed and also proofread (that is, read by people who have not read the original version to make sure the translation by itself makes sense and is coherent). We might want to start with a simpler workflow though.

[OriolAbril]

I am opening yet another thread to discuss not what other projects do but specific proposal for our case.

The cookbook uses the sphinx-book-theme which is based on the pydata-sphinx-one and supports a version switcher with dynamic version metadata so older versions will show newer ones on their switcher. On the other hand, I think it is ok to prevent adding new languages on older versions, and if we add new languages do so only on the most recent version. Consequently, the language switcher can be static like cpython docs do (kind of global version switcher but version specific language switcher).

I would still use branches for releases so ongoing translations can be finished but we could even use tags. If we really want to enforce released branches not getting their content updated we could use pre-commit or github actions on PRs to check only files within the locale folder have been updated when on a release branch.

We would therefore have a versions.json that gets passed to the pydata-sphinx-theme to use as source to dynamically populate the version dropdown (that is, that switcher gets populated when a user visits the site) and a languages.json with a similar format but not necessarily the same which gets read by conf.py and added to the html_context for sphinx jinja templates to build a static html switcher (that is, the switcher gets populated when the website is built). This could be similar for example to how the pydata-sphinx-theme generates the icons at the top right of the navbar from a user provided dictionary defined in conf.py (docs, source).

Implementation wise, that would represent switching from config.yml+jupyterbook to conf.py+sphinx-build (only these two files would need to change, no changes to content or even the toc.yml), a couple tens of html-jinja lines for the html template, the jsons with the metadata and ~5 lines in conf.py to configure the theme provided switcher. I propose this to be part of the version project.

We would then need github actions to 1) build and deploy the website into the corresponding subfolders (update existing one, if possible by using the languages.json to repeat the build step for each language which seems possible and 2) generate po/pot files and push/pull them from whatever service we use for collaborative translations. I propose 1 to be part of the version project, 2 (along with defining when to do that and what service to use, if at all) to be part of the translations project.

In addition, it will also be necessary to properly style the switchers to look nice and similar to each other and to be responsive/available somewhere in all screen sizes. As it depends on the html elements being there, it should be assigned to the project that adds the switchers.

It would also be extremely helpful to get something similar to 1 (something that builds all languages for the current version/branch) to do the same for each PR and add a link to a preview. Not sure which project should take care of that

---

In my opinion, the rest of the tasks are better separated:

• Thoroughly documenting the corresponding github actions.
• Configure and test the old version warning banner -> version project
• Writing documentation for release managers -> version project
• Writing a guide for translators and language managers -> translation project
  • Includes explanation and handling of fuzzy matching for translations as well as making sure the translation history is preserved across versions
• Try and potentially choose a collaborative translation service -> translation project
• 404 page, ideally language aware, otherwise a general single page with messages in the multiple available languages -> translation project
• Upstream translations of theme elements (https://explore.transifex.com/12rambau/pydata-sphinx-theme/ has the existing translations) -> translation project
• Other things I might have missed

[speco29]

• 404 page, ideally language aware, otherwise a general single page with messages in the multiple available languages -> translation project

Does this mean, building a 404 error page if any link breaks on the website?

[OriolAbril]

language manager

reviewing the translated content could be one of their tasks, we'll need to define a bit the different roles which would also be part of that bullet point. When I wrote it I was thinking more about more logistic tasks:

• how do you invite/accept a new translator to the translation platform
• how do you add a new language or change its active/hidden/disabled status
• how do you warn translators when a new version of the content comes in
• I used a work that it completely innocent in Spain but it is an insult in Mexico and it somehow passed the review and was published how (if at all) can the process to pull translation updates and rebuild the website be triggered manually

404 page

Yes, it doesn't need to be anything super complicated but it is nice to have, especially if you are reading the content in a different language and there is a broken cross-reference or any other issue and you end up at a url where there is no page.

This is what the cookbook gets as of now: https://discover-cookbook.numfocus.org/bad-page.html whereas my website (also build with sphinx and served on github pages) gets this page: https://oriolabrilpla.cat/bad-page.html (still much more room for improvement but potentially less scary, especially if the 2 sentences in the page were available in the different languages)

[speco29]

This is what the cookbook gets as of now: https://discover-cookbook.numfocus.org/bad-page.html

Can you send me the github link of this page?

[OriolAbril]

Can you send me the github link of this page?

The page doesn't exist. the bad-page.html part can be replaced with anything that doesn't match any page and you'll see the 404 page. There are ways for us to configure the 404 page so it shows custom more informative content instead of the current page:

[AR21SM]

This is what the cookbook gets as of now: https://discover-cookbook.numfocus.org/bad-page.html

Can you send me the github link of this page?

some examples

Read the Docs shows default 404 page for missing URLs on hosted sites

[AR21SM]

I was exploring how Spyder handles versioning in their documentation, and I wanted to share my findings about their approach.

Spyder uses a branch-based versioning system rather than tags. Each major version has its own dedicated Git branch:

• master branch → Contains docs for the latest Spyder (currently version 5)
• 4.x branch → Contains docs for Spyder 4
• 3.x branch → Contains docs for Spyder 3

This approach allows maintaining separate documentation versions simultaneously, making version-specific updates easier while keeping older versions accessible. Tag-based versioning creates immutable snapshots at specific points in time, whereas branch-based versioning supports ongoing maintenance of older documentation. With tags, each update would require creating a new version tag, which makes it difficult to fix typos or add clarifications to older versions.

The core of the version switcher is versions.json, that defines all available versions along with their associated metadata (like "version": "5", "preferred": true ).

The version switcher appears in the navigation bar, letting users easily toggle between documentation versions. It's implemented using:

• HTML template components that create the dropdown button ( html file )
• JavaScript that reads the versions.json file to populate the dropdown options ( js file ),(css file)
• URL-based navigation that sends users to the corresponding page in different versions (version.json)

When a user switches versions, the system tries to find the same page in the target version, falling back to the version homepage if that specific page doesn't exist in the other version.

Spyder organizes their documentation URLs clearly:

• https://docs.spyder-ide.org/5/ → Latest version (v5)
• https://docs.spyder-ide.org/4/ → Previous version (v4)
• https://docs.spyder-ide.org/3/ → Older version (v3)

The versioning is handled by Netlify's branch deployment capabilities netlify.toml , ci :

• When changes are pushed to the master branch, CI builds those docs and deploys to /5/
• When changes are pushed to the 4.x branch, CI builds those docs and deploys to /4/
• When changes are pushed to the 3.x branch, CI builds those docs and deploys to /3/

Netlify handles the mapping between branches and URL paths, which keeps each version independent while maintaining them under one domain.

(I think we can also use GitHub Pages to achieve similar versioning in the Cookbook repo using custom GitHub Actions.)

Looking at the noxfile.py file, I can see how it supports the versioning approach .This file handles all the build automation for Spyder's docs.

line 46-47:

LATEST_VERSION = 5
BASE_URL = "https://docs.spyder-ide.org"

These constants define the current version number (5) and base URL. When they want to release a new version, they just need to update this LATEST_VERSION constant (after first creating a new branch from master to preserve the previous version documentation)

There's a function called _prepare_multiversion() (around line 507) that handles creating the version directory structure:

This function basically:

• Takes the built docs from the regular build process
• Creates a version-specific directory (like /5/)
• Creates a current directory that points to the latest version
• Generates redirect files

For local development, after setting up a virtual environment, these commands can be run to view documentation in the local browser: nox -s build and nox -s serve. The nox -s build-deployment .

This only builds the current branch's documentation. When I clicked the version switcher during local development, it redirected me to the live site (rather than any local version) because the URLs in versions.json point to the production site (e.g., https://docs.spyder-ide.org/4/). Since the other version directories don't exist locally, and the versions.json file uses absolute URLs rather than relative paths.

The language switcher works similarly to the version switcher, using a language-specific JSON configuration.

If anything I’ve shared is inaccurate or could be improved, I’m absolutely open to feedback and happy to make corrections.

i am currently exploring GitHub Actions and the scikit-learn repository to understand how they implement versioning. @OriolAbril Could you please suggest relevant documentation links that i should explore for my project?

[OriolAbril]

Thanks! I didn't realize only the latest version had translations.

This makes it possible for the language metadata to hardcode the urls including the version which otherwise wouldn't be possible. Did you find any comment or issue about this by any chance?

The version switcher appears in the navigation bar, letting users easily toggle between documentation versions. It's implemented using:

I did skim through the resources you linked plus conf.py and between the number of elements in there and my lack of javascript knowledge I am not completely sure what is going on. In conf.py you can see that they define the elements that go into the navbar:

https://github.com/spyder-ide/spyder-docs/blob/98004cb46d97fa9fa1cfc3f469919928761e7054/doc/conf.py#L142C63-L142C71

and one of them is version-switcher but there is no version switcher template in their repo so it must be using the one from the theme, plus they configure the version switcher included in the PST theme:

https://github.com/spyder-ide/spyder-docs/blob/98004cb46d97fa9fa1cfc3f469919928761e7054/doc/conf.py#L147

But the javascript in language-switcher.json has a lot of version related elements aparently. Do you know if these are legacy no longer used things or are they patches to the theme provided switcher to allow for the multiple switchers? I do know the theme switcher is newer than spyder docs having multiple switchers.

The 1000 lines of javascript in language-switcher.js makes me worry about maintainability of this approach, it looks like not all 1000 lines would be necessary but still several hundreds, and it is still unclear if all that is only to get the simplified case with only languages for the latest version to work or if it is directly useful to our usecase. @aterrel are there any constraints on this maintenance wise?

I think we can also use GitHub Pages to achieve similar versioning in the Cookbook repo using custom GitHub Actions.

I am sure it is possible. It also looks spyder is using github pages to serve the website, maybe netifly only for the PR previews?

i am currently exploring GitHub Actions and the scikit-learn repository to understand how they implement versioning. @OriolAbril Could you please suggest relevant documentation links that i should explore for my project?

AFAIK, scikit-learn uses the pydata-sphinx-theme like spyder and uses the theme provided switcher. So I'd probably recommend going over the CD setup from scikit-learn mostly and looking at the theme provided elements to see if they can be used in our scenario with both versions and languages or if we'll need to build them from scratch.

---

Now that I have explored the spyder docs a bit more I noticed that there are seemingly two versions of the 5+english docs: https://docs.spyder-ide.org/5/index.html and https://docs.spyder-ide.org/5/en/index.html. Do you know how this happens? Is main build once but copied to the two folders? Seeing this makes me worry we might end up in a similar situation but with one version becoming outdated eventually.

[AR21SM]

This makes it possible for the language metadata to hardcode the urls including the version which otherwise wouldn't be possible. Did you find any comment or issue about this by any chance?

some of the related discussion i found :

• Issue #268
• Issue #179
• PR #367 - Commits
• PR #388

and one of them is version-switcher but there is no version switcher template in their repo so it must be using the one from the theme, plus they configure the version switcher included in the PST theme:

you're absolutely correct I can see they're using the theme's built-in version-switcher (with the "switcher" option) but had to create their own custom language-switcher component. (when i commented out the language-switcher related files in conf.py https://github.com/spyder-ide/spyder-docs/blob/master/doc/conf.py#L193C5-L193C31 , https://github.com/spyder-ide/spyder-docs/blob/master/doc/conf.py#L186C5-L186C29
and built it again, the language switcher button disappeared )

But the javascript in language-switcher.js has a lot of version related elements aparently. Do you know if these are legacy no longer used things or are they patches to the theme provided switcher to allow for the multiple switchers? I do know the theme switcher is newer than spyder docs having multiple switchers.

i am not sure why these version related patches are present there .

The 1000 lines of javascript in language-switcher.js makes me worry about maintainability of this approach, it looks like not all 1000 lines would be necessary but still several hundreds, and it is still unclear if all that is only to get the simplified case with only languages for the latest version to work or if it is directly useful to our usecase. @aterrel are there any constraints on this maintenance wise?

i think not all lines are necessary but some of them are .

I am sure it is possible. It also looks spyder is using github pages to serve the website, maybe netifly only for the PR previews?

hmm... i got confused . they have gh-pages branch in their repo . i think they use netlify for pr previews only .

AFAIK, scikit-learn uses the pydata-sphinx-theme like spyder and uses the theme provided switcher. So I'd probably recommend going over the CD setup from scikit-learn mostly and looking at the theme provided elements to see if they can be used in our scenario with both versions and languages or if we'll need to build them from scratch.

i will look into it , will share my finding soon .

Now that I have explored the spyder docs a bit more I noticed that there are seemingly two versions of the 5+english docs: https://docs.spyder-ide.org/5/index.html and https://docs.spyder-ide.org/5/en/index.html. Do you know how this happens? Is main build once but copied to the two folders? Seeing this makes me worry we might end up in a similar situation but with one version becoming outdated eventually.

i am not sure how this is working , but i think if one get updated and other not , it will create problem.

[OriolAbril]

Thanks for the links to issues and PRs. The ones related to sphinx-multiversion can be ignored as it can't be used for our use case (also the reason why they moved away from it eventually). The PRs are very illuminating though. It looks like They added multiple version support through the theme provided switcher in August 2024, then in November 2024 they added support for the language switcher with the whole 1000 lines of javascript file being added in this second PR. This means that the whole file is needed and it is not something legacy but patches to the existing javascript in the theme so it doesn't stop working when introducing the new switcher.

[AR21SM]

i think they just copy pasted this file
https://github.com/pydata/pydata-sphinx-theme/blob/main/src/pydata_sphinx_theme/assets/scripts/pydata-sphinx-theme.js
,
renamed it to language-switcher.js and comment out unnessary component ,added a new section at the end for language switching:

---

they renamed version --> language