Resilience project: the target mapper generator

[Kidev]

The idea is simple, and is a way to answer last year's call from sir @miurahr! Let's make aqtinstall less high maintenance

What's the data, theta? 🐬

The main problem is very obviously that at its core, aqtinstall is a two teams project that don't communicate (us and Qt). So, is the solution communication? Of course not, I'm an autistic engineer, the solution is automation.
The main maintenance requirements is the changes done by the Qt team in their remote repository. It is not all update requirements, but removing this from our concerns would significantly reduce the update rate. It is because, even when those changes seem like 'a simple edge case to add', it quickly gets out of hand when it becomes the heart of a project that is serving tons of packages and that needs to be reliable for its users. Some versions have special rules, some miss things, then tools are removed from versions altogether... Sum this over a few months/year and you turn the project into a massive fractal of switch statement for each user request.

Speedrunning the steady state

As mentioned, the project is converging to a state where each user request has its own hard-coded answer. Well, let's do that at once instead of doing it case by case. It's not as stupid as it sounds, hear me out.

Definitions

Let's agree on those terms to make discussions easier

• target repositories (or target repos) are the folders that contain everything relevant for a host and a target (e.g. linux x64 + desktop). They hold a Updates.xml file.
  • Ex: online/qtsdkrepository/linux_x64/desktop/ is the target repository Linux x64 on desktop
• packages are the name of the leaves of the folder structure, they hold the tarballs for a host, a target, a version, and module(s)
  • Ex: qt.qt6.683.linux_gcc_64 is a package of the target repository linux_x64-desktop for the module root (aka the base module, '') of version 6.8.3 on architecture linux_gcc_64.
  • Ex: extensions.qtpdf.683.win64_msvc2022_64 is a package of the target repository windows_x86-extensions for the module qtpdf of version 6.8.3 on architecture msvc2022_64
• download paths of packages are the URL pathname that, when appended to a base server URL, point to the folder of a package
  • Ex: online/qtsdkrepository/linux_x64/desktop/qt6_660/qt.qt6.660.gcc_64 is the download path of qt.qt6.660.gcc_64

Needs

• Some tool that can create a JSON that maps each packages to a download path, and that is automated. Let's call this the Target Mapper
• A way to way to extract a packet name from a user CLI request. Let's call this the Target Finder

Mandatory requirements

• The system must be reliable, at least as well as it is right now
• It is not acceptable to rewrite everything as it would create too much of a risk that some obscure use case that some client depend on becomes unusable from one day to the next
• For similar reasons, the usage of the tool must remain the same, it's only a back end change

Solutions

Target Mapper

Description

This tool is running entirely on the Github Actions servers. This tool automatically crawls through the downloat.qt.io tree, and keeps a map associating each package with their URL on the remote. It is easier than it seems, as it only has to pull each Updates.xml file it finds and correctly map everything in a JSON

Features

• Can be run manually using workflow_dispatch, ideally some project contributor should be able to manually start this specific workflow
• Ran automatically each day (or less? depends on who can manually run it in case of changes by Qt). Could also be run automatically by a robot whenever a user posts a new issue with a given tag (missing package): there's a lot of automation possible using Github Actions
• Creates a qt-packages.json file. Looks like:

{
  "metadata": {
    "generated_at": "2025-06-06T10:30:00Z",
    "generator_version": "1.2.0",
    "total_packages": 1247,
    "qt_versions": ["6.8.0", "6.7.2", "6.6.3", "5.15.18"],
    "crawl_duration_seconds": 45,
    "base_url": "https://download.qt.io"
  },
  "packages": {
    "extensions.qtpdf.680.linux_gcc_64": {
      "url": "online/qtsdkrepository/linux_x64/extensions/qtpdf/680/x86_64/extensions.qtpdf.680.linux_gcc_64/",
      "qt_version": "6.8.0",
      "platform": "linux_x64",
      "component": "qtpdf",
      "architecture": "gcc_64",
      "size_bytes": 45231616,
      "last_modified": "2025-05-15T09:20:00Z"
    },
    "qt.qt6.660.gcc_64": {
      "url": "online/qtsdkrepository/linux_x64/desktop/qt6_660/qt.qt6.660.gcc_64",
      "qt_version": "6.6.0",
      "platform": "linux_x64",
      "component": "",
      "architecture": "gcc_64",
      "size_bytes": 23445632,
      "last_modified": "2025-05-15T09:20:00Z"
    }
    ...
  }
}

• Lives in aqtinstall repo, on its own branch (unusual git usage): each generation automatically commit the changes to the generated target map file to the special branch

Notes

• Is base server agnostic
• The hash is by design not saved here, but can be deduced from the url field of the JSON and it works no matter the base mirror chosen: https://download.qt.io/${url}/../Updates.xml since those files contain the hash of the package
• Can use tags on the branch to flag certain versions, making clients able to use the tree by default (latest), or whichever they want using a new aqtinstall flag

Estimated performance

• File size of the target map: ~2-5MB
• Parse time: <50ms for most
• Memory usage: Negligible for automated tools
• Transfer speed: Very fast the first time, immediate the successive calls (git update)

Target Finder

Description

Just an alias for... well aqtinstall.

Features

• No changes are made to aqtinstall interface, except for the addition of flags:
  • --target-map TAG_NAME that defaults to latest and is used to know what target map version to pull
  • --refresh-target-map that updates the local target map using git, making this super quick
  • --disable-target-map that basically reverts aqtinstall to its current behavior
  • --map [PACKAGE:URL] that allows for run-time additions to the target map (useful to debug users with special issues)
  • --target-package [NAME] to install a specific package without going through aqtinstall internal logic
• At first run, or each time the user adds the flag --refresh-target-map (discuss? a update each run is possible but maybe overkill) the target mapper branch repo is cloned to the cache directory of the user
• For subsequent runs, the repo branch is pulled if already present. Uses git making this super quick as it pulls only diffs
• Whenever aqtinstall is ran without --disable-target-map inputted:
  • If --map was inputted, create a priority temporary map with user specified data, that takes priority whenever the program needs to read the target map. If one package is described in the target map, but the user choses to change it using --map, then the user inputted update is prioritized, but then the target map is used if needed: that means that the temporary user map is merged onto the official target map.
  • For non aqt list-* command:
    • Runs as usual
    • In case one or more package are not found and the feature is not deactivated, looks into the target map merged with the temporary map, and try to download there
    • (maybe ?) If fails again, updates the target map and retries to download there once
  • For aqt list-* command:
    • Runs as usual, and runs the listing feature on the target map merged with the temporary map
    • Shows a list by taking the current list (let's call it the vanilla output), and adds it to the entries from the target map merged with the temporary map that are not present in the vanilla list

Conclusions

This should solve all future issues of Qt repo architecture. It is safe for the clients because that list may be third party, it is not hosting URLs, only pathnames. Moreover, each feature of this update is designed so it can only add entries to the current ones, never remove, making sure the software is at least as good as it is now.
It makes updating repo issues much faster as a workflow manual start is all it takes, and with automatic periodic workflow runs, we are guaranteed that if Qt breaks something, the users are at most locked out for 24h (0h if they chose to use the official installer feature).
The actual updates requiring real programming work are now easier to prioritize. Plus, for special issues, we can share a --map command to users that immediately fixes their issue without having to update the code. It can be used in wild ways like python -m aqt install-qt --map "notqt.special:https://noqtsite.com/specialpackage" --target-package "noqt.special" and enable virtually any addition/fix live without having to touch the code. This being a CLI value make it possible to use from install-qt-action as well

Tell me what you think about this, give me your feedback on the idea!

[Kidev]

Working implementation of the target mapper here
The qt-packages.json generated can be seen here
The qt-updates.json (contains all the Updates.xml, only for testing for now) generated can be seen here
The working PR is here

Some stats for the crawl, using a sequential cooldown of 0.2s and addition to some wait here and there at some steps:

Qt Repository Target Mapper
Base URL: https://download.qt.io
Starting crawl of https://download.qt.io/online/qtsdkrepository/
------------------------------------------------------------
Crawl completed in 753.1 seconds
Found 24720 packages
Found 2171 Updates.xml files
Explored 2420 directories

Package mapping saved to: qt-packages.json (6.3 MiB)
Updates.xml content saved to: qt-updates.json ( MiB)

So the crawl takes 10mn, and you probably request data faster going through the folders on the Qt repo when doing it manually! 0.2s is fast enough and not too straining. The user agent is also very open about who/what it is.

{"User-Agent": "aqtinstall-mapper/1.0 (https://github.com/miurahr/aqtinstall)"}