Refactor repository structure for easier maintenance

[Arcitec]

As discussed in #413:

Problem:

The index.yml is very often forgotten by people (even core contributors). Some people edit the manifest, some people edit the index, but 50% of the time one of them is forgotten. There's lots of metadata repetition, leading to maintainer and contributor mistakes.

Proposal:

• Delete: index.yml (almost all of its fields are copy-pasted duplicates of the per-app manifests, causing a maintenance burden)
• Add: meta.yml with the following contents, to tell Bottles which sub-directories to scan for *.yml manifests:

categories:
- Games
- Software

• Modify <app name>.yml format as follows:
  • Name: Change this to be the "pretty display name" that used to only exist in the index, such as "GOG Galaxy". This field used to be redundant in the manifest and was always the same name as the manifest file minus the .yml extension (so for example, epicgamestore.yml had a Name: epicgamestore which is pointless and a source of potential typos). So let's instead track that internal "name" via the actual .yml filename, and use the Name field to carry the UI display name.
  • Category: Do not add this to the manifest. Instead, automatically detect it based on the name of the directory the manifest is inside, such as "Games" or "Software". That's one less source of typos.
  • Icon: Add this to the manifest. It's the filename of the icon to display during installation.
  • Event: If it's useful, add it to the manifest. I don't know the purpose. Only a few entries in the index currently have this value, and it's always a filename pointing at an installer file. It may be possible to just remove this entirely. Edit: Include it, I found its purpose and it looks like a great idea: Refactor repository structure for easier maintenance #414 (comment)
  • Hidden: Add this optional field. If missing, assume false. Otherwise allow the manifest to contain true/false here. If hidden is true, do not display this manifest in the UI. The purpose is to allow an easy way to hide broken manifests without having to manually move them out of the auto-scanned manifest directories.

The scanning process for the client to index all manifests is then as follows:

for category in categories:  # "categories" comes from meta.yml
    category_dir = programs_dir / category  # assumes pathlib.Path
    for manifest_yml in category_dir.glob("*.yml"):
        name = manifest_yml.stem
        # We have now automatically derived the category and manifest name:
        # category: Games
        # name: epicgamestore

        # Now read the manifest YAML file, and immediately skip the manifest
        # if `Hidden` exists and is `true`. Do that check before any other parsing,
        # to avoid trying to parse other possibly-outdated manifest structures.

With this repo structure, maintenance is only a matter of editing/moving a single .yml file per application.

Target: Bottles-Next, or possibly even the current public release. If targeting the current release, a simple migration solution is to put up a dummy index.yml file which is empty, so that outdated Bottles clients won't have any fatal errors and simply don't display any programs at all until the user has updated Bottles.

[Arcitec]

I suddenly realized that Event is documented at the top of index.yml:

# Notes:
# - [Event] not implemented yet
#   Will be used by Bottles to trigger an installer when the pattern matches

Yeah that sounds like a great idea to auto-trigger the correct installer script if the user manually drags and drops an installer file onto Bottles. So the Event field should be moved into the manifest too. 👍

[jntesteves]

The denormalized index is useful for performance. I think it's better generated in the repository by CI instead of the client, that avoids the user having to wait for the download of a bunch of files they won't use. So I agree the current lack of normalization is an issue, but I wouldn't do away with the index. We could add all fields to each program's file as you propose, and have the index be generated by CI. We already have something similar in https://github.com/bottlesdevs/components where a GitHub Action generates all the indexes.

Maybe the script in #413 could serve as a base for writing the CI script, but I've never used GH Actions, so I don't know how that works.

[jntesteves]

Actually, just having a script in the repository and allowing contributors to run it instead of manually editing the index may be required anyway because you need to regenerate the index to be able to test your changes locally, so users would have to do it anyway.

So, a CI Action isn't really required. We just need to finish the script, move all fields to the individual files, and regenerate the index.
Edit: …and update contributing documentation.

[Arcitec]

@jntesteves Good points.

I checked the performance for loading 10 000 manifest files, with and without the C-based loader:

Execution time for 10,000 calls with CLoader=True: 1.0478 seconds
Execution time for 10,000 calls with CLoader=False: 6.8975 seconds

Bottles uses the C-based loader (rather than the pure Python loader) by the way.

Bottles-Next should be even faster since it uses Rust.

Benchmark program:

import time
from pathlib import Path
from typing import Any

import yaml


SCRIPT_DIR = Path(__file__).resolve().parent


def read_yaml(yml_file: Path, c_loader: bool) -> dict[str, Any]:
    with yml_file.open("rt", encoding="utf-8") as f:
        # Use the C-based or pure Python loaders on demand.
        data = yaml.load(f, Loader=yaml.CSafeLoader) if c_loader else yaml.safe_load(f)
        if data is None:
            raise ValueError(f'Empty YAML file: "{yml_file}".')

        return data

yml_file = SCRIPT_DIR / "Games/steam.yml"
count = 10_000

for c_loader in [True, False]:
    start_time = time.time()

    for _ in range(count):
        data = read_yaml(yml_file, c_loader)

    end_time = time.time()

    print(f"Execution time for {count:,} calls with CLoader={c_loader}: {end_time - start_time:.4f} seconds")

[Arcitec]

This means it takes around 0.1 milliseconds per manifest file that is being loaded.

The repository currently contains 36 manifests: find Games Software -iname "*.yml" | wc -l

Meaning that if we don't have an index, it takes 36 * 0.1 = 3.6 milliseconds to load the metadata from the manifests instead.

At 10 000 app manifests, we'd still only take 1 extra second to load the metadata from those 10k manifest files instead of an index.

At 1 000 app manifests, we'd take 0.1 seconds to load the metadata from the manifest files.

At our current 36 manifests, we're at 3.6 milliseconds to load the metadata from the manifests.

It seems unlikely that Bottles would even reach 500 manifests by the way. And even that high number would still only take 0.05 seconds to fetch metadata from the manifest files.

So I think we should skip the index for now. Alternatively (or later if it ever becomes a problem), implement the index as a entirely user-local cache file (built on startup by Bottles on the user's local machine) which contains a list of manifests, their file modification timestamps, and the cached data. Whenever a file is modified, added or deleted, the index would be rebuilt. But so far that seems like it's not really important since the lack of an index is not going to cause any noticeable delays until we get into having thousands of manifests. We're far away from that stage. :D

[jntesteves]

I was talking about download overhead only, which is already an issue in Bottles currently as it should reasonably be an offline-first app, instead of currently having a loading screen.

Of course the cost of parsing yaml is negligible, I wouldn't expect different.

[Arcitec]

Oh. I see that the programs are hosted at https://proxy.usebottles.com/repo/programs/:

https://github.com/bottlesdevs/Bottles/blob/de044ac00605b0cef30ffdca0b70d8f9967e3d07/bottles/backend/managers/repository.py#L47

I always thought that it just did a git pull of https://github.com/bottlesdevs/programs.git on startup to fetch the entire latest repo, since the first startup is really slow and looked like it was git cloning something.

Well, if it turns out that manifests are actually being individually fetched on-demand via normal cURL, then it makes sense to have an index. But with a script to auto-build the index, as you mentioned. And improving the manifests with the changes suggested here (moving all data into them, and adding the ability to mark manifests as hidden so that they won't be included in the index).

---

Edit: Of course, it might be better to actually make it a git clone --depth=1 https://github.com/bottlesdevs/programs.git shallow repository clone instead. That way the user has offline access to the installers. And we don't need an index, which simplifies development.

To do updates we'd use git pull --depth=1 to maintain the shallowness (quick download of the updates). Updates can be done asynchronously in the background at startup.

[Arcitec]

@mirkobrombin What are your thoughts on using git clone instead of the HTTP manifest proxy? Look at the speed of the commands in the previous message. It's very fast. Bottles could do that on the first startup. Then run the subsequent git pull on every other startup, which is even faster (practically instant, since it just compares the master REF id).

• Run one of those commands at startup asynchronously in the background. It's very fast.
• After the git command has finished (whether online or offline), allow access to the Installers feature.
• When the user clicks the Installers section or drops an event ".exe" on Bottles, do a quick indexing of all local manifests. Basically lazy local indexing.
• This technique results in an offline manifest index, which is a very desirable side effect since it finally lets people install software in Bottles without an internet connection (assuming they have the .exe file, when the "Event" .exe matching feature exists).
• No more proxy.usebottles.com hosting cost for you. :)
• No more need for any index maintenance.

There could be downsides, but I'm not seeing any. The purpose of "depth 1" is to just get the latest commit immediately without caring about git history, so it's very fast at syncing.