Include wordnet in app

[cassidyjames]

• Wordbook version: 0.3.1
• Python version: unknown, but I'm using Wordbook via Flatpak
• Operating System: Endless OS

Description

Currently, Wordbook requires an Internet connection when first running in order to download word definitions. However, this limits its utility in offline environments, as you cannot redistribute just the app itself to offline environments. (For example, an offline dictionary app would be great on Endless OS for schools with limited or no Internet connectivity.)

Instead, it would be helpful to include the dictionary/wordnet data in the app itself, e.g. via the build system. For Flatpak specifically, I believe the data could be referenced in the manifest and downloaded on demand at build time, then bundled in the app.

[mufeedali]

Thank you for bringing this up. I will be doing this when I get around to working on Wordbook again. I just haven't had the time to.

[goodmami]

It should be possible to pre-build and distribute the Wn database, rather than building on the user's device. Wn's DB is just an SQLite file and is portable.

[mufeedali]

@goodmami Nice to see you here! Actually, it's perfect that you're here!

My current plan is something like this:

• I ship zst compressed versions of the db as here
• Packager or flatpak manifest picks up that and installs it into a system directory.
• Wordbook decompresses into a user directory on first launch (should take less than a seconds thanks to zstd).

Then, everything just works as before.

However, something I'm concerned about is distro packages like Arch. Imagine wn gets a new release 0.14.0 and that breaks schema compatibility (which is totally fair from a 0.x.x versioned library's perspective btw). Then, Arch (for example) ships 0.14.0 of wn, but Wordbook hasn't had a release yet, so it's still using the old db file. Currently, this isn't an issue because Wordbook just re-generates the db. But that wouldn't be possible anymore in a truly offline version of Wordbook, so the app would simply be broken until the package maintainer does something manually.

Realistically speaking, this is probably unsolvable and I should just submodule wn or something like that... Any other ideas?

[goodmami]

@mufeedali Good that you're thinking about this, because I'm (slowly) preparing a v1.0 release and it will likely break backward compatibility with some parts of the Python API and the database schema will change.

Why don't you just pin your dependency to a specific version of Wn?

Wordbook/pyproject.toml

Line 8 in 9598d99

"wn>=0.11.0",

I.e., instead of a >= operator, use ==. Each time you update the Wn dependency, you can build and package a new version of the database. Users updating your app would retrieve the new db alongside the new code.

Does this seem like a good plan to you?

After Wn v1.0 is released, you should feel more confident to use the ~= operator for compatible releases, at least regarding the Python API. You may still choose to pin to a specific version, however, because Wn may need to make database changes to accommodate new features in the WN-LMF schema. I hope in the future to have some way to do database migrations without a full rebuild.

[mufeedali]

Does this seem like a good plan to you?

It would be fine for anyone building from source and for the flatpak (since the python modules are bundled in). I'm just worried about introducing maintenance burden to packagers.

@yochananmarqos Sorry for pinging you here, but I'm curious about your opinion as a packager :)