Replace aria2c with libtorrent

[kelson42]

We want to use more BitTorrent to allow Kiwix users to download and share efficiently ZIM files (and reduce our bandwidth costs).

Traditionally the plan has been to rely on Metalink/Mirrorbrain and aria2c to do that. But:

• It has been implemented only on Kiwix-Desktop/libkiwix because we had difficulties with libaria2c (aria2c binary instrumentation)
• BiTorrent Seeding has not been implemented
• Server side BitTorrent infrastructure (superseeder + Tracker) was not ready

Now we have the server side infrastructure and we clearly need to have a library to handle BitTorrent download/seeding. We could probably use libaria2c but:

• libaria2c is not as easy as libtorrent.
• libaria2c supports metalinks which is ultimatively maybe not something we need anymore because of Webseeds

This issue superseeds #1172

[aviralgarg05]

I would like to solve this issue

[kelson42]

@aviralgarg05 Thank you for proposing yourself. Would you be able to describe how you plan to do that?

[aviralgarg05]

Plan: Replace aria2c with libtorrent for BitTorrent Downloads

Replace the current aria2c binary integration with libtorrent to enable efficient BitTorrent downloads and seeding of ZIM files, improving the download infrastructure while simplifying the library implementation.

Steps:

Remove aria2c dependencies - Delete [src/aria2.cpp], [src/aria2.h], update [src/meson.build] to remove aria2.cpp, and remove aria2 runtime dependency from documentation.

Create libtorrent wrapper - Implement new src/libtorrent.cpp and src/libtorrent.h with similar interface to Aria2 class (methods: addUri, tellStatus, tellActive, tellWaiting, pause, unpause, remove, saveSession, shutdown) plus new seeding functionality.

Update Downloader class - Modify [src/downloader.cpp]) and [include/downloader.h] to replace shared_ptr with shared_ptr, maintaining the same public API for Download and Downloader classes.

Add seeding support - Extend Download class with seeding-related methods (startSeeding, stopSeeding, getSeedingStatus) and update status tracking to include seeding states.

Update build system - Add libtorrent dependency to [meson.build], update [README.md] dependencies section, and ensure all platform-specific build configurations work.

Update tests - Modify download-related tests to work with libtorrent, add new tests for seeding functionality, and ensure all existing download workflows continue working.

Open Questions:

1. Session persistence format - Should we maintain aria2 session file format for backward compatibility, or migrate to libtorrent's format?
2. Metalink support - The current implementation supports metalinks through aria2. Do we still need this with webseeds, or remove it entirely?
3. Configuration options - Which aria2 options (DHT, LPD, port ranges, etc.) should map to libtorrent settings?

[kelson42]

@aviralgarg05 Thank you for your proposal.

First the answers to your questions:

Session persistence format - Should we maintain aria2 session file format for backward compatibility, or migrate to libtorrent's format?

no

Metalink support - The current implementation supports metalinks through aria2. Do we still need this with webseeds, or remove it entirely?

no, our .torrent files have webseeds (and it has to work as we can not assume to have (super-)seeder). We would definitely abandon the Metalink file.

Configuration options - Which aria2 options (DHT, LPD, port ranges, etc.) should map to libtorrent settings?

Difficult to tel you quickly about all the options. We can assume that all the features used by aria2c should be used. But probably this should be re-challenged one by one.

Here my remarks:

• Please don't touch anything related to aria2c, this will be removed once the new Bittorrent feature is fully working
• Focus should be to have as quickly as possible a MVP (then we will implement the rest)
• The "project" should be splitted in dedicated issues and PR, one per task.

I have taken your proposal of agenda/task and tweaked it:

• Add libtorrent dependency to libkiwix meson.build, update README.md dependencies section, and ensure all platform-specific build configurations work. This might be a bit challenging at first, as part of the job is in Python in kiwix-build (but we will help you on this).
• Implement new src/bittorrent.cpp and src/bittorrent.h with similar interface to Aria2 class (methods: addUri, tellStatus, tellActive, tellWaiting, pause, unpause, remove, saveSession, shutdown). Update downloader.cpp
• Modify download-related tests to work with bittorrent/libtorrent, add new tests for seeding functionality, and ensure all existing download workflows continue working.
• Update Kiwix-desktop to use the new BitTorrent downloading capacity
• Add seeding support - Extend Download class with seeding-related methods (startSeeding, stopSeeding, getSeedingStatus) and update status tracking to include seeding states.
• Update Kiwix-desktop (more settings related to the seedings)
• Remove aria2c dependencies - Delete [src/aria2.cpp], [src/aria2.h], update [src/meson.build] to remove aria2.cpp, and remove aria2 runtime dependency from documentation.

Warning: We don't allow IA generated work straight. Use whatever tools who can help you but if we have the feeling we have to review the AI instead of reviewing your work, we will just close the PR. My recommendation is: better don't use it OR you know exactly what your are doing!

@veloman-yunkan @aviralgarg05 Does this sounds reasonable? This project is quite a bit of work!

[aviralgarg05]

Cool!, I dont use AI that much to code, so will keep everything clean and straight. And make sure to complete it perfectly and accurately without any problems

[kelson42]

@aviralgarg05 Great, I have created the first task to start with. This one should be pretty easy ;)

[chapmanjacobd]

One thing that you'll want to test carefully, or perhaps the first thing you'll want to test before proceeding with development, is support of webseeds.

A few weeks ago I tested GetRight-style torrents across qBittorrent-nox (which uses libtorrent), Transmission-cli, and aria2c. aria2c is the only one that seems to handle webseeds with HTTP redirects--unseeded Internet Archive torrents only worked on aria2/aria2c.

Maybe it's specific to something else that IA is doing, eg. cross domain redirects. But it seems like this could be a blocker, just FYI.

[kelson42]

One thing that you'll want to test carefully, or perhaps the first thing you'll want to test before proceeding with development, is support of webseeds.

A few weeks ago I tested GetRight-style torrents across qBittorrent-nox (which uses libtorrent), Transmission-cli, and aria2c. aria2c is the only one that seems to handle webseeds with HTTP redirects--unseeded Internet Archive torrents only worked on aria2/aria2c.

Maybe it's specific to something else that IA is doing, eg. cross domain redirects. But it seems like this could be a blocker, just FYI.

Indeed, webseed is a sensitive topic. Will do that at second task.