Implement reusable dataset downloder

[kurotych]

Linked PR: https://github.com/novalabsxyz/mobile-rewards-estimator/pull/30

[Copilot]

Pull Request Overview

This pull request implements a reusable dataset downloader and integrates it into various parts of the mobile_verifier project. Key changes include updating database transaction handling in dataset assignment, refactoring integration tests to use TaskManager with the new daemon, and introducing the new dataset_downloader package with corresponding dependency updates.

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
mobile_verifier/tests/integrations/common/mod.rs	Uses a transaction for dataset assignment save operations
mobile_verifier/tests/integrations/boosting_oracles.rs	Refactors the dataset downloader integration with TaskManager and updates poll duration
mobile_verifier/src/rewarder.rs	Adjusts the import to use the dataset_downloader's check_for_unprocessed_data_sets
mobile_verifier/Cargo.toml	Adds a new dependency for dataset_downloader
dataset_downloader/tests/downloader_test.rs	Introduces tests covering new dataset downloader functionalities
dataset_downloader/Cargo.toml	New package manifest for the dataset_downloader
Cargo.toml	Updates workspace members and adds dependencies for regex and async-compression

Comments suppressed due to low confidence (2)

mobile_verifier/tests/integrations/boosting_oracles.rs:147

• [nitpick] Consider using a more descriptive variable name instead of 'dsdd' to improve code clarity.

let dsdd = DataSetDownloaderDaemon::new(

mobile_verifier/tests/integrations/boosting_oracles.rs:134

• Verify that a 25ms polling duration is sufficient under expected load and does not introduce flakiness in asynchronous tests.

let poll_duration = std::time::Duration::from_millis(25);

[Copilot]

[nitpick] Consider specifying a more explicit version for async-compression to prevent potential dependency resolution issues.

Suggested change

	async-compression = { version = "0", features = ["tokio", "gzip"] }
	async-compression = { version = "0.3.8", features = ["tokio", "gzip"] }