Skip to content

Conversation

@kleeedolinux
Copy link

When submitting this pull request, I confirm the following (please check the boxes):

  • I have read and understood the Contributor Guidelines.

  • I have checked that there are no duplicate pull requests related to this request.

  • I have considered, and confirm that this submission is valuable to others.

  • I accept that this submission may not be used and the pull request may be closed at the discretion of the maintainers.

Fill in the PR content:

Summary

This PR refactors the Python RPC server to use async/await architecture, significantly improving performance, reducing memory and CPU usage, and enabling better concurrent request handling.

Changes

Framework Migration

  • Replaced Flask with Quart: Migrated from synchronous Flask to async-compatible Quart framework, maintaining API compatibility while enabling async request handling
  • Added uvloop support: Integrated uvloop event loop policy for Linux/Mac systems, providing up to 2-4x performance improvement over the default asyncio event loop

Async Conversion

  • HTTP Downloaders: Converted HttpDownloader and HttpMultiLinkDownloader to fully async operations using asyncio.to_thread() for blocking I/O
  • Torrent Downloader: Refactored TorrentDownloader to use async methods with proper thread pool execution
  • Image Processor: Made ProfileImageProcessor async with proper context managers for memory-efficient image handling

Performance Optimizations

  • Concurrent Operations: Implemented asyncio.gather() for parallel status checks in multi-link downloads
  • Optimized Locking: Improved lock usage to prevent serialization of concurrent operations
  • Lazy Initialization: Added lazy initialization for aria2p API clients to reduce startup overhead
  • Memory Optimization:
    • Converted trackers list to immutable tuple
    • Proper cleanup of completed downloads
    • Context managers for image processing

Code Quality Improvements

  • Type Hints: Added comprehensive type hints throughout all modules
  • Error Handling: Improved exception handling with silent failures where appropriate
  • Resource Management: Better cleanup and resource management patterns

Benefits

  • Performance: Significantly improved request handling throughput with async architecture
  • Resource Usage: Reduced memory footprint and CPU usage through optimizations
  • Scalability: Better handling of concurrent requests and downloads
  • Maintainability: Improved code quality with type hints and better structure

Testing

  • All existing endpoints maintain API compatibility
  • Async operations properly handle concurrent requests
  • Memory usage optimized for long-running operations

Dependencies

  • Updated requirements.txt:
    • Replaced flask with quart
    • Added uvloop (Linux/Mac only, conditional import)

Files Changed

  • python_rpc/http_downloader.py - Full async conversion
  • python_rpc/http_multi_link_downloader.py - Async with concurrent status checks
  • python_rpc/torrent_downloader.py - Async operations with optimized trackers
  • python_rpc/profile_image_processor.py - Async image processing
  • python_rpc/main.py - Complete rewrite using Quart with uvloop
  • requirements.txt - Updated dependencies

@sonarqubecloud
Copy link

@thegrannychaseroperation
Copy link
Contributor

@kleeedolinux apologies for the huge delay on this, but could you please fix the conflicts? I'll be testing it ASAP, this looks great. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants