feat: Add object storage support for PDF and Image extraction - #88
Conversation
Refactors /extract_pdf and /extract_text endpoints to support processing files from S3-compatible object storage (S3, Minio, OCI). Introduces tenant configuration via tenants.json and a storage service using boto3. Co-authored-by: gomesrocha <3893269+gomesrocha@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
Caution Review failedThe pull request is closed. 📝 WalkthroughWalkthroughThis pull request introduces multi-source file extraction capabilities for both PDF and text extraction endpoints, adding support for retrieving files from AWS S3-compatible object storage alongside direct file uploads. It includes new tenant configuration management, an S3 storage service module, and comprehensive test coverage for the new functionality. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant API as API Endpoint
participant TenantMgr as Tenant Config
participant Storage as Storage Service
participant S3 as S3 Object Storage
participant OCR as OCR Processor
participant Response
Client->>API: POST /extract_pdf (source='object_storage', client_id, object_key)
API->>TenantMgr: get_tenant_config(client_id)
TenantMgr-->>API: TenantConfig
API->>Storage: download_file_from_storage(client_id, object_key)
Storage->>S3: Download file using tenant credentials
S3-->>Storage: PDF file
Storage->>Storage: Validate PDF header
Storage-->>API: temp_file_path
API->>OCR: process_pdf(temp_file_path)
OCR-->>API: extracted_text
API->>Storage: Cleanup temp file
API-->>Response: TextExtractDocument
Response-->>Client: HTTP 200
sequenceDiagram
participant Client
participant API as API Endpoint
participant Validation as Input Validation
participant FileHandler as File Handler
participant OCR as OCR Processor
participant Response
Client->>API: POST /extract_pdf (source='upload', input_file)
API->>Validation: Check source='upload' & input_file present
Validation-->>API: Validation passed
API->>FileHandler: Validate PDF header & save to temp
FileHandler-->>API: temp_file_path
API->>OCR: process_pdf(temp_file_path)
OCR-->>API: extracted_text
API->>FileHandler: Cleanup temp file
API-->>Response: TextExtractDocument
Response-->>Client: HTTP 200
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| region_name=config.region_name | ||
| ) | ||
|
|
||
| s3_client.download_fileobj(config.bucket_name, object_key, tmp_file) |
Check failure
Code scanning / SonarCloud
S3 operations should verify bucket ownership using ExpectedBucketOwner parameter High
| return tmp_file.name | ||
|
|
||
| except ClientError as e: | ||
| log.error(f"S3 ClientError downloading {object_key}: {e}") |
Check notice
Code scanning / SonarCloud
Logging should not be vulnerable to injection attacks Low
| log.error(f"S3 ClientError downloading {object_key}: {e}") | ||
| raise e | ||
| except Exception as e: | ||
| log.error(f"Unexpected error downloading {object_key}: {e}") |
Check notice
Code scanning / SonarCloud
Logging should not be vulnerable to injection attacks Low
2 participants
Refactors /extract_pdf and /extract_text endpoints to support processing files from S3-compatible object storage (S3, Minio, OCI). Introduces tenant configuration via tenants.json and a storage service using boto3.
PR created automatically by Jules for task 7752691853998699012 started by @gomesrocha
Summary by CodeRabbit