IPED BFAC#2836
Open
fmpfeifer wants to merge 32 commits into
Open
Conversation
…file upload - Create new iped-bfac subproject with Maven configuration - Implement BfacDialog for user authentication and submission management - Add BfacApiClient for REST API communication with BFAC backend - Login/logout with token management - Session validation and renewal - Batch hash submission (MD5, SHA-1, SHA-256, ED2K) - Segmented file upload with resume support - Implement SubmissionWorker for background processing - Efficient bookmark-based item collection using IPEDSearcher - Progress tracking for both hash and file upload phases - Add BfacConfig for credential storage - Integrate BFAC menu option in iped-app MenuClass/MenuListener - Add localization strings for all supported languages
…pload interruption - Convert BfacDialog to singleton with getInstance() method - Change dialog from modal to non-modal for better user experience - Add window closing handler with confirmation when upload is in progress - Implement showDialog() method to restore and focus existing instance - Add static confirmApplicationClose() method for application-level checks - Integrate upload progress check in App.windowClosing() event
…arate files - Move LoginResult, ValidationResult, SendHashResult, SubmissionResult, and FileUploadStatus from BfacApiClient inner classes to standalone classes in iped.bfac.api package - Update imports in BfacDialog and SubmissionWorker to reference new class locations - Improve code organization and maintainability by separating API result types
…ad operations - Add TOKEN_RENEWAL_INTERVAL_MS constant (12 hours) for periodic token refresh - Implement renewTokenIfNeeded() method with force parameter for token validation - Track last token renewal time with lastTokenRenewalTime field - Force token renewal before starting file upload phase - Add periodic token renewal check during file segment uploads - Import LoginResult class for token renewal response handling
…d of dialog itself - Change showDialog() to restore parent frame state when iconified
…ication exit - Change setDefaultCloseOperation to DO_NOTHING_ON_CLOSE to ensure window closing handler is always invoked - Allows proper cleanup and confirmation dialogs (e.g., BFAC upload progress check) before application exit
…ion expiration recovery - Add onAuthenticationError() callback method to SubmissionCallback interface for handling 401 errors - Implement authentication error detection in SubmissionWorker with authenticationError flag - Add HTTP status code tracking to API result classes (SendHashResult, SubmissionResult, FileUploadStatus) - Add isUnauthorized() helper methods to check for 401 status codes - Implement session expiration dialog
… naming convention - Change Maven name from "IPED BFAC Integration" to "iped-bfac" to align with other IPED subprojects
- Add ActionListener to passwordField to trigger login on ENTER key press
- Change uploadFilesCheckBox constructor to initialize with checked state (true)
- Add BFACConfig.txt configuration file with baseUrl property - Load base URL from config file in BfacConfig class - Search for config in conf/, ../conf/, and classpath - Update BfacApiClient to use configured URL from BfacConfig - Fall back to default localhost:8000 if config not found
- Add USER_AGENT constant with IPED version and extension info - Create newRequestBuilder() helper for common headers - Create newAuthenticatedRequestBuilder() for authenticated requests - Replace all HttpRequest.newBuilder() calls with helper methods - Ensures consistent User-Agent across all API endpoints
- Change serverUrlField from JTextField to JLabel - Display configured URL from BfacConfig instead of editable field - Remove server URL modification logic from onLogin() - Remove serverUrlField enable/disable calls during login - Style URL label with monospaced font and dark gray color - Reorder imports for consistency
- Update default baseUrl from http://10.61.86.217:8000/ to https://10.61.86.217:443/ - Add createHttpClient() method to configure SSL context that trusts all certificates - Import SSL-related classes (SSLContext, TrustManager, X509TrustManager) - Add exception handling for SSL configuration failures with fallback to default client - Replace direct HttpClient.newBuilder() calls with createHttpClient() helper
- Add enableBFACClient property to IPEDConfig.txt with default false - Create BFACClientConfig class to read enableBFACClient and baseUrl properties - Move baseUrl configuration from BFACConfig.txt to BFACConfig.txt in conf/ - Update BfacApiClient to read baseUrl from BFACClientConfig instead of BfacConfig - Remove baseUrl loading logic from BfacConfig class - Add BFACClientConfig to Configuration manager - Conditionally show
- Add static block to set jdk.internal.httpclient.disableHostnameVerification property - Property must be set before HttpClient class is loaded - Allows connections to HTTPS servers with mismatched hostnames
…C client - Add maxConcurrentUploads property to BFACConfig.txt with default value of 5 - Create FileUploadResult class to communicate upload task results - Create FileUploadTask as Callable for parallel file uploads in thread pool - Refactor SubmissionWorker to use ExecutorService for concurrent uploads - Add shared atomic counters and flags for progress tracking across threads - Implement thread-safe callbacks for logging and token
- Add progressCallback parameter to FileUploadTask constructor - Call progressCallback after uploading each segment in upload loop - Call progressCallback when skipping already-uploaded files - Call progressCallback when resuming uploads from startOffset - Move progress calculation from SubmissionWorker result processing to callback - Create progressCallback lambda in SubmissionWorker that updates progress bar - Pass progressCallback to FileUploadTask instances
…cessing - Replace List<Future<FileUploadResult>> with ExecutorCompletionService - Submit tasks via completionService.submit() instead of collecting futures - Track submittedCount to know how many results to collect - Use completionService.take() to get results in completion order - Process results as they complete instead of in submission order
Implement batch upload functionality to group multiple small files into a single HTTP request, reducing overhead and improving upload performance. Add getUploadStatusBatch() to query status for multiple files at once and uploadFileSegmentsBatch() to upload segments for multiple files in one multipart request. Modify SubmissionWorker to group eligible small files based on upload status and size limits before submitting individual upload
Add syncCategoryWithSelectedSubmission() to automatically update the category dropdown when selecting an existing submission. This ensures the category matches the selected submission's category when switching between submissions or loading existing submissions. Add action listener to existingSubmissionComboBox to trigger sync on selection changes.
…itialization Relocate the addActionListener() call for existingSubmissionComboBox to immediately after setting its enabled state. This ensures the listener is attached after all component properties are configured, preventing potential issues with event handling during initialization.
Pass the FileUploadStatus obtained from batch queries directly to FileUploadTask instead of making individual getUploadStatus() calls. Change filesToUpload map to LinkedHashMap to preserve insertion order for consistent batch processing. Add initialStatus parameter to FileUploadTask constructor to accept pre-fetched status, falling back to individual API call only when batch status is unavailable.
Add validation to skip FileHashInfo entries with empty names or non-positive file sizes before processing. Track and report the count of skipped invalid entries to inform users about items excluded from hash submission due to missing or invalid metadata.
Member
|
Thank you very much @fmpfeifer! @wladimirleite, since this is related to a task you have been done for more than one decade, when you have available time, could you help reviewing this? |
Member
|
I used the BFAC client interface to upload two sets of files successfully.
|
Member
|
I am uploading 3 more submissions, each with multiple files. Another question: what is the expected way to report misclassified files (a.k.a. false positives)? |
Member
|
A few minor suggestions (during the upload):
|
Member
Author
|
Thank you for the review, @wladimirleite. Regarding the false positives, the current approach is to submit the hashes as "Non-ASI". This isn’t ideal, and I agree we should look into a better solution. Thanks again |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR integrates the iped-bfac module into IPED, enabling BFAC submission workflows directly from the application.
At a high level, it adds support for: