Support for large file upload in directory assets

[chr-pok]

Description

Currently, large file upload in directory assets does not support multi-part transfer and fails for files larger than a few GB. Since this functionality is required when uploading new circuits (e.g., human circuit in prod-human-data-and-models#29) or when extracting large subcircuits (e.g., long slice from nbS1 in prod-build-circuit#28), the upload functionality for directory assets should be extended to support large files.

Here an example from prod-build-circuit#28 (Job ID: 019db5bf-d6ce-74b4-9950-75c6f95d0a34):

entitysdk.exception.EntitySDKError: Uploading these files failed: {'morphologies/merged-morphologies.h5': 'https://entitycore-data-staging.s3.amazonaws.com/private/e6030ed8-a589-4be2-80a6-f975406eb1f6/2720f785-a3a2-4472-969d-19a53891c817/assets/circuit/1edc30fd-c785-49c3-ad2e-50851d07887a/sonata_circuit/morphologies/merged-morphologies.h5?AWSAccessKeyId=[REDACTED]&Signature=[REDACTED]&x-amz-security-token=[REDACTED]&Expires=1776904867'}

Technical implementation & specs

Proposed file size limit: 200GB (which is the max. disk space in the launch-system)

Acceptance criteria

• Directory assets with files up to 200GB can be uploaded

[evakennyobi]

Can you add a new start date for this task please @GianlucaFicarelli

[GianlucaFicarelli]

Moving now the ticket to in progress

[evakennyobi]

Should be in code review later today

[GianlucaFicarelli]

The changes are deployed to staging with entitycore 2026.5.4.
Needs entitysdk >= 0.16.0, but note that the function client.upload_directory() in this version isn't compatible with entitycore 2026.5.1 currently in production.

[chr-pok]

Hi @GianlucaFicarelli! Thanks for implementing this! Just wondering, are there any specific arguments to upload_directory() required to enable this new feature?

[mgeplf]

@chr-pok > [...] are there any specific arguments to [...]

You can now (openbraininstitute/entitysdk@916fb44#diff-3a8a79c8e7e607d0ca4e79ca03f1b78d3d6eeaff4f763ac31fa150dbe76b79ddR468) have an optional MultipartDirectoryUploadTransferConfig which allows you to control the parallelism, but otherwise it should be the same as before.

[chr-pok]

Thanks! I am now trying to use it and I noticed you have to import it through from entitysdk.schemas.asset import MultipartDirectoryUploadTransferConfig.

Would it be possible to add it to the __init__.py as well so we can import it directly from entitysdk (i.e., same way as MultipartUploadTransferConfig)?