Files uploading via JSON Requests

[ibilux]

Hello @ignatz,

I feel a bit lost here. When uploading a file via JSON requests, one working approach looks like this:

// Get file from input element or file picker    
const fileInput = document.querySelector('input[type="file"]') as HTMLInputElement;    
const file = fileInput.files?.[0]; // This is a File object    
    
if (file) {  
  // Convert File to ArrayBuffer, then to Uint8Array  
  const arrayBuffer = await file.arrayBuffer();  
  const uint8Array = new Uint8Array(arrayBuffer);  
    
  await api.create({    
    // other fields...    
    file_column: {    
      filename: file.name,    
      content_type: file.type,    
      mime_type: file.type,    
      data: Array.from(uint8Array) // Convert Uint8Array to regular array  
    }    
  });    
}

But I recall you previously mentioning that files could also be uploaded via base64. Was that an actual feature, or just a suggestion/idea?

For example, changing the FileUploadInput struct to use #[serde(with = "base64")] would allow Serde to automatically decode base64 strings into a Vec<u8> during JSON deserialization (or use a serde custom deserializer?):

#[derive(Debug, Default, Clone, PartialEq, Serialize, Deserialize)]  
#[serde(deny_unknown_fields)]  
pub struct FileUploadInput {  
  pub name: Option<String>,  
  pub filename: Option<String>,  
  pub content_type: Option<String>,  
  #[serde(with = "base64")]  
  pub data: Vec<u8>,  
}

This seems neat, but it would be a breaking change since existing code currently expects the data field to be Vec<u8>.

Just to confirm, is base64 upload currently supported, or if it was mentioned more as a potential alternative approach we could consider?

Thank you,

[ignatz]

But I recall you previously mentioning that files could also be uploaded via base64. Was that an actual feature, or just a suggestion/idea?

Let's call it brain rot from jumping domains too often. The intention was there and wishful thinking that I had already come around to it :hide:

This seems neat, but it would be a breaking change since existing code currently expects the data field to be Vec.

Right. I suspect this would also break efficient multi-part form encoding.

I do think it makes sense to use base64 in the JSON case, it just feels consistent and leads to more density (note that everything else requires URL-safe base64 encoding, which may be costly for larger files). I took a first stab at how conditional decoding could look like (it also falls back to standard base64 if url-safe fails). WDYT?

I should really add some e2e integration tests for uploads in both formats (just a bit strapped for time)

Thanks for keeping me honest 🙏

[ignatz]

Did you have a PR out before that I missed? I'm very sorry. I very much appreciate your effort 🙏

[ibilux]

Did you have a PR out before that I missed? I'm very sorry. I very much appreciate your effort 🙏

Your implementation is much cleaner. #154 includes file upload tests for the Rust and TS clients, which might come in handy.

[ignatz]

Awesome! I merged your tests: b4f4a20. Thanks so much 🙏

[ignatz]

Building on your foundation and following your example, I also added a test for multipart forms: e618d0f. We should be in a much better place now. Thank you 🙏

[ibilux]

Thank you @ignatz for the insight and swift fix 🙏 I really appreciate you making the time for this when you already have so much on your plate.