Bulk job renaming

[jtherrmann]

We want to support bulk job renaming. Specifically, Vertex should not have to make one API request per job, as this results in potentially thousands of API requests for bulk renaming operations. In discussions with @williamh890, we decided that the new API endpoint should accept a list of job IDs and a new name value. I recommended that we use one TransactWriteItems call per HyP3 API request, so that our API endpoint is guaranteed to be transactional. I believe I incorrectly stated that this would limit us to 25 items per request, but it's actually 100.

Unfortunately, I'm not sure TransactWriteItems is appropriate for bulk item updates. Efficient bulk operations says of TransactWriteItems:

Trade-off – Additional throughput is consumed, 2 WCUs per 1 KB write.

where WCU is a write capacity unit (see here). I'm not sure whether this means that TransactWriteItems updates are twice as slow, or just twice as expensive (billed at twice the cost in AWS). I think it would at least make it more likely that we get throttled, which would result in the operation being slower.

That page also lists BatchExecuteStatement with PartiQL for updating 25 items, though I'm not sure how much efficiency we'd gain or if it would be worth the additional code complexity.

There's also BatchWriteItem for writing 25 items, but does not support updates (only overwrites existing items) and that page recommends UpdateItem instead.

I'm leaning toward just looping through and calling UpdateItem one item at a time. It seems like this should work for renaming thousands of jobs, not sure beyond that. I'm also not sure if we have to manually handle throttling (see the Errors section), but as far as I know we don't worry about this when submitting thousands of jobs, so it's probably fine?

I think the main issue with an update loop is that the HyP3 API request won't be transactional, so we'll have to report partial updates somehow and the client will have to handle that. But I don't think we can get around that if we're not using TransactWriteItems? (Is PartiQL's BatchExecuteStatement transactional?)

[jtherrmann]

Started an implementation at #2970

[jtherrmann]

If the main goal is reduce the number of HyP3 API requests that Vertex has to make, and we don't care as much about efficiency for large updates, or about making the API request transactional, I suppose we could go with the update loop implementation and just limit the number of job IDs per request to something that should always complete in a reasonable amount of time (e.g. a few hundred items). Then if the API request returns an error, the client just retries the request, with the understanding that some of the items may have already been updated, but it's fine to update the entire batch again.

[jtherrmann]

After discussion with @asjohnston-asf it sounds like throttling is probably not a big concern for TransactWriteItems given our table configuration. However, the implementation is still simpler with an update loop that uses our current dynamo.jobs.update_job_for_user function, so I think I'll stick with that for now and we can revisit TransactWriteItems if non-transactional updates are causing problems.

[jtherrmann]

Fixed by #2984