MONGOCRYPT-763 add in-place retry API #1011
Conversation
Co-authored-by: Kevin Albertson <[email protected]>
| d. Feed the reply back with `mongocrypt_kms_ctx_feed`. Repeat | ||
| > until `mongocrypt_kms_ctx_bytes_needed` returns 0. | ||
| d. Feed the reply back with `mongocrypt_kms_ctx_feed` or `mongocrypt_kms_ctx_feed_with_retry`. Repeat | ||
| > until `mongocrypt_kms_ctx_bytes_needed` returns 0. If the `should_retry` outparam returns true, |
There was a problem hiding this comment.
| > until `mongocrypt_kms_ctx_bytes_needed` returns 0. If the `should_retry` outparam returns true, | |
| until `mongocrypt_kms_ctx_bytes_needed` returns 0. If the `should_retry` outparam returns true, |
| return true; | ||
| } | ||
|
|
||
| bool mongocrypt_kms_ctx_feed_with_retry(mongocrypt_kms_ctx_t *kms, mongocrypt_binary_t *bytes, bool *should_retry) { |
There was a problem hiding this comment.
| bool mongocrypt_kms_ctx_feed_with_retry(mongocrypt_kms_ctx_t *kms, mongocrypt_binary_t *bytes, bool *should_retry) { | |
| bool mongocrypt_kms_ctx_feed_with_retry(mongocrypt_kms_ctx_t *kms, mongocrypt_binary_t *bytes, bool *should_retry) { | |
| BSON_ASSERT_PARAM(kms); | |
| BSON_ASSERT_PARAM(bytes); | |
| BSON_ASSERT_PARAM(should_retry); |
Suggest asserting required args are non-NULL to abort early. I expect the abort would only occur due to a driver bug (not during normal execution).
| @@ -1180,7 +1180,25 @@ MONGOCRYPT_EXPORT | |||
| bool mongocrypt_kms_ctx_feed(mongocrypt_kms_ctx_t *kms, mongocrypt_binary_t *bytes); | |||
|
|
|||
| /** | |||
| * Indicate a network-level failure. | |||
| * Feed bytes from the HTTP response. | |||
There was a problem hiding this comment.
| * Feed bytes from the HTTP response. | |
| * Feed bytes from the KMS response. |
Notably: KMIP is not HTTP. Suggest also updating the comment in mongocrypt_kms_ctx_feed to match.
| // Expect no sleep is requested before any error. | ||
| ASSERT_CMPINT64(mongocrypt_kms_ctx_usleep(kms_ctx), ==, 0); | ||
| // Feed a retryable HTTP error. | ||
| ASSERT_OK(mongocrypt_kms_ctx_feed_with_retry(kms_ctx, TEST_FILE("./test/data/rmd/kms-decrypt-reply-429.txt"), &should_retry), kms_ctx); |
There was a problem hiding this comment.
Run ./etc/format-all.sh to fix the check-format Evergreen task.
| if (kms->should_retry) { | ||
| // This happens when a KMS context is reused in-place | ||
| kms->should_retry = false; | ||
| } |
There was a problem hiding this comment.
With mongocrypt_kms_ctx_feed_with_retry, I expect it would be a driver bug to call mongocrypt_kms_ctx_feed directly on a KMS context needing retry. Calling mongocrypt_kms_ctx_feed suggests the driver is not doing in-place retry.
Consider making this an error:
if (kms->should_retry) {
CLIENT_ERR ("KMS context needs retry. Call mongocrypt_kms_ctx_feed_with_retry instead");
return false;
}And setting kms->should_retry to false in mongocrypt_kms_ctx_feed_with_retry.
|
|
||
| If any step encounters a network error, call `mongocrypt_kms_ctx_fail`. | ||
| If `mongocrypt_kms_ctx_fail` returns true, continue to the next KMS context. | ||
| If `mongocrypt_kms_ctx_fail` returns true, retry the request by continuing to the next KMS context or by feeding the new response into the same context. | ||
| If `mongocrypt_kms_ctx_fail` returns false, abort and report an error. Consider wrapping the error reported in `mongocrypt_kms_ctx_status` to include the last network error. | ||
|
|
There was a problem hiding this comment.
The note below might be confusing, since it mentions fanning out requests with the older retry behavior:
Note, the driver MAY fan out KMS requests in parallel. More KMS requests may be added when processing responses to retry.
Consider adding a section describing retry and iteration. Idea:
Retry and Iteration
Retry behavior is enabled by calling
mongocrypt_setopt_retry_kms.There are two options for retry:
- Lazy retry: After processing KMS contexts, iterate again by calling
mongocrypt_ctx_next_kms_ctx. KMS contexts needing a retry will be returned.- In-place retry: If a KMS context indicates retry, retry the KMS request and feed to the response to the same KMS request. Use
mongocrypt_kms_ctx_feed_with_retryand check the return ofmongocrypt_kms_ctx_failto check if a retry is indicated.The driver MAY fan out KMS requests in parallel. It is not safe to iterate KMS contexts (i.e. call
mongocrypt_ctx_next_kms_ctx) while operating on KMS contexts (e.g. callingmongocrypt_kms_ctx_feed). Drivers are recommended to do an in-place retry on KMS requests.
Add in-place retry API to better support drivers that fan out KMS requests.