out_bigquery: add documentation for configurable buffer_size option #1915
Conversation
|
Can you resolve the DCO requirement as that is required to be able to merge? |
pipeline/outputs/bigquery.md
Outdated
| @@ -60,6 +60,7 @@ You must configure workload identity federation in GCP before using it with Flue | |||
| | provider\_id | GCP workload identity provider. Used to construct the full resource name of the identity provider. Currently only AWS accounts are supported. | | | |||
| | google\_service\_account | Email address of the Google service account to impersonate. The workload identity provider must have permissions to impersonate this service account, and the service account must have permissions to access Google BigQuery resources (e.g. `write` access to tables) | | | |||
| | workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | | |||
| | buffer_size | Sets the HTTP client buffer size (in bytes) for reading API responses from BigQuery. | `4192` | | |||
There was a problem hiding this comment.
Is this a fixed size, the lower or upper bound?
There was a problem hiding this comment.
Thanks for the review!
The buffer_size is a fixed-size allocation for the HTTP client buffer. It is not a lower or upper bound. If a response exceeds this size, the plugin logs a warning.
We’ve updated the documentation to clarify this.
tsubaron
force-pushed
the
out_bigquery-buffer-size-configurable
branch
2 times, most recently
from
0a097fe to
c626388
Compare
pipeline/outputs/bigquery.md
Outdated
| @@ -60,6 +60,7 @@ You must configure workload identity federation in GCP before using it with Flue | |||
| | provider\_id | GCP workload identity provider. Used to construct the full resource name of the identity provider. Currently only AWS accounts are supported. | | | |||
| | google\_service\_account | Email address of the Google service account to impersonate. The workload identity provider must have permissions to impersonate this service account, and the service account must have permissions to access Google BigQuery resources (e.g. `write` access to tables) | | | |||
| | workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | | |||
| | buffer_size | Sets the HTTP client buffer size (in bytes) for reading API responses from BigQuery. This is a fixed-size allocation; responses larger than this size will trigger a warning. | `4192` | | |||
There was a problem hiding this comment.
Well, I presume the warning is that they won't be handled right or is this saying it will auto-grow?
There was a problem hiding this comment.
I initially thought the buffer was fixed and allocated based on the provided size, but after checking the internal logic, it turns out the client dynamically increases the buffer as needed.
Here is the part where it grows the buffer, and just below it, you can see that if the allocation fails, a warning is logged and an error is returned:
https://github.com/tsubaron/fluent-bit/blob/out_bigquery-buffer-size-configurable/src/flb_http_client.c#L1470
I will update the documentation to reflect this behavior.
tsubaron
force-pushed
the
out_bigquery-buffer-size-configurable
branch
from
c626388 to
751b7b8
Compare
Signed-off-by: Tsubasa Miyamoto <[email protected]>
tsubaron
force-pushed
the
out_bigquery-buffer-size-configurable
branch
from
751b7b8 to
e71cf43
Compare
Add documentation for the
buffer_sizeoption in the BigQuery output plugin.This complements fluent/fluent-bit#10596