GCS client library migration in Java SDK - part 1 #36876
Conversation
… of gcsio in GcsUtil
| import com.google.api.services.storage.model.StorageObject; | ||
| import com.google.auth.Credentials; | ||
| import com.google.auto.value.AutoValue; | ||
| import com.google.cloud.hadoop.gcsio.CreateObjectOptions; |
There was a problem hiding this comment.
References to com.google.cloud.hadoop.gcsio.* and com.google.cloud.hadoop.util.* have been removed from GcsUtil.java. They are kept in GcsUtilLegacy.java though during the migration process.
| } | ||
|
|
||
| @VisibleForTesting | ||
| void setCloudStorageImpl(GoogleCloudStorage g) { |
There was a problem hiding this comment.
While the two setCloudStorageImpl functions are no longer surfaced for testing in GcsUtil.java, we still keep them in GcsUtilLegacy.java.
| } | ||
| } | ||
|
|
||
| GoogleCloudStorage createGoogleCloudStorage( |
There was a problem hiding this comment.
This function is also removed from GcsUtil.java, but will remain in GcsUtilLegacy.java.
| * @return a SeekableByteChannel that can read the object data | ||
| */ | ||
| public SeekableByteChannel open(GcsPath path) throws IOException { | ||
| return open(path, this.googleCloudStorageOptions.getReadChannelOptions()); |
There was a problem hiding this comment.
Comparing with the previous implementation
we will invoke the open() function below to include the IO request counters similar to the create() functions below.
|
cc'ed @clairemcginty Context: gcsio is sunsetting (https://s.apache.org/beam-gcsutil-modernization), and we are on the way to migrate to the GCS client library for Beam Java SDK. |
| import org.apache.beam.runners.core.metrics.MonitoringInfoMetricName; | ||
| import org.apache.beam.sdk.extensions.gcp.auth.TestCredential; | ||
| import org.apache.beam.sdk.extensions.gcp.options.GcsOptions; | ||
| import org.apache.beam.sdk.extensions.gcp.util.GcsUtil.BatchInterface; |
There was a problem hiding this comment.
BatchInterface and RewriteOp are only used in Tests, so it is fine to get rid of them in GcsUtil and keep them in GcsUtilLegacy.
|
Assigning reviewers: R: @Abacn for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
1 participant
This PR lays the groundwork for the GCS client library migration in the Java SDK. It refactors the existing GCS client implementation to prepare for a new implementation while maintaining backward compatibility.
Key changes include:
GcsUtil.java.GcsUtilLegacy.javaandGcsOptions.javawill be removed in a future PR.openAPI in the legacy module has been rerouted to include previously implemented but unused IO request count metrics, as discussed in [BEAM-11980] Java GCS - Implement IO Request Count metrics #15394 (comment).