Fix bigquery cdc storage api issue 31422 #36877
Conversation
This commit adds support for escaping and unescaping glob wildcard characters
in file path specifications, addressing the issue where files with literal
glob metacharacters (*, ?, {, }) in their names cannot be matched.
Changes:
- Added escapeGlobWildcards(String spec) method to escape glob metacharacters
by prefixing them with backslash
- Added unescapeGlobWildcards(String spec) method to remove backslash prefixes
from escaped glob characters
- Added comprehensive test cases for both methods including round-trip testing
These utilities provide the foundation for allowing users to treat glob
metacharacters as literals when they appear in actual filenames.
Fixes BEAM-13231
When CDC (Change Data Capture) is enabled with STORAGE_WRITE_API method, the system was incorrectly using PENDING streams instead of the default stream, causing an IllegalStateException due to checkState validation. Changes: - Fixed StorageApiWriteUnshardedRecords to use default stream when CDC is enabled - Added comprehensive test case to prevent regression CDC requires default streams because PENDING streams don't support the RowMutationInformation functionality needed for upserts and deletes. Fixes apache#31422
Summary of ChangesHello @Arunodoy18, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request primarily resolves an Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
…AT_LEAST_ONCE - Updated JavaDoc to reflect that triggeringFrequency applies to FILE_LOADS, STORAGE_WRITE_API, and STORAGE_API_AT_LEAST_ONCE methods - Fixed validation logic in expand() to require triggeringFrequency for STORAGE_API_AT_LEAST_ONCE when writing unbounded PCollections - Removed conflicting warning that incorrectly stated STORAGE_API_AT_LEAST_ONCE ignores triggeringFrequency - Added comprehensive test cases to verify proper validation behavior: * Test failure when triggeringFrequency is missing for unbounded collections * Test success when triggeringFrequency is provided for unbounded collections * Test success for bounded collections without triggeringFrequency requirement - Updated error messages to include all three supported methods This ensures consistent behavior across all BigQuery write methods that support triggered writes for unbounded collections.
|
Hi, I'm a first-time contributor. |
|
@Arunodoy18 tests are triggered whenever you commit something. if there are flakes type the action name from ( ) brackets. |
|
Also @Arunodoy18 , please check https://www.apache.org/legal/generative-tooling.html When providing contributions authored using generative AI tooling, a recommended practice is for contributors to indicate the tooling used to create the contribution. This should be included as a token in the source control commit message, for example including the phrase “Generated-by: ”. This allows for future release tooling to be considered that pulls this content into a machine parsable Tooling-Provenance file." |
|
please also split those 3 commits into 3 seperate PRs, it looks like you are trying to fix 3 different issues, which is great, but it shouldn't be one PR. |
2 participants
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.