-
Notifications
You must be signed in to change notification settings - Fork 4.8k
HIVE-29016: Disable caching on the Iceberg REST Catalog #5871
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
|
@deniskuzZ If the +1 stands for the approval, I will merge this pull request |
@okumin, I'd like to loop in @henrib and @dengzhhu653 to reach a consensus |
Thanks @deniskuzZ :-) May be worth trying overriding HMSCachingCatalog:buildTable(): |
good idea with setting the property |
@henrib What client did you use while testing the feature? |
@okumin , setting the property hive.metastore.catalog.cache.expiry=-1 in the configuration disables the cache which solves the issue of creating/updating tables through the catalog triggering an error. The actual coding solution to that error might just be overriding buildTable() as mentioned above. Removing the caching code will just make querying - which is expected to be the most common operation - slower and less efficient; caching is already necessary at scale. |
OK. Although I haven't tested the proposed solution, it's not a bad idea to disable it at this point. Is this expected? |
What changes were proposed in this pull request?
Remove caching on the Iceberg REST API server. It's mainly because the current code didn't work with caching correctly, as stated in HIVE-29016.
Also, I believe the REST Catalog service is not as mature as considering caching. Caching objects without proper consideration can be harmful, such as when users update an Iceberg table with both Thrift and REST APIs in parallel.
Why are the changes needed?
Remove caching.
Does this PR introduce any user-facing change?
No. This feature is not yet shipped.
How was this patch tested?
Unit test