Skip to content

Persistent Parquet Metadata Cache for Object Storage #694

Open
@Selfeer

Description

@Selfeer

Describe the new feature

Currently, we lose all cached data when the server restarts, which impacts performance and requires recalculating or refetching data. To improve reliability and efficiency, we need a mechanism to persist the cache across restarts.

Expected Behavior

  • Cached data should be retained even after a server restart.
  • The system should be able to reload the cache from a backup storage (e.g., disk, or external caching service).

Possible Solutions

The usage of Redis might be beneficial here, as when a new node starts or an existing one restarts, it can already access the cached metadata from Redis.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions