Skip to content

Conversation

jayfeather9
Copy link
Collaborator

Must use with Radix Cache, will store all inputs into disk right after prefill (parallel), and when new request arrive, if disk cache len > gpu cache len && gpu cache len <= 0.5 * input_length, then pull from disk and use the disk cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant