-
Notifications
You must be signed in to change notification settings - Fork 12.2k
llama : save and restore kv cache for single seq id #6341
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
662aaea
llama : save and restore kv cache for single seq id
kaetemi 5462817
remove trailing whitespace
kaetemi ab1c46a
respond error in case there's no space in the kv cache
kaetemi 02a1840
add kv seq save restore to test case
kaetemi b8e8fac
add --slot-save-path arg to enable save restore and restrict save loc…
kaetemi b182f8f
Returning 0 for some cases, instead of asserting.
martindevans a2b48b9
cleanup error cases
kaetemi c4443d7
rename sequence state functions
kaetemi 4d5356b
rename state get set functions
kaetemi bbcbf47
add previous function names back in with DEPRECATED notice
kaetemi 8b5ae29
update doc
kaetemi a71ec3d
adjust endpoints to preferred style
kaetemi bf1d493
fix restoring zero cell count
kaetemi 8ab1a17
handle seq rm return value
kaetemi 0d22136
unused param
kaetemi 29f18c2
keep in the size check
kaetemi f2e41b3
fix return types
kaetemi 92c4681
add server test case for slot save restore
kaetemi 60f685f
cleanup
kaetemi d38eef4
add cake
kaetemi ea717f7
cleanup style
kaetemi b509b8b
add special
kaetemi 129b6ff
removing a whole sequence never fails
kaetemi 8af7211
move sequence state file functionality from server to llama to match …
kaetemi 3d6fa5b
catch exceptions on save as well
kaetemi b3f6da3
error log messages
kaetemi be714a0
check types for stricter restore
kaetemi 0ccfbf2
update server doc
kaetemi 205c44c
readme : update API changes date
ggerganov d9fd0d7
Merge branch 'master' into feature/save-restore-seq
kaetemi f2a4777
strict filename validation
kaetemi 4a4f399
move include, reject bom as well
kaetemi 2fbf0c3
also reject empty filename
kaetemi bf94e9f
reject whitespace and trailing dot
kaetemi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
update doc
- Loading branch information
commit 8b5ae299ecd625d793591f7e14fa1e3014f84d8a
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.