Skip to content

llama : refactor get / set state + remove redundant kv cache API #1143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 24, 2023

Conversation

ggerganov
Copy link
Member

@ggerganov ggerganov commented Apr 23, 2023

  • Normalize the code style
  • Move the definitions at the correct place in llama.cpp
  • Retire llama_get_kv_cache(), llama_get_kv_cache_size() and llama_set_kv_cache()

Not sure how to test this - maybe we need to add an example, or extend main with store/load state functionality

@ggerganov ggerganov added the refactoring Refactoring label Apr 23, 2023
@ggerganov ggerganov requested a review from xaedes April 23, 2023 16:08
Copy link
Collaborator

@xaedes xaedes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me, I love the readability improvements. The save_load script from #730 (comment) works as well.

I have converted this to an example with cmake already, will pull request it.

@ggerganov ggerganov merged commit c4fe84f into master Apr 24, 2023
@ggerganov ggerganov deleted the refactor-state branch April 24, 2023 04:40
@ejones
Copy link
Collaborator

ejones commented Apr 25, 2023

If it's helpful, I put up a take on save/load state in main in #1169 (mostly due to my impatience with 65B on the chat-13B prompt, hah).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
refactoring Refactoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants