Skip to content

llama : add struct llama_vocab to the API #11156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jan 10, 2025
Merged

llama : add struct llama_vocab to the API #11156

merged 3 commits into from
Jan 10, 2025

Conversation

ggerganov
Copy link
Member

@ggerganov ggerganov commented Jan 9, 2025 •
edited
Loading

target #11110

patch df1c467

This change decouples the llama_vocab from the llama_model.

API changes

  • llama_n_vocab() now accepts struct llama_vocab instead of struct llama_model
  • llama_sampler_init_dry() now accepts struct llama_vocab instead of struct llama_model
  • The tokenization API now accepts struct llama_vocab instead of struct llama_model
  • The sampler API now accepts struct llama_vocab instead of struct llama_model

@ggerganov ggerganov requested a review from ngxson as a code owner January 9, 2025 13:31
@github-actions github-actions bot added testing Everything test related examples server android Issues specific to Android python python script changes labels Jan 9, 2025
@ggerganov ggerganov added the breaking change Changes that break ABIs, APIs, file formats, or other forms of backwards compatibility. label Jan 9, 2025
@ggerganov ggerganov mentioned this pull request Jan 9, 2025
Merged
@ggerganov ggerganov force-pushed the gg/llama-refactor-7 branch from c89e808 to bfe781a Compare January 10, 2025 08:25
@ggerganov ggerganov force-pushed the gg/llama-refactor-8 branch from 06ae9ae to b48d763 Compare January 10, 2025 09:07
@ggerganov ggerganov merged commit 1d1f264 into gg/llama-refactor-7 Jan 10, 2025
54 of 57 checks passed
ggerganov added a commit that referenced this pull request Jan 10, 2025
@ggerganov
llama : add struct llama_vocab to the API (#11156)
1439bad 
ggml-ci
@ggerganov ggerganov deleted the gg/llama-refactor-8 branch January 10, 2025 09:21
ggerganov added a commit that referenced this pull request Jan 10, 2025
@ggerganov
llama : add struct llama_vocab to the API (#11156)
c725f69 
ggml-ci
@ggerganov ggerganov mentioned this pull request Jan 11, 2025
Merged
ggerganov added a commit that referenced this pull request Jan 12, 2025
@ggerganov @slaren
llama : add llama_vocab, functions -> methods, naming (#11110)
afa8a9e 
* llama : functions -> methods (#11110)

* llama : add struct llama_vocab to the API (#11156)

ggml-ci

* hparams : move vocab params to llama_vocab (#11159)

ggml-ci

* vocab : more pimpl (#11165)

ggml-ci

* vocab : minor tokenization optimizations (#11160)

ggml-ci

Co-authored-by: Diego Devesa <slarengh@gmail.com>

* lora : update API names (#11167)

ggml-ci

* llama : update API names to use correct prefix (#11174)

* llama : update API names to use correct prefix

ggml-ci

* cont

ggml-ci

* cont

ggml-ci

* minor [no ci]

* vocab : llama_vocab_add_[be]os -> llama_vocab_get_add_[be]os (#11174)

ggml-ci

* vocab : llama_vocab_n_vocab -> llama_vocab_n_tokens (#11174)

ggml-ci

---------

Co-authored-by: Diego Devesa <slarengh@gmail.com>
tinglou pushed a commit to tinglou/llama.cpp that referenced this pull request Feb 13, 2025
@ggerganov @slaren @tinglou
llama : add llama_vocab, functions -> methods, naming (ggml-org#11110)
e6c83d3 
* llama : functions -> methods (ggml-org#11110)

* llama : add struct llama_vocab to the API (ggml-org#11156)

ggml-ci

* hparams : move vocab params to llama_vocab (ggml-org#11159)

ggml-ci

* vocab : more pimpl (ggml-org#11165)

ggml-ci

* vocab : minor tokenization optimizations (ggml-org#11160)

ggml-ci

Co-authored-by: Diego Devesa <slarengh@gmail.com>

* lora : update API names (ggml-org#11167)

ggml-ci

* llama : update API names to use correct prefix (ggml-org#11174)

* llama : update API names to use correct prefix

ggml-ci

* cont

ggml-ci

* cont

ggml-ci

* minor [no ci]

* vocab : llama_vocab_add_[be]os -> llama_vocab_get_add_[be]os (ggml-org#11174)

ggml-ci

* vocab : llama_vocab_n_vocab -> llama_vocab_n_tokens (ggml-org#11174)

ggml-ci

---------

Co-authored-by: Diego Devesa <slarengh@gmail.com>
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Feb 26, 2025
@ggerganov @slaren @NeoZhangJianyu
llama : add llama_vocab, functions -> methods, naming (ggml-org#11110)
4501233 
* llama : functions -> methods (ggml-org#11110)

* llama : add struct llama_vocab to the API (ggml-org#11156)

ggml-ci

* hparams : move vocab params to llama_vocab (ggml-org#11159)

ggml-ci

* vocab : more pimpl (ggml-org#11165)

ggml-ci

* vocab : minor tokenization optimizations (ggml-org#11160)

ggml-ci

Co-authored-by: Diego Devesa <slarengh@gmail.com>

* lora : update API names (ggml-org#11167)

ggml-ci

* llama : update API names to use correct prefix (ggml-org#11174)

* llama : update API names to use correct prefix

ggml-ci

* cont

ggml-ci

* cont

ggml-ci

* minor [no ci]

* vocab : llama_vocab_add_[be]os -> llama_vocab_get_add_[be]os (ggml-org#11174)

ggml-ci

* vocab : llama_vocab_n_vocab -> llama_vocab_n_tokens (ggml-org#11174)

ggml-ci

---------

Co-authored-by: Diego Devesa <slarengh@gmail.com>
mglambda pushed a commit to mglambda/llama.cpp that referenced this pull request Mar 8, 2025
@ggerganov @slaren @mglambda
llama : add llama_vocab, functions -> methods, naming (ggml-org#11110)
65f1c66 
* llama : functions -> methods (ggml-org#11110)

* llama : add struct llama_vocab to the API (ggml-org#11156)

ggml-ci

* hparams : move vocab params to llama_vocab (ggml-org#11159)

ggml-ci

* vocab : more pimpl (ggml-org#11165)

ggml-ci

* vocab : minor tokenization optimizations (ggml-org#11160)

ggml-ci

Co-authored-by: Diego Devesa <slarengh@gmail.com>

* lora : update API names (ggml-org#11167)

ggml-ci

* llama : update API names to use correct prefix (ggml-org#11174)

* llama : update API names to use correct prefix

ggml-ci

* cont

ggml-ci

* cont

ggml-ci

* minor [no ci]

* vocab : llama_vocab_add_[be]os -> llama_vocab_get_add_[be]os (ggml-org#11174)

ggml-ci

* vocab : llama_vocab_n_vocab -> llama_vocab_n_tokens (ggml-org#11174)

ggml-ci

---------

Co-authored-by: Diego Devesa <slarengh@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ngxson ngxson ngxson approved these changes

Assignees
No one assigned
Labels
android Issues specific to Android breaking change Changes that break ABIs, APIs, file formats, or other forms of backwards compatibility. examples python python script changes server testing Everything test related
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ggerganov @ngxson