Error loading mixtral-8x7b-v0.1.Q6_K.gguf

[dlucr]

Code:

• https://github.com/microsoft/kernel-memory/tree/main/examples/105-dotnet-serverless-llamasharp

Model:

• https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/blob/main/mixtral-8x7b-v0.1.Q6_K.gguf
• file size 36G, md5 sum: 2f9a17b2a486b18e7b96f5de1248c4d6

System:

• Apple MBP M1 32GB

Perhaps not enough memory?

Error:

error loading model: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
llama_load_model_from_file: failed to load model
## Error ##
* Message:  Failed to load model /tmp/mixtral-8x7b-v0.1.Q6_K.gguf.
* Type:     RuntimeError [LLama.Exceptions.RuntimeError]
* Location: at LLama.Native.SafeLlamaModelHandle.LoadFromFile(String modelPath, LLamaModelParams lparams)
   at LLama.LLamaWeights.LoadFromFile(IModelParams params)

[dlucr]

Same error with mixtral-8x7b-v0.1.Q2_K.gguf, file size 15G

error loading model: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
llama_load_model_from_file: failed to load model

[SignalRT]

LLamaSharp is a llama.cpp wrapper. When a model fails in LLamaSharp the best approach is to check if llama.cpp supports the model.

If I'm right this is a new model with MoE and it's not supported yet in llama.cpp as described in TheBloke download page:

[dluc]

I see, thanks for the info! Given the PR mentioned, it might just be about a few weeks for the update to drip down to LlamaSharp :-)

(feel free to close this issue)

[SignalRT]

The expectations on this model are very high. For sure that it will a good idea to update llama.cpp binaries once the model is supported in llama.cpp master

[martindevans]

I was already planning to start a binary update later this week anyway, since it's been about a month since the last set. So that should pick up support for mixtral moe :)