Features Request: Ollama support

[tysonchamp]

Please add ollama support

[fatihbaltaci]

Hello @tysonchamp, its on our roadmap. I'll let you know, when we add the ollama support.

Thanks.

[mugoosse]

+1

[wagnerfnds]

+1

[richlysakowski]

Anything done on this yet? This is a REALLY important, high priority requirement for business use.

[fatihbaltaci]

Hi folks! We’re exploring Ollama support for Gurubase. Could you share your specific use case? For example, is it for offline usage, privacy concerns, or to try other open-source models as the base LLM? Your input may help us prioritize this feature request.

[Xoeseko]

I would say for me it is mostly a question of privacy to support ollama or vllm. I wouldn't be looking to self-host a RAG system otherwise. It feels like a core tenant of the offering as a whole.

Neither of which need to be fully integrated but at least, if the now standard openai API is supported and I had the option to select the API URL and perhaps the model I would already be happy as a starting point.

[kay0ramon]

+1

[wagnerfnds]

Hi folks! We’re exploring Ollama support for Gurubase. Could you share your specific use case? For example, is it for offline usage, privacy concerns, or to try other open-source models as the base LLM? Your input may help us prioritize this feature request.

In my case, the company I work for has internal process documents that they do not allow to be sent to OpenAI for privacy reasons. So we run Ollama on a local server.

[joaopalma5]

@fatihbaltaci
I didn't setup the project on my machine but I will let you know how to do:

Go to src/gurubase-backend/backend/core/models.py L1178

and modify to:
client = OpenAI(api_key=self.openai_api_key, timeout=10, base_url= self.openai_api_base_url)

from openai import OpenAI

client = OpenAI(
    base_url = 'http://localhost:11434/v1',
    api_key='ollama', # required, but unused
)

response = client.chat.completions.create(
  model="llama2",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The LA Dodgers won in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ]
)
print(response.choices[0].message.content)

source: https://ollama.com/blog/openai-compatibility
Add url to settings

DONE!

[fatihbaltaci]

@joaopalma5 we're already aware of the OpenAI-compatible endpoint Ollama exposes. We use multiple models behind the scenes (OpenAI for base llm, gte-large for embeddings, bge-reranker for reranking, Gemini for summarization, etc.), so plugging Ollama in isn't just a one-line change. We're working on proper Ollama support to ensure the self-hosted experience matches the performance of Gurubase.io. We'll share updates as soon as it's ready.