local LLM inference / serving
Ollama is a local LLM runner and model manager that downloads quantized open models and serves them through a CLI and REST API on top of llama.cpp.
Same problem, same approach. Swapping one for another is a config change, not a rewrite.
MIT-licensed local model runner with the same get-models-running job; pick it if you want a C++ alternative to Ollama's Go binary.
Get up and running with GLM-5.3-flash, DeepSeek, Qwen, Gemma and other open source frontier models.
Solves the same problem with a different architecture or at a different layer. Expect to rewrite the integration.
TypeScript/Node local runner CLI; pick it if you want a JS-ecosystem toolchain instead of a Go binary.
Super lightweight Ollama + Qwen Code alternative to run Llama 3.3, DeepSeek-R1, Phi-4, Gemma 3, Mistral Small 3.1 and other large language models.
GPU serving framework optimized for throughput; choose it for datacenter batch serving rather than a laptop CLI.
SGLang is a high-performance serving framework for large language models and multimodal models.
Distributed serving across a self-built cluster; choose it when one machine cannot hold the model.
Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere
Python GPU serving engine with far higher throughput but no laptop-friendly model registry; pick it for server deployment.
A high-throughput and memory-efficient inference and serving engine for LLMs
A .NET GGUF inference engine exposing Ollama/OpenAI-compatible HTTP APIs; choose it to stay in the .NET stack.
A native .NET LLM inference engine for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability
An earlier or more limited way to do the job. Still the right call when you need something small, proven, or CPU-only.
On-device X-bit quantized inference limited to small models; pick it for embedded/edge devices where Ollama is too heavy.
On-device LLM Inference Powered by X-Bit Quantization
Does the same job, but ships as an app. Useful to a person, not swappable into a codebase.
Desktop GUI app (LM Studio-style) for point-and-click offline models; not embeddable in a codebase.
Kolosal AI is an OpenSource and Lightweight alternative to LM Studio to run LLMs 100% offline on your device.
Flutter desktop/mobile app that bundles llama.cpp; choose it for a cross-platform GUI client rather than a server.
Maid is a free and open source application for interfacing with llama.cpp models locally, and with Anthropic, DeepSeek, Ollama, Mistral and OpenAI models remotely.
A packaged local-AI desktop app; pick it for an all-in-one GUI instead of a CLI/server.
Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V
These projects were analysed and named ollama among their alternatives. The relationship is not symmetric — how ollama rates them is a separate judgement, made when ollama is analysed in its own right.
calls ollama “Choose when you only need to run open-weight models locally via CLI/API and do not need fine-tuning or diffusion support.”
Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4, MiniMax-H3, Gemma 4, FLUX and more.