oMLX

Running large language models locally on a Mac can be impressive, but long context windows and coding agents often introduce an annoying problem: the model spends too much time rebuilding information it has already processed. oMLX takes a different approach by combining a native macOS MLX server with persistent SSD-based KV caching and continuous batching.

Designed specifically for Apple Silicon, the software turns a compatible Mac into a local AI inference server that can work with popular development clients and locally stored models. It is particularly interesting for developers who use AI coding assistants throughout the day and want more responsive interactions without relying entirely on a remote inference service.

The project supports OpenAI-compatible and Anthropic-compatible APIs, making it possible to connect existing tools without rebuilding an entire workflow. It also supports model management, tool calling, MCP integration, and a browser-based dashboard for monitoring the local server.

Key Features

• Paged SSD KV caching: Cache blocks can be persisted to SSD rather than being kept only in memory. Frequently used blocks remain available for later requests, including after server restarts.
• Continuous batching: Multiple concurrent requests can be processed together, improving throughput when several clients or tasks are active.
• Native macOS application: The menu bar application provides controls for starting, stopping, and monitoring the local server without requiring an Electron-based interface.
• Multi-model serving: LLM, VLM, embedding, and reranker models can be loaded and managed from the same environment.
• OpenAI and Anthropic compatibility: Existing clients can connect through compatible API endpoints, including the Anthropic messages endpoint.
• Tool calling and MCP: Multiple tool-calling formats are supported, along with Model Context Protocol integration and configurable handling of large tool results.
• Hugging Face integration: Models can be downloaded through the dashboard, while existing Hugging Face caches and compatible model directories can be reused.
• Real-time monitoring: The web dashboard provides model management, chat functionality, and performance metrics.

User Interface

The interface is designed around the way a developer actually uses a local inference server. A native menu bar application handles everyday server controls, while the web dashboard provides a broader view of models, conversations, configuration, and performance.

This separation is practical. You do not need to keep a large management window open just to know whether the server is running, while more detailed operations remain available through the dashboard.

The setup process is also straightforward. The macOS application can be dragged into Applications, followed by a guided setup for the model directory, server, and initial model download. Developers who prefer the command line can install the project from source instead.

Accuracy & Performance

Performance depends heavily on the Apple Silicon chip, available memory, model size, quantization, and context length. The project's published benchmarks are based on real hardware rather than theoretical specifications.

For example, testing on an M3 Ultra with 512GB of memory shows Qwen3-Coder-Next running at 58.7 tokens per second for a 1K context in a single request. With eight concurrent requests, the reported throughput reaches 243.3 tokens per second, representing a 4.14× speedup over the single-request baseline.

The underlying caching strategy is especially relevant for long-running agent sessions. Instead of discarding previously processed KV cache blocks whenever the context changes, cached blocks can be recovered from SSD. According to the project's own measurements, this can reduce time to first token substantially on long contexts.

These numbers should not be treated as universal benchmarks. A Mac with different memory capacity or a different model can produce very different results. The community benchmark section provides a much broader collection of results from different Apple Silicon configurations.

Capabilities

The main strength is not simply running one language model locally. It is the combination of local inference with the surrounding developer ecosystem.

Developers can serve MLX-format models from Hugging Face, including models from families such as Qwen, LLaMA, Mistral, Gemma, DeepSeek, MiniMax, and GLM. Vision-language models are also supported, extending the platform beyond conventional text-only inference.

Multiple models can be served at the same time, with LRU-based eviction available when memory becomes constrained. This is useful for developers who want one model for coding, another for general conversation, and additional models for embeddings or other specialized tasks.

Tool calling is another important capability. JSON-based formats and several model-specific tool-calling formats are supported, while MCP integration allows compatible agents to work with external tools through a local setup.

Compatibility with OpenAI-style clients and Anthropic-compatible clients also makes the software more flexible than a standalone chat application. A developer can use the local server as a backend while continuing to work with familiar tools.

Security & Privacy

One of the biggest advantages of local inference is control over where model processing takes place. The software is designed to run the inference server on the user's own Apple Silicon Mac rather than requiring every prompt to be sent to a hosted AI provider.

This can be particularly useful when working with source code, private development projects, internal documents, or other information that a developer would prefer to keep within their own environment.

The project is distributed under the Apache 2.0 license, and the macOS application is signed and notarized according to the project's website. As with any self-hosted AI system, however, privacy ultimately depends on the user's machine, network configuration, connected clients, models, and local security practices.

Use Cases

• AI-assisted coding: Run compatible coding agents against locally hosted models instead of depending exclusively on remote inference.
• Long coding sessions: Persistent KV caching is particularly useful when an agent repeatedly revisits earlier context.
• Local AI experimentation: Developers can test different MLX-compatible models directly on Apple Silicon hardware.
• Private development workflows: Local inference can be useful when source code or other sensitive project material should remain on the developer's machine.
• Multiple model workflows: Different LLM, VLM, embedding, and reranker models can be served within the same environment.
• MCP-based applications: Developers experimenting with tool-enabled agents can use the built-in MCP support.
• OpenAI-compatible applications: Existing software that supports compatible API endpoints can connect to the local server without requiring a completely different integration.

Pros and Cons

Pros

• Built specifically for Apple Silicon Macs.
• Persistent SSD-based KV caching is well suited to long-running agent sessions.
• Supports OpenAI-compatible and Anthropic-compatible APIs.
• Works with popular developer clients such as Claude Code, OpenClaw, and Cursor.
• Supports multiple model types, including LLMs and vision-language models.
• Includes continuous batching for better concurrent throughput.
• Offers tool calling and MCP integration.
• Can reuse existing Hugging Face and compatible model caches.
• Open-source distribution under the Apache 2.0 license.

Cons

• It is designed for Apple Silicon, so Intel Macs are not the intended hardware platform.
• Large models can require substantial unified memory.
• Performance varies considerably between Mac configurations and model sizes.
• Users who simply want a hosted chatbot may find the server-oriented architecture more technical than necessary.
• Running larger models locally can still require expensive Mac hardware.

Pricing Plans

The project is available under the Apache 2.0 license rather than being presented as a conventional subscription-based AI service. There is a downloadable macOS application as well as an option to install the project from source.

The absence of a conventional monthly AI subscription makes it particularly appealing to developers who already own suitable Apple Silicon hardware and want to experiment with local inference without adding another recurring API bill.

How to Use It

Getting started is relatively simple if you already have a compatible Mac. The recommended route is to install the macOS application, move it into the Applications folder, and follow the initial setup process.

Choose the directory containing your models, start the local server, and download a compatible model if necessary. Existing models in supported Hugging Face caches and compatible directories can be reused, which can save considerable time and disk activity.

Once the server is running, open the dashboard to manage models, start conversations, inspect metrics, and configure integrations. For development tools, the dashboard can generate the configuration command needed to connect compatible clients.

Advanced users can install the project from source. The source installation requires Python 3.10 or newer and Apple Silicon, with the server available to OpenAI-compatible clients through the local environment.

Comparison with Similar Tools

Ollama and LM Studio are familiar choices for running language models locally, but the design priorities here are somewhat different. Instead of focusing mainly on making local model execution convenient, this project puts considerable emphasis on persistent KV caching and workloads driven by coding agents.

That distinction matters when a session repeatedly changes context. Traditional in-memory caching can lose useful state when the context shifts, forcing computation to start again. Persistent SSD caching is intended to make previously processed prefixes recoverable instead.

It also stands apart through its combination of a native macOS application, web dashboard, multi-model serving, OpenAI and Anthropic API compatibility, continuous batching, tool calling, and MCP support.

For someone who only wants a simple local chatbot, a more conventional local model application may be easier. For a developer building an agent-heavy workflow on Apple Silicon, the architecture here is considerably more compelling.

Conclusion

Local AI becomes much more interesting when the goal is not simply to chat with a model but to use models as part of an ongoing development workflow. Persistent KV caching, concurrent request handling, compatible APIs, model management, and tool support make this project particularly well suited to that scenario.

The strongest case is for developers with an Apple Silicon Mac who regularly use coding agents and want local inference without repeatedly paying for hosted model usage. The hardware requirements mean it is not for everyone, but on a sufficiently capable Mac, the combination of MLX performance and intelligent caching can make local AI feel considerably more responsive.

For developers who want control over their inference environment and enjoy experimenting with open models, it is a practical project worth exploring.

Frequently Asked Questions (FAQ)

What operating systems are supported?

The project is designed for macOS 15 or newer running on Apple Silicon hardware, starting with M1-series Macs.

How much RAM is recommended?

16GB is listed as the minimum, while 64GB or more is recommended for comfortable use with larger models. The amount of memory required ultimately depends on the model and quantization being used.

Can it work with Claude Code and Cursor?

Yes. OpenAI-compatible and Anthropic-compatible endpoints allow supported clients such as Claude Code and Cursor to use locally served models.

Does it support vision-language models?

Yes. Vision-language models are supported alongside standard language models, embeddings, and rerankers.

Do existing Hugging Face models need to be downloaded again?

Not necessarily. The server can use the standard Hugging Face cache and can also recognize compatible existing model directories, including supported LM Studio locations.

Can multiple models run at the same time?

Yes. Multiple LLM, VLM, embedding, and reranker models can be served together, with memory management available when the system needs to remove less recently used models.

Does it support MCP?

Yes. MCP integration is included, along with tool calling and configurable handling of tool results.

Is it free?

The project is distributed under the Apache 2.0 license and does not present a conventional paid subscription plan on its website.

Is local AI faster than cloud AI?

There is no universal answer. Performance depends on the Mac, model, quantization, context length, and workload. The project's benchmarks demonstrate strong local performance on high-end Apple Silicon hardware, while cloud services may still be preferable for users without powerful local hardware.