Local AI on Apple Silicon

What you install

One command downloads models, serves them, generates, quantizes, trains and transcribes. The map above is the whole product.

• Inference engine

  Runs the model graph on the GPU through MLX and Metal. The Neural Engine programs a model uses are compiled by the installer, or before the first model load that needs them. Validated configurations reproduce mlx-lm’s logits bit for bit.

• Local server

  Chat and text completions, Anthropic Messages, OpenAI Responses, embeddings, audio transcription and decisions on localhost:8080, over one continuous batching scheduler with paged and quantized KV caches, speculative decoding and runtime LoRA adapters. Every route is in the HTTP API reference.

• Browser app

  Chat with tools and saved sessions, manage and download models, transcribe audio, quantize checkpoints, train adapters, generate datasets, and watch metrics and benchmarks. It opens when the server starts.

• Libraries

  @mlx-bun/mlx, inference, quantize, training and hub on npm: loading and generation, plus the kernels, layers, graphs, state, sampling and training they compose. Start with the library guide.

• Terminal

  get, serve, generate, embed, convert, train, fuse, transcribe and fit. Every command and flag is in the CLI reference, generated from the source.

Install

Requires an Apple Silicon Mac running macOS 14 or later. Every option provides the mlx-bun command.

Installer

curl -fsSL https://mlx-bun.dev/install.sh | sh

Downloads the signed, notarized bundle with its runtime and native libraries, then compiles the Neural Engine programs your Mac’s models use. See the installer options.

Homebrew

brew install joshuarossi/tap/mlx-bun

Installs the same bundle and links the command. The app compiles the Neural Engine programs it lacks before the first model load. See the Homebrew notes.

npm

bun install -g mlx-bun   # or run once: bunx mlx-bun

Requires Bun 1.4.2 or later; use Bun rather than Node’s npx. See the npm notes.

1. Serve a model. Download one from Hugging Face and serve it, or run mlx-bun with no arguments to serve a cached model, downloading a starter on first run.

   mlx-bun get mlx-community/gemma-4-e4b-it-OptiQ-4bit
   mlx-bun serve e4b

2. Send a request. The browser app opens on port 8080, and the API answers beside it.

   curl http://localhost:8080/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -d '{"messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 128}'

3. Or generate once, without a server.

   mlx-bun generate e4b "Write a haiku about Apple Silicon."

The quickstart continues from here.

Why mlx-bun

Built to be trusted first, then embedded.

• Bit-exact numerics

  Validated configurations reproduce mlx-lm’s logits bit for bit, and mlx-optiq’s for mixed-precision KV. Correctness is a published contract with evidence, not a benchmark table.

• Native on your Mac

  Metal does the math. The standalone bundle is signed, notarized and ships its own runtime, so there is no Python environment to manage and no API key to keep.

• Nothing to rewrite

  Chat Completions, Messages, Responses, embeddings and transcription keep their wire formats. Existing agents, tools and SDK code keep working after changing one URL.

• Composable

  Every layer is a public subpath: kernels, layers, graphs, caches, sampling, generation. The caller owns weights and state, and using disposes them. The whole server embeds with mlx-bun/server.

• Memory aware

  Fit estimates before a model loads, residency by memory budget across models, paged and quantized KV, prompt caches that persist to SSD, and continuous batching for every request.

• Private by default

  Downloaded models stay in the Hugging Face cache on your Mac; converted models, adapters, chats and KV state live under ~/.mlx-bun. Nothing leaves your Mac unless you fetch or upload something yourself.

Decisions, scored on your Mac

Use Jev or OpenAI Decisions-compatible endpoints with any supported local language model, sharing the same loaded weights as generation.

POST /v1/decisions accepts OpenAI’s Decisions request. POST /v1/systemone accepts the System One format of Jev, TypeSafe AI’s decision model. Ask predicate questions (yes or no; Jev calls them noul), choice questions and score questions about text, or about an image with a vision model, and get the answer, a probability for every option and a confidence.

Every answer is scored from the loaded model’s own likelihood of that answer. No token is sampled and nothing needs tuning. Keep the OpenAI or TypeSafe SDK and change baseURL; name a model that GET /v1/models lists with capabilities.decisions.

OpenAI

import OpenAI from "openai";

const openai = new OpenAI({ apiKey: "local", baseURL: "http://localhost:8080/v1" });

const { answers } = await openai.decisions.create({
  model: "mlx-community/gemma-4-e4b-it-OptiQ-4bit",
  input: "I was charged twice for my order.",
  questions: [{
    type: "choice", name: "department",
    instructions: "Which department should handle this complaint?",
    choices: [
      { value: "billing", description: "Payments, invoices, and refunds." },
      { value: "technical", description: "Problems using the product." },
      { value: "shipping", description: "Delivery and tracking." },
    ],
  }],
});

TypeSafe

import { TypeSafeClient, choice } from "@typesafe-ai/sdk";

const jev = new TypeSafeClient({ apiKey: "local", baseURL: "http://localhost:8080" });

const result = await jev.systemOne({
  model: "mlx-community/gemma-4-e4b-it-OptiQ-4bit",
  state: { document: "I was charged twice. Please fix this ASAP." },
  questions: {
    category: choice("What is this ticket about?", { billing: null, technical: null, other: null }),
  },
});

HTTP

curl http://localhost:8080/v1/decisions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mlx-community/gemma-4-e4b-it-OptiQ-4bit",
    "input": "I was charged twice for my order. Please refund the duplicate payment.",
    "questions": [
      { "name": "refund", "type": "predicate", "instructions": "Does the customer ask for a refund?" },
      { "name": "severity", "type": "score", "instructions": "How severe is the problem?",
        "levels": [{ "label": "Minor inconvenience" }, { "label": "Customer payment problem" },
                   { "label": "Service completely unavailable" }] }
    ]
  }'

curl http://localhost:8080/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mlx-community/gemma-4-e4b-it-OptiQ-4bit",
    "state": "I was charged twice for my order. Please refund the duplicate payment.",
    "questions": {
      "refund": { "type": "noul", "instructions": "Does the customer ask for a refund?" },
      "severity": { "type": "score", "instructions": "How severe is the problem?",
        "criteria": ["Minor inconvenience", "Customer payment problem", "Service completely unavailable"] }
    }
  }'

Answers

// One answer per question. OpenAI returns them in order; Jev keys them by name.
{ type: "predicate", name, probability }
{ type: "choice", name, choice, confidence, probabilities: [{ value, probability }] }
{ type: "score", name, score, confidence, probabilities: [{ value, label, probability }] }

Probabilities are a softmax over the candidates’ average answer-token log likelihood; confidence follows Jev’s documented formulas. The response names the model that ran and counts the tokens scored.

The HTTP example and the SDK examples run against a local server; the HTTP API reference lists both routes.

Build with the libraries

The @mlx-bun/ packages are TypeScript on npm. They run on Bun 1.4.2 or later, with their native libraries inside the archives.

bun add @mlx-bun/inference

Generate

packages/inference/examples/qwen3-generate.ts

import { bindGraphCaches } from "@mlx-bun/inference/models";
import { phaseRowLimit } from "@mlx-bun/inference/models/graph";
import { loadModelConfig, Weights, loadTokenizer, generate, resolveComposition } from "@mlx-bun/inference";
import { Qwen3Model } from "@mlx-bun/inference/models/qwen3";

/** The caller chooses a Qwen3 checkpoint and owns its weights. */
export async function generateText(directory: string, prompt: string, maxTokens = 32) {
  const config = await loadModelConfig(directory);
  const tokenizer = await loadTokenizer(directory);
  const weights = await Weights.open(directory);
  try {
    const graph = bindGraphCaches(new Qwen3Model(weights, config, phaseRowLimit(resolveComposition(config))), resolveComposition(config));
    const generation = generate(graph, tokenizer.encode(prompt), {
      maxTokens, temperature: 0, eosTokenIds: config.eosTokenIds,
    });
    const tokens: number[] = [];
    for await (const { token } of generation) tokens.push(token);
    return { text: tokenizer.decode(tokens), tokens, stats: generation.stats };
  } finally {
    weights.dispose();
  }
}

The tested example from the inference package, rendered from its source at build time. The caller owns the model, the inputs and every resource’s lifetime.

Arrays

packages/mlx/examples/arrays.ts

import { MlxArray, ops } from "@mlx-bun/mlx";

export function double(values: number[]): number[] {
  using input = MlxArray.fromFloat32(new Float32Array(values), [values.length]);
  using output = ops.mulScalar(input, 2);
  return [...output.toFloat32()];
}

The MLX bindings own native memory explicitly: using disposes an array at the end of its scope. Tested example.

Embed the server

The server entry ships in the app package, not in @mlx-bun/inference:

bun add mlx-bun

import { createServer, loadContext } from "mlx-bun/server";

const context = await loadContext("/path/to/hf-snapshot", "my-model");
const server = await createServer(context, 8080, { hostname: "127.0.0.1", memoryBudgetBytes: 12e9 });
try {
  console.log(`API http://127.0.0.1:${server.port}/v1`);
} finally {
  const { stopped, durability } = await server.close();
  if (stopped) context.dispose();
  if (!durability.durable) console.warn("cache persistence incomplete", durability);
}

The full app in your process: every HTTP route, the browser app, chat, jobs and the hub, over a model you loaded. It parses no arguments, installs no signal handler and downloads nothing. See the server entry.

• @mlx-bun/mlx

  Arrays, operations and native resource ownership over MLX.

• @mlx-bun/inference

  Graphs, loading, preprocessing, generation, embeddings, transcription and memory fit.

• @mlx-bun/quantize

  Calibration, sensitivity, mixed-precision allocation, rotation and Trellis packing.

• @mlx-bun/training

  LoRA and preference training, optimizers, losses and adapter production.

• @mlx-bun/hub

  The local model registry and Hugging Face downloads and uploads.

• mlx-bun/server

  The app’s server, engine and model selection as importable entries.

The library API lists every public subpath, export and signature.

Models it runs

Download any of them from Hugging Face with mlx-bun get. Compatibility claims apply to named artifacts and configurations.

Targeted graphs

Gemma 4, DiffusionGemma, GLM 5.2, Qwen 3, Qwen 3.5, Qwen 3 MoE, MiniCPM 5 and Whisper.

Generic dense graph

The architectures mlx-lm’s universal path knows: Llama, Mistral, SmolLM 3, Qwen 2, Gemma, Gemma 2, Phi 3, OLMo 2, GLM 4, Granite and StarCoder 2.

Our quant on Hugging Face

We publish Qwen3.8-27B-Trellis-3.2bpw, a 27B Qwen model in mlx-bun’s packed Trellis format. The full download is about 12.25 GB; running it needs additional memory for inference. This is an optional larger model, separate from the first-run starter. See the model card for requirements, evaluations and usage.

Reference

The CLI, HTTP and configuration references are generated from the code at build time, and the library API from the public exports.

• Quickstart

  Start the app, pick a model, send the first request.

• Installation

  Installer, Homebrew, npm, or a source checkout.

• Using the libraries

  Choose a component and start from a tested example.

• Library API

  Every public package subpath, export and signature.

• CLI reference

  Every command and flag, from the command table.

• HTTP API reference

  Every route each server mode mounts.

• Configuration

  Serve defaults, flags and every runtime key.

• Correctness

  The numerical contract, its evidence and its limits.