Skip to content
vivek pal

wiki / tiny llms

small models on a mac with mlx

Install mlx-lm, run a model, quantize one, and fine-tune it with LoRA on Apple silicon.

updated · 4 min read

why apple silicon fits this

Apple silicon has unified memory: the CPU and GPU read and write the same pool. MLX is built around that. You create an array without saying where it lives, then pick the device per operation (stream=mx.cpu or stream=mx.gpu), and nothing gets copied between them. A model can use most of the machine’s RAM instead of being capped by a separate VRAM size.

MLX on PyPI needs Apple silicon, a native Python 3.10 or newer, and macOS 14.0 or newer. If pip can’t find a matching build, you are probably on an x86 Python under Rosetta:

python -c "import platform; print(platform.processor())"   # should print: arm

install

pip install mlx-lm
pip install "mlx-lm[train]"   # extra dependencies for fine-tuning

run a model

mlx_lm.generate --prompt "How tall is Mt Everest?"
mlx_lm.chat

Both default to mlx-community/Llama-3.2-3B-Instruct-4bit. Pass --model for anything else: a Hugging Face repo or a local path. mlx_lm.generate -h lists every option.

From Python:

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")
messages = [{"role": "user", "content": "Explain unified memory in two sentences."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
text = generate(model, tokenizer, prompt=prompt, verbose=True)

stream_generate yields tokens as they arrive if you want streaming output.

For long inputs, three flags matter:

  • --max-kv-size n uses a rotating fixed-size KV cache. Small values save memory and cost quality.
  • --kv-bits quantizes the KV cache.
  • --prefill-step-size sets how many prompt tokens are processed at once (default 2048). Smaller steps lower peak memory and slow down prompt reading.

If you reuse one long context with different questions, mlx_lm.cache_prompt saves the processed prompt to a file and mlx_lm.generate --prompt-cache-file reuses it.

convert and quantize

mlx_lm.convert --model Qwen/Qwen3-4B -q --q-bits 4 --q-group-size 64 --mlx-path qwen3-4b-4bit

-q turns quantization on. In the default affine mode, --q-bits accepts 2, 3, 4, 5, 6, or 8 and --q-group-size accepts 32, 64, or 128, with 4 bits and groups of 64 as the defaults. --q-mode also offers mxfp4, nvfp4, and mxfp8. Add --upload-repo <name> to push the result to Hugging Face. The memory math note covers what each setting costs.

fine-tune with lora

LoRA trains small low-rank matrices next to frozen weights. If the base model is quantized, mlx-lm trains with QLoRA automatically. The LoRA docs list Mistral, Llama, Phi2, Mixtral, Qwen2, Gemma, OLMo, MiniCPM, and InternLM2 as supported families.

The data directory needs a train.jsonl for training, an optional valid.jsonl for validation loss, and a test.jsonl for evaluation. One example per line, in one of four formats:

{"messages": [{"role": "user", "content": "Hello."}, {"role": "assistant", "content": "Hi."}]}
{"prompt": "What is the capital of France?", "completion": "Paris."}
{"text": "Plain text for continued pretraining."}

The fourth format, tools, adds tool calls to the chat format.

mlx_lm.lora \
  --model mlx-community/Llama-3.2-3B-Instruct-4bit \
  --train \
  --data ./data \
  --iters 600

Adapters land in adapters/ unless you set --adapter-path. --fine-tune-type accepts lora (default), dora, or full. --mask-prompt computes loss on the completion only, for chat and completion data.

When memory runs short, the docs suggest these knobs:

flagdefaulteffect
--batch-size4lower it to 2 or 1
--grad-accumulation-stepskeeps the effective batch size up without more memory
--num-layers16fine-tune fewer layers
--grad-checkpointoffrecompute activations instead of storing them

Evaluate and generate with the adapter:

mlx_lm.lora --model mlx-community/Llama-3.2-3B-Instruct-4bit --adapter-path adapters --data ./data --test
mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit --adapter-path adapters --prompt "..."

fuse the adapter

mlx_lm.fuse --model mlx-community/Llama-3.2-3B-Instruct-4bit

By default this reads adapters from adapters/ and writes the merged model to fused_model/. --upload-repo with --hf-path publishes it with attribution to the original model. --export-gguf writes a GGUF file, but only for Mistral, Mixtral, and Llama style models in fp16.

when a model is slow

On macOS 15 or newer, mlx-lm wires the memory used by the model and cache. If you see a warning that the model needs more memory than is available, generation will likely be slow. If the model still fits in RAM, raising the wired limit can help:

sudo sysctl iogpu.wired_limit_mb=N   # N: above the model size in MB, below total RAM

sources

← all notes