The memory
bottleneck for on-
device AI. Solved.

VeloxQuant-MLX compresses the KV cache your Mac builds during LLM inference — up to 98% less peak memory across 43 methods, accelerated with hand-written Metal kernels, integrated in three lines of code.

Get started — pip install veloxquant-mlx SEE HOW IT WORKS

In Development VeloxQuant Studio, the native Mac app, is currently in development.

Open source · MIT License · Apple Silicon M1+

VeloxQuant Studio — Apple Silicon Inference Optimizer

● Apple M4 Metal Ready

⚡ VeloxQuant Studio

Configure Job

Pick a model and compression method, then optimize in memory.

ℹ Metal GPU kernels compile zero-copy in Apple Silicon RAM

Target Model

mlx-community/Qwen2.5-32B-Instruct-4bit

Presets

Compression Algorithm

turboquant_rvq 2-bit Quantized

Residual vector quantization with bit-exact attention preservation.

LIVE KV MEMORY TELEMETRY ● OPTIMIZED

3.8 GB -86.6% RAM

Original uncompressed KV: 28.4 GB

3.8 GB Active 24.6 GB Headroom Reclaimed

Metal GPU Speedup 14.7×

Sign Accuracy 100%

Throughput 68.2 tok/s

Perplexity Gap < 0.8 pt

Dashboard

Your Mac, your models, and what's been running.

THIS MAC



Chip

Apple M4

10-Core Apple GPU

⚡

Unified Memory

24 GB

4P + 6E Cores · High Bandwidth

🖥

macOS Version

macOS 26.5.2

Metal 3.2 Shaders Loaded

✓

VeloxQuant-MLX

Ready

v0.67.0 · Metal Linked

RECENT JOBS (4 RUNNING NOW)

mlx-community/Qwen2.5-32B-Instruct-4bit

vecinfer · 1-bit · 24s runtime

Cancelled

mlx-community/Qwen3-4B-4bit

kivi · 4-bit · Metal accelerated

Running

mlx-community/Llama-3.2-3B

turboquant_rvq · 2-bit · 729 MB → 12 MB

Running

Hardware Benchmarks & Recommender

Compare methods and see measured speedup from past jobs.

METHOD RECOMMENDER

Chip

Apple M4 ▾

Memory

24 GB ▾

Model Scale

7B - 8B ▾

Priority

Everyday Serving ▾

RECOMMENDED FOR APPLE M4 (24GB)

turboquant_rvq (2-bit)

Cuts memory consumption to approximately 1/8th with near-zero perplexity deviation. Delivers 14.7× faster attention evaluation via hand-written Metal GPU kernels.

Model Library

Manage local MLX HuggingFace weights and compiled cache checkpoints.

INSTALLED MODELS (1)

mlx-community/Qwen2.5-32B-Instruct-4bit

18.4 GB on disk · MLX Native · Metal KV Cache Ready

Installed

mlx-community/Llama-3.3-70B-Instruct-4bit

Requires 38 GB RAM · Recommended for Mac Studio M4 Max

Available

mlx-community/DeepSeek-R1-Distill-Qwen-14B-4bit

8.9 GB on disk · Fits 16GB & 24GB Unified Memory Macs

Available

Job History

Audit trail of compression runs, reclaimed unified memory, and throughput.

PAST SESSIONS (5 TOTAL)

turboquant_rvq (2-bit) on Qwen2.5-32B

Reclaimed 24.6 GB RAM · 14.7× Speedup · Metal Shaders

Completed

kivi (2-bit) on Qwen3-4B-4bit

Reclaimed 2.8 GB RAM · 11.2× Speedup · 4096 ctx

Completed

vecinfer (1-bit) on Qwen2.5-32B-Instruct-4bit

vecinfer · 1-bit · 24s runtime (Cancelled by user)

Cancelled

WHY IT MATTERS

Context is infinite. Memory is not.

LLMs like Llama, Mistral, and Qwen store past context in a KV cache that grows linearly with sequence length. A 7B model at 8k context can consume 14 GB on an 18 GB MacBook M3 Pro — leaving almost no room for anything else.

VeloxQuant-MLX compresses that cache on-the-fly with Metal GPU kernels, making long-context inference practical on consumer hardware.

EXPLORE THE METHODS

98%

Peak memory reduction

729 MB → 12 MB, Falcon3-7B shape

16×

Max key-cache compression

VecInfer-1bit, head_dim=128

13×

Metal kernel speedup

quantize_vq at S=2048

Compression algorithms

Quantization + token eviction

HOW IT WORKS

One API. Six strategies.

Thirty-three compression algorithms behind a single drop-in API. Change one method string to switch between them.

01 QUANTIZERS

Shrink every number

Fit bigger models and longer chats in the RAM you have — most methods need no calibration step.

02 TOKEN EVICTION

Drop what you don't need

The only methods that free RAM you'll see drop in Activity Monitor in real time.

03 CROSS-LAYER MERGING

Share memory across layers

Layers share memory instead of duplicating it. No code changes required.

04 METAL KERNELS

Hand-written for Apple Silicon

Up to 13× faster and 98% less peak memory than plain MLX. Built for your GPU, not ported from elsewhere.

05 DROP-IN API

Three lines, not a rewrite

Change one method string to switch algorithms. Your model-loading code stays exactly the same.

06 MULTI-MODEL SUPPORT

Works with what you already run

Tested on Llama, Mistral, Qwen, Phi, Gemma, Falcon, and vision-language models.

INSTALLATION

Get started in seconds.

The added lines are all you need. Your model-loading code stays exactly the same.

APPLE SILICON M1+ · PYTHON 3.11–3.12 · MLX 0.18+ · NUMPY 1.26+

Requires Apple Silicon and an existing mlx_lm setup.

$ pip install VeloxQuant-MLX

from veloxquant_mlx import KVCacheBuilder, KVCacheConfig

config = KVCacheConfig(method="turboquant_rvq", bit_width_inlier=1, seed=42)
caches = KVCacheBuilder.for_model(model, config)

READ THE DOCS VIEW ON GITHUB

BROWSE ALL 43 METHODS ON GITHUB

FAQ

Common questions.

Is this actually free? What's the catch?

Yes — it's free and open source under the MIT license, which means you can use it (including commercially) at no cost, and anyone can read the code to verify what it does. There's no account, no subscription, and no data collected: it runs entirely on your Mac, and nothing about your conversations or files is sent anywhere.

How is this different from what Ollama or LM Studio already do?

Ollama and LM Studio are built on llama.cpp, which already compresses this memory — but with one fixed setting applied the same way to the whole model. VeloxQuant-MLX offers 43 different methods to choose from, some of which can be tuned per part of the model rather than one setting for everything. It also includes 15 token eviction methods that can drop old, low-value parts of a conversation entirely to save even more memory.

Does this actually free up memory I'll see in Activity Monitor?

For the methods that drop old conversation history (like token eviction) — yes, today. For most other compression methods, they compress keys and values on-the-fly during Metal GPU execution, freeing up massive throughput and preventing out-of-memory crashes on large context windows.

What hardware do I need?

An Apple Silicon Mac (M1 or later; M2/M3/M4/M5 recommended), Python 3.11 or 3.12, and Apple's MLX framework (version 0.18 or later — installed automatically). It's a pure-Python tool with hand-written Metal GPU kernels built right in.

Do I need to re-download or convert my models?

No. VeloxQuant-MLX works directly with any model you've already loaded with mlx_lm — same weights, same file format, zero conversion step. You add the compression with three lines of code; your model file itself remains untouched.

Which method should I start with?

turboquant_rvq — no calibration or setup step required. It's the default for a reason: it cuts memory use to roughly a third with near-zero effect on perplexity. If you need the smallest possible footprint, vecinfer reaches about 16× compression on keys with a one-time calibration pass.

EVERYWHERE YOU BUILD

Run VeloxQuant across your entire workflow

Official Swift, Kotlin, NPM, Rust, and Go SDK runtimes and a VS Code extension available today — with native macOS app coming soon.

VeloxQuant Studio · Apple Silicon Menu Bar & Desktop App

Apple Silicon M-Series · Metal 3

Falcon3-7B · 128k context

KV Memory 729 MB → 12 MB (-98%)

Context Headroom +128k ctx

Metal Speedup 14.7×

Attention Quality Bit-Exact

+ Compress KV cache with Metal RVQ...

Native Mac Application In Development

VeloxQuant Studio for macOS

The flagship visual workspace engineered exclusively for macOS and Apple Silicon. Monitor local model memory in real-time, benchmark hand-written Metal kernels, and reclaim up to 98% of KV-cache footprint in a single click — without touching a terminal.

Zero-latency Metal RVQ & KIVI kernels

100% on-device inference with zero telemetry

VeloxQuant Assistant

Quantized key/value attention tensors to 2-bit with zero perplexity drift.

2 files optimized +128k ctx -92% Review ↗

generate.py +3 -1 •

mlx_cache.py +40 -12

+ Looks great, quantize 32 layers.

VS Code Extension

Install for VS Code

Package.swift · SwiftPM

// Package.swift

dependencies: [

.package(url: "https://github.com/rajveer43/veloxquant-swift", exact: "0.1.0-alpha")

]

// iOS 16+, macOS 13+, visionOS

import VeloxQuantCore

// Native Swift KV-cache compression

Swift Package (SPM)

View on GitHub

Terminal · zsh

model: mlx-community/Llama-3.2-3B

cache: quantized (KIVI 2-bit)

memory: 1.2 GB → 180 MB

Tip: Use /bench to measure Metal kernel speedup

Hey VeloxQuant, compress KV cache| 100% context left

VeloxQuant NPM SDK & CLI

Cargo · Rust 1.80+

// Cargo.toml

[dependencies]

veloxquant = "{ version = \"0.74\", features = [\"metal\"] }"

// main.rs

use veloxquant::prelude::*;

let cache = KvCache::quantize_2bit(&k, &v)?;

// Resident KV: 12.4 MB (98% reduction)

Rust SDK (Crates.io)

pkg.go.dev · Go 1.22+

// go.mod

module myapp

require github.com/rajveer43/veloxquant-go v0.74.0

// inference.go

import "github.com/rajveer43/veloxquant-go/cache"

c, err := cache.New(cache.DefaultConfig(cache.MethodTurboQuantRVQ))

// KV footprint compressed to 1/8th

Go SDK (pkg.go.dev)

build.gradle.kts · Kotlin 2.0+

// build.gradle.kts (Multiplatform)

dependencies {

implementation("com.veloxquant:mlx-kotlin:0.74.0")

}

// Apple Silicon Metal Native

val kv = VeloxQuant.compress(tensors, TurboQuantRVQ)

// KV footprint compressed to 1/8th

Kotlin Package

View on JitPack

The memorybottleneck for on-device AI. Solved.

Configure Job

Dashboard

Hardware Benchmarks & Recommender

Model Library

Job History

Context is infinite. Memory is not.

One API. Six strategies.

Shrink every number

Drop what you don't need

Share memory across layers

Hand-written for Apple Silicon

Three lines, not a rewrite

Works with what you already run

Get started in seconds.

Will your model fit?

Common questions.

Run VeloxQuant across your entire workflow

VeloxQuant Studio for macOS

VS Code Extension

Swift Package (SPM)

VeloxQuant NPM SDK & CLI

Rust SDK (Crates.io)

Go SDK (pkg.go.dev)

Kotlin Package

The memory
bottleneck for on-
device AI. Solved.