Announced on July 14, 2026, the model supports a 262K token context window while delivering impressive local inference performance: 163 tokens/second on an NVIDIA RTX 5090 and 11 tokens/second on an iPhone 17 Pro.
Benchmark results show the ternary version preserves around 95% of the original Qwen3.6-27B performance, while the 1-bit model retains roughly 90%. However, tool calling and AI agent capabilities drop significantly in the 1-bit variant (80.0 → 66.0), making it 4.6× more affected than mathematical reasoning tasks. For AI agents, coding assistants, and tool-using workflows, the ternary Bonsai 27B is the recommended choice, while the 1-bit version is best suited for chatbots, text generation, summarization, and other lightweight on-device AI applications.
Contents
- What Bonsai 27B actually is
- Binary and ternary weights
- Where the compression savings come from
- Performance retention and practical trade-offs
- Running locally on phones, laptops, and browsers
- Agentic workflows and offline deployment
- The part most announcements skip
- FAQ
What Bonsai 27B actually is
Bonsai 27B is a post-training compressed model family derived from Qwen3.6 27B, retaining the original model’s approximate 27-billion-parameter scale while changing how its weights are represented.
The important distinction is that Bonsai is not a small model trained from scratch with fewer parameters, because it is a compact representation of a much larger dense model.
PrismML offers two notable formats: a binary 1-bit variant around 3.9 GB and a ternary variant around 5.9 GB, with the latter retaining more benchmark performance.
Bonsai 27B is a compressed Qwen3.6 27B model that replaces conventional high-precision weights with binary or ternary representations, shrinking deployment size from roughly 54 GB in FP16 to approximately 3.9–5.9 GB.
PrismML Website – Carousel I: End-to-end agentic workflow with Hermes, powered by our Ternary Bonsai 27B model on NVIDIA GeForce RTX 5090.
Binary and ternary weights
A standard neural-network weight is a numerical coefficient that alters the strength and direction of a signal flowing through each layer of the model.
Bonsai’s binary format reduces each weight conceptually to one of two states, generally represented as -1, +1 instead of a full FP16 number.
The ternary format permits three values, −1, 0, and +1, so it can remove some connections entirely while retaining a slightly richer approximation than binary weights. This reduction appears brutal at first glance, and it is lossy—well, not exactly random loss, because the compression pipeline is designed to preserve behavior important to the original model.
| Representation | Possible weight values | Approximate deployed size | Primary objective |
|---|---|---|---|
| FP16 Qwen3.6 27B | Many floating-point values | About 54 GB | Highest original precision |
| Bonsai 27B 1-bit | -1, +1 | About 3.9 GB | Minimum device |
| Bonsai 27B ternary | -1, 0, +1 | About 5.9 GB | Better quality-to-size balance |
The ternary version reportedly uses about 1.71 effective bits per weight, while the binary release uses about 1.125 effective bits per weight after practical packaging considerations.
Where the compression savings come from
The raw storage math explains most of the headline, because reducing a parameter representation from 16 bits toward roughly one bit produces an order-of-magnitude size reduction.
But disk size is only half the story, since inference repeatedly reads model weights from RAM or VRAM while generating each next token. Smaller weights reduce memory bandwidth demand, which can make consumer hardware more viable even when the number of underlying model parameters remains unchanged.
Model-size compression lowers both storage demand and memory traffic, but it does not eliminate KV-cache growth, token-generation latency, or hardware-specific kernel limitations.
Performance
The ternary Bonsai release reportedly retains about 94.6 percent of the source model’s aggregate benchmark performance, while the 1-bit variant retains about 89.5 percent. Those figures are impressive, although engineers should interpret them as reported aggregate retention rather than proof of equivalent behavior in every workload.
A model can score well on selected reasoning and knowledge evaluations while still showing noticeable degradation in multilingual fluency, structured tool calls, rare programming environments, or long-horizon agent reliability.
| Hardware | 1-bit | Ternary |
|---|---|---|
| NVIDIA RTX 5090 | 163 tok/s | 134 tok/s |
| Apple M5 Max | 87 tok/s | 58 tok/s |
| iPhone 17 Pro | 11 tok/s | — |
| Variant | Weight scheme | Effective bits/weight | File size | Baseline retained |
|---|---|---|---|---|
| Ternary Bonsai 27B | {−1, 0, +1} + FP16 group-wise scaling | 1.71 | 5.9 GB | ~95% |
| 1-bit Bonsai 27B | {−1, +1} + group-wise scaling | 1.125 | 3.9 GB | ~90% |
Running locally on phones, laptops, and browsers
PrismML documents a 1-bit MLX variant around 3.92 GiB and presents iPhone 17 Pro deployment as a target configuration for local inference.
On Apple Silicon, MLX provides a natural route for local execution, while GGUF-oriented workflows can use compatible runtimes such as llama.cpp on supported systems. Browser deployment uses WebGPU, meaning the browser can run supported GPU compute workloads locally after the necessary model assets have been loaded. That still involves downloading gigabytes of model data, so browser execution is local after loading rather than magically free of bandwidth, storage, or hardware requirements.
PrismML also advertises a 262,144-token context capacity and speculative decoding support, but the usable context on a given device depends heavily on KV-cache memory and implementation details.
Which benchmark actually collapses under quantization?
Tool calling is where 1-bit quantization shows its biggest weakness.
While mathematical reasoning loses just 3.8% of its benchmark score compared to the original Qwen3.6-27B, tool calling drops by 17.5%—a decline that is 4.6 times larger. The impact is even more significant when you look at the starting point. Math begins with an impressive 95.3 score, while tool calling already starts lower at 80.0. After 1-bit quantization, that score falls to 66.0.
In other words, a model designed to enable agentic workflows ends up scoring just 66/100 on the benchmark that specifically measures its ability to use tools and interact with external functions.
This doesn’t mean the 1-bit model isn’t useful, for chatbots, text generation, summarization, and lightweight on-device AI, it remains an impressive achievement thanks to its tiny memory footprint and excellent speed.
However, if your application relies on AI agents, function calling, MCP servers, coding assistants, or multi-step automation, the results suggest a different conclusion: the ternary version of Bonsai 27B is the better choice, offering much stronger reliability while preserving almost all of the original model’s capabilities.
| Benchmark | Baseline | Ternary | 1-bit | 1-bit degradation |
|---|---|---|---|---|
| Overall | 85.0 | 80.5 | 76.1 | −10.5% |
| Math | 95.3 | 93.4 | 91.7 | −3.8% |
| Coding | 88.7 | 86.0 | 81.9 | −7.7% |
| Tool-calling | 80.0 | 74.0 | 66.0 | −17.5% |
Why does tool calling suffer much more than math?
The most likely explanation is that math and coding are naturally more resilient to quantization. Even when a model loses precision, there are many different internal paths that can still lead to the correct answer. In other words, the knowledge is distributed across the network, so reducing the precision of the weights has only a limited impact.
Tool calling works very differently: here, the model isn’t just generating text—it has to make an exact decision. It must select the correct function from several similar options and produce perfectly formatted JSON with the right parameter names and values.
This is where 1-bit quantization becomes a problem: imagine an AI assistant that has access to two functions:
get_weather(city="Paris")get_forecast(location="Paris")
To a human, they look almost identical, to an AI agent, however, choosing the wrong one means the task fails completely.
Extreme quantization makes these fine distinctions harder to preserve, the model becomes slightly less precise at separating very similar choices, and that small loss of precision has a disproportionate effect on function calling.
This helps explain why tool-calling performance drops by 17.5%, while mathematical reasoning loses only 3.8% – for chat, summarization, or general text generation, a 1-bit model can still perform remarkably well. But for AI agents, MCP integrations, and workflows that depend on reliable function calling, the ternary version of Bonsai 27B is the safer and more capable choice.
How to run Bonsai 27B locally
Getting Bonsai 27B up and running is straightforward, but there are a few details that aren’t immediately obvious—especially if you’re planning to demo the model.
Start by cloning the official repository and installing the required dependencies:
git clone https://github.com/PrismML-Eng/Bonsai-demo
cd Bonsai-demo
./setup.sh # macOS / LinuxCode language: Bash (bash)Next, choose which model you want to run. The project supports multiple sizes (27B, 8B, 4B, and 1.7B) as well as two quantization families:
- ternary (recommended for agents and tool use)
- bonsai (the ultra-compressed 1-bit models)
Configure them through environment variables:
export BONSAI_MODEL=27B
export BONSAI_FAMILY=ternary
export BONSAI_TOKEN="hf_..."Code language: Bash (bash)The Hugging Face token is mandatory for the 27B models, as the repositories are gated. This is an important limitation that many tutorials overlook. While the smaller Bonsai models can be downloaded immediately, 27B requires requesting access first, so don’t expect to deploy it in minutes if you haven’t already been approved.
Once configured, you can launch the model directly from the command line:
./scripts/run_llama.sh -p "Explain ternary quantization in two sentences."Code language: Bash (bash)Or start an OpenAI-compatible API server:
./scripts/start_llama_server.shCode language: Bash (bash)By default, the server is available at http://localhost:8080/v1, making it compatible with existing applications built on the OpenAI SDK.
Running Bonsai 27B on Apple Silicon
If you’re using a Mac with Apple Silicon, the repository also includes an MLX backend, which generally provides the best performance.
source .venv/bin/activate
./scripts/run_mlx.sh -p "What is the capital of France?"Code language: Bash (bash)Useful performance settings
The project exposes several options that are worth enabling from day one.
A 4-bit KV cache reduces memory usage while keeping performance high:
BONSAI_KV4=1 ./scripts/start_llama_server.shCode language: Bash (bash)On NVIDIA GPUs, you can enable speculative decoding to increase inference speed:
BONSAI_SPECULATIVE=1 ./scripts/start_llama_server.shCode language: Bash (bash)And if you’re using reasoning models, you can limit the number of reasoning tokens to improve latency:
./scripts/start_llama_server.sh --reasoning-budget 2048Code language: Bash (bash)Tool calling and AI agents
If your goal is to build AI agents or test function calling, the repository recommends using Open WebUI, which comes preconfigured with tools and vision support.
./scripts/start_openwebui.shCode language: Bash (bash)Open WebUI runs on port 9090, while the inference server continues to expose an OpenAI-compatible API on port 8080.
Because of this compatibility, you can point any existing OpenAI SDK application directly to:
http://localhost:8080/v1Code language: Bash (bash)Benchmarks provide a useful reference, but the real test is how the model performs with your own function schemas, JSON formats, MCP tools, and agent workflows. Since the largest performance drop under 1-bit quantization occurs in function calling, testing against your production tools is far more meaningful than relying solely on published benchmark scores.
Resources
- Bonsai Demo: https://github.com/PrismML-Eng/Bonsai-demo
- PrismML Website: https://prismml.com
FAQ
Is Bonsai 27B a newly trained one-bit model?
No, Bonsai 27B is presented as a compressed derivative of Qwen3.6 27B, rather than a separate 27B model originally trained using binary or ternary weights.
Does a 3.9 GB model always require only 3.9 GB of memory?
No, model weights are only one part of runtime consumption, because the KV cache, inference engine, context length, images, and operating system add memory requirements.
Is ternary Bonsai usually preferable to the 1-bit variant?
For most technical use cases, ternary is the safer default because reported retention is higher, while 1-bit is compelling when memory and portability constraints dominate.
Does local inference mean a browser demo works without downloading anything?
No, WebGPU enables local execution after loading the application and model assets, but the model data must still be downloaded and accommodated by local hardware.
Can Bonsai replace cloud models for autonomous agents?
It can enable capable offline agents for bounded workflows, but teams should validate long-running tool use and recovery behavior before replacing stronger hosted models in critical systems.
Start your free 30-day trial at regolo.ai and deploy LLMs with complete privacy by design. If need help you can always reach out our team on Discord 🤙
👉 Talk with our Engineers or Start your 30 days free →
- Discord – Share your thoughts
- GitHub Repo – Code of blog articles ready to start
- Follow Us on X @regolo_ai
- Open discussion on our Subreddit Community
Built with ❤️ by the Regolo team. Questions? regolo.ai/contact or chat with us on Discord