---
title: "Needle 3 Turns Model Depth Into a Deployment Knob"
description: "Cactus Compute’s September 16 release uses nested 2-to-20-layer subnetworks so one open-weight checkpoint can target different edge budgets without maintaining separate models."
date: 2026-09-16
topic: open-source
tags: ["needle","on-device-ai","tool-calling","edge-ai","model-compression"]
production: ai-assisted
source: https://notes.suffro.com/posts/needle-3-model-depth-ladder
---

# Needle 3 Turns Model Depth Into a Deployment Knob

Cactus Compute's Needle 3 is interesting for a reason that is easy to miss if it is treated as another small-model release. The useful idea is not simply that the model is small. It is that one checkpoint is structured as a family of nested models whose depth can be chosen at deployment time.

On September 16, the project populated its [Needle 3 model repository](https://huggingface.co/Cactus-Compute/needle3) with the checkpoint, tokenizer files, a model card, and benchmark material. The repository marks the weights as Apache 2.0. At the same time, the public [Needle source tree from September 16](https://github.com/cactus-compute/needle/tree/3413a930f6af03951c1e3e6db6ef3964e4770435) already contained the code needed to select a depth, fine-tune that subnetwork, and export it for inference.

That combination makes the release more technically substantial than a collection of separately quantized checkpoints. Needle 3 turns model depth itself into part of the deployment interface.

## The model is a nested ladder

The full configuration in the September 16 source has 20 layers. Its [architecture code](https://github.com/cactus-compute/needle/blob/3413a930f6af03951c1e3e6db6ef3964e4770435/needle/model/architecture.py) defines a deterministic ladder over those layers.

The selection rule keeps the first and last blocks, then repeatedly fills the largest remaining gap. A smaller rung therefore uses a stable subset of the larger model rather than an unrelated network trained from scratch. The code carries the selection order into sliced configurations so that further slicing remains consistent.

This matters operationally. A device fleet rarely has one memory and latency budget. A phone, a Raspberry Pi-class computer, and a smaller embedded target may all need different model sizes. The conventional answer is to maintain different models or at least different distilled variants. Needle's design instead lets the same base checkpoint be cut to a requested depth.

The corresponding [fine-tuning and export code](https://github.com/cactus-compute/needle/blob/3413a930f6af03951c1e3e6db6ef3964e4770435/needle/model/finetune.py) exposes that mechanism through a --layers option. The code slices the parameters and configuration before local LoRA fine-tuning, records the selected depth in the adapter, and preserves it when building the deployable archive.

This is more than documentation. The September 16 [build tests](https://github.com/cactus-compute/needle/blob/3413a930f6af03951c1e3e6db6ef3964e4770435/tests/test_build.py) check that an exported three-layer archive reports three layers. More importantly, they verify that taking a two-layer rung directly from the original model produces exactly the same parameter leaves as taking a three-layer rung first and then reducing it to two layers. That test directly exercises the nesting property on which the deployment scheme depends.

## It is not a conventional tiny Transformer

The architecture also explains why Cactus is targeting constrained devices.

Needle 3 is implemented as a Simple Attention Network, but the released code is not merely the attention-only architecture from the team's earlier paper. It combines grouped-query attention with a parameter-light Hadamard-based mixing module, hashed n-gram "engram" tables, local and global attention patterns, and multi-lane residual connections.

The motivation traces back to the team's July paper, [*A Controlled Study of Attention-Only Transformers*](https://arxiv.org/abs/2607.18363). That work tested whether the large feed-forward network in a standard Transformer is essential when parameter count, training compute, and depth are controlled separately. Its central result was that attention-only models could close most of the pretraining-loss gap when the freed parameter budget was reallocated into additional depth, although the paper also found a remaining weakness in parametric recall.

Needle 3 takes that line of work in a more deployment-oriented direction. The September source replaces a conventional large feed-forward block with a much smaller structured mixer and moves some capacity into lookup-style n-gram memory. It also includes explicit machinery for sliding-window attention, selected global layers, and quantized export.

The result is an architecture designed around the cost of inference rather than around being a general-purpose chat model. The intended jobs are structured ones such as tool selection and argument generation, where constrained decoding can be useful and where a small local model can remove a network round trip.

## The open part is unusually inspectable

The release is best described precisely as an open-weight model accompanied by an open-source implementation. The Hugging Face repository labels the model Apache 2.0, and the [September 16 Python package metadata](https://github.com/cactus-compute/needle/blob/3413a930f6af03951c1e3e6db6ef3964e4770435/pyproject.toml) applies the same license to the source package.

The public code exposes considerably more than an inference wrapper. It includes the architecture, JAX reference inference, LoRA fine-tuning, quantization and export code, the ladder slicing logic, and tests for the generated archive format. A developer can inspect how a rung is chosen and verify that the exported artifact corresponds to that choice.

There is still an important reproducibility boundary. The materials available on September 16 are sufficient to inspect the architecture, use the released checkpoint, fine-tune a selected rung, and rebuild deployment artifacts. They do not provide, in the sources reviewed here, the original pretraining corpus and a complete recipe for independently recreating the base checkpoint from raw data. The release is therefore reproducible at the inference, slicing, fine-tuning, and export layers, not as an end-to-end reproduction of the original training run.

The benchmark material deserves the same caution. Cactus published benchmark updates in the [Needle 3 repository history](https://huggingface.co/Cactus-Compute/needle3/commits/main) on September 16, but those numbers are first-party results. Without an independent reproduction, they are evidence about what the maintainers measured, not proof of a general performance advantage.

## The release was technically ahead of its packaging

The source snapshot also shows signs of a project in transition. On September 16, pyproject.toml still identified the Python package as version 2.0.12, and the top of the README still introduced Needle 2 even though the CLI, model code, checkpoint handling, and later sections already contained Needle 3 functionality.

That is a real caveat. A model can have an interesting architecture while its public packaging is still catching up. Anyone evaluating the September 16 state should treat the exact installation and deployment surface as less settled than the underlying ladder implementation.

It does not diminish the core technical point. The unusual part of Needle 3 is already visible in code and tests: model size is not represented as a set of unrelated checkpoints. It is a deterministic property of one nested model.

If that approach holds up across real devices and independent evaluations, it offers a useful pattern for edge AI. Hardware-aware deployment can become a choice of how much of one trained network to execute, rather than a separate model-development project for every target.

That is the part of Needle 3 worth watching. The strongest evidence on September 16 is not a leaderboard number. It is that the mechanism exists, is licensed permissively, and can be inspected down to the functions that decide which layers survive.

## Sources

- [Cactus Compute, Needle 3 model repository](https://huggingface.co/Cactus-Compute/needle3)
- [Cactus Compute, Needle 3 Hugging Face commit history](https://huggingface.co/Cactus-Compute/needle3/commits/main)
- [Needle source tree at the September 16 snapshot](https://github.com/cactus-compute/needle/tree/3413a930f6af03951c1e3e6db6ef3964e4770435)
- [Needle ladder and model architecture implementation](https://github.com/cactus-compute/needle/blob/3413a930f6af03951c1e3e6db6ef3964e4770435/needle/model/architecture.py)
- [Needle fine-tuning and export implementation](https://github.com/cactus-compute/needle/blob/3413a930f6af03951c1e3e6db6ef3964e4770435/needle/model/finetune.py)
- [Needle build and ladder tests](https://github.com/cactus-compute/needle/blob/3413a930f6af03951c1e3e6db6ef3964e4770435/tests/test_build.py)
- [Ndubuaku et al., "A Controlled Study of Attention-Only Transformers"](https://arxiv.org/abs/2607.18363)