Two ways to make small language models cheaper

1. The Dual Bottleneck in Agentic Small Language Models

Deploying small language models (21M to 150M parameters) for multi-turn tool and code execution exposes two orthogonal bottlenecks: compute inefficiency during prefill caused by static subword tokenization, and HBM memory bandwidth saturation during decoding caused by linear KV-cache growth across long agent trajectories.

2. Axis 1: From Tuple-Hashed BPE to Shannon Conditional-Entropy Patching

In our shipped 21M PyTorch baseline, we implemented a custom Byte-Pair Encoding (BPE) tokenizer from scratch using O(1) tuple-hashed pair counting paired with Rotary Position Embeddings (RoPE). While static BPE compresses frequent substrings, it allocates identical transformer layer FLOPs to highly predictable tokens (e.g., boilerplate JSON keys) and high-surprise decision tokens.

Following the Byte Latent Transformer formulation (Pagnoni et al., arXiv:2412.09871), our active Axis-1 R&D models local conditional entropy H(X_t | X_{<t}) over raw bytes/characters and triggers patch boundaries only when information surprise exceeds a threshold τ_entropy. Compute scales dynamically with semantic uncertainty rather than corpus frequency.

3. Axis 2: Cross-Layer KV & Sparse Index Reuse (CSA2)

To compress inference memory without degrading long-horizon retrieval, we evaluate downscaled variants of Compressed Sparse Attention 2 (CSA2, arXiv:2609.19969) across three layer execution modes:

• Full Mode: Computes latent compressed KV projections and selects top-k sparse attention indices.
• Reindex Mode: Reuses the parent layer's KV cache directly from HBM while recomputing layer-specific sparse routing indices.
• Reuse Mode: Reuses both the parent KV cache and sparse attention indices, incurring zero additional KV HBM footprint.

4. Empirical Takeaways

Pairing dynamic entropy patching with periodic Full/Reindex/Reuse layer blocks cuts HBM KV allocation by over 60% across 12-layer decoders while preserving structural JSON and tool-calling fidelity under teacher-student distillation.