Stories by mezark

A running list of reasons to move to open source

Moe inference optimizations: 15% lower expert load by request reordering

Tensor Network Attention

Redundant Information in LLM Weights

Tans: Precomputing RANS

Also-RANS: Asymmetric Numeral Systems for Entropy Coding

70x faster cold(ish) starts for SGLang

QueueSpec – drafting speculation tokens while a request queues

ZeroDP: Just-in-Time Weight Offloading over NVLink for Data Parallelism

Parallel Primitives for Multi-Agent Workflows

New fastest AI Model Gateway – 450x less overhead than LiteLLM

Should GPUs Make Free Trade Agreements?

Controlled generation of OS LLMs – without impacting latency

Takeoff Inference Server Is Now Open Source

Falcon 7B running real time on CPU