Research, releases, and perspectives from the Zen LM team.
Zen 5 is training right now on Hanzo Network — decentralized compute with verifiable execution. Here's how the community can submit agentic training data and contribute to the next generation of open AI.
Zen 5 brings 1T+ parameter scale, 2M context windows, and state-of-the-art performance across reasoning, code, and multimodal tasks.
How BitDelta (arXiv:2402.10193) compresses fine-tuned behavioral deltas to 1-bit precision, enabling the full Zen model family — nano through ultra — to share a single GPU cluster.
Deep dive on Surprise-Driven Prioritized Replay (SuRe) and Orthogonal Projection Continual Merging (OPCM) — the two SOTA techniques we use for catastrophic-forgetting-free LLM adaptation in the Zen model family.
How Drop-Upcycling (arXiv:2502.19261) transforms dense checkpoints into MoE models at 1/4 training cost, and how it shapes Zen MoDE — Mixture of Diverse Experts with genuine expert specialization.
Why standard representation engineering fails on Mixture-of-Experts models, and how Gate-Targeted QLoRA solves the expert routing problem at 1 trillion parameters.
An inside look at the Mixture of Diverse Experts architecture that powers the Zen model family.
The case for radical openness in AI — why releasing weights is the right thing, strategically and ethically.
Zen4 Ultra is our most capable model: 480B total parameters, 35B active per token, 1M token context window. Benchmark results and use cases.
Hanzo AI, Zoo Labs Foundation, and Lux Partners jointly release over 100 open-weight models. Generations 1 through 5, spanning text, code, vision, audio, and safety research. Apache 2.0.
How Hanzo AI, Zoo Labs Foundation, and Lux Partners Limited build the Zen model family together — and why the three-org structure is not an accident.
GITHUB HUGGING FACE MODELSCOPE DISCORD
Announcing Zen 3.0, our most capable open model family yet.
Reflections on where open AI development is heading and what it will take to get there.
How we're building a decentralized compute network for training large AI models.
How Zoo Improvement Proposals enable community-driven governance of open AI development.
Today we formally launch Zoo Labs Foundation, an open research network dedicated to decentralized AI and decentralized science.
Introducing Agent NFTs, a framework for giving AI agents persistent identity and enabling ownership of their capabilities.
Announcing Training Gym, our open platform for collaborative large model training.
How we're bringing cryptographic verification to AI inference, enabling trustless machine learning.
Introducing the Zen Reranker, a cross-encoder model that dramatically improves retrieval quality in two-stage pipelines.
Intro Generalist Models are hot! We all see an opportunity towards a real generalist model by multimodal multitask learning. We previously release an opensourced unified multimodal pretrained model OFA for this goal. However, we actually met a lot of difficulties in our implementation. For example, it is hard to set up multiple tasks concerning multiple modalities, and it is hard to organize multitask learning, e.g., how to batchify your data and how to make your training stable.
CLIP1 is a phenomenal playmaker in vision and multimodal representation learning. It plays not only as a foundation model but also a bridge between vision and language. It has triggered a series of research in different fields, especially text-to-image generation. However, we find that there is a necessity for a language-specific CLIP for applications, especially cross-modal retrieval, and there is no opensourced Chinese CLIP with good performance. We therefore launched this project to promote the Chinese multimodal representation learning.
Why we trained embedding models with 7680 dimensions and what we learned about the relationship between dimensionality and retrieval quality.
Exploring high-dimensional embedding spaces for semantic search and retrieval.
2022 is a year of generalist models! With the bloom of multimodal pretraining, especially the unified model, we have witnessed the opportunity to building a generalist model that is capable of processing tasks of different modalities or multi-modalities! Thus, we propose OFA1, namely One-For-All, a unified multimodal pretrained model that unifies understanding and generation tasks concerning modalities into a single framework, and we pretrain OFA with the instruction-based multitask-pretraining that endows it with multiple capabilities.
Introducing GRPO, a new approach to reinforcement learning from human feedback that improves sample efficiency and alignment stability.
A companion post to our GRPO paper, explaining group relative policy optimization for language model alignment.
Privacy-preserving machine learning that maintains model quality through novel aggregation protocols.
How federated learning enables collaborative model training while preserving data privacy.
Introducing experience ledgers, a framework for giving AI agents persistent, verifiable memory.
A manifesto for decentralized science (DeSci) and its application to AI research.
Why scientific research needs decentralization, and how blockchain can help.
How we're approaching training data curation to capture humanity's collective intelligence.
We're launching Zen, an open research initiative to build AI that serves everyone.