Distillation case studies: Alpaca, Vicuna, Orca, Phi, Zephyr, and DeepSeek-R1-Distill
The case studies on this page span 2019-2025 and cover the full arc: DistilBERT proving the technique on BERT, the Alpaca/Vicuna era showing you could distill conversational ability for hundreds of dollars, the Orca line demonstrating that reasoning is distillable, the Phi line showing that data quality can substitute for scale, Zephyr demonstrating alignment distillation without human labels, and DeepSeek-R1-Distill proving that reasoning chains transfer across model families.
DistilBERT (2019)
What it was: HuggingFace’s distillation of BERT-base into a 6-layer model.
Method: Feature-based distillation with triple loss (KL on output logits, MLM cross-entropy, cosine alignment on hidden states). Initialized from every other layer of BERT-base.
Result:
| Metric | BERT-base | DistilBERT |
|---|---|---|
| Parameters | 110M | 66M |
| Layers | 12 | 6 |
| Inference speed | 1x | 1.6x faster |
| GLUE average | ~79.6 | ~77.0 |
| GLUE retention | 100% | 97% |
What it proved: You can remove 40% of a BERT model’s parameters and retain 97% of its performance by using the remaining model as a teacher. The cosine hidden-state alignment was as important as the output distillation.
Significance: DistilBERT became one of the most downloaded models on HuggingFace. It established that distillation was a practical compression technique, not just a research curiosity.
Alpaca (2023)
What it was: Stanford’s LLaMA-7B fine-tuned on 52,002 instruction-response pairs generated by text-davinci-003.
Method: Black-box SFT. Used Self-Instruct to prompt text-davinci-003 to generate new instruction-response pairs from 175 seed tasks. Fine-tuned LLaMA-7B on the result.
Cost:
- Data generation: ~$500 in API calls (text-davinci-003 pricing at the time)
- Fine-tuning: <$100 on cloud compute (~3 hours)
- Total: approximately $600
Important clarification: The teacher was text-davinci-003 (OpenAI’s InstructGPT), not GPT-3.5-turbo or GPT-4. This is frequently misreported.
Result: In blind pairwise evaluation (252 instruction pairs), human annotators found Alpaca-7B competitive with text-davinci-003 on a wide range of everyday instruction-following tasks.
What it proved: Instruction-following ability — a capability that required massive RLHF training to develop in text-davinci-003 — could be transferred to a 7B model for $600. The gap between proprietary and open models was not fundamental. It was a data problem.
Limitations: Alpaca hallucinates, doesn’t reliably follow safety guidelines, and is not safe for production deployment. The weights were taken down by Stanford after misuse concerns. Its significance is methodological, not product-related.
Vicuna (2023)
What it was: UC Berkeley (with CMU, Stanford, UCSD, and MBZUAI) distillation of LLaMA-13B using real ChatGPT conversations scraped from ShareGPT.com.
Method: Black-box SFT on ~70,000 multi-turn ChatGPT conversations that users had voluntarily shared online. Unlike Alpaca, the training data was real user-ChatGPT exchanges rather than synthetically generated instructions.
Cost: ~$300 on 8x A100 GPUs for approximately one day.
Evaluation: GPT-4-as-judge on 80 questions across 9 categories. GPT-4 scored responses from both Vicuna and the baseline on helpfulness, relevance, accuracy, and detail.
Result: Vicuna-13B scored ~92% of ChatGPT quality per the GPT-4 judge. The blog title rounded to “90%.”
What it proved: Real multi-turn conversation data transfers more than synthetically generated QA pairs. Vicuna’s conversations had the natural follow-up, clarification, and context-dependent response patterns that real ChatGPT users had elicited. This was closer to what users actually need from a chatbot.
What it introduced: GPT-4-as-judge became the standard evaluation method for open-ended model comparison after Vicuna. MT-Bench (which Zephyr and Orca were later evaluated on) uses the same paradigm.
Caveats: GPT-4-as-judge has well-documented verbosity bias and position bias. The “92% of ChatGPT” framing should be read as “comparable on the specific 80-question set evaluated.” It’s evidence, not a certified benchmark.
Orca and Orca 2 (2023)
What they were: Microsoft’s progression from output imitation to reasoning distillation.
Orca 1
Method: Explanation tuning. Prompting GPT-4 with “explain your reasoning step by step” system prompts, then training LLaMA-13B on the full reasoning traces.
Two-stage approach:
- Stage 1: Distill from ChatGPT on ~5M FLAN-v2 examples (broad coverage)
- Stage 2: Distill from GPT-4 on a higher-quality subset with reasoning traces (deep reasoning)
Results:
| Benchmark | Vicuna-13B | Orca-13B |
|---|---|---|
| BigBench Hard | 24.1% | 42.9% |
| AGIEval | 28.4% | 36.0% |
What it proved: The reasoning process is teachable, not just the final answer. A model that trained on “here is how GPT-4 thinks through this” learned to think through problems more like GPT-4, not just to produce GPT-4-sounding text.
Orca 2
Method: Teaching five reasoning strategies and when to use each. System-prompt erasure: GPT-4 receives strategy-specific prompts during training; the student receives no system prompt at inference.
Results: Orca 2-7B matched or outperformed models 5-10x its size on reasoning benchmarks. Competitive with GPT-3.5 on several tasks.
What it proved: Strategy selection is a learnable skill. A 7B model that knows when not to reason (and just answers directly) and when to reason (and does so systematically) outperforms a 7B model that always reasons.
Phi-1, Phi-2, Phi-3 (2023-2024)
What they were: Microsoft’s line of small models trained on “textbook quality” synthetic data.
The hypothesis: Data quality can substitute for data quantity. A model trained on less data that is more educational and well-structured outperforms a model trained on vastly more data scraped from the web.
Phi-1 (1.3B, 2023)
Method: ~6B tokens of GPT-3.5-generated synthetic Python textbooks, plus ~8B tokens of filtered web code. Total: ~14B tokens (vs. 80B+ for StarCoder).
Results (HumanEval pass@1):
| Model | Params | HumanEval |
|---|---|---|
| StarCoder | 15B | 33.6% |
| Codex | 12B | 28.8% |
| Phi-1 | 1.3B | 50.6% |
A 1.3B model outperformed a 15B model on the primary coding benchmark. The explanation: standard code training data is dominated by copy-paste code, incomplete snippets, and example code with no explanation. Textbook-style data teaches the model how code works, not just what code looks like.
Phi-2 (2.7B, 2023)
Extended textbook quality to general reasoning. ~250B tokens. Selected benchmark performance (Microsoft blog, self-reported):
- GSM8K: 57.2% (vs LLaMA-2-13B at 35.5%)
- ARC: 59.8% (vs Mistral-7B at 55.5%)
Phi-3-mini (3.8B, 2024)
Training: 3.3 trillion tokens using curriculum: synthetic textbook data first, filtered web data introduced gradually.
Results:
- MMLU: 68.8%
- GSM8K: 82.5%
- HumanEval: 58.5%
- MT-Bench: 8.38 (competitive with Mixtral-8x7B at 8.30, a model with ~12x the active parameters)
What the Phi line proved: The standard “train on all of the internet” approach is inefficient. A small model trained on high-quality synthetic data can match models many times its size. This challenged the scaling-laws orthodoxy that more data is always better.
Zephyr (2023)
What it was: HuggingFace’s demonstration that alignment can be distilled without human preference labels.
Method:
- dSFT: Fine-tune Mistral-7B on GPT-4 outputs (response-based SFT)
- dDPO: Generate preference pairs using GPT-4: for each prompt, GPT-4 produces a “chosen” (thorough, reasoned) and “rejected” (minimal) response. Apply DPO on these pairs.
No human annotators. No reward model. Pure teacher distillation.
Result: Zephyr-7B scored 7.34 on MT-Bench. Competitive with 70B-scale models on this benchmark.
What it proved: RLHF alignment — the property that makes models helpful and safe to interact with — is not just a matter of massive human preference data. It can be distilled from a teacher that already has alignment, at a fraction of the cost.
DeepSeek-R1-Distill (2025)
What it was: Six smaller models fine-tuned on 800,000 reasoning traces from DeepSeek-R1.
Background: DeepSeek-R1 is a large reasoning model trained with reinforcement learning (GRPO). It produces long <think>...</think> blocks before answering, exploring multiple solution approaches. These reasoning chains are the training signal for the distilled models.
Method: Pure SFT on 800K reasoning traces. No RL training. The student sees (problem, full reasoning trace + answer) and learns via imitation.
Models released:
| Model | Params | Base |
|---|---|---|
| R1-Distill-Qwen-1.5B | 1.5B | Qwen2.5-1.5B |
| R1-Distill-Qwen-7B | 7B | Qwen2.5-7B |
| R1-Distill-Qwen-14B | 14B | Qwen2.5-14B |
| R1-Distill-Qwen-32B | 32B | Qwen2.5-32B |
| R1-Distill-LLaMA-8B | 8B | LLaMA-3-8B |
| R1-Distill-LLaMA-70B | 70B | LLaMA-3-70B |
Results (pass@1):
| Model | AIME 2024 | MATH-500 |
|---|---|---|
| R1-Distill-Qwen-32B | 72.6% | 94.3% |
| R1-Distill-LLaMA-70B | 70.0% | 94.5% |
| R1-Distill-Qwen-14B | 69.7% | 93.9% |
| R1-Distill-Qwen-7B | 55.5% | 92.8% |
| R1-Distill-LLaMA-8B | 50.4% | 89.1% |
| GPT-4o (comparison) | 9.3% | 76.6% |
| o1-mini (comparison) | 63.6% | 90.0% |
R1-Distill-Qwen-7B at 92.8% on MATH-500 beats GPT-4o (76.6%) with no RL training — just SFT on reasoning traces.
What it proved: Reasoning capability — the specific skill that RL training produces — transfers via SFT on reasoning traces alone. You don’t need to run RL yourself. You need a teacher that already has the capability and enough of its reasoning traces. This dramatically lowered the barrier to producing capable reasoning models.
Important note on metrics: The paper reports both pass@1 and consensus@64 (majority voting over 64 samples). The AIME numbers above are pass@1. Consensus@64 is significantly higher. Check which metric a source is citing before comparing.
What the case studies collectively show
ASCII summary of cost vs. capability over time:
Distillation cost vs. capability trajectory=============================================
Capability (vs. teacher) High | [Phi-3-mini] | [DeepSeek-R1-Distill] | [Orca 2] | [Orca 1] | [Vicuna] | [Alpaca] Low | +-------+--------+--------+--------+-------- $100 $300 $500 $1,000 $5,000+ Cost to produce
[DistilBERT is off this chart -- different domain (BERT-class)]The trajectory: each generation got more capability for less money by improving the quality of the distillation signal. The jump from Alpaca to Orca was not about more data — it was about richer data (reasoning traces). The jump from Orca to Phi was not about better fine-tuning — it was about better pretraining data quality.
References
- Taori, R. et al. (2023). “Alpaca: A Strong, Replicable Instruction-Following Model.” Stanford CRFM.
- Chiang, W-L. et al. (2023). “Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality.” LMSYS blog.
- Mukherjee, S. et al. (2023). “Orca.” ArXiv 2306.02707.
- Mitra, A. et al. (2023). “Orca 2.” ArXiv 2311.11045.
- Gunasekar, S. et al. (2023). “Textbooks Are All You Need (Phi-1).” ArXiv 2306.11644.
- Abdin, M. et al. (2024). “Phi-3 Technical Report.” ArXiv 2404.14219.
- Tunstall, L. et al. (2023). “Zephyr.” ArXiv 2310.16944.
- DeepSeek-AI (2025). “DeepSeek-R1.” ArXiv 2501.12948.
Related topics
- Theory: the theoretical basis for why these methods work
- LLM methods: the training methods these case studies applied
- Distillation prompts: the prompts used in Alpaca, Orca, and Phi