How Public Evals Work and Fail, and What Should Change
How to read eval scores and what to question, from a teardown of 12 benchmarks.
eugeneyan.com · AI & ML · en
Writing about applied machine learning, recommender systems, LLM evaluation, and engineering leadership.
Writing about applied machine learning, recommender systems, LLM evaluation, and engineering leadership.
How to read eval scores and what to question, from a teardown of 12 benchmarks.
Sharing what we learned from Project Glasswing.
A sandboxed target, inputs that influence task difficulty, tools, and a grader.
Build a threat model, discover vulnerabilities, verify, triage, and patch.
Context as infra, taste as config, verify for autonomy, scale via delegation, close the loop.
An eventful year of progress in health and career, while making time for travel and reflection.
Label some data, align LLM-evaluators, and run the eval harness with each change.
Based on what I've learned from role models and mentors in Amazon
An LLM that can converse in English & item IDs, and make recommendations w/o retrieval or tools.
Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.
Recsys & search are converging with LLMs via semantic IDs, data augmentation, and unified foundation models.
What makes a good leader? What do good leaders do? And commando, soldier, and police leadership.
Learning to automate simple agentic workflows with Amazon Q CLI, Anthropic MCP, and tmux.
Applying the scientific method, building via eval-driven development, and monitoring AI output.
How I started, why I write, who I write for, how I write, and more.
Chip Huyen and I share what we've learned, best practices, and insights at NVIDIA GTC 2025.
Model architectures, data generation, training paradigms, and unified frameworks inspired by LLMs.
Exploring how an AI-powered reading experience could look like.
How the sharing of 1M Bluesky posts uncovered the strong anti-AI sentiment on Bluesky.