Topic
Evaluation & Monitoring
Measuring an LLM system instead of arguing about it: evals, tracing, regression detection, and what to put on a dashboard.
3 posts · ← all posts
Temperature zero is not deterministic
Same prompt, same model, different answer. The mechanism behind non-determinism, the arithmetic that kills agent pilots, and what to do about it.
Your RAG doesn't have a retrieval problem. It has a ranking problem.
The document was in the index and the answer was still wrong. How to tell a retrieval failure from a ranking failure, what to fix first, and what each costs.
The gates are the product, not the orchestration
The orchestration in an AI content pipeline is the boring part. What decides whether text ships is a set of model-free functions with an exit code.