Complete Course
Testing AI Applications
Test what you cannot assert: build eval suites for LLM and agent-based applications, judge non-deterministic output, and red-team your AI features before your users do.
~4 hours of content
Course Content
10 comprehensive lessons covering everything you need to know
Introduction
Introduction
Course Overview
Learn what this course covers, who it is for, and how testing AI differs from using AI to test.
Introduction
Why AI applications break traditional QA
Why exact-match assertions, snapshot tests, and record-and-replay all fail here, and what replaces them.
Understanding the system under test
Lesson
Lesson 1LLMs, agents, and RAG in plain language
The vocabulary a tester needs: prompts, context windows, temperature, embeddings, retrieval, tool calls, and agent loops. No maths.
Lesson
Lesson 2Non-determinism and what it does to assertions
From pass or fail on one run to a score across many, plus thresholds, variance, and bug reports that survive it.
Lesson
Lesson 3The failure modes of AI features
Hallucination, wrong refusal, prompt injection, tool misuse, context leakage, cost drift, and silent regression after a model upgrade.
Building an evaluation suite
Lesson
Lesson 4Golden datasets
Curating cases, why 30 good ones beat 300 generated ones, recording expected behaviour instead of strings, and versioning the set.
Lesson
Lesson 5Deterministic checks first
Everything you can still assert exactly: schema, citations, tool calls, forbidden strings, refusals, and latency and cost budgets.
Lesson
Lesson 6LLM-as-a-judge
Writing a rubric, scoring on a scale, calibrating the judge against human labels, and the trap of a judge as wrong as the system it grades.
Lesson
Lesson 7Metrics that matter
RAG metrics, agent metrics, and cost and latency as first-class numbers, with one rule: a metric earns its place by changing a decision.
Shipping and keeping it safe
Lesson
Lesson 8Evals in CI
Turning scores into a gate: two lanes, honest sampling, thresholds derived from measured noise, and what to do when it goes red.
Lesson
Lesson 9Red teaming and adversarial testing
Prompt injection, jailbreaks, data exfiltration, PII leakage, and an adversarial suite you re-run on every model change.
Lesson
Lesson 10Observability and production feedback
Tracing calls in production, spotting failures users never report, and curating them back into the golden dataset.
Bonus & Conclusion
Bonus
Bonus: the AI testing tooling landscape
Every concept from the course mapped onto DeepEval, RAGAS, Promptfoo, LangSmith, Braintrust, and rolling your own.
Conclusion
Congratulations!
You have completed the Testing AI Applications course!