Course Overview

Welcome to Testing AI Applications, a tool-agnostic course about testing software that does not give you the same answer twice.

Something has changed in the products we test. A feature used to be a function: same input, same output, and a test was a bet that the output would not move. Then your company shipped a chatbot, or a RAG-based search, or a summarizer, or an agent that calls tools on a user's behalf, and the bet stopped paying. expect(output).to.equal("...") is now a test that fails for reasons that are not bugs and passes while the feature is quietly getting worse.

This course is about what replaces it.

Who this course is for 👤

1. Testers and QA engineers whose company has shipped, or is about to ship, an AI feature and who need a testing strategy for it by next sprint
2. Automation engineers who can automate anything deterministic and are stuck the moment the expected result becomes "a reasonable answer"
3. Test leads and managers who have to answer the question "is this AI feature good enough to ship?" with something better than a demo

You do not need to be a data scientist. There is no maths in this course, no model training, and no fine-tuning. You need to know how to test software and be willing to trade a green tick for a score.

What this course is not ⚠️

This is not a course about using AI to help you test. That course already exists on the platform, it is AI-Assisted Testing for QAs, and it covers using AI to analyze requirements, generate test ideas and data, write better bug reports, and support your automation.

The distinction matters enough to be blunt about it:

• AI-Assisted Testing for QAs points AI at your testing work. The system under test is ordinary software
• This course points your testing work at AI. The system under test is itself non-deterministic

They complement each other and neither is a prerequisite for the other. If what you want is a smarter workflow, take the other one. If what you want is a way to test the LLM feature your team just shipped, you are in the right place.

What this course is

A concepts course. There is no repository to clone, no library to install, and no code to write. Every lesson is deliberately portable, because the tooling in this space is a year old and will be replaced. What will not be replaced is the underlying practice: curate a dataset, check what can be checked exactly, score what cannot, gate on the score, and attack your own feature before somebody else does.

Those ideas already have a name in the industry: evals. By the end of this course you will know what an eval suite is, how to build one, and how to defend the numbers it produces.

How this course is organized 🗺️

Twelve lessons, grouped into three parts, then a bonus:

1. Understanding the system under test: the vocabulary a tester actually needs, what non-determinism does to assertions, and the failure modes specific to AI features
2. Building an evaluation suite: golden datasets, the deterministic checks that still work, LLM-as-a-judge, and the metrics worth tracking
3. Shipping and keeping it safe: evals in CI, red teaming, and closing the loop with production observability

The bonus lesson tours the current tooling landscape (DeepEval, RAGAS, Promptfoo, LangSmith, Braintrust, and rolling your own) and maps every concept from the course onto what each tool calls it.

If you need more ground 📚

The lessons assume you have tested software before and can read a JSON payload. If APIs feel shaky, the HTTP, REST, and JSON fundamentals topic covers what you need, since almost every AI feature you will test is reached through an API.

👨‍🏫 The Fundamentals section is included with your account. Open it in another tab whenever a term goes past you, then come back.

Quizzes

At the end of every lesson there is a short quiz about the concepts covered.

👨‍🏫 You must score 100% in the quiz to unlock the next lesson.

The questions are about judgment, not recall. You will be asked what a number means and what you should do about it, not which metric belongs to which library.

Need help? 🆘

If you get stuck at any point during the course, you can reach out for support directly from the platform.

Look for the chat bubble with a question mark button in the bottom-right corner of any course or lesson page. Clicking it takes you to the Support page, where you'll find my email address, a template to help you describe your issue clearly, and a link to join our private Telegram group - where you can connect with fellow students and get direct support from the instructor whenever you hit a snag.

Share your feedback 💬

Got an idea to make this course - or the platform - even better? A topic you wish we covered, or a whole new course you'd love to see? I'd love to hear it.

Look for the chat bubble with a plus sign button in the bottom-right corner of any course or lesson page, just above the support button. Clicking it takes you to the Feedback page, where you can send your suggestions, improvement ideas, or course requests through a quick form.

💡 Every bit of feedback helps shape what comes next. Thank you for taking the time to share it!

Let's get started 🚀

That is everything you need to know to begin.

This overview is free to read without an account, so feel free to share it with anyone on your team who is about to inherit an AI feature and has no idea how to test it.

👨‍🏫 Head over to the next lesson to see exactly which of your testing habits stop working, and why.