Testing AI

AI testing and AI quality requires unlearning some comfortable habits, then learning a new set of tools that are rarely handed to you in one clean list. copilots, agents, LLM-powered workflows, model-backed features, adaptive interfaces, RAG apps, generated code, and software whose behaviour depends on prompts, tools and non-deterministic outputs. when systems are non-deterministic, quality is not proven by one good run…quality is measured across samples, slices, traces, risks, failures, behaviour, user-experience, root-cause visibility and time.

In the old world, much of the work was aimed at making software behave exactly the same it behaved last time. That was the comfort…We built tests to make behavior idempotent, repeatable, and predictable. Pos 54 The arrival of LLMs has changed software engineering dramatically…Much software engineering work will eventually be done by AI. AI can generate code quickly. It can also generate buggy code quickly. It can produce plausible answers, plausible plans, and plausible explanations. The hard question is whether any of it is actually good. Pos 74 Testing becomes central again… as a discipline that decides whether AI-generated behaviour can be trusted. Pos 84 AI now touches almost every parting software and, increasingly, every part of human life. testing whether it works correctly, safely, fairly, securely, and usefully…can affect companies, institutions…