Data & AI
AI Engineering
AI engineers build products using foundation models. Calling a model is easy and getting a demo working takes an afternoon; making it reliable, affordable and safe enough that strangers can use it is the actual discipline.
Start here
If you can already build a web application, you can start at Stage 2. If you cannot, do Stage 1 first — this path produces products, and a model call with nowhere to live is not a product.
Before you begin
- Comfortable programming in Python or TypeScript
- Understand HTTP APIs
Foundations
Enough understanding of what these models are to predict how they will fail.
RequiredHow foundation models actually work
Without this, model behaviour looks random and you will debug by superstition.
How foundation models actually work
Without this, model behaviour looks random and you will debug by superstition.
What to learn
- Tokens and tokenisation
- Context windows
- Why models confabulate
- Temperature and sampling
- Training cutoffs
- Capability limits versus prompting problems
Practice
Next: Calling models properly.
RequiredWorking with model APIs
The mechanics: cost, latency and failure handling, all of which behave differently from an ordinary API.
Working with model APIs
The mechanics: cost, latency and failure handling, all of which behave differently from an ordinary API.
What to learn
- Chat and completion APIs
- Streaming responses
- Token cost accounting
- Rate limits and retries
- Timeouts on slow generations
- Model selection by task
Tools
- An LLM provider API
- An SDK in your language
Next: Getting reliable output.
Getting reliable output
The gap between a demo and a product is almost entirely in this stage.
RequiredDeliberate prompting
Prompting is not a trick. It is interface design for a probabilistic system, and it is testable.
Deliberate prompting
Prompting is not a trick. It is interface design for a probabilistic system, and it is testable.
What to learn
- Clear instructions and roles
- Few-shot examples
- Decomposing a task into steps
- Prompt versioning
- Testing a prompt across many inputs rather than one
Practice
Next: Structured output.
RequiredStructured output and validation
Free text cannot be consumed by software safely. This is what makes a model a component rather than a chatbot.
Structured output and validation
Free text cannot be consumed by software safely. This is what makes a model a component rather than a chatbot.
What to learn
- JSON schema and structured output modes
- Validating every response
- Retry on invalid output
- Tool and function calling
- Never trusting model output as code or SQL
Tools
- Zod or Pydantic
- Provider structured-output features
Project
beginnerA structured extraction service
An API that takes unstructured text and returns validated structured data, with schema validation, retries on failure, and a measured accuracy figure on a test set you built.
- An LLM API
- Pydantic or Zod
- FastAPI or Next.js
You can state how often your extraction is correct, with a number.
Next: Giving the model your own knowledge.
Retrieval over your own data
The most common real application, and the one with the most opportunities to get subtly wrong.
RequiredEmbeddings and vector search
Semantic search is the mechanism behind almost every 'chat with your documents' product.
Embeddings and vector search
Semantic search is the mechanism behind almost every 'chat with your documents' product.
What to learn
- What an embedding represents
- Similarity metrics
- Vector databases
- Chunking strategy, which matters more than the database
- Metadata filtering
Tools
- An embedding model
- pgvector, Qdrant or similar
Next: Assembling retrieval into a system.
RequiredRetrieval-augmented generation
Most RAG systems fail at retrieval, not generation. Knowing that is most of the debugging.
Retrieval-augmented generation
Most RAG systems fail at retrieval, not generation. Knowing that is most of the debugging.
What to learn
- Chunking and overlap
- Hybrid keyword and semantic search
- Reranking
- Citing sources
- Handling 'the answer is not in the documents'
- Evaluating retrieval separately from generation
Practice
Project
intermediateA grounded question-answering system
A system answering questions over a document collection, citing its sources, admitting when it does not know, with retrieval and generation evaluated separately.
- A vector database
- An LLM API
- An evaluation set you wrote
You can say what proportion of answers are correct and what proportion are correctly refused.
Next: Evaluation, which is the real discipline.
Evaluation and guardrails
The stage that separates a demo from something you would let strangers use. Most people skip it.
RequiredEvaluation
Without evaluation you cannot tell whether a prompt change improved things or broke them. You are guessing with extra steps.
Evaluation
Without evaluation you cannot tell whether a prompt change improved things or broke them. You are guessing with extra steps.
What to learn
- Building a test set from real inputs
- Automated scoring
- Model-as-judge and its limits
- Regression testing prompts
- Tracking quality over time
- Human review sampling
Tools
- An evaluation framework
- A spreadsheet, honestly, to start
Practice
Next: Safety.
RequiredGuardrails and safety
Your system will be given inputs you did not anticipate, some deliberately hostile.
Guardrails and safety
Your system will be given inputs you did not anticipate, some deliberately hostile.
What to learn
- Prompt injection and why it is not fully solvable
- Never granting a model authority it should not have
- Input and output filtering
- Human in the loop for consequential actions
- Privacy of the data you send to a provider
- Refusal and escalation paths
Practice
Next: Agents, carefully.
Agents and production
Powerful, frequently oversold, and genuinely useful in narrow, well-bounded cases.
AdvancedTool use and agents
Letting a model take actions multiplies both its usefulness and its blast radius.
Tool use and agents
Letting a model take actions multiplies both its usefulness and its blast radius.
What to learn
- Tool definitions
- Multi-step loops and termination conditions
- Error recovery
- Cost and latency of long loops
- Scoping permissions narrowly
- Knowing when a normal program is better
Next: Running it in production.
RequiredProduction concerns
Model-backed features have cost and latency profiles unlike anything else you have deployed.
Production concerns
Model-backed features have cost and latency profiles unlike anything else you have deployed.
What to learn
- Caching responses
- Streaming for perceived latency
- Cost monitoring and budgets
- Fallback models
- Logging inputs and outputs for debugging, within privacy limits
- Provider outages
Project
advancedA production AI feature
Ship a model-backed feature to real users with an evaluation suite in CI, guardrails, cost monitoring, a fallback path, and a written note on its limitations.
- An LLM API
- Your application framework
- An evaluation suite
Real people use it, you know how often it is wrong, and you know what happens when the provider is down.
Where this leads
You do not have to pick one now. These are the directions this path opens up once you are working.
Continue your journey
The paths closest to this one. Skills overlap more than the job titles suggest.
AI Application Development
Build products people use, that happen to have a model inside.
ViewMachine Learning Engineering
Build models that run in production, not only in notebooks.
ViewMLOps
Make machine learning repeatable, deployable and observable.
ViewBackend Development
Build the systems that store, protect and serve everyone's data.
ViewYou do not have to do this alone
Our programs are free, taught live, and built around the same progression. Join one and work through it alongside other people.