Data & AI
Data Engineering
Data engineers move data from where it is created to where it is useful, reliably and on a schedule. Analysts and data scientists get the attention; without data engineers, neither has anything trustworthy to work with.
Start here
You need programming and SQL before this path makes sense. If you have neither, do Data Analysis first — it builds the SQL, and you will meet the problems data engineering exists to solve.
Before you begin
- Comfortable with SQL
- Can write Python beyond a script
- Basic command line
Foundations
Data engineering is software engineering applied to data. Both halves matter.
RequiredSQL and Python at engineering depth
Analysts write queries. Data engineers write queries that run every night for three years without being watched.
SQL and Python at engineering depth
Analysts write queries. Data engineers write queries that run every night for three years without being watched.
What to learn
- Advanced SQL and window functions
- Query performance and execution plans
- Python for data manipulation
- Testing data code
- Git and code review
Tools
- PostgreSQL
- Python
- pandas
Next: Data modelling.
RequiredData modelling
The design decision that determines whether analysts can answer their own questions or must ask you.
Data modelling
The design decision that determines whether analysts can answer their own questions or must ask you.
What to learn
- Normalisation versus denormalisation
- Star and snowflake schemas
- Fact and dimension tables
- Slowly changing dimensions
- Grain, and being precise about it
Practice
Next: Warehouses.
Where data lives
The storage layer, and why the choice matters more than the tooling above it.
RequiredData warehouses
The centre of gravity for analytics in almost every organisation.
Data warehouses
The centre of gravity for analytics in almost every organisation.
What to learn
- Columnar storage and why it is fast for analytics
- Partitioning and clustering
- Cost control on query-priced platforms
- Loading strategies
Tools
- BigQuery
- Snowflake
- Redshift
- DuckDB for local practice
Next: Lakes and lakehouses.
RecommendedData lakes and lakehouses
Where raw and semi-structured data lands before it is modelled, and where cost is controlled at volume.
Data lakes and lakehouses
Where raw and semi-structured data lands before it is modelled, and where cost is controlled at volume.
What to learn
- Object storage as a data layer
- File formats and why Parquet wins
- Table formats
- Partitioning strategy
- Schema evolution
Tools
- Object storage
- Parquet
- Delta Lake or Iceberg
Next: Moving the data.
Pipelines
The core craft: getting data from source to destination, correctly, on time, every time.
RequiredBatch pipelines and orchestration
Most data work is batch, and orchestration is what makes it survivable when step four of nine fails at 3am.
Batch pipelines and orchestration
Most data work is batch, and orchestration is what makes it survivable when step four of nine fails at 3am.
What to learn
- ETL versus ELT and when each fits
- Directed acyclic graphs
- Scheduling and dependencies
- Idempotency and safe reruns
- Backfilling
- Retry and alerting
Tools
- Airflow
- Dagster or Prefect
Practice
Project
intermediateAn automated daily pipeline
A pipeline that pulls from an API daily, lands raw data, transforms it into a modelled table, runs quality checks, and alerts on failure.
- Airflow or Dagster
- A warehouse
- Python
You have a pipeline that has run unattended for a week and you trust its output.
Next: Transformation as code.
RequiredTransformation and testing
Transformation logic buried in scripts is unreviewable. Treating it as version-controlled, tested code is what made data engineering a serious discipline.
Transformation and testing
Transformation logic buried in scripts is unreviewable. Treating it as version-controlled, tested code is what made data engineering a serious discipline.
What to learn
- Modular SQL transformations
- Testing data, not only code
- Lineage and documentation
- Incremental models
- Environments for data
Tools
- dbt
Next: Streaming, if you genuinely need it.
AdvancedStreaming
Powerful and considerably harder than batch. Most organisations adopt it before they need it and regret it.
Streaming
Powerful and considerably harder than batch. Most organisations adopt it before they need it and regret it.
What to learn
- Event streams and topics
- Consumer groups and offsets
- Windowing and late data
- Exactly-once semantics and their cost
- When hourly batch is genuinely the right answer
Tools
- Kafka
- A managed streaming service
Next: Quality and governance.
Quality and governance
The stage that decides whether the organisation trusts your data. Trust lost here takes years to rebuild.
RequiredData quality
A pipeline that runs successfully while producing wrong numbers is worse than one that fails loudly.
Data quality
A pipeline that runs successfully while producing wrong numbers is worse than one that fails loudly.
What to learn
- Freshness, volume and distribution checks
- Schema change detection
- Reconciliation against the source
- Alerting on data rather than infrastructure
- Failing loudly rather than silently
Tools
- dbt tests
- Great Expectations
Next: Governance.
RequiredGovernance and privacy
You will handle personal data. Handling it carelessly is a legal problem, not only an engineering one.
Governance and privacy
You will handle personal data. Handling it carelessly is a legal problem, not only an engineering one.
What to learn
- Cataloguing and lineage
- Access control on data
- Personal data identification
- Retention and deletion
- Anonymisation versus pseudonymisation
Next: Running it at scale.
Scale and operations
What changes when the data no longer fits on one machine or in one nightly window.
RecommendedDistributed processing
Necessary at genuine volume, and frequently used at volumes where a single machine would have been faster and cheaper.
Distributed processing
Necessary at genuine volume, and frequently used at volumes where a single machine would have been faster and cheaper.
What to learn
- Partitioning and shuffles
- Spark fundamentals
- Recognising when data fits in memory after all
- Cost per query
Tools
- Spark
- DuckDB for the single-machine case
Next: Operating the platform.
RequiredData platform operations
Pipelines are production systems. They deserve the same discipline as any other.
Data platform operations
Pipelines are production systems. They deserve the same discipline as any other.
What to learn
- CI/CD for data code
- Infrastructure as code
- Monitoring pipelines and cost
- On-call for data
- Documentation analysts can read
Project
advancedA complete data platform
Ingest from at least two sources including one API, land raw data, transform with tested models, expose a modelled layer to a dashboard, and add quality checks with alerting. All defined in code and deployed by CI.
- Airflow or Dagster
- dbt
- A warehouse
- GitHub Actions
An analyst could use your platform and trust the numbers without asking you how they were produced.
Where this leads
You do not have to pick one now. These are the directions this path opens up once you are working.
MLOps
Extend the same pipeline discipline to models and features.
View roadmapData Science
Move from serving data to modelling with it.
View roadmapMachine Learning Engineering
Build the systems that train and serve models.
View roadmapBackend Development
Broaden into general services and APIs.
View roadmapContinue your journey
The paths closest to this one. Skills overlap more than the job titles suggest.
Data Analysis
Turn messy data into a decision someone actually makes.
ViewData Science
Use statistics and modelling to answer questions data alone cannot.
ViewMLOps
Make machine learning repeatable, deployable and observable.
ViewBackend Development
Build the systems that store, protect and serve everyone's data.
ViewYou do not have to do this alone
Our programs are free, taught live, and built around the same progression. Join one and work through it alongside other people.