Data Engineering with Python and AI/LLMs – Data Loading Tutorial

Channel:

freeCodeCamp.org

Subscribers:

10,700,000

Published on April 16, 2025 5:39:21 PM ● Video Link: https://www.youtube.com/watch?v=T23Bs75F7ZQ

Duration: 0:00

44,727 views

1,603

Master data ingestion for data engineering with Python. Learn to tackle common pipeline failures like schema changes and API limits by adopting the mindset and practices of a senior platform engineer. This course covers essential techniques including extracting data from APIs, automatic schema management, incremental loading, and orchestrating scalable, automated workflows using modern tools.

Course developed by Alexey Grigorev & Adrian Brudaru.

💻 Code: https://github.com/dlt-hub/dlthub-education/tree/main/courses/freecodecamp/de_with_dlt_2025

🏗 ️ dlthub.com provided a grant to make this course possible.

⭐ ️ Contents ⭐ ️
Alexey's part
0:00:00 1. Introduction
0:08:02 2. What is data ingestion
0:10:04 3. Extracting data: Data Streaming & Batching
0:14:00 4. Extracting data: Working with RestAPI
0:29:36 5. Normalizing data
0:43:41 6. Loading data into DuckDB
0:48:39 7. Dynamic schema management
0:56:26 8. What is next?

Adrian's part
0:56:36 1. Introduction
0:59:29 2. Overview
1:02:08 3. Extracting data with dlt: dlt RestAPI Client
1:08:05 4. dlt Resources
1:10:42 5. How to configure secrets
1:15:12 6. Normalizing data with dlt
1:24:09 7. Data Contracts
1:31:05 8. Alerting schema changes
1:33:56 9. Loading data with dlt
1:33:56 10. Write dispositions
1:37:34 11. Incremental loading
1:43:46 12. Loading data from SQL database to SQL database
1:47:46 13. Backfilling
1:50:42 14. SCD2
1:54:29 15. Performance tuning
2:03:12 16. Loading data to Data Lakes & Lakehouses & Catalogs
2:12:17 17. Loading data to Warehouses/MPPs,Staging
2:18:15 18. Deployment & orchestration
2:18:15 19. Deployment with Git Actions
2:29:04 20. Deployment with Crontab
2:40:05 21. Deployment with Dagster
2:49:47 22. Deployment with Airflow
3:07:00 23. Create pipelines with LLMs: Understanding the challenge
3:10:35 24. Create pipelines with LLMs: Creating prompts and LLM friendly documentation
3:31:38 25. Create pipelines with LLMs: Demo

🎉 Thanks to our Champion and Sponsor supporters:
👾 Drake Milly
👾 Ulises Moralez
👾 Goddard Tan
👾 David MG
👾 Matthew Springman
👾 Claudio
👾 Oscar R.
👾 jedi-or-sith
👾 Nattira Maneerat
👾 Justin Hual

--

Learn to code for free and get a developer https://www.freecodecamp.org/p.org

Read hundreds of articles on programmhttps://freecodecamp.org/news/news