Data Pipelines with Python and PostgreSQL

Channel:
United Kingdom SBCODE
Subscribers:
25,200
Published on ● Video Link: https://www.youtube.com/watch?v=QF-qHWekhxw


Duration: 13:23
24,828 views
629

Code : https://github.com/Sean-Bradley/Stream-Data-From-Flask-To-Postgres
I show how to use streaming techniques to build a data pipeline which pulls data from an external API that returns massive amounts of data, and insert it straight into a PostgreSQL data base ASAP.
The Python process that reads the huge amount of data, before inserting into Postgres is able to process the incoming chunks by use of the stream=True option on the Requests module, and the iter_content method of the Request response. The mock 3rd party API, which hosts the potentially petabytes of data, makes use of the Flask stream_with_context method.



Other Videos By SBCODE


2019-05-18100 Percent Test Coverage in Python
2019-05-18SH3D : Install Sweet Home 3D on Windows10
2019-05-18SH3D : Install Sweet Home 3d on Mac OSX
2019-05-08Call Slack Hooks With Zabbix 4.2 Triggers
2019-05-07Setup Zabbix Proxy on a Raspberry Pi behind a firewall : Zabbix 4.2
2019-05-07Install Zabbix Agent 4.2 on Windows Host Behind Firewall
2019-05-01Send Only SMTP Server using Postfix and Mailutils
2019-04-23Installing Zabbix 5 Part2 - Agent, Domain Name, SSL, Swap File
2019-04-23Install Zabbix 4.2 on Ubuntu 18
2019-04-20NETGEAR EX3700 WIFI Range Extender Instructions
2019-04-19Data Pipelines with Python and PostgreSQL
2019-04-10Command Design Pattern In Python
2019-04-09Abstract Factory Design Pattern In Python
2019-04-09Factory Design Pattern In Python
2019-04-05Super Speed HomeIdea3D
2019-03-2910000 Messages in 2.18 seconds with Python and gRPC
2019-03-27Linting a Python Project with Pylint
2019-03-19Parallel Processing CPU Bound Tasks with NodeJS
2019-03-11CI/CD with Github and Heroku
2019-03-08Add Swagger UI to Python Flask API
2019-03-06Hosting React/Redux Apps on GitHub Pages for Free


Tags:
python
flask
postgres
postgresql
streaming
stream_with_context
iter_content
data pipelines