Tech Talk: Obtaining and Validating Big Data
3In this video Konstantinos Pouliasis presents a toolkit combining bash scripting, a refined postgres database design and Node.js libraries for effectively fetching, validating and storing big data publicly available from governmental resources.
Basic unix scripting programming constructs and specific commands are being presented. Curl command is utilized to fetch the zipped CSV files, the unzip command is coupled with sed to extract data and achieve some first data cleanup. A database side validation of CSV data integrity is being exposed and, next, a database design technique utilizing Postgres partitioning is being elaborated as appropriate for large seasonal data sets. Finally, the library pg-pool with its multithreaded connection capabilities is presented as an ideal complementary to automatizing data insertion using Node.js.
----
Fullstack Academy was recently ranked the #1 coding bootcamp in the U.S. Learn more at https://www.fullstackacademy.com.
Other Videos By Fullstack Academy
| 2017-03-28 | Everyone's a Critic |
| 2017-03-28 | It's MakanTime! |
| 2017-03-28 | Meal Memories |
| 2017-03-28 | Basic Retirement Calculator |
| 2017-03-28 | Daily Fantasy Football Filter |
| 2017-03-28 | Archery Strike: Medieval Offensive |
| 2017-03-28 | State Driven Actors |
| 2017-03-21 | Pymetrics with Dr. Julie Yoo |
| 2017-03-20 | Velocity.js Tutorial - An Introduction to Velocity.js |
| 2017-03-20 | Phaser Tutorial - How to Make a Game Using Phaser |
| 2017-03-20 | Tech Talk: Obtaining and Validating Big Data |
| 2017-03-20 | Stackathon Presentation: Maze Hunter |
| 2017-03-20 | Stackathon Presentation: Advanced Wars Redux |
| 2017-03-20 | Stackathon Presentation: Lanterns |
| 2017-03-20 | Stackathon Presentation: HTML5 Space Shooter |
| 2017-03-20 | Stackathon Presentation: Inspo-Generator |
| 2017-03-20 | Stackathon Presentation: nearest |
| 2017-03-20 | Stackathon Presentation: OurPhotos |
| 2017-03-20 | Stackathon Presentation: Faces of Twitch Chat |
| 2017-03-20 | Stackathon Presentation: Blooming Ice |
| 2017-03-20 | Stackathon Presentation: Machi Koro |