Top 10 Data Engineering Mistakes
334A large fraction of big data projects fail to deliver return of investment, or take years before they do so. The reasons are typically a combination of project management, leadership, organisation, available competence, and technical failures. In this presentation, I will focus on the technical aspects, and present the most common or costly data engineering mistakes that I have experienced when building scalable data processing technology over the last five years, as well as advice for how to avoid them. The presentation includes war stories from large scale production environments, some that lead to reprocessing of petabytes of data, or DDoSing critical services with a Hadoop cluster, and what we learnt from the incidents.
EVENT:
#bbuzz 2018
SPEAKER:
Lars Albertsson
PERMISSIONS:
Original video was published with the Creative Commons Attribution license (reuse allowed).
CREDITS:
Original video source: https://www.youtube.com/watch?v=mv7PLnwzLpM&t=132s
Other Videos By Coding Tech
| 2018-06-30 | Solving Pokemon Blue With a Single, Huge Regular Expression |
| 2018-06-29 | The Cost Of JavaScript |
| 2018-06-28 | Knowledge Graphs & Deep Learning at YouTube |
| 2018-06-27 | Cryptography For Beginners |
| 2018-06-25 | Fearless Interview |
| 2018-06-24 | A Webpack Survival Guide |
| 2018-06-23 | A Case for Oxidation: The Rust Programming Language |
| 2018-06-22 | Strategies for Better UX |
| 2018-06-20 | When Fast is Faster Than Fastest |
| 2018-06-17 | If You Don’t Succeed At Beating HQ Trivia, Try Cheating! |
| 2018-06-16 | Top 10 Data Engineering Mistakes |
| 2018-06-13 | Microservices: How To Build Systems That Survive |
| 2018-06-11 | Introduction To TypeScript |
| 2018-06-10 | JS Callback Heaven |
| 2018-06-10 | Styled Components For Your React Apps |
| 2018-06-09 | [JavaScript] Master the Art of the AST and Take Control of Your JS |
| 2018-06-09 | Secure Coding Best Practices |
| 2018-06-07 | VS Code Can Do That?! VS Code Tips and Tricks |
| 2018-06-05 | Words of Wisdom on Coding |
| 2018-05-30 | Mathematics of Animation |
| 2018-05-26 | Building AI Products: From Paper to Prototype to Production |