High Performance Data Engineering in Rust - Efficient Deduplication Example

Channel:

Pragmatic AI Labs

Subscribers:

17,700

Published on January 29, 2023 11:25:30 PM ● Video Link: https://www.youtube.com/watch?v=G2pg9tG97f0

Duration: 8:58

2,542 views

0

I demonstrate using Rust's speed and efficiency for a real-world data engineering use case - building a fast deduplication tool. Includes walkthrough of the thread pools, progress bars, command line interface, and checksumming logic with code examples.

https://github.com/noahgift/rdedupe

✨I build courses: https://insight.paiml.com/bzf


📚LLMOps Specialization: https://insight.paiml.com/a8e
📚Introduction to Generative AI: https://insight.paiml.com/ee2
📚Operationalizing LLMs on Azure: https://insight.paiml.com/e2u
📚Databricks to Local LLMs: https://insight.paiml.com/i6k
📚Advanced Data Engineering: https://insight.paiml.com/uvi

📚Rust Programming Specialization: https://insight.paiml.com/qwh
📚Rust for DevOps: https://insight.paiml.com/x14
📚Rust LLMOps: https://insight.paiml.com/g3b
📚Rust Fundamentals: https://insight.paiml.com/qyt
📚Data Engineering with Rust: https://insight.paiml.com/zm1
📚Python and Rust with Linux Command Line Tools: https://insight.paiml.com/jot


📚Applied Python Data Engineering Specialization: https://insight.paiml.com/5r9
📚Data Visualization with Python: https://insight.paiml.com/y9p
📚Virtualization, Docker, and Kubernetes for Data Engineering: https://insight.paiml.com/xtp
📚Spark, Hadoop, and Snowflake for Data Engineering: https://insight.paiml.com/f6j


📚MLOps | Machine Learning Operations Specialization: https://insight.paiml.com/l5u
📚Python Essentials for MLOps: https://insight.paiml.com/uvm
📚DevOps, DataOps, MLOps: https://insight.paiml.com/ggi
📚MLOps Tools: MLflow and Hugging Face: https://insight.paiml.com/y2v
📚MLOps Platforms: Amazon SageMaker and Azure ML: https://insight.paiml.com/ymb


📚Python, Bash and SQL Essentials for Data Engineering Specialization: https://insight.paiml.com/2or
📚Linux and Bash for Data Engineering: https://insight.paiml.com/d31
📚Scripting with Python and SQL for Data Engineering: https://insight.paiml.com/n3b
📚Python and Pandas for Data Engineering: https://insight.paiml.com/nz7
📚Web Applications and Command-Line Tools for Data Engineering: https://insight.paiml.com/o86


📚Building Cloud Computing Solutions at Scale Specialization: https://insight.paiml.com/hrt
📚Cloud Computing Foundations: https://insight.paiml.com/zrb
📚Cloud Data Engineering: https://insight.paiml.com/75t
📚Cloud Machine Learning Engineering and MLOps: https://insight.paiml.com/jjh
📚Cloud Virtualization, Containers and APIs: https://insight.paiml.com/ce5

📝 Guided Projects:
📝Object-Oriented Programming in Python:https://insight.paiml.com/n4h
📝MySQL-for-Data-Engineering: https://insight.paiml.com/e1k
📝Python Generators: https://insight.paiml.com/i9l
📝Build a Static Website with Rust and Zola: https://insight.paiml.com/a2h
📝Building Rust AWS Lambda Microservices with Cargo Lambda: https://insight.paiml.com/8ed
📝Rust Secret Cipher CLI: https://insight.paiml.com/zzr