Live-coding an asynchronous Rust crate for short EC2 jobs (part 3)

Channel:

Jon Gjengset

Subscribers:

97,800

Published on April 1, 2018 2:33:17 AM ● Video Link: https://www.youtube.com/watch?v=1qv1L8t2-W0

Duration: 5:08:56

3,054 views

27

Third part of a live-coding stream series where we design and build a Rust crate for running short-lived jobs and experiments on EC2 spot block instances. It's intended for users who are already somewhat familiar with Rust, but who want to see something larger and more involved be built. You can see the earlier videos over at https://www.youtube.com/playlist?list=PLqbS7AVVErFgY2faCIYjJZv_RluGkTlKt.

This video covers converting the previously entirely synchronous tsunami crate we've built thus far into one that uses futures/tokio to multiplex the setup of many machines onto a single thread. In the first part of the video, we replace all the innards of the library with asynchronous parts (including getting rid of rayon) while maintaining a (mostly) synchronous external API. In the second part, we start rewriting the library so that it presents an asynchronous interface to users, which would allow tsunami to be used in a larger asynchronous system.

Note that my internet connection cut out at around 2:56:42, so there's a somewhat jarring jump there.

Some useful chunking points:

- First: recap + discussion of futures-based APIs.
- Setting up infrastructure for using async rusoto: 11:22
- Multiplexing machine setup (and rm rayon!): 26:22
- Using async_ssh for multiplexed setup: 42:07
- Fighting the borrowchecker: 1:44:01
- Fought borrowchk and won. Cleaning up the futures API: 2:26:33
- Retrying operations with futures: 2:35:44
- Making the whole thing asynchronous: 2:50:09
- Cancelling requests on a timeout: 3:30:10

From this point forward, there's *a lot* of fighting with the compiler, and basically iterating through lots and lots of similar errors. You *may* not find this particularly interesting, and may at least want to watch at 2x speed. You've been warned. Brief ending at 5:07:01.

- Threading variables through *everywhere*: 3:54:01
- Making all the future error types match up: 4:09:54
- Oh no... borrowck + long futures... 4:36:05

The code is available here: https://github.com/jonhoo/tsunami. If you enjoyed this, go follow me on Twitter, or on Patreon at https://www.patreon.com/jonhoo to see announcements for new videos!