Optimizing ECS with Pulumi and Python | "glowing telegram" - Episode 133
3In this video, I navigate through the process of optimizing our Glowing-Telegram project using Pulumi and Python to manage AWS ECS Fargate resource allocation. We start by pushing our build to AWS Elastic Container Registry (ECR) and delve into monitoring ECS using CloudWatch. One of our goals is to improve task performance by reconfiguring CPU and memory allocations within ECS for more efficient processing. This involves checking and adjusting ECS cluster settings, particularly the usage of container insights for better performance metrics.
I experiment with various CPU and memory settings in an attempt to find the sweet spot for our resource needs. Interestingly, we encounter diminishing returns as doubling resources doesn't quite halve the task times, indicating potential bottlenecks unrelated to computation, such as network IO while pulling video data from S3.
Throughout, we aim to enhance our silence detection code, adjusting configurations to accurately process video files. Even though tasks aren't executing as fast as hoped, they're yielding correct outputs, as demonstrated by the successful integration of silence segments into DynamoDB.
Finally, I discuss my thought process on further enhancements, such as implementing trace logging for a deeper analysis of task durations, and potentially restructuring how data is streamed into processing to cut down on IO wait times.
Join the journey of refining serverless workflows for the Glowing-Telegram project and enhancing Python-on-Pulumi deployments.
🔗 Check out my Twitch channel for more streams: https://www.twitch.tv/saebyn
GitHub: https://github.com/saebyn/glowing-telegram
Discord: https://discord.gg/N7xfy7PyHs