High-Throughput Data-Intensive Computing: Shared-Scan Scheduling in Scientific Databases & the Cloud

Channel:
Microsoft Research
Subscribers:
344,000
Published on ● Video Link: https://www.youtube.com/watch?v=NwwWnryd5Z8


Duration: 1:00:30
189 views
0

Data-intensive computing consists of batch-processing workloads that scan massive data sets in parallel. The focus on data access, data movement, data ingest, and data production means that these workloads overwhelm the network and I/O capabilities of data centers and supercomputers. Major improvements in throughput are available by co-scheduling tasks that access the same data so that multiple tasks complete processing based on accessing and transferring the data a single time. Multiple tasks share I/O, network data transfer, cache space, and even computing with SIMD or vector processing. This talk will review the evolution of co-scheduling in data-intensive computing systems, including shared-scan scheduling for map/reduce workloads (Agrawal et al., VLDB 2008), data-driven batch processing for scientific databases (LifeRaft and JAWS), shared streaming-I/O for spatial workloads, and shared join processing for Pig programs and Nova workflows.



Other Videos By Microsoft Research


2016-08-16Developing cloud enabled applications for WP7 using Hawaii SDK
2016-08-16Butterfly Analysis: Adapting Dataflow Analysis to Dynamic Parallel Monitoring
2016-08-16Mini-course around Event-B and Rodin
2016-08-16How to force unsupervised neural networks to discover the right representation of images
2016-08-16On Optimal Multidimensional Mechanism Design
2016-08-16Mini-course around Event-B and Rodin, lecture 3
2016-08-16Intelligent Multicore Processor for Attention-based Object Recognition
2016-08-16Immersive Visual Communication with Depth
2016-08-16Applications 4
2016-08-16CFA2: Pushdown Flow Analysis for Higher-Order Languages
2016-08-16High-Throughput Data-Intensive Computing: Shared-Scan Scheduling in Scientific Databases & the Cloud
2016-08-16Grace: a new object-oriented educational programming language
2016-08-16Extreme Learning Machine: Why Tuning Is Not Required in Learning?
2016-08-16Systems 3
2016-08-16Security and Software Development
2016-08-16Identifying Peer Effects through Randomized Trials in Networks
2016-08-16Social networks, institutions, and the process of globalization
2016-08-16Keynote - Inside Windows Azure, MicrosoftΓÇÖs Cloud OS
2016-08-16The TPTP World ΓÇô Infrastructure for Automated Reasoning
2016-08-16Competitive Contagion in Networks
2016-08-16Mini-course around Event-B and Rodin, lecture 2


Tags:
microsoft research