Atrax, a distributed web crawler

Channel:
Microsoft Research
Subscribers:
344,000
Published on ● Video Link: https://www.youtube.com/watch?v=i5qLt0ShJSg


Duration: 1:05:39
4,144 views
16

This talk describes Atrax, a distributed and very fast web crawler. Running Atrax on a cluster of four DS20E Alpha servers saturates our internet connection. During a recent crawl, we were able to download about 115 Mbits/sec, or about 50 million web pages per day, over a sustained period of time. Atrax has been used to collect the raw data for numerous web studies performed at Compaq Research.



Other Videos By Microsoft Research


2016-09-15Microsoft Pix - a smarter camera app
2016-09-15Microsoft Pix - Take Better Photos of People, Automatically
2016-09-15Microsoft Pix - Have More Fun with Live Images
2016-09-13Programming Cells
2016-09-13Speech Processing on Multi-Genre Broadcast Media
2016-09-13Explorations on Multi-lingual Neural Machine Translation
2016-09-13Design Expo
2016-09-093D User Interface Techniques for Interactive Content
2016-09-09Hypervisor Scheduler Enhancement Using OS-Hardware Interactions
2016-09-09AI Theory and Practice: Hard Challenges and Opportunities Ahead
2016-09-09Atrax, a distributed web crawler
2016-09-09Around Borsuk's Problem
2016-09-09Is there an model for noise which reduces quantum computation to classical computation?
2016-09-09A topological colorful Helly theorem
2016-09-09Introduction to LUIS
2016-09-09The Tinkerers: The Amateurs, DIYers, and Inventors Who Make America Great
2016-09-0910 PRINT CHR$(205.5+RND(1)); : GOTO 10
2016-09-09Tenth of December: Stories
2016-09-09The Photography of Modernist Cuisine
2016-09-09Engineering Artificial Intelligence into the Human Immune System for Cancer Therapy
2016-09-09Dogfight: How Apple and Google Went to War and Started a Revolution


Tags:
microsoft research