Battling Unfair Demons in Peer Review

Channel:
Microsoft Research
Subscribers:
351,000
Published on ● Video Link: https://www.youtube.com/watch?v=kKXL65kZU2g


Category:
Review
Duration: 59:00
922 views
21

Peer review is the backbone of scholarly research. It is however faced with a number of challenges (or "demons") such as subjectivity, bias/mis-calibration, noise, and strategic behavior, which cause unfairness in the process. This talk will present principled and practical approaches to battle these demons in peer review:

(1) Subjectivity: How to ensure that all papers are judged by the same yardstick?

(2) Mis-calibration: How to use ratings in presence of arbitrary or adversarial mis-calibration?

(3) Noise: How to assign reviewers to papers to simultaneously ensure fair and accurate evaluations in the presence of review noise?

(4) Strategic behavior: How to insulate peer review from strategic behavior of author-reviewers?

(5) Bias: How to rigorously test for existence of (gender/fame/race/...) biases in single-blind peer review?

The work uses tools from statistics and learning theory, social choice theory, information theory, game theory and decision theory. No prior knowledge on these topics will be assumed.

See more at https://www.microsoft.com/en-us/research/video/battling-unfair-demons-in-peer-review/



Other Videos By Microsoft Research


2019-09-26Resource-Efficient Redundancy for Large-Scale Data Processing and Storage Systems
2019-09-26Optimizing Declarative Graph Queries at Large Scale
2019-09-25SILK: Preventing Latency Spikes in Log-Structured Merge Key-Value Stores
2019-09-25Coverage Guided, Property Based Testing
2019-09-25Efficient Robot Skill Learning: Grounded Simulation Learning and Imitation Learning from Observation
2019-09-25Towards Secure and Interpretable AI: Scalable Methods, Interactive Visualizations, & Practical Tools
2019-09-25Sequential Estimation of Quantiles with Applications to A/B-testing and Best-arm Identification
2019-09-25Reproducible Codes and Cryptographic Applications
2019-09-25Inside AR and VR, a technical tour of the reality spectrum with Dr. Eyal Ofek [Podcast]
2019-09-24Verifying Web Page Layouts
2019-09-23Battling Unfair Demons in Peer Review
2019-09-19Engaging with Students and Parents in Bellevue School District in Multilingual Settings
2019-09-19Internship Program - MSR Montreal
2019-09-19Recent Advances in Unsupervised Image-to-Image Translation
2019-09-19Modeling User Experience in Games: Lessons Learned
2019-09-18HCI, IR and the search for better search with Dr. Susan Dumais [Podcast]
2019-09-17Efficient and Perceptually Plausible 3-D Sound For Virtual Reality
2019-09-17Dashboard Mechanisms for Online Marketplaces
2019-09-13Fly-through in the AirSim simulation Team Explorer created for the DARPA SubT Challenge
2019-09-12Compacting the Uncompactable: The Mesh Compacting Memory Allocator
2019-09-12Privacy-Preserving Statistical Learning and Testing


Tags:
peer review
scholarly research
social science
AI
microsoft research