Who Said What: Modeling individual Labelers Improves Classification
Data are often labeled by many different experts, with each expert labeling a small fraction of the data and each sample receiving multiple labels. When experts disagree, the standard approaches are to treat the majority opinion as the truth or to model the truth as a distribution, but these do not make any use of potentially valuable information about which expert produced which label. We propose modeling the experts individually and then learning averaging weights for combining them, possibly in sample-specific ways. This allows us to give more weight to more reliable experts and take advantage of the unique strengths of individual experts at classifying certain types of data. We show that our approach performs better than three competing methods in computer-aided diagnosis of diabetic retinopathy.
Melody is a deep learning resident at Google Brain. Previously she interned as a trader at D. E. Shaw and conducted stem cell research in Doug Melton's lab. She received an M.A. in Statistics and B.A. in Chemistry and Physics from Harvard University, where she graduated with highest ranking. In her youth she medalled at the International Physics, Chemistry, and Biology Olympiads and was invited to the Canadian International Math Olympiad training camp. Melody has published in The Huffington Post, Harvard Political Review, and The Harvard Crimson. She is constantly fawning over music, reading rationality and psychoanalysis blogs, and enjoying long random walks along undirected paths and other outdoor activities.