One of the trending topics on ScienceMag is AI conference use AI to assign papers to reviewers, some of the imperative highlights of which are given here. The researchers working on Artificial Intelligence (AI) are developing tools to meet the growing challenge of identifying and selecting a reviewer who can, intelligently screen through the huge number of the submitted paper to compute conferences.
Particularly when it comes to the computer sciences in contrast to the scientific fields, where the journals serve as a sight for peer review and publications, giving editors ample time to allocate papers to suitable reviewers based on their professional opinion. This isn’t the case for computer science, reviewers are track down in a hurry and often by necessity. During the annual conferences, the organizers receive the manuscripts concurrently, leaving them with only a week or less for assigning a multitude of papers to thousands of reviewers. And this workload crush has quadrupled over the last 5 years.
According to the annual AL conference on Neural Information Processing Systems (NeurIPS), it has received 40% more submissions for an event of December 2020 than the previous year. Leaving organizers with the herculean effort of assigning 31,000 reviews to 7000 reviewers. Luckily, they took help from Toronto Paper Matching System (TPMS) for assigning papers to reviewers. It compares the submitted text and reviewer’s papers and estimates the affinity between them, also letting the reviewers bid on the paper of their choice.
Open Review, a paper-reviewing platform uses Neural Network (a machine learning algorithm inspired by the brain’s wiring) for generating a more affluent depiction of content by, scrutinizing titles of paper and abstracts. Both Melisa Bok and Haw-Shiuan Chang of Open Review and the University of Massachusetts believe that computer science conferences like Neur IPS will use this technology in the coming year along with TPMS.
Organizers of AI conferences believe that improvement in the quality of matches will significantly improve the quality of peer review and published literature. Ivan Stelmakh of Carnegie Mellon University created an algorithm by the name of PeerReview4All, which magnifies quality of least good match by, evading poor matches and enhancing fairness. Stelmakh used PeerReview4All at the International Conference of Machine Learning (ICML) and reported to the Association for Advancement of Artificial Intelligence (AAAI) that the algorithm remarkably improved fairness without compromising average match quality, he inferred that Open Review is also offering a system for enhancing fairness by the name of FairFlow. Alina Beygelzimer is a computer scientist at Yahoo who believes NeurIPS will use one of these this year.
Stelmakh experimented to explore other ways, without relying on AI (that marches known sets of papers to the known set of reviewers). He invited students and recent graduates to review unpublished papers of colleagues and assigned them to mentors and concluded that novice ICML reviews were at least as good as experienced reviewers, which means that organizers could scale up recruiting process without drain.
To reduce manipulation by suspicious bids and preventing hacking of algorithms, a machine learning filter is used to form a countermeasure. It limits the chances of a single reviewer getting assigned multiple manuscripts, making it unlikely for a friend on an author from bidding on it. Laurent Charlin of the University of Montreal who directed the development of TPMS affinity-measure toolsbelieves that these tools are new therefore are difficult to evaluate on outperformance in a real-world setting. She believes “we are quite comfortable as a field with some level of automation. We have no reason not to use our tools”
To read in further detail, visit the remarkable post by the science journalist at Science.
Keyword:
Artificial Intelligence (AI), Computer science, Neural Information Processing Systems (NeurIPS), Toronto Paper Matching System (TPMS), neural network, PeerReview4All, International Conference of Machine Learning (ICML), FairFlow.
COMMENTS