Can AI Pick the Best Reviewers Without Bias?

Imagine having to assign 40 projects to 40 reviewers, but without judgment errors or favoritism. AI can do it! The new system uses a transformer language model to compare a proposal’s content with reviewers’ CVs and Google Scholar profiles. In practice, AI reads and understands every word’s meaning, so it can tell how “similar” a reviewer is to a project. Then, with an optimization algorithm, it finds the best combination: who writes, who is the lead reviewer, who is just a reviewer, while considering workload and conflicts of interest. The results are impressive: projects receive a high similarity score (from 0.65 to 0.89) when the reviewer is truly competent, and low when not. The algorithm is faster and fairer because it doesn’t rely on subjective preferences. In short, AI makes the review process more transparent, scalable, and suitable for large scientific events. It’s an example of how AI can improve science itself, helping put the right ideas in the right hands.

The core of this approach is the combination of two key technologies: the transformer language model and integer linear programming (ILP). The transformer model, specifically all-mpnet-base-v2, was trained on a vast corpus of scientific texts, enabling it to capture complex semantic relationships between words and phrases. When a proposal is entered into the system, the model generates a 768-dimensional embedding vector representing its content. Similarly, reviewers’ CVs and Google Scholar profiles are transformed into embeddings. The cosine similarity between these vectors provides a score indicating how “in tune” a reviewer is with a proposal. These scores are then normalized and turned into rankings, feeding the ILP. The ILP is formulated with binary variables indicating whether a reviewer is assigned to a proposal. Constraints include: (1) workload balance, ensuring no reviewer receives more than a set number of proposals; (2) conflicts of interest, excluding reviewers who have collaborated with the author; (3) role-specific assignments, designating a lead reviewer, a scribe, and a secondary reviewer for each proposal. The objective is to maximize the sum of similarity scores, ensuring proposals are evaluated by the most relevant experts. Testing on 40 researchers across two disciplines showed that similarity between proposals and reviewers is much higher when content is related (0.65–0.89) and much lower when unrelated (from –0.21 to 0.08). Moreover, the system proved robust to the type of reviewer document: CV or Google Scholar profile, similarity scores are comparable. This indicates the algorithm can extract key expertise regardless of document format. In conclusion, integrating advanced NLP with mathematical optimization offers a scalable, transparent, and bias-free solution for managing scientific reviews, with potential applications in conferences, journals, and grant agencies.