I am an AI Research Engineer at Duolingo, where I advance evaluation science and infrastructure for AI-enabled learning experiences. In August 2026, I graduated with my Ph.D. from the School of Computer Science at Carnegie Mellon University. There, I was extremely fortunate to be advised by Ken Holstein and Steven Wu.

I design tools and frameworks for translating diverse forms of human expertise into rigorous, scalable AI evaluations. I leverage these evaluations to inform decisions about when and how AI systems should be deployed, and to shape their behavior through post-training.

Luke Guerdan
lukemguerdan[at]gmail[dot]com

Recent News

Oct 2026 This fall, I am co-organizing the AI Measurement Science Workshop at CoLM and the Predictive AI Evaluation Competition at NeurIPS.
Aug 2026 I defended my PhD dissertation and joined Duolingo as an AI Research Engineer in NYC!
Jan 2026 Our paper Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas was accepted at ICLR 2026.
Dec 2025 I am excited to present Validating LLM-as-a-Judge Systems under Rating Indeterminacy at NeurIPS. Check out the code release, tutorial, and blog post about the work to learn more.
Aug 2025 I am organizing a webinar series on AI Evaluation Science, hosted through the AI Measurement Science & Engineering Research Center at CMU. Watch the recordings here!

Selected Work

(*) Co-first Author; (**) Co-senior Author
  1. Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas Luke Guerdan*, Justin Whitehouse*, Kimberly Truong*, Kenneth Holstein, and Zhiwei Steven Wu International Conference on Learning Representations (ICLR), 2026 [arXiv]
  2. Validating LLM-as-a-Judge Systems under Rating Indeterminacy Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Zhiwei Steven Wu, and Alexandra Chouldechova Advances in Neural Information Processing Systems (NeurIPS), 2025 [arXiv] [Blog] [Code]
  3. Bridging Prediction and Intervention Problems in Social Systems Lydia T. Liu*, Inioluwa Deborah Raji*, Angela Zhou*, Luke Guerdan, Jessica Hullman, Daniel Malinsky, and 30 Additional Authors arXiv, 2025 [arXiv]
  4. Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks Luke Guerdan*, Devansh Saxena*, Stevie Chancellor**, Zhiwei Steven Wu**, and Kenneth Holstein** Conference on Computer-Supported Cooperative Work and Social Computing (CSCW), 2025 [arXiv]
  5. Predictive Performance Comparison of Decision Policies Under Confounding Luke Guerdan, Amanda Coston, Kenneth Holstein, and Zhiwei Steven Wu International Conference on Machine Learning (ICML), 2024 [arXiv] [Code]
  6. Counterfactual Prediction Under Outcome Measurement Error Luke Guerdan, Amanda Coston, Kenneth Holstein, and Zhiwei Steven Wu Conference on Fairness, Accountability, and Transparency (FAccT), 2023 [PDF] [Video] [Code]  Best Paper Award
  7. Training Towards Critical Use: Learning to Situate AI Predictions Relative to Human Knowledge Anna Kawakami, Luke Guerdan, Yang Cheng, Kate Glazko, Matthew Lee, Scott Carter, Nikos Arechiga, Haiyi Zhu, and Kenneth Holstein Conference on Collective Intelligence, 2023
  8. Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making Luke Guerdan, Amanda Coston, Zhiwei Steven Wu, and Kenneth Holstein Conference on Fairness, Accountability, and Transparency (FAccT), 2023 [PDF] [Video]