CV

Harrie Oosterhuis

Curriculum Vitae

Employment

2020-present: Assistant Professor - Radboud University - Nijmegen, The Netherlands
2023-2025: Visiting Research Scholar - Google DeepMind - Amsterdam, The Netherlands
2022: Staff ML Researcher - Twitter - Remote
2018: Visiting PhD Student - RMIT University - Melbourne, Australia
- Visited the ISAR research group to work with J. Shane Culpepper.
2016-2020: PhD Student - University of Amsterdam - Amsterdam, The Netherlands
2017: Research Intern - Google Brain - Mountain View, California, U.S
2016: Research Intern - Google Strategic Technologies - Mountain View, California, U.S
2015: Research Intern - Google Strategic Technologies - Mountain View, California, U.S
2013–2014: Research and Development, Software Engineer - MediaSynced - Amsterdam, The Netherlands.
2013: Research Intern - National Aerospace Laboratory - Amsterdam, The Netherlands

Education

PhD in Machine Learning for Information Retrieval, University of Amsterdam, 2016-2020
- Received Cum Laude.
- Research Group: Information Retrieval Lab Amsterdam
- Promotor: prof. dr. Maarten de Rijke
- Thesis: Learning from User Interactions with Rankings: A Unification of the Field
- Thesis won an Andreas Bonn medal in 2024.
MSc in Artificial Intelligence, University of Amsterdam, 2014-2016
- Graduated Cum Laude with an average of 9.1.
- Completed Honours program by performing and publishing research in the ILPS research group headed by prof. dr. Maarten de Rijke.
BSc in Artificial Intelligence, University of Amsterdam, 2011-2014
- Graduated Cum Laude and Cum Honore with an average of 8.3.

Grants and Awards

2024: Early Career Researcher Award - ACM SIGIR - Washington DC, USA
- Award received in the research category with the following motivation: “For exceptional research contributions introducing both theoretical and empirical innovations with extensive impact on research and practice.”
2024: Andreas Bonn Medal - Het Genootschap ter bevordering van Natuur-, Genees- en Heelkunde - Amsterdam, The Netherlands
- Award received for PhD thesis: Learning from user interactions with rankings: A unification of the field
2023: Radboud Science Award - Wetenschapsknooppunt Radboud University - Nijmegen, The Netherlands
- Proposal title: Zoekende computers
- Funding amount: €5.000 EUR
2023: Veni Grant - NWO Talent Programme (Round 2022) - The Netherlands.
- Proposal title: Search Systems with a Bayesian Understanding of Their Users (BAYES-SEARCH)
- Funding amount: €280.000 EUR
2022: Best Paper Award - ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR ’22) - Madrid, Spain.
- Paper: Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank
- Author: Harrie Oosterhuis (solo-author)
2021: Best Paper Award - International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’21) - Montreal, Canada (Online Event).
- Paper: Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness
- Author: Harrie Oosterhuis (solo-author)
2021: The Google Research Scholar Award - The Google Research Scholar Program.
- Proposal title: Search and Recommendation Systems that Learn from Diverse User Preferences
- Funding amount: $60,000 USD
2021: Best Paper Award - ACM International Conference on Web Search and Data Mining (WSDM ’21) - Jerusalem, Israel (Online Event).
- Paper: Unifying Online and Counterfactual Learning to Rank
- Authors: Harrie Oosterhuis and Maarten de Rijke
2019: Best Reproducibility Paper Award - European Conference on Information Retrieval (ECIR ’19) - Cologne, Germany.
- Paper: Optimizing Ranking Models in an Online Setting
- Authors: Harrie Oosterhuis and Maarten de Rijke
2017: Outstanding Reviewer Award - International Conference on Information and Knowledge Management (CIKM ’17) - Singapore.

Conference and Workshop Organization

2024: Workshop Co-organizer - Third Edition of the CONSEQUENCES Workshop at the Conference on Recommender Systems (RecSys) - Bari, Italy DC, USA.
2024: General Chair - ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR) - Washington DC, USA.
2023: Workshop Co-organizer - Second Edition of the CONSEQUENCES Workshop, Conference on Recommender Systems (RecSys), Singapore.
2022: Workshop Co-organizer - First Edition of the CONSEQUENCES Workshop at the Conference on Recommender Systems (RecSys - Seattle, USA.

PhD Candidate Supervision

2019-2024 (PhD 2024): Jin Huang - Bias Mitigation Methods for Sequential Recommendation.
- Jin Huang. Learning Recommender Systems from Biased User Interactions. PhD thesis, University of Amsterdam, February 2024.
- Co-supervised by Maarten de Rijke.
2021-2025 (expected): Norman Knyazev - Bias Mitigation for Recommendation Algorithms
2021-2025 (expected): Shashank Gupta - Debiasing Methods for Conversational Recommendation
- Co-supervised by Maarten de Rijke.
2023-2027 (expected): Jingwei Kang - Full-Page Personalization for Recommendation
- Co-supervised by Maarten de Rijke.
2024-2028 (expected): Oscar Ramirez Milian - Bayesian Learning to Rank.

Publications

According to my Google Scholar profile my publications have received over 1100 citations and my h-index is 21.

Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank

Gupta, S., Oosterhuis, H., & de Rijke, M. (2024, October). Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM ’24).

Optimal Baseline Corrections for Off-Policy Contextual Bandits

Gupta, S., Jeunen, O., Oosterhuis, H., & de Rijke, M. (2024, October). Optimal Baseline Corrections for Off-Policy Contextual Bandits. In Proceedings of the 18th ACM Conference on Recommender Systems (RecSys ’24).

Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I.

Oosterhuis, H., Jagerman, R., Qin, Z., Wang, X., & Bendersky, M. (2024). Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative AI. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’24), August 25–29, 2024, Barcelona, Spain. ACM, New York, NY, USA, 11 pages.

Local Feature Selection without Label or Feature Leakage for Interpretable Machine Learning Predictions

Oosterhuis, H., Lyu, L., & Anand, A. (2024). Local Feature Selection without Label or Feature Leakage for Interpretable Machine Learning Predictions. In Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024.

Going Beyond Popularity and Positivity Bias: Correcting for Multifactorial Bias in Recommender Systems

Huang, J., Oosterhuis, H., Mansoury, M., van Hoof, H., & de Rijke, M. (2024, July). Going Beyond Popularity and Positivity Bias: Correcting for Multifactorial Bias in Recommender Systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 416-426).

Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted Trees

Kang, J., de Rijke, M., & Oosterhuis, H. (2024, July). Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted Trees. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 2390-2394).

Is Interpretable Machine Learning Effective at Feature Selection for Neural Learning-to-Rank?

Lyu, L., Roy, N., Oosterhuis, H., & Anand, A. (2024, March). Is Interpretable Machine Learning Effective at Feature Selection for Neural Learning-to-Rank?. In European Conference on Information Retrieval (pp. 384-402). Cham: Springer Nature Switzerland.

Unbiased Learning to Rank: On Recent Advances and Practical Applications

Gupta, S., Hager, P., Huang, J., Vardasbi, A., & Oosterhuis, H. (2024, March). Unbiased Learning to Rank: On Recent Advances and Practical Applications. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining (pp. 1118-1121).

A First Look at Selection Bias in Preference Elicitation for Recommendation

Shashank Gupta, Harrie Oosterhuis, and Maarten de Rijke. 2023. A First Look at Selection Bias in Preference Elicitation for Recommendation. In CONSEQUENCES Workshop at RecSys ’23, September 18-22, 2023, Singapore. ACM, New York, NY, USA, 6 pages.

A Lightweight Method for Modeling Confidence in Recommendations with Learned Beta Distributions

Knyazev, N., & Oosterhuis, H. (2023, September). A lightweight method for modeling confidence in recommendations with learned beta distributions. In Proceedings of the 17th ACM conference on recommender systems (pp. 306-317).

A Deep Generative Recommendation Method for Unbiased Learning from Implicit Feedback

Gupta, S., Oosterhuis, H., & de Rijke, M. (2023, August). A Deep Generative Recommendation Method for Unbiased Learning from Implicit Feedback. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval (pp. 87-93).

Recent Advances in the Foundations and Applications of Unbiased Learning to Rank

Gupta, S., Hager, P., Huang, J., Vardasbi, A., & Oosterhuis, H. (2023, July). Recent advances in the foundations and applications of unbiased learning to rank. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 3440-3443).

Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization

Gupta, S., Oosterhuis, H., & de Rijke, M. (2023, July). Safe deployment for counterfactual learning to rank with exposure-based risk minimization. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 249-258).

Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank

H. Oosterhuis. "Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank." ACM Transactions on Information Systems 41.3 (2023): 1-33.

VAE-IPS: A Deep Generative Recommendation Method for Unbiased Learning from Implicit Feedback

S. Gupta, H. Oosterhuis, and M. de Rijke. "VAE-IPS: A Deep Generative Recommendation Method for Unbiased Learning from Implicit Feedback." CONSEQUENCES+REVEAL Workshop at RecSys ’22, September 23, 2022, Seattle, USA.

Closing the Gender Wage Gap: Adversarial Fairness in Job Recommendation

C. Rus, J. Luppes, H. Oosterhuis and G. Schoenmacker. "Closing the Gender Wage Gap: Adversarial Fairness in Job Recommendation." RecSys in HR’22: The 2nd Workshop on Recommender Systems for Human Resources, in conjunction with the 16th ACM Conference on Recommender Systems, September 22, 2022, Seattle, USA.

Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness (Extended Abstract)

H. Oosterhuis. "Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness (Extended Abstract)." In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence, 2022.

Learning-to-Rank at the Speed of Sampling: Plackett-Luce Gradient Estimation With Minimal Computational Complexity

H. Oosterhuis. "Learning-to-Rank at the Speed of Sampling: Plackett-Luce Gradient Estimation With Minimal Computational Complexity." In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2022.

State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study

J. Huang, H. Oosterhuis, B. Cetinkaya, T. Rood and M. de Rijke. "State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study." In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2022.

Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank

H. Oosterhuis. "Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank." In Proceedings of the 2022 ACM SIGIR International Conference on the Theory of Information Retrieval. ACM, 2022.

The Bandwagon Effect: Not Just Another Bias

N. Knyazev and H. Oosterhuis. "The Bandwagon Effect: Not Just Another Bias." In Proceedings of the 2022 ACM SIGIR International Conference on the Theory of Information Retrieval. ACM, 2022.

FOCUS: Flexible Optimizable Counterfactual Explanations for Tree Ensembles

A. Lucic, H. Oosterhuis, H. Haned and M. de Rijke. "FOCUS: Flexible Optimizable Counterfactual Explanations for Tree Ensembles." In AAAI 2022: Thirty-Sixth AAAI Conference on Artificial Intelligence. 2022.

It Is Different When Items Are Older: Debiasing Recommendations When Selection Bias and User Preferences Are Dynamic

J. Huang, H. Oosterhuis and M. de Rijke. "It Is Different When Items Are Older: Debiasing Recommendations When Selection Bias and User Preferences Are Dynamic." In Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM ’22). ACM, 2021.

Unifying Online and Counterfactual Learning to Rank (Extended Abstract)

H. Oosterhuis and M. de Rijke. "Unifying Online and Counterfactual Learning to Rank (Extended Abstract)." In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence, 2021.

Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness

H. Oosterhuis. "Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness." In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2021.

Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank

H. Oosterhuis and M. de Rijke. "Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank." The World Wide Web Conference. ACM, 2021.

Unifying Online and Counterfactual Learning to Rank

H. Oosterhuis and M. de Rijke. "Unifying Online and Counterfactual Learning to Rank." In Proceedings of the 14th ACM International Conference on Web Search and Data Mining (WSDM ’21). ACM, 2021.

Learning from User Interactions with Rankings: A Unification of the Field

Harrie Oosterhuis. "Learning from User Interactions with Rankings: A Unification of the Field." PhD thesis, University of Amsterdam, November 2020.

When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank

A. Vardasbi, H. Oosterhuis, M. de Rijke. "When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank." In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. ACM, 2020.

Keeping Dataset Biases out of the Simulation: A Debiased Simulator for Reinforcement Learning based Recommender Systems

J. Huang, H. Oosterhuis, M. de Rijke, and H. van Hoof. "Keeping Dataset Biases out of the Simulation: A Debiased Simulator for Reinforcement Learning based Recommender Systems." In Fourteenth ACM Conference on Recommender Systems, pp. 190-199. ACM, 2020.

Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking

H. Oosterhuis, and M. de Rijke. "Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking." In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval. ACM, 2020.

Policy-Aware Unbiased Learning to Rank for Top-k Rankings

H. Oosterhuis, and M. de Rijke. "Policy-Aware Unbiased Learning to Rank for Top-k Rankings." In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2020.

Unbiased Learning to Rank: Counterfactual and Online Approaches

H. Oosterhuis, R. Jagerman, and M. de Rijke. "Unbiased Learning to Rank: Counterfactual and Online Approaches." In Companion Proceedings of the Web Conference 2020. ACM, 2020.

Learning to Rank in Theory and Practice: From Gradient Boosting to Neural Networks and Unbiased Learning

C. Lucchese, F. M. Nardini, R. K. Pasumarthi, S. Bruch, M. Bendersky, X. Wang, H. Oosterhuis, R. Jagerman, M. de Rijke. "Learning to Rank in Theory and Practice: From Gradient Boosting to Neural Networks and Unbiased Learning." In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2019.

To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions

R. Jagerman, H. Oosterhuis and M. de Rijke. "To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions." In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2019.

Optimizing Ranking Models in an Online Setting

H. Oosterhuis, M. de Rijke. "Optimizing Ranking Models in an Online Setting." In European Conference on Information Retrieval. Springer, Cham, 2019.

The Potential of Learned Index Structures for Index Compression

H. Oosterhuis, J. S. Culpepper, M. de Rijke. "The Potential of Learned Index Structures for Index Compression." In Proceedings of the 23rd Australasian Document Computing Symposium. ACM, 2018.

Differentiable Unbiased Online Learning to Rank

H. Oosterhuis, M. de Rijke. "Differentiable Unbiased Online Learning to Rank." In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. ACM, 2018.

Ranking for Relevance and Display Preferences in Complex Presentation Layouts

H. Oosterhuis, M. de Rijke. "Ranking for Relevance and Display Preferences in Complex Presentation Layouts." In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM, 2018.

Optimizing Interactive Systems with Data-Driven Objectives

Z. Li, A. Grotov, J. Kiseleva, M. de Rijke, H. Oosterhuis. "Optimizing Interactive Systems with Data-Driven Objectives." In arXiv preprint. 2018.

Sensitive and Scalable Online Evaluation with Theoretical Guarantees

H. Oosterhuis, M. de Rijke. "Sensitive and Scalable Online Evaluation with Theoretical Guarantees." In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 2017.

Balancing Speed and Quality in Online Learning to Rank for Information Retrieval

H. Oosterhuis, M. de Rijke. "Balancing Speed and Quality in Online Learning to Rank for Information Retrieval." In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 2017.

Query-level Ranker Specialization

R. Jagerman, H. Oosterhuis, M. de Rijke. "Query-level Ranker Specialization." In Proceedings of the 1st International Workshop on LEARning Next gEneration Rankers, co-located with the 3rd ACM International Conference on the Theory of Information Retrieval (ICTIR ’2017). ACM, 2017.

Semantic Video Trailers

H. Oosterhuis, S. Ravi, M. Bendersky. "Query-level Ranker Specialization." In ICML 2016 Workshop on Multi-View Representation Learning. 2016.

Multileave Gradient Descent for Fast Online Learning to Rank

A. Schuth, H. Oosterhuis, S. Whiteson, M. de Rijke. "Multileave Gradient Descent for Fast Online Learning to Rank." In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining. ACM, 2016.

Probabilistic Multileave Gradient Descent

H. Oosterhuis, M. de Rijke. "Probabilistic Multileave Gradient Descent." In European Conference on Information Retrieval. Springer, Cham, 2016.

Probabilistic Multileave for Online Retrieval Evaluation

A. Schuth, R. Bruintjes, F. Büttner, J. van Doorn, C. Groenland, H. Oosterhuis, C. Tran, B. Veeling, J. van der Velde, R. Wechsler, D. Woudenberg, M. de Rijke. "Probabilistic Multileave for Online Retrieval Evaluation." In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2015.

Talks

Unifying Online and Counterfactual Learning to Rank

November 26, 2021

Invited Seminar Talk at Search Engines Amsterdam (SEA), University of Amsterdam (Online Event), Amsterdam, The Netherlands (Online Event)

SIKS Course: Unbiased Learning to Rank

November 26, 2021

Workshop Course at SIKS Course: Advances in Information Retrieval, Netherlands Research School for Information and Knowledge Systems, Berg en Dal, The Netherlands

Recent Advances in Unbiased Learning to Rank from Position-Biased Click Feedback

November 22, 2021

Invited Seminar Talk at University of Glasgow, Information Retrieval Seminar, Glasgow, United Kingdom (Online Event)

Recent Advances in Unbiased Learning to Rank from Position-Biased Click Feedback

October 21, 2021

Invited Talk at Twitter Research, London, United Kingdom (Online Event)

Recent Advances in Unbiased Learning to Rank from Position-Biased Click Feedback

October 08, 2021

Invited Seminar Talk at CIIR talk series, UMass Amherst, United States (Online Event)

Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness

July 12, 2021

Conference Proceedings Talk at the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’21), Online Event

Recent Advances in Unbiased Learning to Rank from Position-Biased Click Feedback

June 09, 2021

Invited Talk at Google Mountain View, Mountain View, California, United States (Online Event)

Optimizing Search and Recommender Systems based on Position-Biased User Interactions

April 30, 2021

Invited Seminar Talk at the Florence Nightingale Colloquium at the Leiden University, Leiden, The Netherlands (Online Event)

Unifying Online and Counterfactual Learning to Rank

March 11, 2021

Conference Proceedings Talk at the 14th ACM International Conference on Web Search and Data Mining (WSDM ’21), Online Event

Optimizing a Ranking System with User Interaction Logs: Counterfactual Learning to Rank

December 15, 2020

Guest Lecture at Radboud University, Nijmegen, The Netherlands

Policy-Aware Unbiased Learning to Rank for Top-k Rankings

December 03, 2020

Invited Workshop Talk at the 19th Dutch-Belgian Information Retrieval Workshop (DIR ’20), Online Event

Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking

September 17, 2020

Conference Proceedings Talk at the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval (ICTIR ’20), Online Event

Policy-Aware Unbiased Learning to Rank for Top-k Rankings

July 10, 2020

Conference Proceedings Talk at the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’20), Online Event