Advances in Multi-Stage Recommender Systems for Large Scale Applications

Recommender systems have become indispensable tools for enhancing user experiences in today's digital landscape. As online platforms continue to gain success, users are faced with an overwhelming array of choices. Recommender systems address this challenge by offering personalised suggestions that align to individual preferences and needs. This not only facilitates decision-making but also fosters user engagement and satisfaction, driving customer loyalty and platform success. In real-world applications, the complexity of recommender systems often demands a multi-stage approach to strike a balance between efficiency and effectiveness. The first stage of this architecture focuses on candidate item retrieval, where algorithms aim to narrow down the vast pool of items to a more manageable subset. Scalability is fundamental in this phase, algorithms have to deal with huge catalogue of items which can reach the size of billions. Subsequently, in the re-ranking phase, a more sophisticated model comes into play. Here, a diverse set of features associated with users and items is leveraged to re-rank the recommendations, tailoring them to specific user preferences. This multi-stage architecture allows recommender systems to efficiently deal with vast amounts of data while delivering high-quality recommendations that resonate with users. This PhD thesis embarks on a comprehensive exploration of both stages within real-world recommender systems, placing a distinct emphasis on innovative algorithmic strategies. Within the context of candidate item retrieval, the thesis delves into the domain of graph-based algorithms, particularly focusing on models utilising graph convolution techniques. By leveraging the inherent graph structure of user-item interactions, these algorithms exploit the underlying connections between users and items. Firstly, an advancement is presented in enhancing the scalability and practicality of graph convolution-based algorithms for candidate item retrieval. Secondly, the thesis delves into a theoretical investigation of the performance enhancement offered by graph convolution-based models in candidate item retrieval where a novel connection is established between spectral models and graph convolution based algorithms. Moving forward, the thesis shifts its focus to the re-ranking phase, where it takes on a reinforcement learning perspective. In the re-ranking phase, many algorithms prioritises myopic metrics, neglecting the optimisation of long-term user engagement an imperative element for the enduring prosperity of online platforms. Within the realm of reinforcement learning, the exploration centres on slate recommendation models, predominantly employed in real-world scenarios. These models specialise in suggesting sequences of items, by considering the intricate interconnections that bind them. In this context, an innovative approach is proposed, aimed at amplifying both scalability and serving efficiency when compared to conventional models while retaining the optimisation of long-term user engagement. Through a balance of theoretical exploration and empirical validation, this research contributes valuable insights to the fields of graph-based algorithms and reinforcement learning in the context of recommender systems.