Summary
In the Season 2 Ep. 19 of Dr. Pawd, Gustav Söderström, Chief R&D Officer at Spotify, discusses the use of machine learning in personalizing user experiences. He explains that machine learning has been at the heart of Spotify since its early days, even when senior management didn't recognize its value. However, the company persisted, and the development of a vector space that captured some of the hidden dimensions of how songs co-occurred with each other in playlists led to the creation of song-to-song similarity and user-to-song recommendations. Gustav reveals that the company's ultimate goal is to deliver the right content to the right listener at the right time, which is why they use consumption-based embeddings in a vector space, a knowledge graph, and sonic analysis to create genre-wise and sonically coherent playlists.
While music recommendations heavily rely on consumption-based embeddings, the process for podcast recommendations is different and focuses more on language models like transformers. Gustav talks about Spotify's expansion into podcasting, which started with user research that revealed people wanted their podcasts with their music. Spotify's tools for musicians, such as Spotify for Artists, have also expanded to cater to podcast creators, with the addition of vertical teams for each creator type.
The episode also delves into the A/B testing platform used by Spotify to test algorithm changes. Gustav explains there are three levels of sophistication involved in the A/B testing process, from correlating past A/B test performance to creating a music simulator that can simulate user behavior based on their listening history and taste graph. The simulator optimizes for long-term retention, which means optimizing for the user's engagement over a more extended period. Gustav acknowledges the importance of understanding algorithms before exploiting them and the trade-offs they entail.
Gustav explains that the media industry is not static, with changes in taste, trends, and non-stationarity affecting it, leading to a shift from it being all about consumers to creators. He shares some challenges the company faced when making recommendations for new podcasts due to a lack of data, but the surprising success in using music taste to predict podcast taste. The conversation closes with the speaker expressing gratitude towards Gustav for his contribution to the topic being discussed.
