Analyzing SiriusXM's The Heat "Now Playing" Data with Apache Spark

8 January 2018

My SiriusXM tracker for ‘The Heat’ radio station has now been running since the summer of 2016 and has collected a good amount of data. It seemed like a great opportunity to make use of this data for getting some insight into what songs/artists/albums have been popular throughout the months, and also get some Spark practice under my belt.

For more information regarding the tracker you can see my previous blog post or view the live web client here. There is also a clone tracking ‘The Highway’ station. The source code for following post is available at the end.

Data Quality

Before a song is stored in the database it is ran through some filters to ensure that only real songs are kept, and things like promos, different channel advertisements, DJ Mixes, etc. are discarded. A song that passes the filters then is checked against the Spotify API, and if it exists there the Spotify metadata is recorded as well. We can say that the stored songs, which also have Spotify data can be regarded as good. Nothing is perfect of course, and sometimes songs are not found on Spotify because of different spellings between SiriusXM and Spotify or some bad data has slipped through the filters.

From all the songs tracked, 97.27% have Spotify metadata. From the remaining 2.73% that do not, the top 3 played songs among them are by BeyoncĂ©. Since BeyoncĂ©’s music is not on Spotify, we can safely assume that those 3 songs can also be counted as good. Bringing the amount of good data to 99.12%. Looking very good!

I remember there was a brief period of time in which the tracker had gone down due to a change in the Spotify API policy, now requiring access tokens. Looking at the tracked songs per day we easily can see the gap, between May 29, 2017 and June 11 2017. Other than, that it looks like smooth sailing and there is consistent data throughout the recorded timeframe.

Overview

  • First tracked song: Nothing Is Promised by Mike WiLL Made-It, Rihanna on June 20th, 2016.

  • Last tracked song: rockstar by Post Malone, 21 Savage on January 8th, 2017.

  • Total songs: 200,871

  • Total different songs played: 1,508

  • Total artists played: 1,041

Top Songs

Most Played:

Rank Artist Song
1 DJ Drama, Chris Brown, Skeme, Lyquin Wishing 2197
2 Tory Lanez LUV 2181
3 PARTYNEXTDOOR, Drake Come and See Me 2003
4 Lloyd Tru 1959
5 Usher, Young Thug No Limit 1949
6 DJ Khaled, Drake For Free 1900
7 ScHoolboy Q, Miguel, Justine Skye Overtime 1875
8 The Game, Jeremih All Eyez 1868
9 Wale MY PYT 1778
10 R.LUM.R Frustrated 1777
11 Chris Brown Grass Ain't Greener 1737
12 Drake Controlla 1736
13 Bruno Mars 24K Magic 1718
14 Bruno Mars That's What I Like 1699
15 Chris Brown, Usher, Gucci Mane Party 1646
16 Russ Losin Control 1631
17 Kevin Hart, Trey Songz Push It On Me 1616
-- Beyonce Sorry 1616
19 Drake Teenage Fever 1599
20 The Weeknd, Daft Punk Starboy 1596
21 French Montana, Swae Lee Unforgettable 1569
22 Kendrick Lamar, Rihanna LOYALTY. 1554
23 Ty Dolla $ign, Lil Wayne, The-Dream Love U Better 1538
24 Drake Fake Love 1496
25 Calvin Harris, Frank Ocean, Migos Slide 1487
26 Rihanna Needed Me 1475
27 Bryson Tiller Sorry Not Sorry 1469
28 Meek Mill, Chris Brown, Ty Dolla $ign Whatever You Need 1468
29 Gucci Mane, Drake Both 1463
30 Kid Ink, Jeremih, Spice Nasty 1459

Most Played per Month:

Year Month Song
2016 June Sorry 103
2016 June Grass Ain't Greener 102
2016 June Needed Me 99
2016 July Exchange 298
2016 July Sorry 297
2016 July Controlla 285
2016 August For Free 302
2016 August LUV 274
2016 August No Limit 267
2016 September LUV 288
2016 September For Free 288
2016 September Wishing 277
2016 October Tru 301
2016 October Come And See Me 259
2016 October Wishing 255
2016 November All Eyez 263
2016 November Wishing 247
2016 November Do You Mind 246
2016 December Come And See Me 289
2016 December Black Beatles 256
2016 December Do You Mind 252
2017 January Push It On Me 292
2017 January Do You Mind 263
2017 January Wishing 254
2017 February Fake Love 240
2017 February Party 239
2017 February Location 236
2017 March Party 270
2017 March Frustrated 268
2017 March Overtime 249
2017 April Frustrated 284
2017 April That's What I Like 271
2017 April Shining 265
2017 May That's What I Like 267
2017 May Location (Remix) 259
2017 May Both 252
2017 June Blessings 200
2017 June Losin Control 189
2017 June I'm The One 181
2017 July Wild Thoughts 317
2017 July Teenage Fever 281
2017 July Love Galore 277
2017 August LOYALTY. 306
2017 August It's A Vibe 296
2017 August Whatever You Need 290
2017 September Unforgettable 292
2017 September LOYALTY. 279
2017 September Love U Better 273
2017 October Something New 268
2017 October Insecure 256
2017 October Love U Better 256
2017 November Love U Better 308
2017 November LOVE. 289
2017 November Something New 276
2017 December Sky Walker 455
2017 December LOVE. 449
2017 December Something New 447
2018 January LOVE. 95
2018 January Sky Walker 95
2018 January No Limit 95

Most Played Song in a Single Day:

Artist Song Date
Kendrick Lamar, Zacari LOVE. December 4, 2017 21
Wiz Khalifa, Ty Dolla $ign Something New December 4, 2017 20
SZA The Weekend December 6, 2017 19
Miguel, Travis Scott Sky Walker December 4, 2017 19
Wiz Khalifa, Ty Dolla $ign Something New December 6, 2017 19
Miguel, Travis Scott Sky Walker December 6, 2017 19
It looks like December 4th and 6th were some very repetitive days for The Heat listeners.

Top Albums

#1
Starboy
4501 times.
#2
Views
3745 times.
#3
24K Magic
3717 times.
#4
ANTI
3647 times.
#5
DAMN.
3576 times.

Source Code

The source code and Spark project for the analysis above can be found on my GitHub.

Spark on!

comments powered by Disqus