Analyzing The Rock Playlist

In New Zealand, there is a radio station name “The Rock“, that vows to never play the same song twice in a single day (Between 9 – 5). They call it “The Rock No Repeat Workday”. Snappy.  It sometimes changes into a competition where they will intentionally play the same song twice in a day and you can call up to win a prize etc.

One of the main criticisms of The Rock, is that even if it doesn’t play the same song between 9 – 5, it still plays the same song everyday, often at the same time. To be fair to them, it’s probably no different to the criticism hurled at any popular radio station really. Anecdotal, I used to listen to the radio as I was getting up in the morning, and I used to swear that for weeks on end, I would be getting up to the same song.

Rather than live with the idea in my head that they “may” be playing the same songs. I sought to see if they really were. What a brilliant use of my free time I thought (/s)! But really, I had some time to kill as work was settling down for the xmas holidays, so let’s do this!

I had this crazy idea that I could stream the radio to my computer, and run some sort of Shazam type API to work out what song they were playing. But as it turned out, it’s much easier than that. The radio station keeps a page updated, that lists songs they have played that day : http://www.therock.net.nz/Shows/Workday.aspx . It’s not completely in real time, but it’s close enough that each day I can download the songs and store them somewhere for later analysis.

Weirdly enough, when I started trying to rip the contents of the page. I noticed that hidden in the page, was the ENTIRE setlist for that day, not just what it showed on the page. Even songs that were played at 2am were actually in the source code, and then they ran this amazing piece of code to decide whether to show it or not.

What that’s basically saying is, if today isn’t a Saturday or a Sunday, and it’s between 9 and 6, then you can show the song. Otherwise, even though the song is written to the webpage anyway, just don’t show it to the user.  If you ever wondered why The Rock website is so f-ing slow, this is the reason. The webpage is over 4000+ lines of code long, but most of it is just repeated junk just to output 20 songs on a page. Crazy stuff.

Anyway, I whipped up a quick app that would start at 11:45PM every night on my computer and hit up the page to download the list of songs. It would then save them down into a CSV, and that would be it. Unfortunately, it seems like TheRock removes some entries every now and again, so I didn’t always get complete days. But still, I recorded 1300 songs played by the radio station.

Awards time!

The award for most replayed song goes to… Mountain At My Gates – Foals. Played 18 times so 1.3% of all songs played.

The award for most played band goes to… Red Hot Chili Peppers. Played 34 times so 2.6% of all songs played. (They were closely followed by Foo Fighters with 33 plays).

And some other numbers that I found rather interesting. In total, there was 610 unique songs in the list. And 242 total unique bands. I would say the total is probably somewhat less than that, because I didn’t bother cleaning the list of “feat” artists.

If you want the CSV file to download for yourself, you can grab it here. Let me know what other cool numbers you come up with in the comments! Initially I wanted to work out if the same band was played roughly the same time per day, (e.g. Foo Fighters in the morning, Pearl Jam for your evening), but my Excel Fu is really not that great.

For those that may be interested (Not many), here is the very simple C# code I wrote to download the list.

Analyzing The Rock Playlist

Blocking Google Analytics Referral Spam

Let’s start with the shitlist so far…

buttons-for-website.com
blackhatworth.com
darodar.com
traffic2cash.xyz
hulfingtonpost.com
top1-seo-service.com
semalt.com
snip.tw
share-buttons.xyz
w3javascript.com
nufaq.com
topseoservices.co

And the list goes on. If you have seen any of these in your Google Analytics referral list, you are the subject of GA referral spam. It’s been going on for years, but only recently has it reached a point where the bulk of your traffic showing up in reports is complete junk.

So how does it work? Many initially believe that it was bots actually visiting your site, and triggering GA to record a visitor. This is not the case however, these bots never actually come to your site at all. You see, on your site you will have a GA code that looks similar to this “UA-12345678-1”. All these spammers do is roll through each GA code by increasing the userid by one each time, and then spamming a few hits to each.

So how can you stop them if they aren’t actually coming to your site? Well really Google should step up to the plate and do something about it since really, it’s making their service completely useless. But support of existing products is not a Google way of thinking. Luckily there is a simple way in GA to block them, and albeit it’s manual and it only blocks them going forward. It’s better than nothing.

1. Once logged into GA, at the top of the window should be an Admin tab. Click it!

google-analytics-admin-tab

2. On the far left column, select your account from the top down. Then select “All Filters”.

google-analytics-all-filters

3. Click the “Add Filter” button. Then fill it out similar to below where the “Filter Pattern” field below is each of the domains you wish to block, separated by a | character, and with a \ character before every full stop.

google-analytics-exclude-campaign-source

4. Further down the page you should “select” the views you wish to apply this to. If you have no idea what this means, you should only see one named “All Website Data”, and you want to select this one.

Wallah! You should now be spam free!

So you’re probably saying to yourself that it seems like a lot of work to have to manually join up all these domains, and put a slash infront of the full stops etc. It’s madness you say! Well, I’ve created a Github repo of all the spam domains I’ve come across, and a small tool to join them all together. I hope that in the future, more people will join in so we have a more collaborative list of blocking spam domains. You do need Node installed to run the tool, but hopefully in the future I’ll get around to making it a bit easier for those non technical people.

You can check out the Github repo here : https://github.com/mindingdata/Google_Analytics_Spam_Stopper

If you are unsure what the heck you are doing on Github, then the easier way is to just grab the output list here : https://raw.githubusercontent.com/mindingdata/Google_Analytics_Spam_Stopper/master/output.txt . Take the second line in that file, and slam it into the block list in GA. And wallah, you have a pretty dang comprehensive spam list. You should check that file often to see if anything new has been added, and just take the entire line again. It makes it a lot more easier than having to maintain your own list!

And of course, for those interested in helping, please create a pull request to add any spam domains to the list so others can benefit from it. The general thought is that if you haven’t been hit my a particular spam domain yet, you still may be in the future, so it’s better to max out your spam list while you can.

Blocking Google Analytics Referral Spam

Debt Crowdfunding In New Zealand Is Getting Crowded

Firstly, I know, terrible pun.

Today I woke up to some interesting news coming out of the team at Pledgeme. They have applied for their Peer to Peer lending license from the Financial Markets Authority. With Harmoney, LendMe, Squirrel Money and as of last month Lending Crowd jumping into the mix, things are going to get very crowded. (Ugh, Sorry, I did it again).

Pledgeme already offer “Rewards” based crowdfunding (Similar to kickstarter) and equity crowdfunding (Invest and you get a % of the company). So to add an interest based crowdfunding on top of that is an easy expansion for them. Interestingly, Pledgeme is not going after consumer debt, but rather it is going after “Organization” debt. It seems like this can range from companies, to social enterprises to non profits. So the scope is very large, but it’s very much different from other offerings currently out there. Ontop of that, Pledgeme seem to be going for the route that these organizations already have their crowd, and are looking to utilize them to get funding rather than the existing model of Harmoney and the like, which is to almost blindly connect lender with borrower through the platform.

It comes as an interesting time as Harmoney is under fire for “double dipping” on investments. That is refinancing a large portion of existing loans to double dip on fees (While the original lenders lose out on interest). This seems to still be happening pretty regularly. At the time of writing, there is a total of 41 borrowers looking for finance, but of that 41, 26 are actually rewrites. So, atleast right now, Harmoney is double dipping on over half of it’s lending. Crazy stuff.

While I’ve been a member of Harmoney since inception, I’ve never bothered to invest. Simply for the reason that you get exceptionally minimal information about the borrowers, often a one line explanation of why they need the money. Often the lending makes zero sense too (People borrowing to repay their student loan and go overseas, but Student Loan interest is currently sitting at 5.3%, MUCH lower than Harmoney’s rates).

As an example, here is a typical borrower you come across. Looking to consolidate existing debt which is pretty normal. But really, you have very very little information about the persons ability to repay. But maybe I’m just a bit of a wimp when it comes to investing 🙂

harmoney-borrower-information

Taking a look at the existing Pledgeme platform gives me hope that borrowers will provide much better info on why they want to borrow.

My only other debt funding experience has been a quick sign up to Lending Crowd. But so far, I haven’t seen any debts actually available to fund. Maybe they are too late to the game? It’s been telling me that there are no loans since signing up, so I’m not sure what the deal is.

lendingcrowd-no-loans-needed

I would love to hear from anyone else that has an experience using the other platforms (Squirrel Money, LendMe).

Debt Crowdfunding In New Zealand Is Getting Crowded