← Nathanael Silverman

A collection of thoughts from one human being

Calculating a Correlation on Android

Calculating correlations is easy—especially when using a math library. If you're like me and you like playing around with data it's a tool you'll want as part of your arsenal. Here we'll be using "Pearson's product-moment correlation coefficient". It's a common way to measure the linear correlation between two variables. But first a quick note about why this article targets Android.

Smartphones are the ideal data collection tool. They're always on us so they're easy to whip out to save any kind of information. And they're equipped with data collecting hardware like GPS, WiFi, and sensors. AND they're now powerful enough to process all that data themselves!... Excited yet? Let's jump right to it!

First we'll import Apache Commons' math library. This will give us access to the PearsonsCorrelation class. Let's add the following dependency to our build.gradle file:

compile 'org.apache.commons:commons-math3:3.5'

Well done! This library let's us use a couple of data formats: RealMatrix or double arrays. Let's use the latter.

BOOM! We're done!

The correlation coefficient will vary from -1 to 1... 1 is a perfect correlation, meaning the values rise and fall together. -1 is a perfect negative correlation, meaning when one value rises the other falls. Here's an easy way to interpret the values between:

  • -0.1 to 0.1 indicates no linear correlation
  • -0.5 to -0.1 or 0.1 to 0.5 indicates a "weak" correlation
  • -1 to -0.5 or 0.5 to 1 indicates a "strong" correlation

So we've found a strong correlation between eating ice cream and happiness. At least for the person observed. I sense a Nobel Prize in our future!

Thank you for learning how to correlate today! And if you use correlations in your code, tell me about it!

Posted

Take My Money and Shut Up, Spotify

I listen to music in phases. Pop songs for a week, queue classical music, then the regular Pink Floyd revival... This month's phase was: "no music whatsoever", and my premium Spotify account was going to waste. Bam, cancelled. What was I paying for anyway? It hit me a few weeks later.

The cycle had gone round and I was listening to my old playlists again. I'm bobbing my head getting into a flow as the first advertisement hits... "I can live with it," I thought, so I tried for a few days. Here's what I've concluded: Spotify, take my money. Anyone who gives me the choice between ads and paying, take my money now. Here's why.

They're Getting My Money Anyway

There's a reason companies pay to run ads. They know they'll make it back, and more, in increased sales. Those sales wouldn't have happened otherwise. That's money I wouldn't have spent—but was convinced to through remarkable perseverance.

Ads sound like someone is doing me a favor by bringing amazing deals to my attention, but really, an ad is just a company's way of asking for money. So in the case of Spotify, I can either give my money upfront, or get constantly asked to do so—with no end in sight.

"Why don't you buy a new car? Look at this sound system, it's better than yours. Come onnnnn, at least let me save you money on insurance! Not now? Fine, listen to this. I'll be back in a few minutes." No free service is worth being harassed.

Posted

The Best Time to Post on Hacker News

For weeks now I’ve been gathering data on Hacker News submissions, looking to answer the question: what is the best time to submit content to Hacker News?

A few hours on the frontpage can bring in tens of thousands of visitors, but only about 13% of all submissions make it (and not always for very long). You need good content for that, of course, but as you’ll see, the timing of the submissions is also worth considering.

Reaching the Hacker News Frontpage

frontpage_likelihood2png

The first thing I did was look at the percentage of submissions that reached the frontpage depending on the hour slot at which they were posted. This value peaks at 18% from 7 to 8 AM EST, which is almost a 40% increase from the all-around average.

Now, if you just want the bragging rights that come with reaching the frontpage at least once, that time slot is your best bet. You might then realize that frontpage status is fickle. With nearly 10% of submissions not making it past the 10 minute mark, and some staying featured for much longer, the average time spent on the frontpage is clearly worth factoring in.

Staying on the Frontpage

frontpage_staypng

So we see submissions from 8 to 9 PM EST will stick to the frontpage for 11 hours on average, in contrast with early morning which bottoms at 6 hours. Now, we could combine this with the likelihood of reaching the frontpage and get the best of both worlds, but, as you might have realized, we’d be neglecting another important factor.

If we submit something at 8 PM and it spends 11 hours on the frontpage, most of that will be night time in the US. Whereas if we post at 7 AM, it’ll spend less time on the frontpage, but all of it will be during daytime.

To figure out which is more interesting we need just one more data set: Hacker News’ hourly traffic. That way we’ll know how much every hour of exposure is worth. Unfortunately, because I don’t have access to that information, this piece of the puzzle will have to be an approximation.

Estimating Hacker News’ Traffic

Luckily we don’t need precise numbers of visitors. Knowing how hour slots fare against each other will suffice. Our best bet then is to look at the voting activity.

activity_estimate1png

It does look like US visitors command the lion share of Hacker News’ traffic, with spikes from morning until lunch, and then back again at the end of the day. The 4 PM dip is rather intriguing: is it the most productive work hour for hackers? They might also have read everything at that point, though I suppose the two aren’t mutually exclusive.

The Best Time to Post on Hacker News

We are now well equipped to answer the question that started all of this. We can look at the average traffic we would get from reaching the frontpage at any given hour, and if we factor in the likelihood of actually reaching the frontpage, we can decide on a best time to post on Hacker News to maximize our expected exposure.

Aaaand, the best time to post on Hacker News is…

expected_exposure1png

9 to 10 AM EST. Generally speaking, 7 AM to noon is better than any other time.

Posted

Don’t Worry About Your Productivity, Worry About You

After years of roaming the Internet, its video clips, its news aggregators (I’m looking at you Reddit and Hacker News), I’ve become very familiar with a feeling of restlessness. Like I want, or need, to constantly feed information to my mind.

Now I think I’m not alone when I say this frequent need to consume content has taken a toll on my productivity. At the very least, if it hasn’t been too detrimental to my work, it might not be the best use of my free time.

But, as with all things enjoyable (and Internet seems to be in good standing with the short-term pleasure centers of our brains), it’s a very tough habit to kick. Ultimately you’ll always wonder why abandon something you enjoy, and you’ll think about going back often.

Today, I’m happy to say I have kicked this particular habit, and I don’t think I’m going back, ever. And the reason is precisely the restlessness I mentioned earlier. I realized, in fact, I was no longer enjoying the moment. I couldn’t pause anymore. I felt averse to things that didn’t hold my attention like my computer screen can.

So I stopped. I stopped using Internet as a time-filler. I stopped time-filling altogether. My overarching philosophy now is to immerse myself completely in what I’m doing at any given moment, to do only what I’ll be glad I did later, when I look back, and to always take my time.

This for me means no more Reddit and Hacker News, no more casual youtubing, no more games for gaming’s sake. I’ve stopped using my Rapidshare account aggressively. I’ve even cut back on listening to music while I work. I already didn’t Facebook or Twitter.

Instead I read carefully selected books. I write, I exercise, I spend time with the people I love. Sometimes I do nothing, because sometimes nothing needs to be done.

I don’t plan my next entertainment fix halfway through the present one. I don’t feel guilty for squandering my time. I’m not as stressed as I used to be. The realization that this was really having an impact came to me in my car, actually. It struck me that I wasn’t paying close attention to my speed anymore, because I didn’t care about going as fast as I could. I didn’t care about passing other cars, or getting to places quickly for no particular reasons. I didn’t mind red lights or small traffic jams. I could coexist with a population of idiot drivers.

As far as I can see, cutting back on constant content consumption has been the root cause for what feels like a very refreshing change of pace. This to me holds infinitely more value than the idea of “being more productive” and it goes to show that these habits can in fact be detrimental to our mental health and to our enjoyment of life in general.

Posted

Subscribe by email