Correlation & causation

The incident is funny to us because we realize that the correlation between the elderly gentleman’s mouse gestures and the video buffering is spurious. We also realize that one does not cause another. In other words, we clearly understand that, in this case, correlation does not imply any causation. Any confusion between the two can be a genesis of a new superstition.

We are exceedingly good pattern recognizers! We see them everywhere. We see faces in clouds, letters in tree barks and whatnot. Not to say that this is bad, though: all our accumulated knowledge is mostly about patterns, how things & people behave.

So why is it we tend to overgeneralize and find more patterns than there seem to be? Why do we develop superstitions? Why do we mistake correlation for causation? After all, it is not very rational, for it is true that small correlation exists between almost any two sets of data.

One poignant example of this is in the stock market where we rely so much on patterns to help us make money. The commentary for Ben Graham’s “The intelligent investor” succinctly warned us of perils of doing so :

If you look at a large quantity of data long enough, a huge number of patterns will emerge—if only by chance. By random luck alone, the companies that produce above-average stock returns will have plenty of things in common. But unless those factors cause the stocks to outperform, they can’t be used to predict future returns.

Wired to be superstitious

Research seems to suggest that we are wired to be an overgeneralizing, superstitious species.  The idea is surprisingly simple. Pardon my use of a simplistic analogy to demonstrate the logic. Imagine there are two people, one who overgeneralizes and sees a lion in the grass when there is none and another who doesn’t overgeneralize and sees a lion only when it is truly staring down at him. It is more likely that the former will survive to reproduce than the latter. Hence, over the eons, we have evolved to overgeneralize.

Science to rescue

While it may be true that being superstitious is better when the there is a large cost for missing something with a relatively small cost of getting it wrong, in our current lifestyles as comfortable denizens of concrete jungles trying to live off the gullibility of others, superstition is more of a hindrance. How do we go about reducing its effect?

I think we can do so by using scientific method, in general. The beliefs we want to adopt should be falsifiable. We need to test those beliefs in a strictly empirical (observation and experiments) and skeptical (with proper controls ) fashion to ascertain their validity.

Philosophical underpinnings

The philosophical underpinning of the whole argument, however, rests on a principle called Occam’s razor. It is a principle of parsimony which can be simply stated as “when you have two competing theories that make exactly the same predictions, the simpler one is the better” or as Einstein is often quoted “Everything should be made as simple as possible, but not simpler”. I’d like to conclude with a question to my readers. What makes Occam’s razor tenable?