Causality vs. Corelation Part II

In my recent post Causality vs. Correlation I wrote about how an incomplete representation of facts can lead to a half truth which is far worse than a lie. Here is another stark example of the irrational use of correlation in statistics. A study recently made a comment that students of history tend to have more active sexual lives than their peers. It goes on to say that students with a more active sexual life also get certain grades. No attempt whatsoever is made to establish the much needed causality. In the absence of any such scientific explanation are we to assume that people who study history just have higher levels of hormones? Should a parent whose child has selected History as a college in subject be more worried about the threat of STDs given the sexual behaviour statistics of their peer group? Sounds irrational right? It is precisely this kind of mindless representation of statistics that give it a bad name. Research for the sake of research does more harm than good to the subject.

The journalism of haste

Yes the media will always give the losers a hard time. The loser has to answer difficult questions and while the winner can just smile and make a few canned statements, the winner has to come up with a damn good reason for the loss and in today’s world- come up with that reason fast. Because our journalists today are in a hurry. They do not want news that is investigative and informative. They just want that sound byte which they can put on at prime time with a fifteen second looping video so that they can give the impression of being right there in the heart of the action.

That is what happened with the senior leaders of the BJP when the results of the Lok Sabha elections were being announced. By the time it was obvious that the UPA ( or the ruling party led by the Congress) would be able to form a majority Government at the centre, people like Sushma Swaraj, Arun Jaitley, Narendra Modi were being put on national television and asked to answer questions they could possibly have no answers to. The verdict was as much a surprise to the BJP as it was to the TV channels who had used exit polls to show a fractured verdict and a strong anti-incumbency factor less the twenty four hours before. The channels were quick to give themselves a moment to be surprised and move on, but they were not willing to wait and get a meaningful statement from the BJP leaders.

Many of those interviews, to my mind, were used to just fill time gaps. Not a single BJP leader could make any comment except that the party would meet in a couple of days and together do an analysis of what had happened for such a surprising outcome. Yet the same questions were asked in several ways and the same responses were given so many times that it was embarrassing. It looked more like a lynching of the BJP leaders rather than a serious journalistic attempt at getting news and information that would otherwise not be available to the common man. The loss of the BJP was being made out into a spectacle and was being served up to an audience hungry for thrills.

The exit polls are a symptom of precisely the same disease that is plaguing our TV media today. They hope to predict what is almost impossible to do- as the actual election results showed- and are an attempt by TV channels to cash in on sensationalism. Today’s journalists do not want news unless it is breaking news. So they will create news and sensation. No one can wait for the actual poll results- they must do and exit poll to tell you what the result will be like. Of course there is the always the unsaid footnote that all of this could turn out to be nothing more than a bunch of useless numbers.

This is such a strong phenomenon that it seems to have affected even the leading news men and women of today. Take for example the Udayan Mukherjee interview with Mr. Pranab Mukherjee. As Editor in chief of CNBC TV 18, Udayan is a highly respected figure, but the interview published on the front page of a national newspaper, sounds just like a TV interview. An attempt to extract statements about the budget before the budget is announced. Trying to make the FM commit to agendas and items in the budget knowing full well that it would be impossible for him to do so. All because Udayan and the newspaper wanted to be the ones that “broke the news” about the budget before anyone else did and before the budget was even put out there. The result- a two-page interview at the end of which you do not feel any more informed than you already were before reading the interview.

While all this maybe deemed necessary at the altar of commercial TV in this hyper-competitive age, journalism in haste has a price tag. It comes at the cost of real news. Sound bites quoted out of context and deceptive headlines leave people with half the information. It builds a culture where people do not have the patience or the taste for the kind of journalism that is truly informational and enlightening.

Looking at the TV and reading some of the stuff in newspapers today it is sad to see the waste of space and time that comes with a journalism of haste.

Causality vs. Co-relation

Anybody who has dreaded algebra and Calculus like I did throughout my years at School and then at Grad School will perhaps agree that a lot of the fear comes from a lack of understanding. Teachers expect students to learn formulas, difficult concepts and apply them to problems printed in text books as part of a syllabus. But how many teachers really take the time to tell you "why" this is relevant for you? Very few. Making it hard for people to acquire a skill that I have come to realise is critical for professional success.

I would have gone through life with a tremendous handicap but I had a bit of last minute luck. During my post-grad, a mathematics teacher decided to finally educate me on the reasons behind mathematics. and once again yesterday when I was reading "Freakonomics" by Levitt and Dubner I learnt some more- about co-relations and causalities. On page 163 of my edition the authors ask "Let’s say that you want to ask the ECLS data a fundamental question about parenting and education: does having a lot of books in your home lead your child to do well in school? Regression analysis can't quite answer that question, but it can answer a subtly different one: does a child with a lot of books in his home tend to do better than a child with no books? The difference between the first and second questions is the difference between causality (question 1) and co-relation (question 2). A regression analysis can demonstrate a co-relation but it doesn't prove cause" In other words there maybe a co-relation between books and a child's grade but causality alone can give that co-relation a meaning beyond just statistics. That is why the authors go on to add that “A regression analysis alone can't tell you whether it snows because it snows because it is cold, whether it’s cold because it snows, or if the two just happen to go together….. What we really want to do is measure two children who are alike in every way except one- in this case, the number of books in his home- and see if that one factor makes a difference in his school performance. “

This is what differentiates a pure statistician from a person who has the skill to look at numbers in a meaningful way. Unfortunately till today this skill is a rare one. And it became even more evident when I was reading an article published in “The Mint” a couple of days ago. It is titled “Wealth effects of the phone ring begin to widen”. The article – covering a good page of the newspaper spends ninety percent of the space throwing out statistics with no effort to establish a causality that would tell us why the cell phones are having a positive effect on smaller town economies. It just makes statements like “Indian states with higher mobile penetration can e expected to grow faster with a growth rate 1.2 percentage) points higher for every 10% increase in the mobile penetration rate” It quotes the MD of India’s largest cellular company saying that 2% of the country’s growth GDP growth has been contributed by the growing telecom sector. The question that needs answering here is – which 2%? Education, rural commerce and consumption, employee productivity? If we had the answer to that question then we would know where to push the mobile penetration and how. If the correlation could be extended to meaningfully establish causality then we could make the growth even faster and more efficient.

After it has quoted every industry sponsored study and the head of every organization with a stake in the growth of the mobile business , does the newspaper get down to giving some explanation of why the mobile phones maybe considered as drivers if growth. “Mobile phones can perform in under served areas and regions in the same as fixed lines did in many developed countries…widen markets , create better information flows”. But the article would have been much more meaningful if it had quantified just how much more this information spread can be.

But then maybe I am expecting too much from a newspaper article. It’s job is to give you an overall picture. They can be absolved of this responsibility but the teachers in our class rooms should never be. They should equip students so that they know when to see statistics for what they are, question them if needed and apply them where most beneficial. Because if they can not do that then it will always be lies, damn lies and statistics.

Manmohan, Mandate and the Market

Indians had barely recovered from the brilliance of what they had done in the recent Lok Sabha elections that they saw the stock market make another historic milestone- it was up by twenty percent before the half day mark and with the Sensex up more than Two Thousand Point it had to be closed because a circuit had come into play.

Yes it is euphoria and no one knows how long this will last but one thing is for sure, the Indian public has displayed a self-belief and a maturity that no one believed possible. Yes the Western Press covered our elections extensively, but most of it was around the sheer magnitude of the election. Of how operationally daunting such a task can be. But no one ever thought the Indian voter would go beyond (basically) caste, creed and past grievances and be “aspirational”.

The demure Singh from Punjab managed to work his magic and was voted back despite a down turn, a sixty per cent drop in stock market, job losses, inflation and a near miss with the Nuclear Deal and the CPI (M). His message of clean politics, consensual growth and putting professionals to work, clicked with the people. The margin with which the Congress was voted back does not merely say that people want to maintain the status quo, it says that they aggressively back a party that is focused on what it can do instead of mud-slinging and what it wants to prevent from happening.


The magic combination of Manmohan Singh and Sonia Gandhi works because while one is the product the other is the absolutely brilliant packaging- consistent, clear and with mass appeal. The good work done by the last UPA government needed to be taken to the people. This is what the threesome of the Gandhi family pulled off successfully. Coming from a legion of politicians they understood that it is not enough to do good- certain processes of a Lok Sabha election must be followed. People’s hand must be shaken, rallies must be held in villages, dust must be eaten and sweat allowed to flow.

The market’s reaction is saying that we had a belief in the India story all along- the fast growing youth population, the huge potential for infrastructure investment and hence returns, stable banking infrastructure- and now the last doubt of whether there would be a stable government or no has also been lifted so we will go and put our money behind India Inc. So now Manmohan Singh and Co, have to do what they all along have planned to do- be pro-growth, deregulate slowly but surely, continue the regulations that prevent the kind of irresponsible behaviour seen in the world’s financial institutions. And as for the foreign money flowing in- why will it not come to India? Where else will people invest? In the US and Europe everything that gets put in is swallowed up in recovery rather than new value creation. So money managers around the world will be happy to come to a country of a billion buyers who need new houses, roads, schools, factories bridges and are willing to borrow money and give returns.

A song worth revisiting

So what makes a song, a movie, a play or a book memorable? I think it’s the impact with which it is able to convey its message. While a movie or a book has the advantage of length on their side, a song is perhaps the most difficult of all because the duration is usually no more than a couple of minutes. So when a song leaves an indelible mark on it’s audience it is all the more laudable.

For generations of movie goers “ Jab Pyar Kiya to Darna Kya” from Mughal-E-Azam remains an all time favorite. No one can forget Madhubala’s defiant stature, Prithvi Raj Kapoor’s blazing eyes and the glass set of Sheesh Mahal, put up at a huge cost just for this one song and not to forget that this was the only sequence in the movie which was shot in Technicolor- again at a huge cost to the already beleaguered producers. The sheer expense of this song tells us that to the makers, this song lay at the very heart of the film.

After seeing the movie there is no denying that the expense and the effort are absolutely worth it. Madhubala is magnificent in the song. She misses no chance of conveying the message- while the powers that be, can have a public victory by killing the love between a prince and a chamber maid, privately it is love alone that will triumph. “Parda nahin jab koi khuda se, bandon se parda karna kya”she sings with an almost militaristic stand and we all nod with her. And here the Director K Asif adds a masterful touch by showing the only time in the film when Akbar is forced to bow his head and acknowledge the win of his opponent.

The camera and editing too are used masterfully in the song. Throughout the sequence we are taken from person to person showing the different reactions of the members of the Royal family. Prince Salim played by Dilip Kumar wears a cautiously proud expression as his lover dares to defy the King in such a public setting. Jodha Bai, the loving mother and proud queen is amazed at the cheek of a chamber maid who can challenge her might husband and King Akbar, outraged and yet helpless sits on his throne seething in the fire of his own ego.

Most people of the previous generation know the lyrics of this song by heart. It is both tragic and victorious at the same time. Anarkali has the courage to show the King that his victory his hollow but as a maiden she can not help but mourn the loss of a love she feels so deeply. So where there is the “Parda nahin jab koi khuda se, bandon se parda karna kya” there is also the “Unki tamanna dil mein rahegi, shamma issi mehfil mein rahegi” that conveys her pain.
In this song the combination of choreography, setting, lyrics and music and creates a masterpiece that should be studied by all film-makers to see why a song sequence is an integral part of hindi cinema and how it can be used masterfully to move the storyline forward and wow the audience.

The new Avtar of match finding

BharatMatrimony.com has been a popular online matchmaking site for some time now. I remember that it was started some 4-5 years ago. But the reason I am writing about it today is the series of online ads it has recently launched. Accompanied by beautiful shots of couples on their wedding day, there are slogans like “ Get married. Your parents did it” or “Get Married. Don’t be afraid to commit”. To me the message is simplistic and makes the fatal mistake of believing that the product is more important than the market demand.

All of us know that the typical online match making website is used primarily for young people (both married and unmarried) to find someone for a casual fling. Just because the name is Shaadi.com or BharatMatrimony, it doesn’t in anyway mean that people will use it to find their soul mates. Also in India majority of the internet users are in the cities and urban areas. These young people are far less likely to simply look for a soul mate online They are precisely the kind of demographic who is liberalized and westernized and will not hesitate to have a fling with someone they meet online.

In the face of this, throwing in a message like, “Commit don’t be afraid” is just too little and too simplistic. You can not change market behaviour like this and will not become popular with the current generation by asking them to ape the previous one!

But all said and done this does not mean that the eternal search for a life partner is not big business. Testimony to this fact are two new shows that are soon to come to the small screen. One is a reality show that will trace the “Swayamvar” of the Bollywood Bimbo Rakhi Sawant and the other a show called “Vivah” that essentially takes sites like Bharat Matrimony and Shaadi.com to TV. Their USP, on the Internet people can fool you, but on TV you will see the reality and meet only candidates who have been pre-screened for their seriousness about marriage.

So even though Bharat Matrimony maybe the wrong platform for it, or maybe simplifying the message too much, the demand for arranged marriage services is alive and kicking. But it has a new face with a dash of excitement and risk. Whatever else these shows or websites might say or do (because this maybe nothing more than a veiled poke at this whole Internet dating thing), one thing is clear, that paradoxically enough in this era of rising divorce rate and where many young people are choosing to be single, marriage and that too “arranged” has enough of an attraction.
But the why should this surprise me? I- an educated city girl with liberal views- was married less than a decade ago to a man who had every liberty to choose his spouse but decided to marry the woman his parents found for him. And on my part I married him because he is the one and only man flattered me with all the love and attention he showed me the first time he met me and also because “daddy said so”. And I didn’t even need the pretense of a website or a reality show.