Exit polls fiasco puts research companies' methodology into question

The advertising and marketing fraternity calls for review of the way market research is conducted by agencies in the wake of drastically inaccurate polling by pollsters.

Exit polls are the ultimate consumer research. They gauge the mood of the public before the final election results are out. Exit polling is the most high profile application of surveying and opinion polling. Brands and marketers use consumer surveys and polls to evaluate if a market has the right indicators for a product to be introduced or assess needs of consumers, and in general use surveys to guide their decisions. 

When exit polls are way off the mark like they have been with the 2024 general elections, people begin to question some fundamentals of opinion polls and the methodology that goes into such surveys. 

Several senior marketers and agency heads that afaqs! spoke to on Wednesday (June 5) said that they were all pondering over what exactly went wrong and if there was cause for concern about market research and market research firms. 

Krishna Rao Buddha, senior category head - marketing, Parle Products, who has been a marketer for three decades, said, “there is a complete shake up in our confidence regarding the quality of market research work especially during the exit polls. While we can discount respondent accuracy to an extent, there is a big question on the credibility of all research firms involved.”

The question marketers are asking is “how could the polls go so wrong” and some are seeing this as a massive let down by research firms. It is especially alarming because exit poll results are telecast to a large audience and when they get it wrong by a huge margin, the difference between projections and real results is very stark.

During exit polls, the scrutiny is as high as it can be and mistakes raise doubts on research methodologies, quality of field work and make one wonder — do pollsters really have their ear to the ground, say marketers. 

One senior marketer from the automotive industry, said that while he is not sure about the level of concern, “surely questions will be asked.” Marketers will likely demand checks and balances to repose trust in research now. 

The CEO of an international advertising network agency, stated in no uncertain terms, “if research on exit polls went horribly wrong, is the other data we are getting on brands completely accurate?” 

Psephologist Yogendra Yadav identified four factors that led to pollsters predicting a two-thirds majority for the BJP-led coalition. He said they did not sample enough voters who may have voted for the INDIA coalition, there was likely under-reporting by those who voted for the Congress and Samajwadi Party in Uttar Pradesh, under aggregation — many undecided voters were allotted more to BJP than to others, and under conversion of votes into seats. 

He further said that while all pollsters make these errors, the exit polls by some agencies made all these errors in the same direction. This suggests a mishap in the research methodology. 

The wrong assessments made by experts who have decades of experience "brings to light the conundrum between expertise and bias" pointed out Aditi Patwardhan, a strategic planner with two decades of experience. "Did cognitive bias i.e. confirmation bias shape the research rather than being objective observers of the pulse of the people," she asked.

Therefore, Gaurav Ramdev, CMO, Protean eGov Technologies, a digital infrastructure company, felt, “it is not the credibility of the market research firms that should  be questioned but the scientific approach to the marketing research that yields any results.” 

Consumer research results, much like exit polls, have forever been a matter of debate in the marketing fraternity. There could be biases, inadequate data, cultural and demographic differences that need to be accounted for, and other pitfalls like serving the agenda of the company that commissions the research. Like the ex-CMO at a fin-tech brand put it, “it is about what the client wants to hear; seeking validation instead of trying to understand.” 

Ramdev here cites response bias — stated response (what the consumer says) and actual action (what the consumer does) have always been divergent. He suggests that an adequate sample size might have perhaps helped ensuring that response bias was as low as possible. 

One marketer gives the example of how market research firms failed some international giants in the streaming and service aggregation categories. “Research told them that India is a market with a lot of potential but these companies even after reducing pricing in India are struggling to attract users.” The diversity and complexity of India as a market needs a combination of qualitative and quantitative research which should finally be analysed with a nuanced lens.  

The challenge, Saurabh Parmar said, “is we are still trying to gauge human response as binary data points on an Excel sheet whereas it's far more nuanced. There are aspects of body language, how different people evaluate different things (your 4 star rating will be different from mine) and even their mindset at the time of survey.” 

Parmar, a fractional CMO who consults with tech brands, adds that many of his clients got their TAM (total addressable market) wrong because of flawed research. He is of the opinion that it is not market research firms or their commitment that are under question but market research as it exists definitely needs to be re-evaluated. 

It is not all doom and gloom. Exit polls and market research work very differently, explains Karishma Gupta, senior vice president strategy, Ogilvy India. She says, “from a brand perspective, market research does not only rely on any one methodology but instead is an amalgamation of data from various sources and with different methodologies across qualitative, quantitative research as well as social listening, in order to build a hypothesis and create a way forward. No one data point works in isolation. So, we shouldnt be alarmed at the inaccuracy of exit polls, and we definitely do not correlate it to questioning the efficacy of research as a whole."

What has unnerved observers in the marketing sector is the number of things that must have gone wrong for the polls to be this inaccurate. Deepthi Karthik, a fractional CMO who works extensively with D2C brands, felt that nothing explains the forecasts by polling firms. “The exit polls are based on a sample set of 1,000 per seat and are usually taken in the initial phases of election. The election has been closely contested in terms of vote share and hence it's possible that the momentum shifted in the later half. This however doesn't help answer the uniformity in all exit poll results because the normalisation of such data should have shown dramatically different results by different agencies.”

Karthik said that research agencies that have a diversified portfolio and carry out political research in addition to other consumer research are more likely to bear the brunt of trust deficit for their performance during elections. 

Patwardhan, the ex-CSO, at Sideways consulting who is also disappointed in the lack of rigour in the research, said, "It is a sure dent in the repute of research."

Only time will tell how the exit polls of 2024 reshaped opinion polls in India. Until then, keep calm and steer clear of biases.