Truth Reconciled

With the rise of statistics and the prevalence of scientific research and data on every subject, one would think that everyone would start to come to a better agreement on what is true. That has certainly been the case in the hard sciences like physics and chemistry, and it is somewhat true in the softer sciences, but it is certainly not true in areas such as business and politics.

Both business and politics are a part of most people’s everyday lives and seem to be the topic of most conversations. Nowadays, anyone with a strong opinion about anything has a “scientific” study or two that they can cite to validate their belief. Whenever an argument breaks out, it is common to see people cite seemingly contradictory studies, no matter what the subject is. How can that be? Why does this disagreement exist? Why don’t the statistics clearly support one position?

In most such cases that I have cared to investigate, the disagreement was due to misleading statistics. The arguers usually borrowed these statistics from political pundits, who in turn borrowed them from articles that were written about some kind of analysis. Usually the statistics were simply misinterpreted and misused. But oftentimes they were wrong from the source because the original analysis was flawed.

The purpose of this post is to provide some practice with recognizing unreliable statistics, and give some insight into how and why it happens. I believe it is something that everyone should be aware of in this day and age. I don’t want to attack any particular person, organization, or ideology, so I won’t provide any real world examples. Instead, I made up a story about a new product called Fattaburn that claims that it can help you lose weight. I’ll start by presenting the ad. See if you can spot the deception. If you want to see more real-world examples, check out the links in the discussion section of this post. 

Can you spot the deception?

Suppose you see an ad for a new weight loss pill called Fattaburn. They start with testimonials from individuals who lost dozens of pounds with Fattaburn. They then present the following data-based arguments to convince you that Fattaburn is the best thing ever:

A recent study demonstrated the incredible weight loss benefits of Fattaburn. Study participants in a randomized controlled trial who included Fattaburn as part of their diet and exercise routine lost significantly more weight than those who did not. Our competitor Rivalburn, on the other hand, failed to produce significant weight loss for participants who took it compared to those who did not.

The vast majority of study participants who lost weight with Fattaburn said they would recommend it to a friend, as you can see in this bar graph.

The effectiveness of Fattaburn for its consumers is seen in how popular it is becoming. Last year, Fattaburn’s sales rose by a massive 25%, while Rivalburn increased sales by only 16%.

In a recent survey of health professionals, 75% recommended Fattaburn as a safe and effective weight loss supplement.

The ad then continues with more testimonials, a sense of urgency, and amazing offers that seem too good to be true.

What is your impression of the ad? Are you convinced that Fattaburn might be worth a try? To make it clear how this data was manipulated, let me tell you my made-up story of how it came to be. 

The story of Fattaburn’s statistics 

Bob was the CEO of the company that sells Fattaburn. Bob was absolutely convinced that Fattaburn works, because he himself used it and lost 50 pounds! Bob knew his product was going to be a hit, and he was ready to ramp up the company’s marketing efforts. In order to make the advertisements more convincing, Bob decided to hire someone who could prove the greatness of Fattaburn with a scientific study.

Bob hired a recently graduated PhD named Verity Truth, because she was willing to accept a low salary. Bob proudly showed Verity the data the company had collected on all the people who had used Fattaburn. The dataset included Bob and his employees and many of their friends and relatives. Most of the people who took the pill lost weight. Verity, being the good scientist that she was, recognized that this data was likely very biased. She recommended conducting a randomized controlled trial or RCT, because that’s the only kind of study that can actually prove that an effect exists. 

A randomized controlled trial is an experiment in which something is done or tried on a set of randomly selected study participants to see if it causes an effect. The first step of an RCT is selecting a random sample of the population. The randomization minimizes the effects of hidden variables, such as the desire and motivation to lose weight. The next step is to split the sample into two groups: the control group and the trial group. The control group does not receive the experimental pill, while the trial group does. This approach provides statistical control over many variables that could make the results unusable. For example, if there were suddenly a health buzz in the population during the study, then both the control group and the trial group might lose weight, but by only considering the difference in results between the two groups, the effect of the health buzz is filtered out.

Verity explained all this to Bob. It sounded like overkill to him, but he wanted to look scientific, so he reluctantly agreed to pay for the study. Verity connected with a company that helps run clinical studies, and they got the job done. They ran trials on both Fattaburn and the rival pill Rivalburn.

A few months later, in a very important meeting with Boss Bob and the whole marketing team of Fattaburn, Verity presented the results of her study. She started with the results of the rival weight loss pill Rivalburn. She found no significant difference between the weight outcomes of the Rivalburn trial group compoared to the control group. Bob smiled with pride as she said this. The marketing team took notes. Their rival company’s pill had no effect!

Verity continued, she also found no significant difference between the weight outcomes of the Fattaburn trial group compared to the control group. The whole team was shocked! Bob was furious. “You did something wrong,” he said, “Find the mistake and fix it!”

That week, Verity thoroughly reviewed her research, but found no errors. Naturally, she was fired. Bob realized that he needed to hire someone with more business experience, even though it might cost him more money. He found a very confident and highly acclaimed consulting data analyst named Froddy Krimes. Froddy assured Bob that he could get the results that Bob wanted. He was hired.

Froddy had knowledge of statistics. He knew from Verity’s study that Fattaburn had no significant effect on the population as a whole. But he was experienced and knew how to manipulate the results of her study to make it say what he wanted it to say. He knew he couldn’t get away with saying “participants who took Fattaburn lost weight compared to a control group.” He also knew, however, that at least some participants lost weight, so he would focus on those people. He found that something most of those people had in common was that they were actively exercising and controlling their diet during the course of the study. Froddy realized he could get away with saying “participants who included Fattaburn as part of their diet and exercise routine lost weight compared to a control group.” This statement gives a false impression that Fattaburn had an effect when in reality it was probably just the diet and exercise routine. But the statement is still technically true. For their rival company, Froddy would not mention diet and exercise. He would simply report Verity’s honest conclusion that Rivalburn had no significant effect.

Having now “fixed” Verity’s study, Froddy looked into other sources of data that could be useful, such as the sales data for Fattaburn and Rivalburn. Fattaburn made 2,000 sales in the past year, which was 400 more than the previous year. Rivalburn made 22,000 sales, which was 4,000 more than their previous year. These numbers made Fattaburn look small and uncompetitive, but Froddy realized that Fattaburn’s meager sales looked better in terms of percentages. Fattaburn increased sales by 25 percent, while Rivalburn increased by only 16 percent. Since Fattaburn was so small, its percentages varied widely from year to year, even going negative sometimes. But at least during the most recent year it was a positive number that was bigger than Rivalburn’s number, and that’s all he needed.

Froddy also thought it would be a good idea to get the backing of some health professionals. He had some connections, so he contacted 4 of his pharmacist buddies from a pharmaceutical company he used to work for. He told them about Fattaburn and asked them if they thought it was safe for people to use. He also asked if they would recommend reaching out to pharmacies, since it could be effective in increasing sales for the pharmaceutical company. Of the 4 pharmacists, 3 responded affirmatively, and the other was too busy. Thus, 75% of the health professionals surveyed thought it was safe and effective and recommended it.

At the next marketing meeting, Froddy showed up and presented his deceptive but technically true data-based conclusions. Everyone was thrilled! They immediately began reaching out to investors and preparing their next round of advertisements. Investors were convinced that Fattaburn was outperforming its main competitor Rivalburn in both sales and efficacy. They decided to invest in Bob’s company. Froddy got a bonus and another happy customer. 

Hundreds of thousands of people saw the advertisements, including you. What did you think of it?

Discussion

Now that you know the story of how the data came to be, go back and read the ad again. See if you can spot the deception now. If you pay close attention, you may find that every sentence of the ad was misleading. 

The comparisons between Fattaburn and Rivalburn measured their effects using different baselines. Fattaburn’s results included diet and exercise while Rivalburn’s did not. The axis of the bar graph was truncated to give the impression that almost all survey participants recommended Fattaburn, when really it was only about half of the ones who lost weight. The sales data presented is meaningless because of the relatively small number of Fattaburn sales. Any small change from year to year can potentially yield a large percentage growth or loss. The supposed approval of health professionals came from a very small non-random sample, who may have responded differently if they knew how their responses would be interpreted. 

This is an extreme example, but it demonstrates a few of the many deceptive tactics of organizations who use data for purposes other than discovering the truth. If you would like to see more examples of misleading statistics from real-world sources, take a look at this small sample of articles on the subject:

• Misleading Statistics & Data: How To Protect Yourself Against Bad Statistics
• Stopping COVID-19 with Misleading Graphs
• Misleading Statistics Examples – Discover The Potential For Misuse of Statistics & Data In The Digital Age
• Are you being misled by statistics?

Usually, you can safely assume that most of the data and statistics presented in any ad, on the news, or in any political debate are biased, unreliable, and misleading. If the data collectors, analyzers, or presenters stand to benefit in any way from a certain result, then the results cannot be trusted. They will inject their bias into it, consciously or unconsciously, either by manipulating the studies with the intent to deceive or by unconsciously ignoring the systematic flaws in the study.

Fortunately, not all statistics are so unreliable. We can generally trust studies carried out by people who receive no benefit for obtaining a particular result. This is the case for most scientists involved in basic research. For example, physicists at CERN don’t get paid to discover new particles; they get paid to do research. You’ve probably seen news articles with titles like Physicists ramp up their particle accelerator and find…nothing. I consider these headlines to be evidence of the credibility of physicists. Physicists know the value of finding nothing in a previously untested space: it can rule out physical theories and refine our understanding of physical laws. They generally don’t care about what the public thinks, and that’s a good thing.

There are several practices that make basic research much more reliable than other kinds of research. These are all part of the modern scientific method. Omitting any one of them should call into question the reliability of a study.

• Transparency: Every detail of the study is described publicly. The reasons for the study and the question that it intended to answer are clearly stated. The methods of data collection and analysis are described in sufficient detail to make the study repeatable. At CERN, most scientific articles describe all these details and are made publicly available on arXiv.org. The datasets are also publicly available here.
• Replication: A scientific study must be able to be replicated. The results of the replicated study should agree with the original study. If not, then both studies must come under scrutiny until the issue is resolved. At CERN, physics experiments are carried out independently by several different collaborations working with a variety of independently built detectors in the LHC ring. The main detectors are CMS, LHCb, ATLAS, and ALICE. Many of the conclusions from studies at CERN can also be replicated in smaller laboratories.
• External Review: Studies should be seen by experts in the field who did not participate in the study. An extra set of eyes will notice things that the analyzers ignored, and catch flaws in the study before it gets published. Papers published by the collaborations at CERN go through several rounds of reviews and revisions before finally reaching publication.

These practices ensure the honesty of the researchers and quality of the results. However, even the highest-quality study can be misused by ignorant or deceptive people. Although a scientist can look at the source of a claim and determine whether or not the claim is true and applicable, the average person cannot. This is a difficult problem to solve. In the past few years, fact-checkers have arisen in an attempt to prevent the spread of misinformation, but the fact-checkers themselves are humans and are known to be biased. So what can we do? How can we reliably communicate the truth to the average person?

Conclusion

Here’s the conclusion that I’ve come to: try not to have a strong opinion about anything that is uncertain. Some things are certain, and many things are not. If you focus on what is uncertain, then you will find yourself in a never-ending war of opinions, making enemies here and there, always fighting but never really winning. You will often feel angry, and rarely feel at peace. But if you focus on what is certain, then you can simply ignore the whole confusing mess.

I realized over time that my opinions were not actually based on recent studies and surveys and news articles. Instead, they were based on my personal beliefs, and those never changed no matter how many articles I read. That fact caused me to question why I was trying so hard to keep up with the times. If it really doesn’t affect my beliefs and my decisions, then why do I even bother. I now try to focus on what I know to be true with absolute certainty, and that’s usually enough to help me with most of life’s decisions.