You might be familiar with this: one study says "Beta-carotene protects against cancer!" and another "Beta-carotene leads to MORE cases of cancer!" So which is it? Which study is actually right? In this Ziel mir keng episode, we look at different types of studies and their reliability. We will be focusing on bio medicine, using examples from nutritional science.
The aim is to give you a bit of guidance through the complex process of research:
By the way, for this video we teamed up with researcher Torsten Bohn from the Luxembourg Institute of Health. He is a nutritional scientist and a specialist in carotenoids.
That is why, in this video, we will use the example of a carotenoid, namely beta-carotene. It’s a molecule found in carrots, for instance, as well as in many other orange fruits and vegetables. When you consume beta-carotene, part of it is converted into vitamin A, which is important for our eyes, for example. The rest remains in the body and has, among other things, an anti-inflammatory effect.
Now, we know that inflammation in the body can be partly responsible for cancer. Could it therefore be that beta-carotene protects against cancer? Let's look into this question systematically. For now, it is just a hypothesis after all!
To find out if beta-carotene really does protect against cancer, we turn to the pyramid of evidence model.
There are different types of studies – different study designs, so to speak. The lower a study type sits on this pyramid, the less conclusive its findings are. The higher up, the more conclusive the evidence.
Studies further down the pyramid are often easier or cheaper to run. They help us to form and test hypotheses. Studies further up, on the other hand, tend to be more expensive, have greater statistical power, and can confirm correlations – or indeed, show that there aren't any. And if they are right at the top, they can sometimes even prove causality: meaning they can tell us whether substance A actually causes disease B – or not.
Let's go back to our question: does beta-carotene protect against cancer?
We will start at the bottom tier of the pyramid: cell and animal experiments.
In lab studies, meaning with cells in a Petri dish, researchers were able to show that beta-cartone has an anti-inflammatory effect. They have also been able to show how the underlying mechanisms of action work.
Does this mean that beta-carotene fights cancer? No!
The effect on cells in a Petri dish isn't necessarily identical to the effect on a whole organism. The concentrations used are often different from what you’d find in humans, too.
But the hypothesis is plausible. At least in cells, the evidence points in that direction.
There are also results from animal experiments, e.g. on mice. One study found that lab mice with lung metastases survived significantly longer when treated with beta-carotene.
Beta-carotene slowed down the spread of cancer cells.
Is this now proof that beta-carotene protects against cancer? No…
For example, it was found that mice convert beta-carotene into vitamin A at a rate of 100%. Humans, however, only manage about 30%. These studies are therefore of limited relevance to humans. But the hypothesis remains plausible.
Moving up the pyramid, we encounter expert opinions and case reports (descriptions of individual cases) or case series (several cases grouped together).
At this level, humans are involved, but there is not yet a systematic study design to provide objective results, nor is there any statistical power behind them yet.
To bring in more objectivity and statistical power, let’s move up to the next two levels of the pyramid: observational studies. Here we have case-control studies on the one hand, where you look back in time, or cohort studies, where you look forward.
In a case-control study, you compare people who, in this example, have cancer (the case group) with people who do not have cancer, i.e., the control group. Then you look back to see how much beta-carotene they consumed. Usually by means of questionnaires or interviews.
In one study, for example, researchers asked 600 women with breast cancer and 600 healthy women about their eating habits. The researchers found that beta-carotene, among other things, was linked to a lower risk of breast cancer.
Does this now mean that beta-carotene protects against cancer? No, still not.
These kinds of studies can show correlations, if there are any, i.e., links, but not causality, i.e., no cause-and-effect relationship.
More on this topic in our video on common mistakes in statistics: Only science.lu: related article: 5 common mistakes in statistics
A disadvantage of this type of study is, among other things, recall bias. Participants cannot always remember details. But the upside is that these studies are easy to conduct and are not very expensive: simply interview two groups at a given point in time, and you already have results.
Now let’s look at cohort studies. Here, researchers take a cohort, i.e., a group, for example, of healthy people, and follow them over many years. During this time, the researchers repeatedly ask how much beta-carotene the people in the cohort are consuming. Over time, statistically speaking, a number of people from the cohort will unfortunately develop cancer. When that happens, the researchers analyse whether – and how – those cases are linked to beta-carotene consumption.
For example, one cohort study recruited over 56,000 women and followed them for years. 196 developed lung cancer during the time of the study. But the researchers found no statistical correlation to beta-carotene.
A general problem with observational studies is that you cannot attribute everything to a single substance. People who consume a lot of beta-carotene usually get it from fruits and vegetables – which are generally healthy. But how many of the health benefits can then be credited specifically to beta-carotene? Could it instead come from the fibres or the vitamins contained in fruits and vegetables? What’s more, people who eat a healthy diet tend to be more health-conscious overall and often have a higher socio-economic status. So there are quite a number of confounding factors!
So where does that leave us with beta-carotene? Some studies are promising, others less so...
Now we come to the type of study that can potentially demonstrate causality: randomised controlled trials. Ideally: randomised, double-blind, placebo-controlled trials!
Here, too, researchers are looking into the future. But the design of these kinds of studies is much more tightly controlled to eliminate any kind of distortion. Randomised means that the participants are assigned to a group at random. One group receives beta-carotene, the other a placebo.
This randomness is statistically important to ensure that the two groups are as homogeneous as possible. That is why the number of participants also plays an important role. Randomised controlled trials with a few thousand people are more conclusive than studies with just a few hundred or fewer participants.
Double-blind means that neither the participants nor the researchers know who is getting what: the placebo or the beta-carotene. This is to eliminate bias on both sides. Because if, for example, participants know they are receiving the active ingredient, they automatically start to feel better. That needs to be prevented.
These studies are the gold standard in biomedical research.
There are actually two major randomised controlled trials that investigated the effect of beta-carotene on cancer. And they found that the study participants who received beta-carotene… developed more cancer… What?? More cancer! Not less!
In fact, both studies had to be stopped early for ethical reasons.
How could this be? Up until now, it looked so promising?!
When you look more closely at the data, you can see that in the studies, beta-carotene had no effect on people who didn’t smoke, but actually increased the risk for people who did smoke.
So, if some people were already thinking that beta-carotene could be a miracle cure: unfortunately, that’s not the case.
But you might say: those are just two studies. Wouldn’t it be better to look at a large number of studies? True to the old adage: one swallow does not a summer make.
That is exactly what so-called meta-analyses do. They look at lots of studies together and analyse whether they all point in the same direction – or not.
In the old pyramid model, they sit above randomised controlled trials. In newer models, meta-analyses sit alongside them. Because, yes, several studies are better than one. But if a randomised controlled trial involves several thousand or even a hundred thousand participants, is well conducted, and delivers clear results, it can sometimes be more insightful than a meta-analysis that compares lots of small studies. Especially since those smaller studies often use different methods and vary in quality. In other words: it is sometimes like trying to compare apples with oranges.
In our example, various meta-analyses show no effect of beta-carotene on cancer risk. However, one meta-analysis, for instance, confirms that beta-carotene leads to more cancer in smokers.
And in another, overall mortality goes up with beta-carotene consumption. This is because smokers and non-smokers were lumped together.
If you find yourself losing track in the jungle of all these studies and discoveries: that’s completely normal! But what can you actually rely on?
This is where bodies like nutrition societies (such as the EFSA) or other organisations like the WHO come in. They specialise in pooling all the existing knowledge on a specific topic and formulate a recommendation. Generally, you can confidently rely on them for guidance.
That's why, in some versions of the evidence pyramid, these "clinical practice guidelines" are placed right at the very top.
Another excellent source is quality science journalism, where specialist journalists dig into a topic, often with the help of researchers.
Ideally, in biomedicine, you want to know: if I consume substance A, does it cause or protect against disease X? But not every study can provide a clear answer. So researchers take a step-by-step approach. They gather evidence, piece by piece like a puzzle, until a finding becomes increasingly confirmed. You can almost never be 100% certain. Nevertheless, researchers can still discover a great deal.
In the case of beta-carotene, there are many studies suggesting that the substance has the potential to protect against cancer. However, this was later refuted in crucial large-scale studies. As a result, the recommendation today is not to take supplements, but simply to eat a healthy diet. And for people who smoke, taking additional beta-carotene is actually advised against.
The pyramid we’ve discussed here applies mainly to biomedicine. It simplifies heavily and is not exhaustive. In other fields like sociology, the humanities, or other natural sciences, there are completely different study designs and methods.
Either way, the golden rule is: be sceptical whenever someone makes overly grand health promises.
Ziel mir keng! is broadcast on Sunday evenings after the programme Wëssensmagazin Pisa on RTL Tëlee and is a collaboration between RTL and the Luxembourg National Research Fund. You can also watch the episodes on RTL Play.
Author: Jean-Paul Bertemes (FNR)
Peer review and advice: Torsten Bohn (LIH)
Editing: Michèle Weber, Lucie Zeches (FNR)
