Two Experts Share AdviCe on HOW to read a Scientiﬁc Study
Consumers rely on the media for accurate health and nutrition information. With such an important educational role, journalists and bloggers have a responsibility to accurately vet and distill the vast amount of scientiﬁc research that appears each year.
Public health depends on correct interpretation of studies. Reporting on a bad study can have more than a neutral eﬀect. It can actually have a negative impact if, confused by all the contradictory health news, people start to tune out even the important information that can help them change their habits for the better.
As part of its Finding Common Ground initiative, an ongoing partnership between top nutrition scientists and leading journalists who care about grounding their work in solid science, the nonproﬁt Oldways solicited advice from two experts on what to consider when reading a scientiﬁc study. Here are 11 tips for reading a scientiﬁc study from our two experts:
- Monica Bertoia, PhD, Visiting Scientist, Department of Nutrition, Harvard TH Chan School of Public Health.
- Stephanie Chiuve, Adjunct Assistant Professor, Department of Nutrition, Harvard T. H. Chan School of Public Health; Assistant Professor of Medicine, Brigham and Womens Hospital.
1. Who are the authors?
First, look at the authors’ names and aﬃliations. Is the ﬁrst author or last author from a respected institution? (The last is the senior author or mentor who is often guiding the project and oﬀering experienced advice to the lead author.) Are the authors considered experts in their ﬁelds?
Have you heard of the journal? Is it a peer-reviewed journal? What is the impact factor? Journals such as JAMA (Journal of the American Medical Association) and the American Journal of Clinical Nutrition, for example, have solid reputations and a high impact factor. However, even reputable journals can publish unsound studies from time to time, though, so look closely at the design of each study.
For background, a journal’s impact factor is a measure of how often articles from that journal are cited by other papers (i.e., how many other researchers are reading and using the paper). Google “[Journal Name] impact factor” to ﬁnd it. Typically, 5 or higher is very good.
2. Is this a single study?
One study by itself rarely provides conclusive evidence. It is essential to look at the study in context and compare the results with the results of other studies. For instance, how does this study relate to the other science collected from similar studies over the last decade? For credibility, you should see a large body of evidence, with the same study results, across various study types and populations. If you look at 40 studies, they will probably not all agree, but you should pay attention to the conclusion of the bulk of the studies before making a public health recommendation. A study that contradicts commonly accepted science should not be discounted, as it could be a signal to go back and reassess by designing diﬀerent studies. Yet, while the contradiction may make a sexy story, without further study, making conclusions or health recommendations from a single study is not based on sound science and is a disservice to consumers.
It is also important to remember that just because a food or nutrient does not have a beneﬁcial eﬀect on one particular disease, it can still be “healthy.” For example, a randomized trial found that multivitamin use did not reduce the risk of cardiovascular disease but it did reduce the risk of cancer.
3. What type of study is it?
There are many types of studies including these most common ones: Lab/Animal, Case-Control, Cohort and Randomized Trial. They all have a role in developing a strong body of evidence on diet and disease studies but some oﬀer stronger evidence than others. The type of study should be described in the abstract or materials and methods section. Here is a summary:
Lab/Animal Study. Studies using animals can provide important data, however, you can’t draw conclusions and generalize ﬁndings to humans. The advantage is that animals, such as mice, are easy to study as the experiment is well controlled. These animals don’t live as long as humans, so it is easier to look at diet over a lifespan. It is also easier to measure disease markers in tissue.
Case-Control Study. This is an observational study that compares patients who have a disease or outcome of interest (case) with those who don’t (control) and looks for diﬀerences that could potentially explain what causes the disease. Say researchers are looking at the connection between heart attacks and diet. They would compare two groups—those who had heart attacks and those who didn’t—and compare case diets to control diets. If there are diﬀerences in diet, it is possible that they could explain health diﬀerences. It is tricky to select the control group, however. People who have had a major health incident, such as a heart attack, often think through events in the past diﬀerently so asking them to “recall your diet over the last 10-12 years” may not yield reliable data. While this is a good way to gather information, it is not conclusive and is not an ideal study design.
Cohort Study. This type of study allows researchers to follow a large group of people over time in order to see which risks factors are associated with a disease or outcome. For instance, one Harvard School of Public Health cohort study has followed 50,000 male health professionals since 1986 (The Health Professionals Follow-up Study). Researchers follow up every two years to ask these men about their health, weight, smoking status, disease, medications, etc. With a 25-year study, researchers can see the development of disease and other outcomes. Evidence from a well-designed cohort study is solid, however, it has drawbacks too. In this case, those in the study select their own diet and lifestyle. Often diet goes hand-in-hand with other lifestyle choices: those who eat better tend to have other healthy habits such as exercising more, not smoking, etc. Statistically researchers try to correct for this, however, it can be diﬃcult to tease out true cause and eﬀect. In this example, the challenge is determining if lower risk of disease is due to speciﬁc eating habits alone or eating habits combined with other related behaviors like a certain level of exercise or other healthy habit.
Randomized Controlled Study. This is often considered the gold standard of studies. This study design randomly divides participants into two groups—an experimental (treatment) group and control (placebo) group. The only expected diﬀerence between the two groups is the experimental treatment being studied (e.g., Vitamin D pills or advice to follow a Mediterranean diet). Because the study is randomized, each person has an equal chance of being in either group. If enough people are in the study, on average, there will be about the same percentage of smokers, exercise level, etc. in the experimental group and control group. This means that, unlike the problem in cohort studies, where healthier eaters tend to exercise more and smoke less which makes it hard to tease these behaviors apart, in a randomized study of diet, exercise levels and smoking habits will be the same in the experimental and control groups. Therefore, theoretically, the only diﬀerence between the two groups is their experimental diet, and thus any diﬀerences in the outcome between these groups will be attributed solely to the diet. These studies can be expensive and time consuming, which is why there are fewer underway. Further, even the most motivated participants may have diﬃculty adhering to an assigned dietary regiment for a long period of time (several years) which is needed to study chronic diseases that develop slowly over many years. Because of lack of adherence and limited duration, randomized trials can easily miss important eﬀects of diet or other factors, and thus be misleading. Additionally, many times randomized trials are simply not possible to conduct due to ethical or practical reasons (for example, to study the eﬀects of occupational exposures or reproductive factors).
One side note: sometimes studies that did not deliver expected results are not published. It is good practice to publish these “null” results as it provides another source of data for scientists and helps with future research directions and pubic health recommendations.
4. How large is the study?
It is important to look at whether a study is made up of 10 people or 10,000. As expected, a larger, well-designed study is typically a more reliable source of data. Still, a high-quality smaller study, taken in context of the whole body of work, can also provide strong evidence. It can signal an interesting ﬁnding that leads to more research. In the abstract, authors will usually comment on study size and put it in context. This is an area you should ask about with the author or outside expert because some study designs only require a few participants.
5. What kinds of people were in the study?
Due to ﬁnancial and time constraints, studies are typically not representative of a whole population. An author should comment on “generalizability” in the paper. For example, a study focused on men may not be generalizable to women, or a study based on healthy people may not be generalizable to those with signs of heart disease. Before reporting on a study, you should understand how applicable the ﬁndings are to the general public.
6. Did the study look at real disease endpoints?
The strongest studies look at “hard” disease end points such as heart attack, stroke, cancer, etc. These diseases typically develop over many years. Studies that look at these outcomes are rare since it is expensive and time consuming to follow participants for many years. Instead, scientists often look at intermediate, or “soft,” endpoints such as blood pressure or cholesterol. While one may assume the intermediates are good markers for the disease endpoints, one can’t be sure. It is essential to look at hard endpoints before making a conclusion about them—and certainly before presenting any public health recommendations.
7. How was diet assessed?
Dietary patterns are diﬃcult to measure. With input provided by the subjects themselves, the information is not always reliable. Documenting food intake is not as easy as, say, keeping track of taking one Vitamin D pill each day.
There are two common measurement methods:
- Food Records. This is the most accurate of the two methods but not foolproof. Patients record everything they put in their mouths during a set period of time (typically 3 to 7 days). However, because information is self-reported, there is room for error based on forgetfulness or reluctance to confess every morsel. Also, because it is tedious to record everything eaten for several days, people can change their eating habits to more simple foods to reduce the work.
- Food Frequency Questionnaire. This method depends on the accurate recall of patients. Participants check oﬀ boxes asking about overall diet and speciﬁc foods. Subjects check oﬀ if they eat a certain food, how much, and the frequency. This questionnaire typically asks about food choices over the last three months to a year so relies on the participant’s memory.
A good study will report validity by comparing data from food records, questionnaires and nutrient biomarkers (measurements taken from body tissues or ﬂuids). A high correlation provides conﬁdence that the dietary measurement is solid. All measures have some error but well-designed food frequency questionnaires and food records used in a cohort study can provide strong information on diet and health. In long-term studies, a single measure of diet (i.e., a single food frequency questionnaire at baseline) is less desirable than repeated measures over time (i.e., participants have completed a food frequency questionnaire every few years during the study.
8. Time frame: how long was the study and what is the time frame of exposure to the dietary factor?
Ideally, a researcher wants to capture a diet before disease develops and a longer study provides a better opportunity for this. For example, diseases like heart disease and cancer develop over decades and therefore researchers are interested in measuring diet over the long-term. Depending on the research question, a shorter study also may be eﬀective. For instance, a study looking at changes in blood pressure may only need to measure diet over the previous few weeks because some dietary factors can change blood pressure quickly.
9. Does this study conﬂict with existing data?
The authors should put the study into context, especially if it contradicts a large body of work. They should compare speciﬁc diﬀerences between the studies and comment on what caused the diﬀerences (i.e., study design, study population, follow up). An outside expert can also help explain this.
10. Is this a meta-analysis?
A meta-analysis is a statistical approach that summarizes the literature around a speciﬁc topic. When done well, a meta-analysis can present a conclusion that is statistically stronger than just one study. The challenge is determining which studies are high quality and should be included in the analysis. Researchers must have a very clear, unbiased search criteria related to what is included/excluded. Look at the study design to see which types of studies are included and identify any limitations. Randomized controlled trials are the strongest, but the best body of evidence comes from a combination of randomized trials and cohort studies. Publication in a respectable journal is an indication of quality as well.
11. Have you checked with the author and an outside expert to make sure you haven’t missed any nuances?
Reporters should always contact the author to review the data and ﬁndings. If the author isn’t available, you can also talk with a co-author who can give an insightful overview. In addition, it is helpful to discuss the study with a third-party expert who can help put it in context in an unbiased way. A good resource is Oldways Science and Media Clearinghouse, which links journalists with scientiﬁc experts on food and nutrition who are available for comment. In addition, make it a habit to always look at the original published study vs. just an abstract or press release. Often journalists are responding to a press release distributed by a journal, so the responsibility rests on the media department to ensure a study is summarized in a credible and responsible way.
As we are learning, science is an evolving process. A body of work, including results from randomized trials and prospective epidemiologic studies, ideally supported by sound mechanistic evidence and animal experiments, must be considered for strong evidence. Asking these 11 questions as you read a scientiﬁc study will ensure that you are providing credible and useful information to people through your reporting, contributing to improved public health.