6 Reproducibility and Open Science
Textbook Chapters (or similar texts)
Journal Articles
Tips
Popular Press Articles
*******************************************************
Inquiry-based Activity: Much ado about methods
Introduction: Some psychological scientists say we are in the middle of a replication crisis, with recent attempts to quantify the problem finding that only 39% of studies were able to be replicated (Aarts et al., 2015). Regardless of whether you believe this phenomenon to be a crisis or merely a side effect of the advancement of a field, it is clear that attempts at replication are important and needed. In order to attempt replications at any scale, it is important for methods sections to be comprehensive, rigorous, and full of details that undergraduate students are rarely trained to look for. In this activity, students will brainstorm and revise what an ideal methods section would look like, using published research as their guide. Depending on your particular aims, there is also room to discuss open science practices such as pre-registration.
Question to pose to students: If you are a scientist trying to see if published findings can be repeated, what details about their experiment would you need to know? And, as a secondary question, do you believe most articles include those details?
Students form a hypothesis: Prior to this activity, you should find several published articles that are accessible to undergraduate students/readers. For each, simplify the research question into a short sentence (e.g., “does mood affect memory?”). Once students are in groups in class, let them pick a research question they want to work with. Then, in their groups, they should answer the question above specifically for their chosen research question. Depending on their background, some prompting may be necessary (e.g., what would you need to know about the participants? About the tasks used?).
Students test their hypotheses: Once groups have developed their required set of characteristics, have them actually look at the methods section of their chosen article. Which characteristics can they find? Which details are only partially available, which are completely missing? Finally, have students discuss how the missing pieces could affect a possible replication attempt. For example, what if details of the sample are not completely described? What effect might that have on the replication attempt? Using the example research question above, what if the article does not describe how they induced a particular mood state? Are there likely to be differences between inducing negative mood via physical pain vs. negative images? The particular questions students consider will depend upon the research question/article they have selected.
Do the students’ hypotheses hold up?: As a class, have groups report out about the proportion of the required details they came up with were actually included in the research article. What effects might this have on the general replication effort in the field of psychology? What ideas do they have about how to fix this problem? If you are knowledgeable about preregistration/registered reports (or motivated to become knowledgeable) this would be a good place to explain their role in this problem.
