Abstract

As members of society, students must be able to evaluate scientific claims across a wide variety of media to make sound decisions about health and wellness. However, students - and most members of society - struggle to evaluate the quality of evidence supporting a scientific claim. The goal of this lesson is to empower students to recognize unethical and/or overstated scientific claims. Towards this end, the lesson plan contains a combination of pre-class work, analysis of a TED video, group discussion and a jigsaw activity. The in-class portion culminates with a critical evaluation of the putative memory enhancer Prevagen®. We find that students who successfully complete the lesson know criteria for evaluating the quality of material that is presented as scientific. They feel empowered to make informed decisions about health and wellness based on their newly acquired practice with identifying valid/invalid scientific reasoning and with recognizing pseudoscience.

Citation

Society Learning Goals

Science Process Skills

• Process of Science
• Pose testable questions and hypotheses to address gaps in knowledge
• Interpret, evaluate, and draw conclusions from data
• Construct explanations and make evidence-based arguments about the natural world
• Modeling/ Developing and Using Models
• Build and evaluate models of biological systems
• Quantitative Reasoning/ Using Mathematics and Computational Thinking
• Apply the tools of graphing, statistics, and data science to analyze biological data
• Communication and Collaboration
• Share ideas, data, and findings with others clearly and accurately

Lesson Learning Goals

• determine whether scientifically presented conclusions were formed from valid scientific reasoning.
• develop the ability to make informed decisions about whether to take different food supplements.
• recognize pseudoscience.
• appreciate the nuances of how the media, companies, and sometimes even researchers present flawed scientific data and results in order to support their desired outcomes.

Lesson Learning Objectives

• create criteria for evaluating information that is touted as scientific.
• apply those criteria to evaluate the claim that Prevagen® enhances memory.
• identify the misleading tactics used on the Prevagen® website and in their self-published reporting.
• decide whether to recommend taking Prevagen® and explain their decisions.

INTRODUCTION

The ability to evaluate scientific claims critically, whether in a research article, commercial, or on a website, is a necessary skill for daily life (1), especially given the amount of pseudoscience present in the media (2, 3). For example, countless health claims about magical cures distract consumers from scientifically valid treatments (4), particularly for products that claim to improve mental acuity (5-7; https://www.ted.com/talks/molly_crockett_beware_neuro_bunk). Science instructors can empower students to make educated decisions about their health care by teaching them how to recognize pseudoscience (2, 4). Students can gain this ability by applying the principle that scientific reasoning results from testing ideas and generating conclusions based on high-quality evidence (3, 8). While students may understand this process at face value (9), it is more challenging to teach them how to evaluate the quality of evidence on their own (e.g., 1).  

Pseudoscience describes information that has been presented in a way that superficially looks like science, but is instead made up of misleading statements, fraudulent data, or outright lies (2). Some universities offer entire courses that develop students’ abilities to recognize pseudoscience (e.g., 2, 3, 10). However, a shorter lesson, such as the one presented here, can be integrated into many courses, thus increasing students’ opportunities to practice identifying pseudoscience (e.g., 4, 11). Our lesson differs from others (e.g., 2-4, 10, 11), by focusing on a class of medical treatments—food supplements—that are sometimes scientifically demonstrated to be beneficial (e.g., folic acid supplements reducing incidents of spina bifida, 12), and sometimes not (this lesson).

This lesson helps students recognize the aspects of pseudoscience (e.g., http://www.compoundchem.com/2014/04/02/a-rough-guide-to-spotting-bad-science/, 13) so that they can make their own, informed conclusions about a well-advertised food supplement. For example, advertisers know that scientific imagery makes advertisements more believable. College students accept erroneous scientific conclusions more readily when newspaper articles that misinterpret information are accompanied by a picture of a brain (14). Here, we focus on the unsubstantiated claims made by Quincy Bioscience about their product Prevagen® (http://www.prevagen.com/), a food supplement that they claim enhances memory. Quincy Bioscience cites studies—some of which are peer-reviewed—on their website that they claim support the efficacy of Prevagen® at improving memory. A close look at the studies, however, reveals a number of ways in which standard scientific practices have been violated:

• some are not controlled;
• inappropriate statistical tests are used and inappropriate statistical conclusions are reached (15);
• Quincy Bioscience paid for the studies, so a strong conflict of interest exists;
• the active ingredient in Prevagen®, the protein apoaequorin, does not pass the blood-brain barrier, so it cannot impact memory (16, 17);
• the primary report listed on the website is self-published and not peer-reviewed;
• two peer-reviewed papers by employees of Quincy Bioscience (16, 18) assessing safety in rodents were both published in Regulatory Toxicology and Pharmacology, a contentious journal that is viewed by many academics as a vehicle for publishing pseudoscience that promotes industry interests at the cost of health and safety (19);
• the authors of the peer-reviewed studies have only published papers about Prevagen® and were funded and/or employed by Quincy Bioscience; and
• a series of recent lawsuits question the efficacy of Prevagen® (e.g., 7).

In this lesson, we ask students to evaluate the claim that manufacturers have made about the efficacy of Prevagen®. Through a combination of pre-class work and in-class analysis of a TED talk (https://www.ted.com/talks/molly_crockett_beware_neuro_bunk), students develop and then apply criteria for assessing the quality of science to discover unethical and overstated claims about a widely-advertised dietary supplement.

Intended Audience

This lesson can be used with a variety of audiences. Because teaching students to recognize pseudoscience is so critical (1-3, 8), it can be used in any science course for science majors and non-science majors at both introductory and advanced levels.

We initially developed this lesson for a second-year course for non-science majors at a regional master’s university, but we have also used it in a third-year course for science majors at the same institution. Although both courses had a maximum enrollment of 24 students, the in-class activities would work for any class in which a jigsaw (20) is implemented, which can include large classes (21). The materials we used for these two audiences are the same. They only differ in the nature of the in-class discussions and the sophistication with and speed at which students complete the pre-class assignment.

Required Learning Time

Pre-class work (Supporting File S1: Pre-class work) takes between 30 and 45 minutes; we distributed the assignment through our learning management system, following the standard practice for the courses we teach. The rest of the lesson can be completed in a 50-minute class period. We have taught it as a 50-minute portion of a 2-hour class meeting.

Pre-requisite Student Knowledge

We expect that students who have been exposed to advertising about food supplements may be more engaged in the lesson. Because we have used this activity successfully in such different courses—second year non-science majors and advanced science majors—we know that specialized scientific knowledge is not required. Students complete a pre-class assignment in which they read and write about ways to assess science. Students will develop criteria for evaluating the quality of science presented to them by building on the foundational knowledge they have when they enter the classroom.

Pre-requisite Teacher Knowledge

Teachers should come to class with criteria for detecting pseudoscience and a basic familiarity with pseudoscience. The materials that we present here to prepare students clearly articulate many common practices in pseudoscientific presentations:

• https://www.ted.com/talks/molly_crockett_beware_neuro_bunk
• Supporting File S1: Pre-class work
• Supporting File S2: In Class Worksheet
• Emily Willingham’s essay about “10 questions to distinguish real from fake science” https://www.forbes.com/sites/emilywillingham/2012/11/08/10-questions-to-distinguish-real-from-fake-science/#6de88e0533bf (13)
• “A Rough Guide Spotting Bad Science” by Compound Interest (http://www.compoundchem.com/2014/04/02/a-rough-guide-to-spotting-bad-science/)

In class, students watch a TED talk by Dr. Molly Crockett called “Beware neuro-bunk” (https://www.ted.com/talks/molly_crockett_beware_neuro_bunk) that deconstructs strategies that advertisers use to sell products that claim to enhance brain function. She defines neuro-bunk as gross exaggerations of conclusions from scientific studies, or even manipulation or lies about scientific data. We prefer the term pseudoscience to neuro-bunk, because it applies to this same suite of misleading approaches but applied to any scientific discipline.

Teachers may also want to prepare to discuss and support the applicability of detecting pseudoscience in larger real-world settings, including misrepresentations of evolution (3) and climate change (22), and non-scientific but popular topics such as astrology (11) and extrasensory perception (ESP, 2). Additional resources for identifying pseudoscience include descriptions of curricula covering the topic (2, 3, 10).

Students ask many questions about Prevagen® and the pseudoscience that Quincy Bioscience has published. In addition to the information we present in the Introduction, it would be helpful for teachers to read Speth’s critique of Quincy Bioscience’s published research on PubMed (17), which helps interpret their peer-reviewed publications (16, 18), which appear in a notoriously unethical journal, e.g. Regulatory Toxicology and Pharmacology (19). It will also be helpful to review one of the legal challenges to Prevagen® (7).

SCIENTIFIC TEACHING THEMES

Active Learning

This lesson uses a variety of strategies for students to engage in active learning, including individual writing, individual brainstorming and whole class discussion, analysis of a TED video, a jigsaw, and small group discussion (Lesson Plan Timeline, Supporting File S1: Pre-class work, Supporting File S2: In Class Worksheet).

A jigsaw is included to facilitate small group discussion (20, Supporting File S2: In Class Worksheet). A jigsaw is a cooperative learning technique in which each student’s part of an assignment is a jigsaw piece essential for complete understanding of the final product, the completed jigsaw puzzle, by all the students. The class divides into small groups of 3-5 students that foster expertise in different figures or tables that the instructor chooses from a Quincy Bioscience report (15) or from the articles Quincy Bioscience has published (16, 18). For example, instructors may want to choose Figures 1 and 2 from the self-published report (15). Then, the groups reshuffle to include one member of each expert group. Each student in the reshuffled group can then share their expertise with the others. The lesson concludes with a whole class discussion that affirms what students have solved on their own and helps them through more troublesome areas.

Assessment

Table 1 presents the way the lesson aligns to the learning goals and objectives.

To prepare for class, students submit a summary of the pre-class reading that outlines ten criteria for evaluating material that is presented as science (13). The students summarize the reading and then describe in more detail three of the criteria that they find particularly helpful to apply (Supporting File S1: Pre-class work). Because we ask students to turn in assignments (through the learning management system) at least one day before class starts, we evaluate their answers to the questions in Supporting File S1: Pre-class work and decide which points need additional clarification during class.

In class, we assess learning by eliciting student feedback during group discussions and by circulating through the classroom to see what students were writing down on the worksheets we provided to guide in-class activities (Supporting File S2: In Class Worksheet). We use these worksheets for formative assessment.

For summative assessment, we collect the worksheets at the end of class, spot-checking each to make sure that each student has met the learning objectives. We assign a grade for participation based on the quality of the completed worksheets, keeping in mind that students learn at different rates and record different amounts of detail. Students who prefer to take more time to complete the worksheet are welcome to turn it in the following class. We have not included any additional assessment of this activity in our courses.

Inclusive Teaching

This lesson was designed for second-year non-science majors, but is has also been used for third-year science majors. It teaches practical and essential scientific literacy skills by having students analyze data that a real company reports on its website about a real—and controversial—food supplement.

We designed the lesson to involve all students. Everyone completes the preparatory pre-class work. We project the TED talk (https://www.ted.com/talks/molly_crockett_beware_neuro_bunk) with closed captioning, and students can refer to the transcript of the talk while they complete the in-class activities. Everyone becomes an expert in the first half of the jigsaw, and everyone shares their expertise in the second half of the jigsaw. The jigsaw format, in particular, promotes inclusive teaching (21). 

The students learn through many different strategies, including individual writing and large and small group discussions. The number of strategies means that students who favor different approaches to learning will benefit from this lesson. Small groups can be formed through a variety of mechanisms that promote inclusion in ways that reflect each instructor’s overarching goals for a course (23).

LESSON PLAN

Pre-class Preparation

Instructor Preparation

The assigned reading (13) and TED talk (https://www.ted.com/talks/molly_crockett_beware_neuro_bunk) help instructors prepare for the lesson. Instructors may find that a table comparing valid scientific approaches to misleading ones complements the assigned reading. Compound Interest has already tabulated these into “A Rough Guide to Spotting Bad Science” (http://www.compoundchem.com/2014/04/02/a-rough-guide-to-spotting-bad-science/). We recommend projecting this chart as a slide or making printouts to distribute to students.

Instructors should prepare 3-5 data figures or tables obtained from the Prevagen® website (http://www.prevagen.com/), and paste these into the in-class handout (Supporting File S2: In Class Worksheet). Although it would have been helpful for us to include the figures or tables in Supporting File S2, we could not do so because they are copyrighted. The number of figures chosen depends primarily on the number of groups desired; the main messages of the figures, as they pertain to pseudoscience, are redundant: improper samples, improper measures, and improper data analysis (15: Figures 1 & 2, 3 & 4, 5 & 6, Table 3). We recommend that instructors choose figures or tables from a self-published study by Quincy Bioscience called the Madison Memory Study (15) or from their papers that are peer-reviewed but published in suspect journals (16, 18). The Madison Memory Study (15) focuses on data from a task the participants of the study were asked to complete. While these data are used as evidence to argue that Prevagen® increases memory, the data are derived from a verbal learning task, rather than a memory task.  Moreover, the people in these samples had normal cognitive function.  In terms of data presentation, error bars are not included. In general, the studies on the Prevagen® website are not properly controlled (17), use inappropriate statistical tests (17), contain major conflicts of interest (17), and, as stated above, are either not peer-reviewed or are published in suspect journals (19).

Student preparation

Students will be assigned a pre-class reading about how to recognize pseudoscience (13). The students will be asked to summarize the reading in 150-200 words and describe three methods for assessing the credibility of scientific reports (Supporting File S1: Pre-class work). We ask students to submit the assignment through the learning management system before class starts, so that we have time to review their work beforehand. With this approach, we can assess whether there are inaccurate ideas that we need to address in class. However, the assignment is straightforward, and most students meet the expectation that they understand common pseudoscience tactics. The general goal of the pre-class preparation is to introduce students to the idea that scientific claims must be critically evaluated and that there are established best practices to use.

Progressing Through the Lesson

The in-class period will consist of four different activities, which are guided by a worksheet (Supporting File S2: In Class Worksheet). The worksheet begins by asking students to brainstorm as individuals about how they recognize pseudoscience, and then engage in a whole class discussion that generates criteria for the whole class to apply later.  After the group discussion, the students watch the TED talk “Beware neuro-bunk” (https://www.ted.com/talks/molly_crockett_beware_neuro_bunk), taking notes about additional aspects of pseudoscience that they might not have considered.  These steps prepare them to begin evaluating the Prevagen® website (http://www.prevagen.com/) with the criteria they generated as a class. Then, they divide into groups to explore the data that Quincy Bioscience presents in a jigsaw. After the groups reshuffle as a jigsaw, they synthesize their learning to decide whether to recommend that their grandmothers take Prevagen® (Lesson Plan Timeline). The instructor and each student will require individual access to a computer with Internet connection in order to complete the activities and worksheet. It is helpful for the instructor to project sections of the Prevagen® website (http://www.prevagen.com/).

1. Developing criteria for recognizing pseudoscience

Students work in groups of two or three to draw from their life experiences and their pre-class work to develop criteria for recognizing pseudoscience (Supporting File S1: Pre-class work). Then, a whole class discussion generates a master list of criteria. Collating student ideas in a shared document, such as a Google Doc, can facilitate list making. More traditional note-taking on a board also works well. The instructor guides discussion, using elements from “A Rough Guide to Spotting Bad Science” by Compound Interest (http://www.compoundchem.com/2014/04/02/a-rough-guide-to-spotting-bad-science/). Ideas that students have generated are presented in Table 2. Once criteria have been generated by the whole class, students can compare their work to the criteria presented by Compound Interest in “A Rough Guide to Spotting Bad Science” (http://www.compoundchem.com/2014/04/02/a-rough-guide-to-spotting-bad-science/).

2. Beware neuro-bunk: TED talk and group discussion

After students generate their criteria for evaluating science, they watch the TED talk “Beware neuro-bunk,” a talk by neuroscientist Dr. Molly Crockett detailing the limits of interpreting neuroscientific data and the ubiquity of modern marketing strategies to neurological functioning (https://www.ted.com/talks/molly_crockett_beware_neuro_bunk). Students use their handout to record the pseudoscience that Dr. Crockett mentions, which primes them for subsequent class discussion about the pseudoscience that the students observed.

3. Evaluation of the Prevagen® website

In this activity, students will use the list of ways to recognize pseudoscience that they generated as a class to assess the credibility of scientific reports included on the Prevagen® website (http://www.prevagen.com/) to convince the public that their supplement is a memory enhancer. Students will begin by exploring the Prevagen® website (http://www.prevagen.com/) individually, taking note of specific instances of failure to meet criteria. Then, they will then share their results with a neighbor.

This activity is student-led, so that students are using their own habits and skills of browsing the Internet to interpret the website. It is not essential which part of the site they visit, so this part of the activity is robust to changes in the website.  The website was redesigned after we began teaching this activity, and this activity continues to work. Most students observe the attractive, cosmetic aspects of the website that promote a scientific theme, including a picture of a jellyfish, a page on the site devoted to research, claims about what Prevagen® can do, and a Q&A page with reasonable-sounding questions. However, the students also recognize asterisked comments that the claims on the site have not been evaluated by the Food and Drug Administration, and that the photographs of people on the site are intended to appeal to a particular demographic. Noticing these details primes students for the jigsaw in which they evaluate the data that Quincy Bioscience presents about Prevagen®.

After perusing the website, students will work in a jigsaw format to evaluate the 3-5 data figures obtained from the material cited on the Prevagen® website (http://www.prevagen.com/). We recommend using figures or tables from Quincy Bioscience’s self-reported Madison Memory Study (15). Students then divide into groups of 3-5, with the number of student groups matching the number of data figures the instructor chooses to assign. We recommend adding the figures to the handout provided in Supporting File S2: In Class Worksheet. After the students become experts at understanding one figure from their first group, they reshuffle into new groups consisting of one team member who is expert at each figure; in other words, each piece of the jigsaw is represented in the second group. The experts teach their group members about their jigsaw piece, so that, by the end of this activity, each student has completed the handout asking questions about all of the figures. After the jigsaw, a whole class discussion highlights the key points about each figure.

4. Synthesis

After the jigsaw, the students synthesize their learning by making a recommendation. Working in the small groups from the reshuffled part of the jigsaw, they discuss a situation in which one of them has a grandmother who has been complaining about memory loss. She asks the student whether to start taking Prevagen®, and the students work collaboratively to make—and explain—a recommendation.

TEACHING DISCUSSION

This lesson teaches students how to evaluate the scientific claims presented on one website, which gives them practice with the essential skill to evaluate critically scientific claims in the popular press. The lesson starts by asking students to learn about pseudoscience and construct a list of the best practices for assessing the credibility of a scientific claim (Table 1). Then, students apply this knowledge to evaluate a company website’s scientific claims and the highly biased evidence they present to support those claims.

The pre-class work assignment prepares students to come to class with knowledge about common strategies used to assess the credibility of a scientific claim. The instructor uses class discussion and activities to encourage the students to expand their knowledge of these unethical strategies. We have found that, after the combination of pre-class work and in-class discussion, students seem eager and excited to evaluate the Prevagen® website. Students have also commented on the fact that they have seen Prevagen® commercials and wondered about the scientific claims made by the company. Thus, students are engaged throughout the class.

We have used this activity in two very different courses to give students practice critically evaluating scientific claims. The first was a second-year seminar for non-science majors about neuroscience and memory. Students attended one class per week. This lesson was taught at the end of the course, after they had learned about how memories form and about neuropathologies. The instructor (author AGS) was able to use this lesson to discuss pseudoscience, which set the stage for subsequent classes about the science of how peoples’ memories change as well as the societal implications for those changes. The second class is a third-year classroom-based undergraduate research experience (25) for majors in which students ask and answer novel questions about the fossil record. The instructor (author RMP) teaches this lesson toward the end of the course, so that students can compare the science they have been conducting to the pseudoscience presented on the Prevagen® website. By completing the lesson, the students practice transferring their knowledge of how science works to a completely different real-world situation in which they need to evaluate the quality of evidence presented to promote a food supplement. Students in both classes were able to generate and apply criteria for recognizing pseudoscience.

Developing Expertise

At the beginning on the lesson, some students are surprised to learn that companies often present misleading claims and data in order to sell products. In other words, some students believe that scientific research produces hard and fast results that are not easily manipulated. By the end of the lesson, however, students know they must evaluate scientific claims. They know the best practices for conducting scientific experiments and evaluating scientific claims, and they appreciate the nuances of how the media, companies, and sometimes, even researchers bias the presentation of data and results to support their desired outcomes. Based on the conversations we have with students after class, as well as comments reported in student evaluations of teaching, they leave the lesson feeling empowered by this newly acquired knowledge. Their in-class worksheets also indicate that they can identify a number of the misleading claims on the Prevagen® website, and they can explain why those claims are misleading. The recommendations they make about whether their grandmothers should take Prevagen® are thoughtful and well reasoned. Most students recommend that Prevagen® should not be taken, but some, interestingly, decide to recommend its use because of the benefit of the placebo effect.

Limitations/Adaptations

This lesson sets the stage for later discussions and examples throughout a course or even a curriculum. In fact, universities are offering entire courses about pseudoscience (2, 3, 10). The advantage of our lesson is that it can be completed in one class meeting, and thus could be incorporated into more courses. Instructors who want to spend slightly more time on the subject could challenge students to find other examples of pseudoscience in the media and then present their evaluation of those claims to the class. Of course, this additional work on the lesson would make it last longer than a single class period.

Another way to expand the lesson would be to brainstorm about scientific research benefits to society. As presented, the lesson only focuses on the way society is vulnerable to manipulation by those interested in profiting from research results. The legal aspects surrounding pseudoscience, and Prevagen® in particular, are just beginning to emerge. Since we first taught this lesson, there have been more legal cases filed against the makers of Prevagen® and other companies selling “cognitive enhancer” supplements (e.g., 5) and “brain games” (e.g., 6). In regards to this particular case, in early 2017, the Federal Trade Commission (FTC) and New York State Attorney General charged the company marketing Prevagen® with making false and unsubstantiated marketing claims (7). In other words, the claims that the product improves memory, provides cognitive benefits, and that these benefits have been “clinically” shown to work are false. The FTC seeks refunds for consumers who had purchased Prevagen®. Therefore, other instructors may want to develop a lesson more directly devoted to bioethics and to exploring potential legal issues surrounding pseudoscience.

SUPPORTING MATERIALS

• S1. Bad Science - Pre-class work: Recognizing the Quality of Information Presented Scientifically
• S2. Bad Science - In Class Worksheet: Recognizing the Quality of Information Presented Scientifically

ACKNOWLEDGMENTS

We thank Drs. Elizabeth X. Kwan, Atom Lesiak, Miriam Meister, Elizabeth A. Morton, the students who participated in this lesson, the Science Teaching Experience for Postdocs (STEP) program at the University of Washington, and anonymous reviewers.

References

1. Gormally C, Brickman P, Lutz M. 2012. Developing a Test of Scientific Literacy Skills (TOSLS): Measuring undergraduates' evaluation of scientific information and arguments. CBE--Life Sciences Education 11:364-377.
2. Lilienfeld SO, Lohr JM, Morier D. 2001. The teaching of courses in the science and pseudoscience of psychology: Useful resources. Teaching of Psychology 28:182-191.
3. Rowe MP, Gillespie BM, Harris KR, Koether SD, Shannon L-JY, Rose LA. 2015. Redesigning a general education science course to promote critical thinking. CBE--Life Sciences Education 14:ar30.
4. Adam A, Manson TM. 2014. Using a pseudoscience activity to teach critical thinking. Teaching of Psychology 41:130-134.
5. Federal Trade Communication. 2015. Supplement marketers will relinquish $1.4 million to settle FTC deceptive advertising charges: Ads claimed Procera AVH would restore 10 to 15 years of memory loss. https://www.ftc.gov/news-events/press-releases/2015/07/supplement-marketers-will-relinquish-14-million-settle-ftc. Accessed July 11, 2018.
6. Federal Trade Communication. 2016. Lumosity to pay $2 million to settle FTC deceptive advertising charges for its "Brain Training" program: Company claimed program would sharpen performance in everyday life and protect against cognitive decline. https://www.ftc.gov/news-events/press-releases/2016/01/lumosity-pay-2-million-settle-ftc-deceptive-advertising-charges. Accessed July 11, 2018.
7. Federal Trade Communication. 2017. FTC, New York State charge the marketers of Prevagen with making deceptive memory, cognitive improvement claims. https://www.ftc.gov/news-events/press-releases/2017/01/ftc-new-york-state-charge-marketers-prevagen-making-deceptive. Accessed July 11, 2018.
8. AAAS. 2011. Vision and Change in Undergraduate Biology Education: a Call to Action. AAAS, Washington, DC.
9. Price RM, Perez KE. 2018. Many paths towards discovery: a module for teaching how science works. Journal of College Science Teaching 47:78-87.
10. Duncan D. n.d. Teaching the nature of science using pseudoscience: A semester-long curriculum to accompany any introductory science course. http://casa.colorado.edu/~dduncan/pseudoscience/. Accessed July 11, 2018.
11. Quinn VS. 2015. Using astrology to confront & discuss pseudoscience in the classroom. The American Biology Teacher 77:542-548.
12. Viswanathan M, Treiman KA, Kish Doto J, Middleton JS, Coker-Schwimmer EJ, Nicholson WS. 2017. Folic acid supplementation: An evidence review for the U.S. Preventive Services Task Force. Agency for Healthcare Research and Quality, Rockville, MD.
13. Willingham E. November 8, 2012. 10 questions to distinguish real from fake science. https://www.forbes.com/sites/emilywillingham/2012/11/08/10-questions-to-distinguish-real-from-fake-science/#6de88e0533bf. Accessed July 11, 2018.
14. McCabe DP, Castel AD. 2008. Seeing is believing: The effect of brain images on judgments of scientific reasoning. Cognition 107:343-352.
15. Lerner KC. 2016. Madison Memory Study: A Randomized, Double-Blinded, Placebo-Controlled Trial of Apoaequorin in Community-Dwelling, Older Adults. Quincy Bioscience L, Madison, WI.
16. Moran DL, Tetteh AO, Goodman RE, Underwood MY. 2014. Safety assessment of the calcium-binding protein, apoaequorin, expressed by Escherichia coli. Regulatory Toxicology and Pharmacology 69:243-249.
17. Speth R. 2016. PubMed Commons: Comment about Moran et al 2014 Adv Mind Body Med ftp://ftp.ncbi.nlm.nih.gov/pubmed/pubmedcommons/commons_archive.csv. Accessed July 11, 2018.
18. Moran DL, Marone PA, Bauter MR, Soni MG. 2013. Safety assessment of Apoaequorin, a protein preparation: Subchronic toxicity study in rats. Food and Chemical Toxicology 57:1-10.
19. Guterman L. 2002. Scientists accuse toxicology journal of industry ties, urge disclosure of conflicts of interest, p In The Chronicle of Higher Education. Washington, D.C. http://www.chronicle.com/article/Scientists-Accuse-Toxicology/115196/. Accessed July 11, 2018.
20. Clarke J. 1994. Pieces of the puzzle: The jigsaw method, p 34-50. In SHaran S (ed), Handbook of Cooperative Learning Methods. Greenwood Press, Westport, CT.
21. Theobald EJ, Eddy SL, Grunspan DZ, Wiggins BL, Crowe AJ. 2017. Student perception of group dynamics predicts individual performance: Comfort and equity matter. PLoS One 12:e0181336.
22. Reardon S. 2011. Climate change sparks battles in classroom. Science 333:688-689.
23. Wilson KJ, Brickman P, Brame CJ. 2018. Evidence-based teaching guides: Group Work. CBE--Life Sciences Education 17:fe1.
24. Compound Interest. 2014. A rough guide to spotting bad science. http://www.compoundchem.com/2014/04/02/a-rough-guide-to-spotting-bad-science/. Accessed July 11, 2018.
25. Brownell SE, Kloser MJ. 2015. Toward a conceptual framework for measuring the effectiveness of course-based undergraduate research experiences in undergraduate biology. Studies in Higher Education 40:525-544.