Abstract

Reforms in undergraduate science education place an increasing emphasis on teaching the skills of practicing scientists, such as reading and interpreting the primary research literature. Many approaches have been proposed to introduce students to the primary research literature. One approach, the C.R.E.A.T.E. (Consider, Read, Elucidate hypotheses, Analyze and interpret data, Think of the next Experiment) method, leads students through a deconstruction of a research paper. However, the C.R.E.A.T.E. approach, as with many other approaches including more traditional journal club discussions, has primarily been implemented in small courses or discussion sections. Yet, at many larger colleges and universities, lecture courses do not have separate discussion sections. As a result, I modified the C.R.E.A.T.E. approach into a small group modified jigsaw activity. Students work in groups of four with each student assigned a role that corresponds with parts of C.R.E.A.T.E. Students complete a draft of their role's work prior to class. During class, students share their role with other members of their group and receive feedback. Then, group members work together to develop a final concept map that links together the different components of C.R.E.A.T.E. Students who used this approach self-reported significant gains in their ability to read and interpret the primary literature. However, their beliefs about science and scientists did not change in the same way as has been reported for the original C.R.E.A.T.E. method. Students report different components of the modified C.R.E.A.T.E. approach to be useful in their understanding of research articles and course material.

Citation

INTRODUCTION

In the context of undergraduate biology education, reading and analyzing the primary literature is essential as students develop skills in the process of science and experimentation (1).  In addition, as the pace of scientific discovery continues to increase, textbooks will no longer be sufficient sources of information on a subject.  Therefore, having the ability to acquire knowledge through reading research articles will only become more important for students.  Given the importance of including the primary literature in undergraduate biology courses, a variety of approaches has been proposed (2-8).   Recently, the emphasis on teaching students to engage with the primary literature has focused on particular aspects of primary research articles, such as the analysis of figures (9-11), rather than more traditional discussion and journal club approaches.  These approaches have resulted in increases in students’ ability to interpret data (9) and critical thinking skills, even in subsequent courses (10).

Hoskins et al. (12) developed the C.R.E.A.T.E. (Consider, Read, Elucidate hypotheses, Analyze and interpret data, Think of the next Experiment) approach to reading the primary literature to lead students through a rigorous analysis of a series of scientific papers and to simulate a laboratory environment. This approach has been widely adopted across diverse types of institutions (13-14). Hoskins et al. (15) Table 1 and Figure 1 present, respectively, an overview of the approach and how it is implemented across a semester, as does Hoskins et al. (12) (also see Figure 1 below). In this approach, instructors give students a paper from the primary literature with the abstract, the discussion, and identifying information removed, such that they only have the introduction, methods, results, and any figures and tables. Students then begin by developing a concept map of the introduction of a paper (Consider). Then, they annotate figures and draw schematic representations of the experimental design or methods that lead to each figure (Read). In addition, students define any terms that they did not know previously and present data that are in tables as graphs. Next, students rewrite the figure titles in their own words and describe the hypotheses that are being addressed in each figure (Elucidate hypotheses). Students then "Analyze and interpret the data" in each figure and develop a concept map for the paper. Finally, students propose two future studies that follow on from the results of the current study (Think of the next Experiment). Based on this preliminary work, students participate in in-class discussions of the study design and results of the paper. This process is repeated for four papers from the same research group related to a particular research question. Between each session on a particular paper, students participate in simulated grant panel during which they critique the proposed future experiments.

The C.R.E.A.T.E. approach has been shown to result in significant gains for students in a variety of domains.  For example, students showed a significant increase in critical thinking skills (12), gains in their views of their ability to read and analyze the primary literature (12,15), a greater interest in science (12), and positive changes in their understanding of and beliefs about science (15). Similar effects were found in an introductory level course, in addition to gains in experimental design skills (16).  Furthermore, these gains are repeatable across diverse types of institutions (13-14).

While the C.R.E.A.T.E. approach and other methods for introducing students to the primary literature are effective in leading to positive gains for students, courses with lower course enrollments have been the primary adopters.  The one exception is the work of Sato et al. (10), which introduced primary literature in large lab sections.  As a result, my goal was to modify the C.R.E.A.T.E. approach so that it could be integrated into an ecology lecture course without a required lab or separate discussion section and with a course enrollments ranging from approximately 40-65 students. 

MODIFICATION OF THE C.R.E.A.T.E. APPROACH

In order to scale up the C.R.E.A.T.E. approach, groups of four students, rather than the class as a whole, do the majority of the analysis and discussion of primary research articles. I give students a paper from the primary literature with the abstract, the discussion, and identifying information redacted. Using a modified jigsaw approach (17), I assign each student in a group of four a role for a particular research article that corresponds roughly to the different components of the C.R.E.A.T.E. approach (Figure 1). Role 1 is to develop a concept map of the introduction section of a paper (Consider). Role 2 annotates the figures and tables in the paper and then draws schematic representations of the experimental design or methods that lead to each figure or table (Read). In many cases, a single schematic of the experimental design is sufficient, as multiple figures and tables result from the same experiment. Role 3 rewrites the figure or table titles in their own words and describes the hypotheses that are being tested in each figure or table (Elucidate hypotheses). The student in each group who has Role 3 also interprets each figure or table in the context of the hypotheses being tested (Analyze and interpret the data). Finally, Role 4 proposes two future studies that follow on from the results of the current study (Think of the next Experiment) by defining the specific research question, drawing a schematic of the experimental design, and providing predicted outcomes of each experiment.

Students complete the work of their individual roles prior to the 50-minute class period set aside for discussion and submit their drafts online through the course management system. This insures that students have completed their work prior to discussion. At the beginning of the discussion period, I provide a brief overview (approximately 5-10 minutes) of the research article and answer any general questions about the article. Then, students divide into their groups to complete the jigsaw. Unlike a traditional jigsaw approach, students with the same role do not meet together before meeting with members of their group that have different roles. Each student presents their role's work to the rest of the group as the expert on their particular section of the paper. Next, each student elicits peer feedback on the work that they have completed prior to class. Finally, the group works as a whole to develop a concept map that integrates all of the roles. (Example concept maps are available from the author on request.) During this group work time, I (and any teaching assistants) circulate among the groups to answer questions and prompt discussion. At the end of the class period, I might address any additional common questions or discussion points with the entire class. As the class period is not long enough for the initial discussion, small-group discussion, and development of the final concept map, each group submits the final concept map online through the course management system two days after the discussion period. This allows students to finalize their concept maps outside of class. One negative aspect of allowing students additional time to complete their final concept map outside of class is that some groups might not be on task for the entire class period because they know that they will have additional time to complete the assignment outside of class. Implementing this approach in a course with longer class sessions (e.g., 1.5 hours), could allow students to complete the assignment during class.

I grade the final concept map submitted by each group according to the general rubric in Table 1. Fifty percent of each student's grade (5 out of 10 points) is based on their individual contribution to the final concept map and 50% (5 out of 10 points) is a group grade. Part of a student's individual contribution grade is based on submitting their draft prior to class (1 point) and attending class on the day of the discussion (1 point). These incentives are necessary so that students are prepared for class and actually participate in the jigsaw discussion with their group in class. Otherwise, based on observations during my initial implementation of this approach, students merely collect their individual contributions and paste them together outside of class time and do not attend class. The remaining three points of a student's individual contribution grade is based on their role as included in the final concept map. The overall quality of the final concept map, using the criteria from the general rubric in Table 1, determines the group portion of a student's grade. Holding each member of the group accountable for the overall quality of the final concept map, rather than just their individual role, creates an incentive to understand all parts of the paper and provide constructive feedback.

Students remain in the same groups for four discussion periods and rotate through all four roles. After four discussion periods, I randomly assign students to a new group. If the course enrollment is not exactly divisible by four, I assign either three students or five students to some groups. For groups of three students, one role is skipped each week. For groups of five students, two students are assigned Role 4 (Think of the next Experiment).

Although the basic components of the C.R.E.A.T.E. approach for deconstructing research papers is the same in this modified approach, some major differences are apparent. In my implementation of this modified approach, I do not select a series of papers from a particular research group. In contrast, I select current papers related to the ecological content we are discussing in class. As a result, the topic of the paper changes with each discussion. So that students are exposed to recent research, I select new papers each semester. Another key difference between the approaches is that in-class discussion is predominantly within the small groups in my approach, rather than with the entire class for the original C.R.E.A.TE. approach. Finally, instead of peer review panels, other members of a student's group critique future experiments.

In my initial implementation of this approach, students read and discussed 12 papers throughout the semester. We held discussions on Fridays and final concept maps were due the following Monday. Based on mid-semester and end-of-semester feedback from students in the course, I reduced the number of discussions to eight papers per semester. In addition, we now hold discussions on Mondays and have final concept maps due on the following Wednesday. Allowing for the completion of the group work to occur during the week rather than over the weekend facilitates easier collaboration among the students within a group.

STUDENT SELF-ASSESSMENT

For three semesters, I assessed changes in student self-reported ability to read and interpret the primary literature and student self-reported attitudes and epistemological beliefs about science and scientists, using the same survey used by Hoskins et al. (15) (see Table 2). In addition, I implemented a mid-semester survey to collect students' perspectives on this approach to discussing the primary literature and ways in which I could improve the process. In particular, I asked which role was most helpful in understanding the paper, which role was most helpful in understanding course material, which role was most difficult, and any suggestions on improving the approach.

Overall, students showed significant increases in their self-reported ability to read and interpret the primary literature for all constructs except "Research in Context" ("I understand why experiments have controls.") (Table 3). However, for "Research in Context," the mean pre-semester score was 4.7 on a 5-point scale, suggesting a possible ceiling effect. Gains for students in my course were similar to those for students using the actual C.R.E.A.T.E. approach (15) (Table 3). In contrast, students in my course showed little gain in self-reported attitudes and epistemological beliefs about science and scientists across all semesters (Table 4). However, the lack of gains for students in my course could be due to high scores at the beginning of the semester. High pre-semester scores might be due to the fact that the majority of students in the course are juniors and seniors, and 87% of the students had taken a previous class that included reading of primary literature. In addition, with the original C.R.E.A.T.E. approach, the series of papers discussed during a semester are all from the same research group. In contrast, in my modification of the approach, we read papers on different ecological topics from different research groups for each discussion. Perhaps, considering how a particular research group refines their understanding over time through a series of research studies is essential in influencing students' attitudes and beliefs about science and scientists. Of particular note, students in my course showed a significant decrease in "Certain Knowledge" and no significant change in "Sense of Motives" from the beginning of the semester to the end of the semester, whereas students who took courses with the actual C.R.E.A.T.E. approach exhibited significant increases in both factors (Table 4). Focusing on the research of a particular research group might allow students to develop a better understanding of scientists' motivations. Furthermore, the progression of research articles in the original C.R.E.A.T.E. approach might help reinforce the idea that knowledge is not certain, as scientists refine their ideas through subsequent studies. A more detailed comparison of student self-assessment of students who took my course with the modified C.R.E.A.T.E. method and students who took courses with the actual C.R.E.A.T.E. approach (reported in Hoskins et al. (15)) are included in the Supporting File 1 (Supporting File S1: Modified C.R.E.A.T.E. Approach - Detailed analysis and interpretation of student self-assessment data).

Based on the mid-semester survey, student perceptions of the different roles varied considerably from semester to semester (Figure 2). However, students generally found Role 1, in which they had to develop a concept map of the introduction of the paper, to be most helpful in understanding the course material (Figure 2A). In Fall 2015, students also reported that this role was most helpful in understanding the paper that we were discussing (Figure 2B). In contrast, in the other two semesters, students found Role 3, in which they analyzed the figures and tables and drew conclusions, to be most helpful in understanding the paper (Figure 2B). In two of three semesters, Role 4, which involves designing new experiments, was considered the most difficult (Figure 2C), even though students self-reports for the item "I know how to design a good experiment," which is a part of the Active Reading factor (Table 1), increased from the beginning of the semester to the end of the semester (Wilcoxon Signed Rank Test, W=2335, P<0.001). The student-perceived differences in the difficulty and helpfulness of the different roles reinforces the importance of students rotating roles within their groups. In addition, what students gain from completing each role is different. For example, in Role 1, students are learning how to integrate basic ecological principles with specific experimental tests of those principles. In contrast, in Roles 2 and 4, students are learning about experimental design, either through dissecting the experimental design in the paper being discussed or developing their own experimental design. Role 4 also develops students' skills in formulating testable research questions. Finally, in Role 3, students are learning how to interpret data from tables and graphs.

IMPLICATIONS AND RECOMMENDATIONS

Together, the results from student self-reports suggest that instructors can effectively integrate a modified version of the C.R.E.A.T.E. method into a lecture course without a separate discussion section and promote students' skills in reading and interpreting the primary literature. In addition, students found different components of the approach to be important in understanding the article being discussed and understanding course material. As a result, focusing on one component or role (e.g., deconstructing figures, (9)) might not be sufficient to result in student gains. Given that the majority of the discussion occurs in small groups, this approach is also likely scalable to larger lecture courses in which separate discussion sections might not be feasible. However, in large courses, teaching assistants will play an essential role in answering questions from the groups during the small group discussion. Therefore, the instructor should meet with the teaching assistants prior to class to discuss the paper and highlight parts of the paper that might be difficult for students (e.g., unfamiliar methods, new graphical methods). Finally, the teaching assistants will need to participate in grading the final concepts maps. As a result, development and use of a detailed rubric for grading is essential.

SCIENTIFIC TEACHING THEMES

ACTIVE LEARNING

The different roles require students to participate actively in a variety of scientific practices, including reading the primary literature, interpreting graphs and tables, and developing new hypotheses and experimental designs. In class, the modified jigsaw approach requires students to work in small groups within which they present their work, review and critique the work of others, and collaboratively synthesize their work in the form of a concept map.

ASSESSMENT

I assessed the individual components of the final concept map based on the requirements for each role (Table 1). I assessed students based on their individual contribution and the overall work of the group. Peer assessment occurred through peer feedback during the small work. Students self-assessed their abilities across the semester through voluntary surveys.

INCLUSIVE TEACHING

Since the majority of the discussion occurred in the smaller groups, rather than with the entire large, lecture class, students were more willing to participate.

SUPPORTING MATERIALS

• Supporting File S1. Modified C.R.E.A.T.E. Approach - Student Self-Assessment Data. Includes detailed analysis and interpretation of student self-assessment data.

ACKNOWLEDGMENTS

Thanks to the students who participated in this study, to Dr. Brian Sato who provided feedback on an earlier version of the manuscript, and to Dr. Lisa McDonnell for helpful editorial suggestions.

References

1. Pelaez, N., T. Anderson, S. Gardner, Y. Yin, J. Abraham, E. Bartlett, C. Gormally, J. Hill, M. Hoover, C. Hurney, T. Long, D. Newman, K. Sirum, M. Stevens. 2017. The basic competencies of biological experimentation: Concept-skill statements. PIBERG Instructional Innovation Materials. Paper 4. HYPERLINK "http://docs.lib.purdue.edu/pibergiim/4" http://docs.lib.purdue.edu/pibergiim/4
2. Janick-Buckner, D. 1997. Getting undergraduates to critically read and discuss primary literature. J Coll Sci Teach 27:29-32. https://pdfs.semanticscholar.org/51cc/40d9b5a81556aaace430ba2b2c7bb981e8c5.pdf
3. Herman, C. 1999. Reading the literature in the jargon-intensive field of molecular genetics. J Coll Sci Teach 28:252-254.
4. Smith, G. R. 2001. Guided literature explorations: Introducing students to the primary literature. J Coll Sci Teach 30:465-469.
5. Mulnix, A. B. 2003. Investigations of protein structure and function using the scientific literature: An assignment for an undergraduate cell physiology course. Cell Biol Educ 2:248-255. https://doi.org/10.1187/cbe.03-06-0025
6. Kozeracki, C. A., M. F. Carey, J. Colicelli, M. Levis-Fitzgerald. 2006. An intensive primary-literature-based teaching program directly benefits undergraduate science majors and facilitates their transition to doctoral programs. CBE Life Sci Educ 5:340-347. https://doi.org/10.1187/cbe.06-02-0144
7. Van Lacum, E. B., M. A. Ossevoort, M. J. Goedhart. 2014. A teaching strategy with a focus on argumentation to improve undergraduate students' ability to read research articles. CBE Life Sci Educ 13:253-264. https://doi.org/10.1187/cbe.13-06-0110
8. Jebanesan, D., A. Ashok. 2016. "Reading groups" in an undergraduate biology course: A peer-based model to help students develop skills to evaluate primary literature. CourseSource doi: 10.24918/cs.2016.2.
9. Round, J. E., A. M. Campbell. 2013. Figure facts: Encouraging undergraduates to take a data-centered approach to reading primary literature. CBE Life Sci Educ 12:39-46. DOI: 10.1187/cbe.11-07-0057
10. Sato, B. K., P. Kadandale, W. He, P. M. N. Murata, Y. Latif, M. Warschauer. 2014. Practice makes pretty good: Assessment of primary literature reading abilities across multiple large-enrollment biology laboratory courses. CBE Life Sci Educ 13:677-686. https://doi.org/10.1187/cbe.14-02-0025
11. Schinske, J. N., K. Clayman, A. Busch, K. Tanner. 2008. Teaching the anatomy of a scientific journal article. Sci Teach 75:49-56.
12. Hoskins, S. G., L. M. Stevens, R. H. Nehm. 2007. Selective use of the primary literature transforms the classroom into a virtual laboratory. Genetics 176:1381-1389. https://doi.org/10.1534/genetics.107.071183
13. Stevens, L. M., S. G. Hoskins. 2014. The C.R.E.A.T.E. strategy for intensive analysis of primary literature can be used effectively by newly trained faculty to produce multiple gains in diverse students. CBE Life Sci Educ 13:224-242. https://doi.org/10.1187/cbe.13-12-0239
14. Kenyon, K. L., M. E. Onorato, A. J. Gottesman, J. Hoque, S. G. Hoskins. 2016. Testing create at community colleges: An examination of faculty perspectives and diverse student gains. CBE Life Sci Educ 15:ar8. https://doi.org/10.1187/cbe.15-07-0146
15. Hoskins, S. G., D. Lopatto, L. M. Stevens. 2011. The C.R.E.A.T.E. Approach to primary literature shifts undergraduates' self-assessed ability to read and analyze journal articles, attitudes about science, and epistemological beliefs. CBE Life Sci Educ 10:368-378. https://www.lifescied.org/doi/10.1187/cbe.11-03-0027
16. Gottesman, A. J., S. G. Hoskins. 2013. Create cornerstone: Introduction to scientific thinking, a new course for stem-interested freshmen, demystifies scientific thinking through analysis of scientific literature. CBE-Life Sciences Education 12:59-72. https://doi.org/10.1187/cbe.12-11-0201
17. Clarke, J. 1994. Pieces of the puzzle: The jigsaw method. In Sharan S (ed), Handbook of cooperative learning methods. Greenwood Press, Westport, CT.