Abstract

Critically reading and evaluating claims made in the primary literature are vital skills for the future professional and personal lives of undergraduate students. However, the formal presentation of intricate content in primary research articles presents a challenge to inexperienced readers. During the fall 2020 semester, I introduced a Collaborative Annotation Project (CAP) into my online 400-level developmental neurobiology course to help students critically read eight research papers. During CAP, students used collaborative annotation software asynchronously to add clarifying comments, descriptions of and links to appropriate websites, and pose and answer questions on assigned papers. Student work was guided and assessed using a CAP grading rubric. Responses to anonymous surveys revealed students found CAP helpful for reading the primary literature and the rubric clarified expectations for the project. Here, I describe how I introduced, used, and assessed CAP in my online class, and I share the detailed CAP instructions and rubric.

Primary image: A moment of levity while annotating primary literature. Sample student annotations from the Collaborative Annotation Project. Student #1 compares immunofluorescence data to Christmas lights, an observation appreciated by student #2. Student names have been removed.

Citation

Primary Literature in the Undergraduate Classroom

The ability to critically read and evaluate claims made in the primary literature is a vital skill for my undergraduate students who plan to apply for medical and allied healthcare professional programs, graduate programs in biological research, and scientific jobs. A survey of 159 faculty members revealed strong agreement of the importance for life science undergraduates to be able to read and evaluate the primary literature (1). However, the formal academic language used and complexity of information in the primary literature presents a challenge for students (2). In addition, graduate students and faculty utilize diverse approaches to read primary literature (see the article "How to (seriously) read a science paper" for examples), and the strategies they use change over the course of their careers (3). Thus, there is no singular “right” approach to reading primary literature that works best for all readers, presenting a challenge for instructors when introducing the primary biological literature to their students.

The variety of approaches used to read primary research articles is echoed by the wide range of methods instructors describe to introduce the primary literature to undergraduate students. For example, Schinske et al. (4) and Round and Campbell (5) employ classroom activities to guide students through specific sections of selected papers. Hoskins et al. (6) designed a method called CREATE (Consider, Read, Elucidate hypotheses, Analyze and interpret the data, and Think of the next Experiment) to present a series of four related papers from the same research group to demonstrate to students how the state of knowledge on a topic changes over time. Anderson (7) and Levine (8) hold discussions of primary research articles with students in small groups while Rawlings (9) presents students the option to contribute to whole-class discussions either in person or in online forums. Gillen et al. (10) have created an online tutorial about the scientific process and the structure of primary research articles to help non-majors read research papers.

Collaborative Learning Promotes Important Skills for Students

The term collaborative learning describes a wide range of educational approaches that involve “groups of learners working together to solve a problem, complete a task, or create a product” (11). In a study of 2,050 second-year students at 23 institutions, Cabrera et al. (12) revealed collaborative learning was associated with gains in personal development, understanding of science and technology, appreciation for art, analytical skills, and openness to diversity. Donelan and Kear (13) showed the design of authentic tasks and skills development relevant to the workplace were key aspects of a successful collaborative learning project. This suggests collaborative learning is a good approach for teaching undergraduate students how to read and critically evaluate the primary literature, as these are vitally important skills for students’ future careers.

In fall 2020, I used two forms of collaborative learning in my online 400-level developmental neurobiology class. First, the entire class employed Hypothesis software to collaboratively annotate assigned primary research papers asynchronously. Second, in groups of three, students took turns presenting these research papers during synchronous class-time in a “Journal Club” style similar to that described by Glazer (14). Here, I will describe in detail the first collaborative learning activity that I called the Collaborative Annotation Project (CAP), including how I introduced, used, and assessed the project in my class.

Use of Hypothesis for Collaborative Annotation

Annotation involves the addition of information to a document while preserving the original work (see Figure 1 for example). A recent study by Kararo and McCartney (15) revealed undergraduate students with little experience reading the primary literature found annotations of selected papers useful for vocabulary and graph interpretation. Kararo and McCartney (15) used the Science in the Classroom website, that serves as a repository of papers annotated by volunteer graduate students, post-doctoral fellows and faculty, as a source of primary research articles. During collaborative annotation activities, students work as a team to contribute their own annotations to a work using annotation software. This allows students to pose questions, respond to the questions of others, and provide insight from their own experiences, making the exercise more personal and interactive than reading already-annotated papers. During CAP, my students used collaborative annotation software to add clarifying comments, descriptions of and links to appropriate websites, and ask and answer questions on assigned papers.

A number of available web-based tools can be used for collaborative annotation, including Perusall and Hypothesis (16). I chose Hypothesis for use in my class as it is used for peer-review and discussion of primary research articles by some scientific publishers (17) and allows users to comment on both webpages (Figure 1) and online PDF files. Working on the webpages of primary research papers allows students to easily download larger data images and supplementary materials, as well as access cited work through direct links. During the fall 2020 semester, Perusall only allowed users to comment upon PDF files uploaded to their servers.

Developmental Neurobiology Course Organization and the Collaborative Annotation Project (CAP)

Following an introduction to methods, animal model systems, and primary biological literature, we studied eight topics in our online developmental neurobiology course (Supporting File S1: I Really Enjoy These Annotations – Detailed schedule for developmental neurobiology). Each topic was examined using a sequence of four activities. First, a topic was introduced to the class with lectures over one or two synchronous class periods to provide a basic framework for the material. Second, students annotated a primary research article on the topic outside of class-time as part of CAP. Third, during synchronous class-time, students individually took a quiz covering content from the topic and article. Fourth, a group of three students presented the primary research article during synchronous class-time in Journal Club style (14). CAP annotations were due two or three days before group presentations to help with preparation of the talks and for students to study for the quizzes.

One synchronous class period early in the semester was dedicated to introducing CAP. Before attending this class, students were provided with instructions (Supporting File S2: I Really Enjoy These Annotations – CAP instructions), a grading rubric (Supporting File S3: I Really Enjoy These Annotations – CAP grading rubric), and were asked to annotate a paper by Sorrells et al. (18). At this point, no further instructions were provided to students. Notably, students had already read and discussed the Sorrells et al. (18) paper during the previous class period. In class, students completed a short ice-breaker activity and followed the prompts in S4 (Supporting File S4: I Really Enjoy These Annotations – Introductory CAP rubric in-class activity) to assess annotations of the Sorrells et al. (18) paper using the CAP grading rubric. This class provided an opportunity for students to consult with each other and myself regarding how to use the rubric to guide their annotations and how the rubric would be used to grade their work throughout the semester. Following class, students were encouraged to adjust their paper annotations before being graded using the CAP rubric.

CAP Rubric Criteria and Assessment of Student Work

Student annotations were assessed using a 15-point rubric with the following criteria: annotation quality, annotation quantity, distribution of annotations within the text, and constructively adding to the comments of others (Supporting File S3: I Really Enjoy These Annotations – CAP grading rubric). Examples of high-quality annotations include students adding to other students’ annotations (Figure 2), describing useful papers and websites that provide clarity to an aspect of a paper (Figures 3 and 4), posing questions that cannot easily be looked up or answering the questions of others (Figure 5), and connecting the content of a paper to other topics from our class, other classes taken, or independent research projects (Figure 6). More detailed examples of high quality annotations are found in the CAP instructions (Supporting File S2: I Really Enjoy These Annotations – CAP instructions) and grading rubric (Supporting File S3: I Really Enjoy These Annotations – CAP grading rubric). Collaborative annotations of eight primary research papers were worth 5% of students’ final developmental neurobiology course grades.

The CAP rubric simplified grading student work and providing constructive feedback. For example, on the first paper by Sorrell et al. (18), most students earned a perfect score of 15/15 on their annotations. However, two students did not make any collaborative annotations to earn scores of 12/15 and another made only seven annotations on the paper to earn a score of 14/15. In these cases, I was easily able to point out which criteria on the rubric were missed, and what should be done on future papers. These students did make collaborative comments and at least eight annotations on the remaining papers in our course.

Assessment of Course-Level Primary Literature Learning Objectives

For each course topic, students individually took a quiz covering content from the topic and corresponding primary research article. The purpose of this quiz was to ensure students had reviewed relevant course material and read the research paper prior to the group presentation and class discussion of the paper. Quizzes were a combination of 6-8 short answer and multiple choice questions, approximately 20 minutes in duration, and assessed the lower-level (19) course learning objective, “students should be able to describe key experiments in primary research papers.” For example, the following two questions about the paper by Sorrell et al. (18) were used on the first quiz:

1. Describe two methods Sorrell et al. (2018) used in today’s paper to examine the proliferation (division) of cells in the hippocampus.

2. Describe two differences Sorrell et al. (2018) found in the process of adult neurogenesis in human and macaque hippocampi.

The higher-level course objective (19), “students should be able to interpret and draw conclusions from experimental data in primary research papers,” was assessed during both the group presentation of primary research papers and a final course project. While both projects involved group presentations of data from the primary literature, students chose topics and papers to present for the final course project. Based on the feedback described below, students found CAP helpful to prepare for quizzes and class presentations and ultimately, meet course-level learning objectives related to the primary biological literature.

Initial Student Impressions and End-of-Course Feedback

Students completed an anonymous exit survey following the in-class CAP introduction (n = 18 survey respondents). This survey, available in S4 (Supporting File S4: I Really Enjoy These Annotations – Introductory CAP rubric in-class activity), revealed that students found in-class use of the CAP rubric helpful. For example, 100% of students responded that using the rubric clarified how to annotate primary research papers for the course, and use of the rubric was either “very easy” or “easy” to score their work. One student described the rubric as being, “simple to use to evaluate the quality of annotations. I liked how detailed the rubric was on what is expected to get a perfect score.” Students also appreciated the collaborative nature of CAP. For instance, another student remarked, “The collaborative paper annotations are really helpful! When other students are able to answer our questions, it’s cool to see how two or three heads working together is better than one.”

Students also completed an anonymous survey following course completion (n = 18 survey respondents) (Supporting File S5: I Really Enjoy These Annotations – CAP end-of-course survey questions). This survey revealed students viewed CAP positively. For example, 94% of students either strongly agreed or agreed with, and 6% felt neutral about the statement, “CAP was a helpful approach to analyze a primary research paper." Students also reported CAP was useful for preparing for other aspects of the course. For example, 94% of students agreed CAP was useful for preparing for their group paper presentation and 88% of students agreed reviewing collaborative annotations was useful when studying for quizzes. Students found the CAP grading rubric helpful, as 100% of students agreed with the statement, “The CAP rubric clarified how to annotate primary research papers for our course." In addition, 94% of students agreed with the statement, “Use of the CAP rubric was helpful to see how my work was evaluated." Many students remarked that they enjoyed learning from their peers during CAP. For instance, one student wrote, “It was immensely useful to see other's thought process in reading the papers and also getting insight from the diverse expertise in this class. For example, often students who have had experience with a specific wet-lab technique or field of research that we discuss in class will often chime in their perspective and help clarify topics.”

Discussion

In spring 2020, universities nationwide moved their classes online mid-semester in response to the COVID-19 pandemic (20). This sudden transition to emergency online learning led to social isolation (21), which had a detrimental effect upon the mental health of many students (22). Thus, when my institution announced the continuation of virtual coursework for the fall 2020 semester, I considered ways to address social isolation. In my introductory biology class, I used small group work where students worked on activities (see 23, 24, and 25 for examples) in teams of 2-3 during synchronous class sessions (26). I introduced CAP in my developmental neurobiology class to, in part, address social isolation during online learning. Many students noted in an end-of-course survey that student-student interaction was a positive feature of CAP (Supporting File S5: I Really Enjoy These Annotations – CAP end-of-course survey questions). For example, one student described, “I found CAP to be helpful as it allowed me to see how others are interpreting the paper. I was also able to interact with my classmates through an asynchronous format. Some of the questions raised are really insightful and I gained a lot by reading others' comments.” Similarly, another student stated, “CAP was very helpful! I liked being able to teach my classmates about content I was confident about and also receive help in areas I struggled with understanding.” In describing the class as a whole, one student exclaimed “this class is more like a family rather than just an elective,” thus collaborative learning activities, including CAP, were successful in building community for some students.

The types of primary research papers that worked well for past group Journal Club presentations in developmental neurobiology were also effective for CAP. Controversial papers, and papers that challenge long-held dogma in textbooks, easily generate group discussion amongst students. Papers utilizing animal model systems that students are familiar with from classes and independent research projects, for example the fruit fly (Drosophila melanogaster), the zebrafish (Danio rerio), or the mouse (Mus musculus), are more assessible to students than papers that use other model systems such as the cat (Felis sylvestris catus) or chicken (Gallus gallus domesticus). Students have an easier time determining which aspects of a paper are most important when a limited number of experimental techniques are used and only 5-7 main figures are presented in a paper. In past semesters, papers with more than seven figures and/or highly complex genetics or computational modeling have generated confusion and shut down class discussion.

One unanticipated benefit of CAP upon the course was the impact on group Journal Club presentations of primary research papers. Oftentimes, presenters would credit and thank classmates for insightful comments made or useful papers shared in their annotations. Some groups also made an effort to answer questions that arose in the paper annotations. These moments were appreciated by all and also contributed to a greater sense of community within the course.

The frequency of contributions to CAP varied widely among students. Approximately half of the class began to annotate papers one week ahead of deadlines and would revisit the paper multiple times to reply to the comments of their peers. One third of the class would begin CAP 2-3 days before the deadline, and revisit the paper 1-2 times to add collaborative comments, while the remaining students would visit the paper once or twice on the day before the deadline to complete the CAP assignment. In a future semester, it would be interesting to dedicate class time throughout the semester to CAP, so that all students could perform annotations together synchronously. This might promote more student-student interaction, active learning, and inclusivity.

Given the fall 2020 offering of developmental neurobiology was moved to an online format in response to the COVID-19 pandemic, it remains unclear if enthusiasm for CAP would be maintained in face-to-face classes. Possibly, students who have been socially isolated for a period of time are more willing to participate in collaborative learning activities. However, given the positive feedback from students for CAP this semester, I will continue the annotation project in future online or in-person offerings of developmental neurobiology. Some students indicated a desire for CAP to be worth a greater percentage of the final course grade, given the amount of time they spent on the project. In future semesters, I will consider raising the value of CAP from 5% of the developmental neurobiology final course grade.

Scientific Teaching Themes

Active Learning

Collaborative annotation allows students to directly interact with the primary literature and their peers in an asynchronous manner. This is consistent with Chi (27) who suggests the most effective forms of active learning are those that engage students cognitively, including when students constructively interact with a partner.

Assessment

Student annotations are assessed with the same rubric used by students during their introduction to the Collaborative Annotation Project (Supporting File S3: I Really Enjoy These Annotations – CAP grading rubric). This rubric is available to students on our learning management system class site, and students are encouraged to refer to this rubric when annotating papers. In addition, assessment of comprehension of primary research papers takes place during regular in-class quizzes.

Inclusive Teaching

The Collaborative Annotation Project (CAP) contributes to an inclusive classroom in a number of ways. First, collaborative learning is associated with gains in openness to diversity (12). Second, providing primary research papers, detailed instructions (Supporting File S2: I Really Enjoy These Annotations – CAP instructions), a schedule with CAP deadlines (Supporting File S1: I Really Enjoy These Annotations – Detailed schedule for developmental neurobiology), and a CAP grading rubric (Supporting File S3: I Really Enjoy These Annotations – CAP grading rubric) on the first day of class all clarify expectations for the assignment (28). Third, Stevens and Levi (29) and Feldman (30) encourage rubric use to make grading practices explicit for both students and instructors. Together, these practices promote the participation of all students in class.

SUPPORTING MATERIALS

• S1. I Really Enjoy These Annotations – Detailed schedule for developmental neurobiology
• S2. I Really Enjoy These Annotations – CAP instructions
• S3. I Really Enjoy These Annotations – CAP grading rubric
• S4. I Really Enjoy These Annotations – Introductory CAP rubric in-class activity
• S5. I Really Enjoy These Annotations – CAP end-of-course survey questions

Acknowledgments

PC would like to thank Dr. Vanessa Auld and Dr. Jane Roskams who first encouraged PC to teach developmental neurobiology (Biology 458) at the University of British Columbia in Vancouver, Canada while he worked as a post-doctoral research fellow. PC would also like to thank “student #2” for inadvertently providing the title for this work. Finally, PC would like to thank Emory University for supporting this work with a Winship Award for Lecturer Track Faculty.

References

1. Coil D, Wenderoth MP, Cunningham M, Dirks C. 2010. Teaching the process of science: faculty perceptions and an effective methodology. CBE Life Sciences Education, 9(4):524-535. doi: 10.1187/cbe.10-01-0005.
2. Snow CE. 2010. Academic language and the challenge of reading for learning about science. Science, 328(5977):450-452. doi: 10.1126/science.1182597
3. Hubbard KE and Dunbar SD. 2017. Perceptions of scientific research literature and strategies for reading papers depend on academic career stage. PLoS One, 12(12):e0189753. doi: 10.1371/journal.pone.0189753.
4. Schinske JN, Clayman K, Busch A, Tanner K. 2008. Teaching the anatomy of a scientific journal article. The Science Teacher, 75(7):49-56.
5. Round JE and Campbell AM. 2017. Figure Facts: Encouraging undergraduates to take a data-centered approach to reading primary literature. CBE Life Sciences Education, 12(1) https://doi.org/10.1187/cbe.11-07-0057
6. Hoskins SG, Lopatto D, Stevens LM. 2017. The C.R.E.A.T.E. approach to primary literature shifts undergraduates’ self-assessed ability to read and analyze journal articles. CBE Life Sciences Education, 10(4) https://doi.org/10.1187/cbe.11-03-0027
7. Anderson KL. 2016. Active learning in the undergraduate classroom: A journal-club experience designed to accentuate course content. The American Biology Teacher, 78 (1): 67–69. https://doi.org/10.1525/abt.2016.78.1.67
8. Levine E. 2001. Reading your way to scientific literacy. Journal of College Science Teaching, 31(2)122-125.
9. Rawlings JS. 2019. Primary literature in the undergraduate immunology curriculum: Strategies, challenges, and opportunities. Frontiers in Immunology, 10:1857. https://doi.org/10.3389/fimmu.2019.01857
10. Gillen CM, Vaughan J, and Lye BR. 2004. An online tutorial for helping nonscience majors read primary research literature in biology. Advances in Physiology Education, 28(3):95-99. https://doi.org/10.1152/advan.00044.2003
11. Laal M and Ghodsi S. 2012. Benefits of collaborative learning. Social and Behavioral Sciences, 31:486–490.
12. Cabrera AF, Crissman JL, Bernal EM, Nora A, Terenzini PT, and Pascarella ET. 2002. Collaborative learning: Its impact on college students development and diversity. Journal of College Student Development, 43(1):20-34.
13. Donelan H. and Kear K. 2018. Creating and collaborating: Students’ and tutors’ perceptions of an online group project. The International Review of Research in Open and Distributed Learning, 19(2). https://www.learntechlib.org/p/183601/
14. Glazer FS. 2000. A successful vehicle to science literacy. Journal of College Science Teaching, 29(5):320-324.
15. Kararo M and McCartney M. 2019. Annotated primary scientific literature: A pedagogical tool for undergraduate courses. PLoS Biology, 17(1):e3000103. https://doi.org/10.1371/journal.pbio.3000103
16. Farber M. 2019 (July 22). Social annotation in the digital age. Edutopia. Available from: https://www.edutopia.org/article/social-annotation-digital-age (Retrieved January 28, 2021)
17. Shaikh-Lesko R. 2019. Web annotation tool Hypothesis hits a milestone. Nature, 569(7755):295 https://doi.org/10.1038/d41586-019-01427-9
18. Sorrells SF, Paredes MF, Cebrian-Silla A, Sandoval K, Qi D, Kelley KW, James D, Mayer S, Chang J, Auguste KI, Chang EF, Gutierrez AJ, Kriegstein AR, Mathern GW, Oldham MC, Huang EJ, Garcia-Verdugo JM, Yang Z, and Alvarez-Buylla A. 2018. Human hippocampal neurogenesis drops sharply in children to undetectable levels in adults. Nature, 555:377–381. https://doi.org/10.1038/nature25975
19. Crowe A, Dirks C, and Wenderoth MP. 2008. Biology in Bloom: Implementing Bloom’s taxonomy to enhance student learning in biology. CBE Life Sciences Education, 7(4)368-381. https://doi.org/10.1187/cbe.08-05-0024
20. Foresman B. 2020 (March 12). Here are the U.S. universities that have closed due to coronavirus. Edscoop. Available from: https://edscoop.com/universities-closed-due-coronavirus-2020/ (Retrieved January 28, 2021)
21. Blankstein M, Frederick JK, and Wolff-Eisenberg C. 2020. Student experiences during the pandemic pivot. Ithaka S+R, doi: 10.18665/sr.313461.
22. Hamza CA, Ewing L, Heath, NL, and Goldstein, AL. 2020. When social isolation is nothing new: A longitudinal study psychological distress during COVID-19 among university students with and without preexisting mental health concerns. Canadian Psychology/Psychologie canadienne. Advance online publication. doi: 10.1037/cap0000255.
23. Cafferty P. 2021. Electrical and chemical communication in the nervous system. HHMI BioInteractive A&P FMN (2019), (Version 2.0). QUBES Educational Resources, https://doi.org/10.25334/8GJ3-YD44
24. Cafferty P. 2021. DNA replication: How does the cell accurately copy the information storage molecule? POGIL Activity Clearinghouse, 2(2). http://pac.chem.pitt.edu/index.php/pac/article/view/158
25. Cafferty P. 2021. CRISPR-Cas9 genome editing: How are components of prokaryotic adaptive immunity used to edit genomes? POGIL Activity Clearinghouse, 2(2). http://pac.chem.pitt.edu/index.php/pac/article/view/165
26. Cafferty P. 2021. Adaptation and facilitation of small group activities in an online introductory biology class. CourseSource. https://doi.org/10.24918/cs.2021.9
27. Chi MT and Wylie R. 2014. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4):219-243. https://doi.org/10.1080/00461520.2014.965823.
28. Sathy V and Hogan KA. 2019. (July 22) How to make your teaching more inclusive. The Chronicle of Higher Education. Available from: https://www.chronicle.com/article/how-to-make-your-teaching-more-inclusive/ (Retrieved May 14, 2021)
29. Stevens DD and Levi AJ. 2005. Introduction to rubrics: An assessment tool to save grading time, convey effective feedback and promote student learning. Sterling, VA: Stylus Publishing.
30. Feldman J. 2020 (January 27). Improved grading makes classrooms more equitable. Inside Higher Ed. Available from: https://www.insidehighered.com/views/2020/01/27/advice-how-make-grading-more-equitable-opinion (Retrieved January 28, 2021)