Abstract

Learning through authentic scientific inquiries is essential for understanding the nature and process of science and for developing critical thinking and communication skills. It is logistically challenging, however, to offer experiences of authentic scientific inquiries in large-enrollment introductory science courses. We developed a web-based ecological inquiry project using archived BearCam photos to provide an authentic scientific inquiry experience for students in a large introductory ecology course. Students conduct web-based individual research projects outside of class over a five-week period with ongoing peer feedback through online group discussions as well as instructor-facilitated discussions in class. They conduct a background study of grizzly bear biology and behavior; observe BearCam photos and generate testable hypotheses; design sampling and collect and analyze data; interpret results and develop inquiry reports guided by a rubric; conduct Calibrated Peer Review; and revise their reports based on peer feedback and self-assessment. Formative and summative assessments are used to facilitate and assess student learning, using direct and indirect measures. A rubric for the inquiry report communicates expectations, facilitates peer feedback and self-assessment, and forms the basis for evaluating student products and learning. Students have consistently reported significant learning gains in interest in ecology, ability to formulate testable hypotheses, understanding of how ecologists conduct research, and ability to evaluate quality of scientific reports.

Citation

Society Learning Goals

Ecology

• Species-Habitat Interactions
• How do species interact with their habitat?
• Populations
• How do populations change over time?

Science Process Skills

• Process of Science
• Pose testable questions and hypotheses to address gaps in knowledge
• Plan, evaluate, and implement scientific investigations
• Interpret, evaluate, and draw conclusions from data
• Construct explanations and make evidence-based arguments about the natural world
• Address novel questions through authentic research experiences
• Modeling/ Developing and Using Models
• Build and evaluate models of biological systems
• Quantitative Reasoning/ Using Mathematics and Computational Thinking
• Apply the tools of graphing, statistics, and data science to analyze biological data
• Communication and Collaboration
• Share ideas, data, and findings with others clearly and accurately

Lesson Learning Goals

• develop a basic understanding and appreciation of the process of science.
• understand the process of peer review and appreciate its necessity to science and benefits to learning.

Lesson Learning Objectives

• conduct an authentic ecological inquiry including
  • generate a testable hypothesis based on observations,
  • design investigation with appropriate sampling selection and variables,
  • collect and analyze data following the design, and
  • interpret results and draw conclusions based on the evidence.
• write a research report with appropriate structure and style.
• evaluate the quality of inquiry reports using a rubric.
• conduct peer review to evaluate and provide feedback to others' work.
• revise the inquiry report based on peer feedback and self-assessment.

INTRODUCTION

Rationale and Origin of Lesson

The Vision and Change in Undergraduate Biology Education report (1) calls for introducing the scientific process to undergraduate students early in their college careers and specifies the ability to apply the process of science as a core competency. The "Engage to Excel" report by the President's Council of Advisors on Science and Technology (2) also calls for improvement of the first two years of undergraduate education in science, technology, engineering, and mathematics (STEM) through engaging students in authentic STEM experiences. These efforts can promote active learning that: enhances performance of all students (3); improves recruitment and retention of STEM majors (2); and helps non-STEM majors develop STEM skills and perspectives, which are critical for developing a STEM literate citizenry. Development of the STEM skills for non-majors can also directly benefit the STEM workforce since about 26% of those working in STEM fields received their bachelor's degrees in non-STEM fields (4,5).

There has been a long history of providing undergraduate research experiences largely through an apprentice model, typically for selected upper-level undergraduate students in biological sciences (6,7). In recent years, however, there have been substantial efforts in course-based undergraduate research experiences (CURES) to scale up the significant impact of authentic science experiences (8,9). The field of ecology has a strong emphasis on field-based observational and experimental approaches, which makes it challenging to implement authentic science experiences in ecology, especially in large-enrollment introductory courses at research institutions. In addition, many two-year and four-year institutions have difficulty implementing CURES, in general, because of limited resources and diverse student/faculty populations. Web-based data sets and virtual learning environments have the potential to provide authentic ecological research experiences in such settings and can also supplement field-based research experiences (10,11).

This web-based BearCam inquiry project enables students in large-enrollment introductory courses to engage in authentic ecological inquiries, including: conducting a background study; making observations and generating testable hypotheses; designing investigations to test the hypotheses; collecting and analyzing data; interpreting results and writing research reports; conducting peer review; and revising reports based on peer feedback and self-assessment. Students can conduct these web-based, individual inquiry projects outside of class, with ongoing peer feedback from online group discussions and support from instructor-facilitated, in-class discussions. A notable strength of the BearCam inquiry project is that it enables students to engage in authentic experiences of science process without requiring training or background in the topical area, which can have significant benefits for engaging diverse students in lower-division introductory courses.

This inquiry project uses a set of photos (stills) collected by Lawrence Griffing and an undergraduate team of researchers. From the campus of Texas A&M University, they remotely controlled a pair of video cameras at McNeil Falls in the McNeil River State Game Sanctuary to focus on particular individuals or behaviors relevant to their studies and took a large number of photos in variable intervals. They archived these photo files, each labeled with date and time information, as well as videos that were sent via satellite to the web. A compressed-image web feed was re-posted on a local server for public viewing and comment. The video cameras, power sources, and transmission were redesigned each year from 2001 to 2007 and installed by Daniel Zatz and his company, SeeMore Wildlife Inc. At the end of local project at Texas A&M University, the cameras were replaced in 2008 by the National Park Service and the National Geographic Society. This remote site was problematic for maintenance of the cameras, resulting in moving them to the Brooks River in Katmai National Park in 2009. There, they continue to provide public access to live web-feed cameras during the summer months and archived video (http://explore.org/live-cams/player/brown-bear-salmon-cam-brooks-falls).

The research using the archived data from McNeil by the Griffing lab focuses on (1) bear-bear distance as it relates to aggression, fishing success, and time of day; (2) mate choice at the falls; (3) behavioral changes in individual bears as they age over a 5 year period; and (4) fishing dynamics, feeding and kin recognition. Archived videos and stills from several of the years are available on the Griffing lab server (http://griffing.tamu.edu/Site/McNeil/index.htm).

The BearCam inquiry project also uses Calibrated Peer Review (CPR, cpr.molsci.ucla.edu/Home.aspx) to help students understand the process and value of peer review in scientific inquiries and enhance their skills in critically evaluating scientific writing. Calibrated Peer Review is a web-based instructional tool that enables submission and rubric-based peer evaluation of writing assignments in classes of any size. Through Calibrated Peer Review, each student submits her/his own work, reviews three "calibration" submissions of various quality to learn how to evaluate using a rubric, reviews three anonymous peer submissions, reviews her/his own submission (self-assessment), and views the results of peer reviews. Studies have shown that Calibrated Peer Review can improve students learning of the materials (12,13,14), their skills in writing (15,16), evaluating materials (12,14), critical reasoning (16), and their confidence in evaluating their own writing (17).

We developed this inquiry project through the activities of the NSF-funded Information Technology in Science (ITS) Center for Teaching and Learning at Texas A&M University (18). As a graduate student participant of the ITS Center, Cheryl Ann Peterson first explored a similar in-class inquiry activity for a middle school class in 2005. In 2006, we modified and expanded that project as a web-based inquiry for a large-enrollment undergraduate ecology course.

Intended Audience

The BearCam inquiry is intended for students in introductory ecology or biology courses for majors and/or non-majors. It is well suited for mixed major and non-major introductory courses because it enables students with little prior training in biology to engage in authentic ecological inquiries while affording opportunities for students with more knowledge and experiences to explore further. It has been successfully implemented as a web-based, outside-of-class project in an introductory ecology course at Texas A&M University every fall semester in 2006-2015. This course is lecture only, meets on MWF with a 50-minute class period, and has a total of 400-500 students (over 50% non-majors) in two sections. However, the BearCam inquiry is suitable and perhaps even more effective for smaller classes.

SCIENTIFIC TEACHING THEMES

Active Learning

Outside of class: Students design and carry out the web-based authentic inquiry, engage in online group discussions and provide feedback to each other's work, participate in Calibrated Peer Review, and revise the inquiry report based on peer feedback and self-assessment.

In class: Students answer clicker questions and participate in Think-Pair-Share discussions of key concepts and procedures related to the inquiry process and evaluate anonymous samples of student work using the rubric.

Assessment

We use a set of formative assessments to gauge student learning and to respond to their needs, including in-class clicker questions, selected online quiz questions, pre-project evaluation of an inquiry report using a rubric (Figure 1), and monitoring of online group discussion postings.

Summative assessments of student learning include evaluations of student inquiry reports using the rubric, student performance in pre- and post-project evaluation of a sample report using the rubric, and student performance in Calibrated Peer Review.

Students are encouraged to self-assess and reflect on their own understanding, inquiry process, and writing through online group discussions, the Calibrated Peer Review process, revision of their reports, and self-assessment questions in the post-project survey. They also conduct peer evaluation of individual group member's contributions to the group discussions.

Inclusive Teaching

The design of the inquiry project supports different modes of learning. There are in-class mini-lectures, instructor-led discussions, clicker questions, and think-pair-shares, which are coupled with out-of-class individual research and online collaborative learning through group discussions and Calibrated Peer Review. The online group discussions provide opportunities for individual students to express their ideas in their own style and pace, to hear and benefit from diverse perspectives of their peers, and engage in discussions to deepen their understanding and improve their work.

LESSON PLAN

The inquiry project is designed to be completed in five weeks (Table 1), within our context of a web-based inquiry project in our large-enrollment introductory ecology course of majors and non-majors. All student work is done outside of class (in our course Blackboard Learn site and in the Calibrated Peer Review system), using personal or open-access-lab computers.

Figure 2 shows the self-reported time spent on individual tasks of the inquiry project by students in our classes.

PRE-PROJECT PREPARATION AND ASSESSMENT

Preparations by instructor

1. Make the directions and supporting materials available in Blackboard Learn:

• Directions for BearCam inquiry project: These include detailed week-by-week descriptions of the tasks and associated prompts, the grading scheme for the inquiry project, Guidelines for the Inquiry Report, and the rubric for reviewing the inquiry report. An example of the directions for the inquiry project is provided in S1.
• Link to website with BearCam photos (stills) for inquiry projects: This website (http://bearcaminquiryproject.weebly.com/) provides a set of BearCam photos organized by hours of the day. The photos are displayed by thumbnails or by photo numbers (arbitrarily assigned to photos within the same hour of day, for identification purposes only), chosen via two pull-down menus. An aerial photo and an associated gridded map of the McNeil River Fall study area are also provided. Students can mark estimated locations of individual bears on the gridded map, in reference to the aerial photo, and estimate the distances between bears using the gridded map.
• Example data files: This resource provides a set of example data files for students (S2).

2. Set up discussion groups and assessments in Blackboard Learn:

• Inquiry project discussion groups: Set up groups of ~10 students in Blackboard Learn. Given the large class size, we randomly assign students to the groups. For each group, a discussion forum is set up, as well as a mail tool and a list of the group members.
• Evaluation of an inquiry report using a rubric: Set this activity up as a quiz, using the middle-quality report for calibration in Calibrated Peer Review and the rubric for the inquiry report (see S5).
• A quiz on the directions for the inquiry project: This quiz helps students become familiar with the tasks, guidelines, and schedule of the inquiry project. This quiz can be optional, but is very helpful for large classes to reduce the amount of missed work and email inquiries.
• Peer evaluations of contributions to group discussions: These evaluations are implemented as online quizzes (one for each group). Each quiz includes a prompt about the three criteria for discussion contributions and a set of questions, one for each student in the group (labeled with the student's name). The four answers for each question indicate whether the discussion contributions of the student meets 3, 2, 1, or 0 of the criteria, respectively. S3 is an example quiz for this peer evaluation.
• Inquiry project feedback survey: Set this survey up as a quiz. An example survey is provided in S4.

3. Set up Calibrated Peer Review for the class.

• Set up a Calibrated Peer Review course: Calibrated Peer Review system is centrally managed on our campus by the Instructional Technology Services who set up each course upon instructor request.
• Activate BearCam Project assignment: In the Calibrated Peer Review course, activate a new assignment for BearCam Project, search for "Report for inquiry project using BearCam" in the Calibrated Peer Review Central Assignment Library, and select the assignment.
• If needed, modify the Guidelines for the Inquiry Report (See S1), the calibration reports and answer keys, and/or the assignment scoring. The three reports ("essays" in Calibrated Peer Review terminology) used for calibrations consist of past student reports with minor modifications. The calibration reports and answer keys are provided in S5. Please refer to "Activating a CPR Assignment" under "Download" after instructor log-in and the CPR Assignment Tutorial on the Calibrated Peer Review web site (cpr.molsci.ucla.edu).
• Set the timing and grading scheme: Set the timing for the assignment (start and end time for text entry, calibrations and reviews) and the grading scheme you prefer - percent of the Calibrated Peer Review grade allocated to the quality of the report and performance in calibrations, peer reviews, and self-assessment, respectively.
• Student performance in the calibrations is evaluated based on comparisons of their grading of the calibration reports to the instructor's grading using the rubric. Their performance in reviewing the reports of peers' and their own is evaluated based on comparisons of their rating of each report to the weighted average of the ratings of the same report by the three reviewers. The score for the report (essay) is the weighted average of the ratings by three peer reviewers. These weights are based on individual reviewers' performance scores in calibrations. The Calibrated Peer Review system processes these assessments of the performance and scoring of the report (essay) automatically. It then generates an overall Calibrated Peer Review score (100 points maximum) based on the grading scheme.

4. Prepare presentations for mini-lecture and class activities. See the "Activities in-class" for each week below. An example presentation is provided in S6.

What students do outside of class

• Students use the rubric (Figure 1) to evaluate an inquiry report (middle-quality report for calibration in Calibrated Peer Review; see S5). This assignment serves as the pre-test for students' ability to evaluate inquiry reports using a rubric.

Week 1: Conduct background study

What students do outside of class

• Introduce oneself (name, class, major, hometown, and a special fact/experience) to the discussion group in Blackboard Learn.
• Conduct an online search for background information on the biology and behavior of grizzly bears. Share one interesting piece of information on bear biology or behavior, which has not been mentioned by other group members, with the discussion group in Blackboard Learn.
• Take an online quiz in Blackboard Learn on the directions for the inquiry project.

Activities in class

• S6 contains an example presentation file that can be used as is or modified for the Week 1 class period. During the first class period, discuss the rationale and the process for the authentic inquiry project. Have students reflect on their career goals and skills needed through Think-Pair-Share. Then use a clicker question asking students to select which thinking skills in Bloom's Taxonomy (19) are most important for achieving their career goals. Share with students the Core Competencies and Disciplinary Practice outlined in the Vision and Change report (1). Discuss the process of the inquiry project and the grading scheme (Table 2).

Week 2: Develop hypothesis and design field investigation

What students do outside of class

• Study the BearCam photos (at http://bearcaminquiryproject.weebly.com/) and search for interesting patterns in bear behavior and spatial distribution. Formulate a hypothesis in terms of specific predictions of the pattern and design the procedure for collecting relevant data, with sufficient sample size and appropriate sampling regime (as described in S1) for testing the hypothesis. Consider the prompts in the BearCam Inquiry Project Directions and Rubric.
• Share the hypothesis and design with the discussion group in Blackboard Learn and provide feedback to the hypothesis of at least two other group members.

Activities in class

• Discuss testable hypotheses, bias, random and stratified sampling, and sample size, through mini-lectures and evaluation of anonymous student work using the rubric with clickers and follow-up discussions (see S6).

Week 3: Collect data, analyze data, and develop inquiry report

What students do outside of class

• Collect data from each photo selected (identified by hour of day and number label for the photo) as part of the sample and record the data in an Excel file (refer to S2). If measuring distances, use the gridded map to record locations of bears and estimate distances between them. Conduct data analysis: generate figure(s) (e.g., bar charts and X-Y scatter plots) or table(s) to represent your results. Submit the data file in Blackboard Learn.
• Develop an inquiry report following the "Guidelines for the Inquiry Report" section of the BearCam Inquiry Project Directions (S1). Refer to the rubric, also in S1. Submit completed report as a PDF in Calibrated Peer Review (cpr2.tamu.edu).
• Make at least one posting to the discussion group in Blackboard Learn to discuss one's data collection, analysis, and interpretations. Respond to at least two postings of other group members by providing feedback related to the hypothesis, experimental design, and/or results and discussions.

Activities in class

• Discuss data collection and analysis, interpretation of graphs, and writing discussions through evaluation of anonymous student work using the rubric with clickers and follow-up discussions (see S6).

Week 4: Conduct Calibrated Peer Review

What students do outside of class

• Complete three calibrations, review three reports of peers, and conduct self-assessment of one's own report in the Calibrated Peer Review system using the rubric.

Activities in class

• Discuss the importance of giving constructive feedback in the peer review, using anonymous student work, and how to arrive at the overall rating in Calibrated Peer Review based on grading using the rubric (the number of items with positive (Yes) rating divided by 3).

Week 5: Revise inquiry report based on peer review feedback

What students do outside of class

• Revise inquiry report based on the peer review comments and self-assessment, conducting additional data collection and analysis if necessary; submit revised report in a Word or PDF file in Blackboard Learn.

Activities in class

• Discuss the peer review and revision process in scientific publishing and how to use review feedback to improve one's writing. An effective approach for this discussion is for the instructors to talk about personal experiences in publishing our work in peer-reviewed journal articles. In addition to the review process, describe what we are expected to do with the revisions and responses to review comments, how we may initially feel about and productive ways to respond to critical comments and comments that we disagree, and how the review comments, often especially the critical ones, help us improve our work.

Post-project assessment

What students do outside of class

• Complete peer evaluation of each group member's discussion participation in Blackboard Learn based on the following criteria: 1) postings were meaningful/helpful for improving hypothesis, sampling design, data collection, analysis, or writing of the inquiry report; 2) postings were made in a timely manner to provide feedback for improvement; and 3) postings were done in a respectful manner (S3).
• Complete the "Inquiry Project Feedback" survey in Blackboard Learn, which includes both self-reflections on the learning through the project and feedback to the project (S4).

Grading and processing by instructor/TA

• Use the rubric in S1 to assess the quality of the revised inquiry report and convert the total score based on the rubric (30 points maximum) to the portion of project grade assigned. For example, since we assigned 40% of the inquiry project grade to the report, we divide the score by 30 and multiply 0.4.
• Download the Calibrated Peer Review scores (100 points maximum) and convert it to the portion of project grade assigned.
• Process the results of peer evaluation quizzes: download the results, calculate the average of peer ratings for each student as the peer evaluation grade for the student, and convert the average rating to the portion of project grade assigned.
• Process the results of the "Inquiry Project Feedback" survey, assign a score based on completion (answering all survey questions), and convert it to the portion of project grade assigned.
• Extract student ratings of the mid-quality calibration report in the Calibrated Peer Review, which is used as a post-test. To assess student-learning gain in ability to evaluate inquiry reports using a rubric, compare these ratings to the student ratings in the "Test evaluation of inquiry report using rubric" completed on Blackboard Learn before the beginning of the BearCam inquiry project.

TEACHING DISCUSSION

We have used the BearCam inquiry project in our large introductory ecology course every fall semester since 2006. Overall student response has been very positive based on the project feedback survey. We asked students to reflect on their learning through the inquiry project as part of the survey. They have consistently reported significant improvement in their interests in ecology, ability to formulate testable hypotheses, understanding how ecologists conduct research, and ability to evaluate the quality of scientific reports from before to after the inquiry project.

The implementations of the BearCam inquiry project in this particular course have been focused on student learning of the nature and process of science. We have not emphasized learning of topical content such as animal behavior or potential implications for conservation practice and policy, although class discussions can involve these topics. These dimensions can be explored more in other course contexts. The McNeil River Fall Wildlife Sanctuary and Refuge web site of Alaska Department of Fish and Game, the citizen web site Friends of McNeil River, as well as the McNeil River Photo ID Books published by the Friends of McNeil River could be useful resources for such explorations.

Practices that worked well for our large classes

Making sure students understand and remember the tasks, expectations, and the deadlines for the inquiry project is critically important for such large classes. We display a slide with current tasks and deadlines before we start each class, include questions in weekly online quizzes about important tasks and timing of the inquiry project, and use clicker questions and in-class discussions to clarify the expectation as well as the rationale.

Throughout the five-week period for the inquiry project, we usually use the first 5 to 15 minutes at the beginning of each class (MWF) to discuss the inquiry project. We ask and answer student questions, have targeted exercises and discussion coupled with clicker questions, and give mini-lectures as needed. Whenever possible, we try to relate what students are doing in their BearCam Project to our own research experiences and examples, which we know well and in which we have special insights. These connections appear to engage students, as they respond positively with interest.

The rubric (Figure 1), which relates to the inquiry report as well as the inquiry process, guides the entire inquiry project. Frequently, we show a piece of anonymous student work for the section of the report they are writing, ask students to evaluate it based on the rubric using clickers, and have small group and class discussions on why the work is evaluated in a certain way and how the work can be improved. We also remind students often to base their peer reviews and revision of their own work on the rubric. Given the level of course and the students, the rubric we use focuses on the basic process of scientific inquiry and the basic elements of a scientific report. For advanced students, a rubric with more refined measures of quality may be appropriate.

Peer evaluation of group members' discussion participation (which accounts for 10% of the inquiry project grade; Table 2) is important to encourage meaningful collaborative learning in the groups. We discuss the criteria for the peer evaluation early on and remind students throughout the inquiry project. The peer evaluation is implemented as online quizzes (one for each group) in Blackboard Learn. Each quiz includes a prompt about the three criteria for discussion contributions and a set of questions, one for each student in the group (labeled with the student's name). The four answers for each question determine whether the discussion contributions of the student meets 3, 2, 1, or 0 of the criteria, respectively. The average of peer ratings for a student is used as the peer evaluation grade for the student. It requires a few hours to set up these online quizzes and process them afterwards, given the large class size (up to 50 groups).

Adapting BearCam inquiry project for different course contexts

For smaller classes with facilitated lab or in-class sessions or students with more prior training, the project could be implemented in shorter formats. For example, a format with two weekly lab periods (2-3 hours) can be used for a small class:

• In the first lab period, (1) have instructor-led discussions on the rationale, background and process of the BearCam inquiry project, (2) have structured activities for students to conduct background research on the biology and behavior of the grizzly bears, (3) have students observe the BearCam photos and generate testable hypotheses, and (4) have students design the investigation (sample and variable selections). It is important to build in structured group and class discussions on the hypothesis and design of investigation developed by individual students to enhance understanding and provide feedback and opportunities to revise.
• In the week between the first and second lab periods, have students carry out data collection and analysis, interpret the results, and develop individual inquiry reports based on the guidelines and the rubric for the inquiry report. Make available an online discussion forum where students can ask questions and give one another feedback.
• In the second lab period, if Calibrated Peer Review is available, have students (1) conduct Calibrated Peer Review in class and (2) participate in group activities to discuss what they have learned from the Calibrated Peer Review process, share their plans for revising the inquiry reports, and give each other feedback. If Calibrated Peer Review is not available, have students (1) evaluate calibration reports using the rubric (or do these as online quizzes before class) followed by class discussion, (2) evaluate the inquiry reports of members of each small group, and (3) discuss the review of each report and plan for revisions in small groups. Have students complete the revisions in or after class. Have a Think-Pair-Share activity to reflect on the most important or useful things learned through the inquiry project.

There could be many variations dependent upon the context of the course and the level and experiences of the students. It would be important to include formative feedback among peers, opportunities for revisions, Calibrated Peer Review process, and reflections on learning.

Calibrated Peer Review and alternatives

Calibrated Peer Review appears effective, and about 70% of the students reported the Calibrated Peer Review process made them think about their study and/or report differently. Overall themes from student evaluations include: (1) the calibration prepared students well and forced them to understand the rubric well and learn what to look for in peers' and their own inquiry reports; (2) giving and receiving feedback benefited student learning and helped them in realizing how to improve their reports; and (3) the process improved student confidence and broadened their perspectives.

In our early implementations of Calibrated Peer Review, some students complained about confusion in the process and expectations of Calibrated Peer Review and lack of confidence in the quality and fairness of the grading of their inquiry reports by the three peers. In addition to improving the way we inform and remind students of the processes and expectations of Calibrated Peer Review, we then modified the grading scheme so that the grades are based only on their performance in the reviews (of the three calibration reports, the three peer reports, and the self-evaluation) and not on the quality of their reports submitted to Calibrated Peer Review, a change which alleviated student concerns. The Inquiry Project Report score is determined by TA evaluation of the students' revised reports following Calibrated Peer Review. A small number of students may feel their Calibrated Peer Review grade, based on their performance in the reviews, is not accurate. We ask them to meet with instructor or TA to present their evaluations based on the rubric and we can re-evaluate and adjust their grades.

If a Calibrated Peer Review license is not available, a similar process can be implemented, especially for smaller classes, using online or clicker quizzes for calibrations, organizing peer reviews and follow-up discussions based on small groups, and using online quizzes for evaluation of peer review performance. Another online peer review system Peerceptiv(R) (http://www.peerceptiv.com/wordpress/) is also available. It does not have a calibration phase but instructors can add "training" review(s). It has a useful "back evaluation" feature that incorporates author feedback in evaluating the performance of the reviewers.

SUPPORTING MATERIALS

• S1. BearCam-Example BearCam inquiry project directions
• S2. BearCam-Example BearCam data files
• S3. BearCam-Example quiz for peer evaluation of contributions to group discussion
• S4. BearCam-Example inquiry project feedback survey
• S5. BearCam-Example Calibrated Peer Review calibration essays and answer keys
• S6. BearCam-Example BearCam presentation file

ACKNOWLEDGMENTS

Partial support for this work was provided by the National Science Foundation through the Centers for Teaching and Learning Program (ESI 0083336) and CCLI/TUSE Program (DUE-0942715). We thank the over 3,700 undergraduate students in our Fundamentals of Ecology course and our Teaching Assistants, Tymon Bloomer, Christopher Cheleuitte, Hannah Ertl, Erin Fierro, Jose Franco, Robert Powell, Carlos Santamaria, and Amy Williams, for their valuable feedback which has helped us improve the BearCam inquiry project. We thank Dr. Robin Wright, the Editor of this manuscript, two anonymous reviewers, and Hannah Ertl for their thoughtful review comments, which led to significant improvement of this work.

References

1. American Association for the Advancement of Science (AAAS). 2011. Vision and Change in Undergraduate Biology Education: A Call to Action. Washington, DC.
2. President's Council of Advisors on Science and Technology (PCAST). 2012. Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering and Mathematics. Washington, DC.
3. Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H, Wenderoth MP. 2014. Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences 111(23): 8410-8415.
4. U.S. Census Bureau. 2012. American Community Survey. https://www.census.gov/programs-surveys/acs/.
5. U.S. Census Bureau. 2014. Where do college graduates work? A Special Focus on Science, Technology, Engineering and Math. https://www.census.gov/dataviz/visualizations/stem/stem-html/.
6. Sadler TD, Burgin S, McKinney L, Ponjuan L 2010. Learning science through research apprenticeships: A critical review of the literature. Journal of Research in Science Teaching 47(3):235-256.
7. Wei, C. A. and T. Woodin. 2011. "Undergraduate Research Experiences in Biology: Alternatives to the Apprenticeship Model." CBE-Life Sciences Education 10(2):123-131.
8. Shaffer CD, Alvarez C, Bailey C, Barnard D, Bhalla S, Chandrasekaran C, Chandrasekaran V, Chung HM, Dorer DR, Du C, et al. 2010. The Genomics Education Partnership: successful integration of research into laboratory classes at a diverse group of undergraduate institutions. CBE-Life Science Education 9:55-69.
9. Auchincloss LC, Laursen SL, Branchaw JL, Eagan K, Graham M, Hanauer DI, Lawrie G, McLinn CM, Pelaez N, Rowland S, Towns M, Trautmann NM, Varma-Nelson P, Weston TJ, Dolan EL. 2014. Assessment of course-based undergraduate research experiences: A meeting report. CBE-Life Sciences Education 13:29-40.
10. Ellwein AL, Hartley LM, Donovan S, Billick I, 2014. Using Rich Context and Data Exploration to Improve Engagement with Climate Data and Data Literacy: Bringing a Field Station into the College Classroom. Journal of Geoscience Education 62(4):578-586.
11. Webb AM, Knight SL, Wu XB, Schielack JF. 2014. Teaching science with web-based inquiry projects: An exploratory investigation. International Journal of Virtual and Personal Learning Environments 5(2):57-68.
12. Margerum LD, Gulsrud M, Manlapez R, Rebong R, Love A. 2007. Application of Calibrated Peer Review (CPR) writing assignments to enhance experiments with an environmental chemistry focus. Journal of Chemical Education 84(2):292-295.
13. Pelaez N. 2002. Problem-based writing with peer review improves academic performance in physiology. Advances in Physiology Education 26: 174-184.
14. Russell AA. 2005. Calibrated Peer Review(TM): A writing and critical-thinking instructional tool. In Invention and impact: Building excellence in undergraduate science, technology, engineering and mathematics (STEM) education. American Association for the Advancement of Science, Washington DC.
15. Gunersel AB, Simpson NJ, Aufderheide KJ, and Wang L. 2008. Effectiveness of Calibrated Peer Review(TM) for improving writing and critical thinking skills in biology undergraduate students. Journal of the Scholarship of Teaching and Learning 8(2):25-37.
16. Heise EA, Palmer-Julson A, Su TM. 2002. Calibrated Peer Review writing assignments for introductory geology courses. Abstracts, Geological Society of America 34(6), A-345.
17. Likkel L. 2012. Calibrated Peer Review(TM) essays increase student confidence in assessing their own writing. Journal of College Science Teaching 41(3):42-47.
18. Schielack JF, Knight SL. 2012. An Information Technology-based Learning Ecology Model to Promote Science Education Leadership. Teachers College Press, New York, NY.
19. Anderson LW, Krathwohl DR (eds). 2001. A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Allyn and Bacon.