Scientific writing instructors need systematically collected local data to guide curriculum decisions. Some data will come directly from writing-related training activities, but there are other data sources that instructors can tap into as well. This page describes data sources that we have used or evaluated. Not every data source will be available or appropriate to every program, but we hope this list encourages instructors to look beyond their own assignments for data.

We invite the community to suggest other data sources that they have used!
 

---

Indirect Data Sources

First Year Writing Program Survey

(Source: STEM Writing Project. Added 5/17/22)

First year college writing programs have student placement procedures that can provide valuable cohort data about incoming students' prior experiences and skills. For example, our institution conducts a directed self-placement survey of all incoming freshmen. It asks 15 reflective questions about students' past experiences with academic reading and writing and their perceptions of academic writing. These are two example questions:

1. Of the following the best description of the reading I used in my writing in high school is:
   • Mostly literature (poems, novels, short stories)
   • Mostly informational/expository texts (historical and scientific articles)
   • Mostly descriptive (non-argumentive) texts
   • Mostly argumentive texts
   • A variety of types (informational, literary, argumentive, multimedia)
      
2. In your last two years of high school, how often did your writing require you to integrate information from expository, historical, or scientific texts?

Our institutional writing program provided actionable insights into incoming students' scientific writing experiences and skills. Their findings were similar to what we suspected anecdotally, but lacked sufficient data to say with any certainty.From their anonymized and aggregated student responses we learned that:

1. Only 4% of our incoming freshmen had any experience reading or using  informational/ expository texts. Anecdotal follow-up data suggested that most of this (already limited) experience was with historical rather than scientific articles.
2. About 10% of students had any experience analyzing and integrating information from expository, historical, and scientific texts. Nearly all of their relevant experiences occurred in literature courses.
3. Most students agreed with the statement that academic writing should be “balanced” and conversational, which is very much at odds with the more formal, argumentive style of scientific literature.
4. Other relevant findings shared with us:
   • Writing program faculty were uncertain how well students are instructed in identifying and assessing outside sources on their own;
   • Students claimed they “lacked preparation” in high school, but it was unclear precisely what this means. It could mean limited writing practice overall, minimal feedback and/or recursive writing process opportunities, a lack of responsibility on the student’s part for improving writing, or a combination of all three challenges.
   • Students were unclear on what the assumptions of each genre are (purpose, audience, expectations, conventions).
   • Students needed more guidance to understand what writing related terms like “organization,” “analysis,” “evidence,” "summary data," etc. mean.
   • Students needed more experience with a recursive writing process: drafting, (peer & instructor) feedback, revision, editing.

 

Independently Collected Writing Samples

(Source: STEM Writing Project. Added 6/13/22)

There are many opportunities to collect short writing samples from students. These can be analyzed using a standard rubric or by computational language processing. Some examples:

• Program assessments. For example our biology program administers a pre/post assessment exam that includes two short open response questions. Students' de-identified answers provide two pairs of responses from ~50% of all majors.
• Pre-lab entries in electronic notebooks, or similar pre-lab writing assignments.
• Homework assignments. One we use is to give students a mock lab report with either no abstract or a poorly written one. Their assignment is to re/write the abstract. 
• Answers to open-response questions on pre-lab or post-lab quizzes.
• Responses to questions posted on class discussion boards.

 

---

Published Evaluation Instruments

Experimental Design Ability Test

(Source: STEM Writing Project. Added 5/27/22)

Students in our introductory biology sequence complete a pre/post assessment exam. One of the questions comes from the Experimental Design Ability Test (EDAT) (Sirum, 2011). The original test prompts asks students to design an experiment to test a claim. Responses are rated on a 10-point scale for presence or absence of specific experimental components. We have also used students' free responses to the EDAT prompts to evaluate their vocabulary range using type-token ratio (ttr).

For example, the pre-test EDAT prompt is below with responses from three different students. Both of the more complete answers (Students A and C) have smaller type-token ratios (ttr) than the incomplete answer (Student B). This ttr metric can be calculated without reading the individual responses. In pilot students we have also classified responses by comparing words used to pre-defined vocabularies of general, academic, and disciplinary terms.

 

Pre-Test Prompt:

The claim has been made that women may be able to achieve significant improvements in memory by taking iron supplements. To determine if the claim is fraudulent and prior to accepting this claim, what type of evidence would you like to see? Provide details of an investigative design.

 

Sample Response from Student A (78 types, 145 tokens, ttr =  0.5379):

    The study size. The length of the study. what type of plants were used in the study, and what time of year did the study take place.   I think an investigative design for an experiment like this would need a control group and an experimental group. the control group would not be treated with herbal supplement, and the experimental group would. Both groups would be place under stress ie harsh sunlight, cold temperature, too much water. the plants would be time to see how long they last under adverse conditions. Both the plants in the control group and the experimental group would need to have the same type of plant. I think the statistical test that I would use would Chi squared goodness of fit test because this experiment would only test for one plant endurance for one individual variable at one time.

 

Sample Response from Student B (40 types, 62 tokens, ttr =  0.6452):

    I would have a control group that ran a mile for 1 week, then I would split the group in half for the experimental group and have them take the herbal supplement. Then they would run for another week, and another, over a period of 5 weeks. I would measure their breathing patterns and see if their endurance got better as evidence.

 

Sample Response from Student C  (79 types, 155 tokens, ttr = 0.5097):

    Perform an experiment using mice to compare the effects of varying levels of ginseng on the endurance of the creature. One possible experiment would be how long a mouse can swim in 10.0 degree Celsius bucket of water.  A control group of mice would be administered a placebo pill with no ginseng and this group would be used to compare the effects of ginseng using multiple other experimental groups. These other groups could be categorized as 5 micro grams, 10 micro grams, 15 micro grams and 20 micro grams of ginseng. It is important to the test that multiple subjects are tested at each of the increments to ensure that the findings are accurate. If it is found that there is an adequate difference between the time that a mouse can swim in the bucket of water when given ginseng and when it is not then it can be deemed true that ginseng promotes endurance.

 

Resources	Links
Original article describing EDAT	Link to ERIC

 

---

 

Biology Self-Efficacy Scale (BSES)

(Source: STEM Writing Project. Added 6/14/22)

BSES is a set of 23 questions probing students' confidence in three dimensions of biological literacy: methods of biology; generalization to other biology/science courses and analyzing data; and application of biological concepts and skills (Baldwin, 1999). Several questions on BSES focus specifically on students' confidence in their ability to communicate in written form.

We include BSES questions about writing skills in our final lab course evaluation each semester.

• After one semester of writing training, aggregated student responses rarely align to their performance; students continue to over-estimate their writing skills.
• After a second semester of training and practice, aggregated responses start to align with observed performance. Students self-rate their skills below the highest scale point, and closer to the observed median.

Given these results, we have concluded that our students need at least 2 semesters of scientific writing training and experience before their internal view of their writing skills starts to align with our assessment.

 

Resources	Links
Original article describing BSES	Link to publisher

 

---

Where to Learn More

1. Baldwin, J. A., Ebert‐May, D., & Burns, D. J. 1999. The development of a college biology self‐efficacy instrument for nonmajors. Science Education, 83(4) 397-408. https://doi.org/10.1002/(SICI)1098-237X(199907)83:4%3C397::AID-SCE1%3E3.0.CO;2-%23

2. Shanks, R. A., Robertson, C. L., Haygood, C. S., Herdliksa, A. M., Herdliska, H. R., & Lloyd, S. A. 2017. Measuring and Advancing Experimental Design Ability in an Introductory Course without Altering Existing Lab Curriculum. Journal of Microbiology & Biology Education, 18(1): 18.1.2. https://doi.org/10.1128/jmbe.v18i1.1194

3. Sirum, K., Humburg, J. 2011. The Experimental Design Ability Test (EDAT). Bioscene: Journal of College Biology Teaching, 37(1): 8-16. https://eric.ed.gov/?id=EJ943887