SMc Curriculum is here to meet the needs of Oregon high school students. Call us at (503) 550-4298. x

Pilot Process

Description of Participants

In May and June of 2013, over 390 students completed a pilot assessment in the McMinnville and Pendleton School Districts in Oregon.  The following characteristics are true of each pilot student:

  • Oregon high school student (freshman, sophomore or junior)
  • Had a previous best OAKS score between 221 and 254
  • Enrolled in a high-school level math class

During the assessment, students were allowed to use OAKS-assessable manipulatives, technology and formula sheet.  Students were each given one attempt on the assessment.  There was no time limit on the assessment.

The data for 390 students was collected.  This data was examined and students were removed from the data set for any of the following reasons:

  • student effort (length of time it took the student to take the test < 10 minutes)
  • students who did not complete the test
  • students who had technical difficulties which they (self-reported) stated affected their results
  • teacher feedback on students who they felt did not take the pilot assessment seriously.

After these students were removed, there were 319 students left who were included in the data seen in this report.

 

Process for Setting Cut Score

Each student was given an identification number which connected the student to his or her best OAKS score and his or her pilot assessment score.  Linear regression calculations were done to determine the coefficient of determination. In statistics, the coefficient of determination, denoted R2, indicates how well data points fit a line or curve. R-square can take on any value between 0 and 1, with a value closer to 1 indicating that a greater proportion of variance is accounted for by the model.

Figure 8 shows an R-square value of 0.5252.  This means the correlation accounts for over 50% of the variation.  A linear regression equation of y = 0.0654x + 224.11 represents the relationship between x, the students pilot assessment score, and y, their previous best OAKS score.

 

Figure 8 - Linear Correlation of Pilot  Assessment and Oaks Scores

Figure 8 – Linear Correlation of Pilot Assessment and Oaks Scores

 

 

Using this linear regression model, a cut score can be determined by using the OAKS cut score of 236 for y in the equation, y = 0.0654x + 224.11.  A cut score of 181.8 (rounded to 182) is determined.  Using a cut score of 182 on this assessment, 53.9% of the students would have passed the pilot assessment while 66.8% of the students actually passed the OAKS assessment.  This differences shows that the assessment provides rigor at or above the OAKS assessment for scoring purposes.  See Appendix D for the raw data that was used.

Figure 9 – Local Assessment Cut Score

Category

OAKS Assessment
Cut Scores

SMc Curriculum Local Assessment
Cut Scores

Meets

236

182

 

Discriminative Efficiency

Each question item involved in the pilot was examined for discriminative efficiency.  This statistical analysis works under the assumption that students who have scored highly on the other parts of the test should also have scored highly on each individual question (or vice versa), so the score for an individual question and the score for the test as a whole should be well correlated. See Appendix E for a description of how discriminative efficiency is calculated.

The data of the pilot students was examined based on the current class students were enrolled in to better understand the discriminative efficiency values.  For example, if students were enrolled in Algebra and had not yet taken Geometry, one could assume that the discriminative efficiency scores were useful in the Algebra domain but not as useful in the Geometry domain.

Items that had little or no discriminative efficiency have been edited for better clarity or removed from the assessment.  This statistic will continue to be examined to continue to better the assessment.