Assessing Learning in Introductory Statistics

MS 150 Statistics is an introductory statistics course with a focus on statistical operations and methods. The course is guided by the 2007 Guidelines for Assessment and Instruction in Statistics Education (GAISE), the spring 2016 draft GAISE update, and the ongoing effort at the college to incorporate authentic assessment in courses. A history of the evolution open data open data exploration exercises and associated presentations as authentic assessment in the course was covered in a May 2017 report.

Andrea and Berlene present their statistical analysis to the class

Three course level student learning outcomes currently guide MS 150 Introduction to Statistics:

• Perform basic statistical calculations for a single variable up to and including graphical analysis, confidence intervals, hypothesis testing against an expected value, and testing two samples for a difference of means.
• Perform basic statistical calculations for paired correlated variables.
• Engage in data exploration and analysis using appropriate statistical techniques including numeric calculations, graphical approaches, and tests.

The course wrapped up coverage of content three weeks prior to the end of the term. The students then engaged in a series of three data analysis, exploration, and presentation exercises. The course then ended with a final examination which was completed as an online test inside Schoology. This term the spring 2017 final examination was posted as a practice test.

Statistics spring 2017 final examination

The use of the spring 2017 final examination as a practice test was made possible in part due to the use of Google Sheets for data sharing back in spring 2017 and the continued use of Google Sheets in the course this term. Spring 2017 was the first term to use Google Sheets as the supporting software for the course from day one.

Data for the spring 0217 statistics final examination

Spring 2018 saw the adoption of Schoology Institutional and the further deepening of integration between the course and Google Sheets via the Google Drive Assignments application in Schoology.

The basic structure of the final examination has been fairly stable over time, and performance on an item-by-item basis is also fairly stable over time.  

Thirty-three students sat the final examination spring 2018. The table depicts the percent success rate on each item on the final examination.

topic	Fall 2015	Spring 2016	Fall 2016	Spring 2017	Fall 2017	Spring 2018
n	0.93	0.97		0.97	0.98	0.91
mode	0.94	1.00	0.96	0.89	0.98	0.85
median	0.96	0.97	0.94	0.95	1.00	1.00
mean	0.94	0.95	0.94	0.89	1.00	0.97
min	0.94	0.95	0.96	0.95	0.95	1.00
max	1.00	1.00	0.96	0.97	0.98	1.00
range	0.94	0.95		0.95	0.98	1.00
quartile 1	0.96	0.95	0.90	0.92	0.98	0.97
quartile 3	0.94	1.00	0.88	0.89	1.00	0.91
boxplot		0.49	0.84	0.92	0.79	0.67
histogram	0.59	0.41	0.88	0.97	0.90	0.79
stdev sx	0.93	0.97	0.94	0.89	0.93	1.00
st error se	0.72	0.80	0.76	0.82	0.81	0.94
t-critical	0.91	0.87	0.78	0.74	0.67	0.82
margin E	0.72	0.69	0.41	0.63	0.50	0.73
lower95	0.59	0.44	0.31	0.42	0.71	0.52
upper95	0.59	0.41	0.29	0.45	0.71	0.52
n paired	0.54	0.77	0.59	0.39	0.43	0.61
slope	0.93	0.95	0.71	0.82	0.83	0.91
intercept	0.93	0.74	0.78	0.84	0.83	0.88
correlation	0.91	0.69	0.71	0.66	0.93	0.91
strength	0.74	0.56	0.63	0.84	0.98	0.76
x predict y	0.44	0.33				0.39
y predict x	0.37

The first twelve questions covered basic one variable statistics. Questions 13 to 17 involved constructing a 95% confidence interval. Questions 18 to 23 covered statistics of two variable dependent variables. Students evidence strength in calculating basic statistics of both one and two variables. Performance of inferential calculations for both one and two variables has remained weaker. The existence of a practice test does not appear to have a marked impact versus prior terms and did not positively impact the calculation of the lower and upper bounds for a 95% confidence interval. Again this term students were hung up on the use of plus or minus two standard errors as an approximation of the 95% confidence interval. This is used to introduce t-critical values. Again the solution would seem to be to omit all mention of plus and minus two standard errors and move from material in chapter eight directly to chapter 9.2 omitting section 9.12. 

The MS 150 Statistics course spring 2018 consisted of two sections, a total of 40 students enrolled as of term end, 22 females and 18 males. The two sections are kept in curricular synchronization during the term. Both sections covered the same material, worked the same assignments, and gave presentations on the same topics. The sections met at 8:00 and 9:00 on a Monday-Wednesday-Friday schedule.

Performance by section on the final and in the course

The 8:00 section is impacted by transportation difficulties which have historically led to weaker performances by the 8:00 section versus the 9:00 section. Both the average on the final examination and course averages differed by section, consistent with the performance differential seen in prior terms. The students in the 8:00 section tend to have more absences and more frequently arrive late to class. There is also potentially a pre-selection factor. More organized students may register earlier and preferentially fill the later class - the 8:00 section is the last to fill during registration. 

Performance differences by gender in the course and on the final exam

Gender differences were not statistically significant given the small underlying sample size. The differences were sufficiently small as to be random.

The final examination in MS 150 Statistics has tracked performance on basic statistics and linear regression statistics since fall 2012. The course has also reported on performance on open data exploration exercises over the same period of time. This term performance on specific learning outcome 3.3 was used to report on performance on open data exploration exercises.

Final examination performance by course learning outcome

Final examination performance on the first course learning outcome, basic statistics, has been generally stable around 80% since fall 2012. Performance Spring 2017 rose to 84% and Fall 2017 performance improved again to 87%. This term performance rose to a 92% success rate. This rise from fall 2017 might be attributable to the practice test that was deployed for the first time this year.

While performance on the second student learning outcome, paired data calculations, had remained stable near 70%, last term saw a rise to 80%. The long term average remains 70% on this course learning outcome and the drop to a 74% success rate this term can be seen as a return to the long term mean.

There is far more inertia in a value than I suspect those in education comprehend. There is a penchant in education for "continuous improvement." These success rates tend to be stable over long periods of time and reflect both the difficulty of the material as well as the many reasons students do not succeed on the material. For every student who did not do well, there is a complex back story.

The third course learning outcome involving open data exploration cannot be compared on a term-on-term basis. The nature of the rubrics and scoring systems used to mark this section have changed over the terms. Spring 2018 the adoption of Schoology Institutional made possible direct measurement of this outcome based on work done during the term. Rubrics included as specific student learning outcome 3.3: "Report results of analysis. Draw conclusions based on statistical analyses and tests, obtain answers to questions about the data, supported by appropriate statistics." This student learning outcome was marked with evaluation of open data exploration and analysis skills in mind. Measured this way, performance appears stronger than in past terms. However, term-on-term comparisons cannot be made as the method of reporting learning for this outcome has varied from term to term.

Long term trend in final examination average Fall 2005 - Fall 2017:
y-axis does not start at zero nor end at 100! vertical range is exaggerated! 

Over the long haul, regression to the mean is as inescapable in statistics as entropy is in physics. Means return to long term means and those long term means return to even longer term means. And in the world of final examination averages, moving those longer term means is very difficult. Term-on-term fluctuations are almost meaningless and should not be viewed as calls to action. Since 2012 the final examination percent has moved in a narrow range of plus or minus four percent from 74%. Fall 2017 was slightly up at 79.3%, but this was projected last fall to be a random aberration. Just as there was a multi-term slump in the averages from Fall 2009 to Fall 2011, so can there be multi-term improvements that do not hold over the longer haul. Spring 2018 saw performance return to towards the mean as projected last fall.

Long term course average and standard deviation

The course average since 2007 has also remained stable and has tended to remain within four percent of 78%. This term's 78.7% course average is statistically indistinguishable from the long term mean of 78.1%.

The standard deviation of the students' individual course averages is also relatively stable around 15% with a slight rise to 18.5% seen spring 2018.

Overall, given a list of numbers and spreadsheet software, students show a strong mastery of basic statistics, good capabilities with linear regressions, and more moderate abilities with confidence intervals.