Assessing Learning in online Introductory Statistics fall 2020

MS 150 Statistics is an introductory statistics course with a focus on statistical operations and methods. The course is guided by the 2007 Guidelines for Assessment and Instruction in Statistics Education (GAISE), the spring 2016 draft GAISE update, and the ongoing effort at the college to incorporate authentic assessment in courses. A history of the evolution open data open data exploration exercises and associated presentations as authentic assessment in the course was covered in a May 2017 report.

The course went online in the summer 2020 with a slightly modified curriculum to fit both the summer calendar and the online nature of the course. Learning gained during the summer run was used to inform the fall 2020 term. 

A list of 34 learning support statistics videos developed for the summer 2020 session was expanded to 64 videos in support of the fall 2020 run of the course. The additional videos were developed to address specific gaps identified summer 2020 along with new needs that arose fall 2020. The course is deployed to run fully asynchronously, allowing the students to work on material at a time that best fits their own schedule. The online course has attracted working students who could not previously obtain time off to attend regular class. The course went online due to the pandemic and as a result of being online met an unmet need.

Three course level student learning outcomes currently guide MS 150 Introduction to Statistics:

• Perform basic statistical calculations for a single variable up to and including graphical analysis, confidence intervals, hypothesis testing against an expected value, and testing two samples for a difference of means.
• Perform basic statistical calculations for paired correlated variables.
• Engage in data exploration and analysis using appropriate statistical techniques including numeric calculations, graphical approaches, and tests.

Final examination details and performance

Assessment of learning was based both on the tracking of performance against student learning outcomes and on an item analysis of the final examination. 

The course began the term in August with 50 students. During the course of the term seven students would either submit no work or stopped submitting work during the term and were withdrawn. Although attempts were made to contact these seven students, none responded nor indicated why they had stopped participating. At term end the course had 43 students in two sections. Section one had 20 students and Section two had 23 students. Of the 43 students, 29 submitted the final examination. The high rate of nonsubmission was reported on in an earlier blog. 

Of the 29 students who submitted a final examination by the deadline, 13 were in section one and 16 in section two. The final examination had 29 questions in the areas of basic statistics, linear regressions, confidence intervals, and hypothesis testing for a difference of means. The content of final examination was substantively the same as in prior recent terms.

The table depicts the percent success rate by item on the final examinations across eleven terms.

Final examination item analysis 

The pattern of student success on the final examination is similar to the pattern of student success seen in the past. The overall average of 81% was identical to the average for all terms. Student performance on calculating the upper and lower bounds for 95% confidence intervals was stronger than in many prior terms. 

Performance by section on the final and in the overall course average along with section size n

In residential course the time of day may have an impact on student attendance and performance. In an asynchronous online class there really is no such thing as a section. Students do not perceive that they belong to a particular section and appear to be relatively uninformed as to who else is in their section in a course such as introductory statistics with its emphasis on individual work.

Student learning outcome performance over multiple terms

The introduction of Schoology Institutional in January 2018 has made possible tracking of performance based on student learning outcomes. Data across nine terms is reported for the following three learning outcomes:

1.0 Perform basic statistical calculations for a single variable up to and including graphical analysis, confidence intervals, hypothesis testing against an expected value, and testing two samples for a difference of means.
2.0 Perform basic statistical calculations for paired correlated variables.
3.0 Engage in data exploration and analysis using appropriate statistical techniques including numeric calculations, graphical approaches, and tests.

Overall percentage performance on course level student learning outcomes over multiple terms

Summer 2020 and fall 2020 the course was asynchronously delivered online. As such comparison of these two terms to prior terms is statistically problematic. The core takeaway is that student learning was achieved at a level essentially on par with historic values. The uptick seen in student learning outcome three is not noteworthy: student learning outcome three was evaluated in the past by in class presentations where students presented to the class, this term the outcome was evaluated by submitted presentations. These are not equivalent, presenting in front of peers was far more challenging.

Long term course average and standard deviation

The course average returned towards the long term overall mean, a known behavior in statistical systems. If there is a signal to be observed, the signal is that the online asynchronously delivered course has delivered course averages on par with the multi-term residential course average. This suggests that the online course can deliver results on par with the residential course.

Long term trend in final examination average

The fall 2020 final examination was structurally identical to the fall 2019 final examination. Note that spring 2020 did not have final examination due to the pandemic. Comparison of the summer 2020 and fall 2020 final examination to past performance is confounded by these finals having been done online from home. The fall 2020 final was available for three days and had no set time limit. This was intentional. Power can go off unexpectedly and may remain off. A student might have to return the following day to finish a final examination. Under such circumstances a time limit is not possible. Final exams from fall 2019 and earlier were taken on campus and had a two hour time limit. The final examination went open book in 2007. 

Student mastery of the three course learning outcomes

Mastery was set at five performances of the outcome with a mark of 70% or higher on the performance for course learning outcomes one and two. Outcome three was calculated at two performances of 70% or higher due to there only being five opportunities to perform that outcome. 

Student mastery of course learning outcomes

The 43 students in two sections of MS 150 encountered course learning outcome one basic statistical calculations 31 times during the term. Course learning outcome two linear regressions were encountered ten times. Course learning outcome three open data exploration was encountered five times. 

Of the 43 students 95% mastered outcome one, 74% mastered outcome two, and 51% mastered outcome three. 

In general the second run of the statistics course as a purely online asynchronous course went well. At term end 38 of the remaining 43 students in the course achieved passing grades in the course. Learning outcome performance was on par with prior residential instruction. This data does exclude the seven students who were withdrawn for non-participation early in the term.

Plans going forward include addressing the challenge of improving mastery of course learning outcome three. Improving submission rates on assignments and the final examination near the end of the term will also be addressed.