Student evaluations fall 2024
Student evaluations of instructor, course, and course materials provide guidance for the institutions on areas of relative strengths and areas where there may be room for improvement.
Leading strengths include awareness of the student learning outcomes, instructor preparation, and clarity of the syllabi.
Trailing weaknesses include the perpetually low score for textbook access, the always anomalous overall effectiveness of the instructor (this score is always significantly lower than the other metrics in this section of the student evaluations), and regularity of student-instructor contact.
Of 30,983 individual responses 27,780 were either a five or a four. That represents 90% of all responses. Of those 16,876 were a five. For a student to choose a four is less common and a choice below a four is a unusual. This suggests that even a four is, in some sense, a negative rating, especially wherein students are marking a mix of responses.
This report presumes familiarity with the student evaluations form in use at the institution. The responses were converted to numeric values:
- Strongly disagree
- Disagree
- Neutral
- Agree
- Strongly agree
Students dominantly mark strongly agree or agree which were assigned scores of five and four respectively. In the following, the numeric scores are usually referenced rather than the actual rating. Every prompt is a positive prompt where higher values are better.
The most common "straight" was for students who marked all fives - 69% of the students marked straight down the first column.
In the past this first column was "strongly disagree." But a check of comments showed that students were not reading the ratings - the students were strongly disagreeing with the prompt and then wrote glowing, positive comments about their wonderful teacher and excellent course. The large number of spurious "strongly disagree" values was eliminated summer 2024 by making the first rating "strongly agree." This led to an increase in the mean rating by term.
The complication with using the raw frequency counts is that the number of student evaluations from residential courses outnumbers the evaluations from online courses.
The other result of the preponderance of fours and fives is that the term means fall within a narrow range around 4.34, with the aforementioned uptick in summer 2024 due to the change in the order of the ratings. This term the overall average was 4.41.
A histogram of the means for the 1360 student submissions also shows the impact of "straights" with spikes at 3, 4, and 5. Note that the above chart is plotting the mean for each of the three sections of the evaluation separately, thus there are 4080 means in the above histogram.
Fall 2024 saw the largest number of student evaluation submissions.
Click to enlarge
When disaggregated by course delivery mode, the frequency distribution indicates that the residential courses saw more strongly agree and agree responses for the regular contact prompt.
Instructor evaluations
Click to enlarge
The overall average for the prompts in the instructor evaluation section was 4.428, just slightly above the overall average of 4.412. The first prompt has always underperformed the other prompts in this section. The reason for this pattern is not known.
By calculating a t-statistic, the small differences can still be used as a first approximation to areas of relative strength and relative weakness. Prompts with a t-statistic between -2 and 2 can be considered to be functionally equivalent to the mean. Areas of relative strength were:
- The instructor was always well prepared
- The instructor made sure that the students were aware of the Student Learning Outcomes (SLOs) for the course
- The instructor gave clear directions and explained activities or assignments that emphasized the course SLOs
The two areas of relative weakness were:
- The instructor initiated regular contact with the student through discussions, review sessions, feedback on assignments, and/or emails
- The instructor presented data and information fairly and objectively, distinguishing between personal beliefs and professionally accepted views
The means for the different course delivery modes for the regular contact prompt show a slightly lower mean for the online courses and a slightly higher mean for the residential courses - much as one might expect.
By controlling for the different number of evaluations, the relative frequency shows that the online and residential courses have very similar distributions in the number of ratings for each rating. This suggests that the weakness in regular contact is not being driven by any one particular mode of course delivery. This weakness in regular contact is emerging from all course delivery modes.
Recommendations for improvement in the regular contact metric might include:
Clearly communicating response time expectations, promptly addressing student inquiries within a reasonable timeframe (usually within 24 - 48 hours), providing timely feedback on assignments (usually prior to the next assignment's due date), maintaining consistent communication through announcements and updates, offering virtual office hours, and being accessible through multiple channels like email and discussion boards, while also clearly indicating potential delays during holidays or weekends.
Course evaluations
The section mean for the course evaluations was 4.429.
There were two areas of weakness in this section of the student evaluations.
Term-by-term z-scores
- The student learning outcomes helped me focus in this course.
- The testing and evaluation procedures were fair.
How students interpret the underperforming prompt, "The student learning outcomes helped me focus in this course." is not clear to this author. From the student's point of view their focus is on getting assignments done, successfully answering questions on a tests, and completing other work as assigned in a course. How the student learning outcomes would help a student focus on these specific tasks seems unclear.
Term-by-term z-scores
This metric has consistently strongly underperformed the other prompts in this section term after term, including this term, which may be an indication that the prompt is asking something that does not necessarily make sense from a student's point of view as to what they need to focus upon.
The other area of weakness, the fairness of testing and evaluation procedures, should almost be expected to rate lower than other prompts.
Course materials evaluations
The section mean for the course materials evaluations was the lowest of the three evaluation sections at 4.371.
The appropriateness of the text and access to the text were the two areas with the most weakness. These two areas have remained unresolved areas of weakness.
Both areas have consistently underperformed the other prompts in this section, with textbook access the area of greater weakness.
The lowest mean for textbook access was seen in hybrid courses. Hybrid courses are often courses where there is an online lecture component and a residential laboratory component of some sort.
For faculty who have developed their own textbook, Moodle provides a Book resource type. If this is an option for a course, then textbook accessibility should not be an issue for that course.
Other metrics
Student evaluation responses have come in from all five campuses that participated in the student evaluations.
Based only on student evaluations, the college remains close to evenly split between being a residential and an online institution.
When asked about their preferred materials for learning, students still report preferring in-class lectures. Online presentations are their second preference followed by the textbook and online videos. The smallest slice of the chart is synchronous online videoconferencing. In general, the students do not favor synchronous online videoconferencing course sessions.
Disaggregating the material preference by the course mode shows that the strongest preference for in-class lecture is, logically enough, in the residential courses. Yet 30% of the students in online course claim to prefer in-class lectures. Online synchronous videoconferencing is the preference of the fewest students in each course mode. The strong preference of students in online courses for asynchronous presentations suggests that this material type is even more important than the textbook. Faculty in online courses should develop and deploy asynchronous presentations when and where appropriate and feasible.
Number of student evaluations submitted per instructor
The averages for an individual instructor on any given prompt will not be statistically meaningful unless there are at least five or more student evaluations submitted, and even five is potentially problematics. 70% of the instructors received five or more evaluations. 30% received 4 or fewer evaluations. The above chart cannot take into account the number of faculty for whom there were no student evaluations submitted.
Interpreting the instructor means
These colors, however, are relative only to the other values for that instructor. The author's overall mean for all three sections (instructor, course, course materials) is 4.66 therefore 4.65 is yellow. Means above 4.66 are increasingly green, means below 4.66 are increasingly orange. This allows each instructor to identify their areas of relative strength and relative weakness with respect to their own overall mean.
To understand how a mean for an instructor relates to institutional values, the distribution of means at the institution are necessary.
Click to enlarge
Summer 2024, fall 2024: the two prompts in red were deleted at some point
The above box and whisker plots provide statistical context. The plots are the means for each evaluation section, 1360 means for each section (slightly less where a student opted to put "Not applicable" for a section). The solid line in the box is the median, the dashed line is the mean for that section. The box plot colors are arbitrary and meaningless.
The author's median for the Courses section was 5 and the author's mean for the courses section was 4.66. These values are above the median and mean for the institution. The one area of weakness for the author in this section was "The SLOs helped me focus in this course." As noted above, this prompt consistently underperforms the other prompts, and the instructor's 4.46 outperformed the institution mean of 4.38. This prompt is an area in which improvement might be sought, but the prompt is already above the institutional average.
On the other hand, this hypothetical example demonstrates the relativity of the color coding. Yellow means that the values are at the 3.33 average for this fictional instructor. Based on the box plot, 75% of the means are at least a four or higher.
A 3.33 mean is equivalent to being among the bottom 10% of evaluation submission means.
Obviously everyone cannot be above average, and there is alway a longer term tendency of any value to return to the mean value. In the lower tail of the distribution, however, there is room for improvement.
The fictional instructor above might, at a first glance at the colors, appear to be merely average, but that is because their mean is close to 4.92. Note that the colors are not based on the mean for each individual section of the evaluation but based on the mean for all three sections. This fictitious instructor will have green ratings in either their instructor or course materials section. If real, this would be an instructor with very high course ratings.
A rain cloud plot of the 1360 student evaluation means provides another indication of where a mean of 4.92 is relative to the rest of the means. Note that the above box plot and rain cloud charts are the means for each individual student evaluation. Thus the two data values at 1 represent two submissions where a student or students marked a straight "Strongly disagree" on their submission.
When looking at individual instructor results, bear in mind that at a very minimum at least five evaluations must have been submitted. No inferences can be meaningfully drawn from less than five evaluations.
Campus level differences
The following is intended to assist campuses in determining which areas an individual campus might want to focus upon. The following tables were generated using the campus the student is physically on data field.
Instructor evaluations
Click to enlarge
The table above illustrates areas of potential strength and areas of potential weakness. Although the differences in the means are small, they can still provide guidance on what areas might be best focused on by campus.
Regular and timely feedback are important to supporting regular and substantive engagement with a course. Moodle has an app that instructors of online courses should install on their mobile devices. Instructors who are unwilling to be available and connected outside of regular business hours probably should not be instructing online courses.
Course evaluations
Course materials evaluations
Resolving textbook appropriateness and access are the two areas of the greatest weakness. The effort to move to open educational resources for online courses should continue to be supported.
Term-on-term differences in the prompts
Summer 2024, fall 2024: the two prompts in red were deleted at some point
Two prompts that had been included in prior runs of the student evaluation instrument were not included in this run of the student evaluation:
- Evaluate the course [Overall, this course was a valuable learning experience]
- Evaluate the course [The student learning outcomes were clear.]
The student evaluation form has a number of editors and Google Forms does not natively provide a version history. While the reason for the deletion is not known, the first deleted prompt did not provide actionable information on how to potentially improve the course. The second deleted prompt asks a question might be construed as being somewhat redundant to the next prompt and which might be better answered by faculty who know what makes a good student learning outcome.
Comment summaries
To anonymize the student comments to permit sharing, I chose to let Google Gemini summarize the comments. There were four prompts at the end of the survey that sought open answer comments. The first three were in a section titled "Changes Needed."
- Please write down specific suggestions you would like to make for this instructor.
- Please write down specific suggestions you would like to make for this course.
- Please write down specific suggestions you would like to make for the course materials.
The fourth open answer prompt was, "Other Comments: If you have any other comments you would like to share about your course, distance learning, your instructors, or your experiences at [the college], please write freely here."
Gemini summary of instructional comments
The feedback is overwhelmingly positive, with many students indicating that no changes are needed for the instructor. Common themes in the positive feedback include the instructor being helpful, kind, clear, and well-prepared. Some students want more activities or examples, and some want assignments to be more flexible or open earlier. A few students also mention wanting more interaction and communication with the instructor.
Gemini summary of course comments
The feedback shows that most students are satisfied with the course. The most common feedback is that the course is "good" "all good" or that there are "no changes needed." Some specific positive feedback mentions that the course was clear, well-taught, and helpful in learning the subject matter.
While most of the feedback is positive, some students offer suggestions. These include incorporating more hands-on activities, providing more real-life examples and images, offering the class face-to-face only, having more in-class time, assigning less work, explaining content more clearly, and giving a second attempt for quizzes.
Gemini summary of course materials comments
The feedback shows that most students are satisfied with the course materials. Many students indicated that the materials were "all good," "good," or that there were "no changes needed."
- Some specific suggestions were given, including:
- Providing a less expensive textbook
- Updating course materials
- Including more video explanations and visual aids
- Offering textbooks instead of online materials
- Making materials easier to get
- Not locking online assignments
Gemini summary of other comments
The comments express a range of student experiences and opinions. Many students express gratitude for their instructors and the knowledge they gained. Some students encountered challenges with distance learning, such as using Canvas or submitting assignments online. Specific instructors were mentioned, with some receiving praise for their teaching and organization, while others were criticized for lack of clarity or support. Some students offered suggestions for improvement, including more hands-on activities for Micronesian History classes and additional support for completing FAFSA forms online. Overall, the feedback highlights both positive and negative aspects of the students' learning experiences.
Tools
The data analysis and charts for this report were produced using Google Sheets and Statisty. Comment summaries used Google Gemini.
Comments
Post a Comment