Student evaluations summer 2024

Summer 2024 student evaluations featured a new ratings response order

Student evaluations of instructor, course, and course materials provide guidance for the institutions on areas of relative strengths and areas where there may be room for improvement. Bearing in mind that old data is not usefully actionable data, the intent of this report is to convey broad themes emerging in the evaluations to decision makers as rapidly as possible.

This report presumes familiarity with the student evaluations form in use at the institution. The responses were converted to numeric values:

Strongly disagree
Disagree
Neutral
Agree
Strongly agree

As a preface, results this term are not directly comparable to prior terms. Data from the past five terms showed that a significant number of students mark the same response to all of the prompts. These students are marking all fives, or all fours, and in some instances, all ones, in response to every rating prompt. These are termed "straights" in this report.

In the spring of 2024 straights comprised 43.9% (288) of 653 responses.

Straights have distributed in a similar manner each term. Straights are dominated by fives and fours with 88% of the straights being a four or five straight. No student marked straight twos in spring 2024. 25 students, however, marked straight ones: the students disagreed with every single prompt.

The prompts are written in a manner that stronger agreement is better, prompts such as "The instructor initiated regular contact with the student through discussions, review sessions, feedback on assignments, and/or emails." Straight ones would suggest that the student strongly disagreed with each and every prompt related to the instructor, course, and course materials.

The puzzle would appear in the comments. Although the student had marked straight ones, the comments were often glowing reviews of the instructor, course, and course materials. "Best instructor ever!" was the gist of the typical comment.

Spring 2024 student evaluation form

What appears to have been happening is that the student was simply marking straight down the first column without reference to the header row characterizations. This layout was originally duplicated from a paper based form five terms ago, the response order was maintained for consistency with the paper based form.

Summer 2024 student evaluation form

For the summer 2024 the response order in the header was reversed.

Summer 2024 the split was statistically identical to spring 2024 with 43.2% of the 447 responses being straights versus 43.9% in the spring of 2024.

Four and five straights, however, have increased from 89% to 96% of the straight responses. This summer there was a single straight comprised of ones and a single straight comprised of twos.

The straight ones was a student (responses are anonymous) who answered "Not applicable" to all of the instructor and course evaluation prompts. The student then marked "Strongly disagree" for every prompt under course materials. The student made no comments.

The straights twos was a student who answered "Disagree" to every prompt in all three ratings sections. The student left a negative comment that made clear the student intended the ratings that they gave.

To provide some context, ones and twos are rare responses when all responses are considered. For 447 student evaluations submitted there were 11098 responses not including "Not applicable." Many students either strongly agree or agree with a prompt. A score of neutral is infrequent, twos and ones are very rare. Where a student is making decisions between a five and a four, a four is effectively a form of rating the prompt down. A three is, in effect, a very negative rating where fives dominate the responses. Teasing out where improvements can be made requires differentiating among a set of relative high scores. Scores that are even higher this summer with the disappearance of all but one straight ones.

In the report that follows all averages will have moved up due apparently to the change in the order of the rating responses. Improvements term-on-term are not comparable this summer, nor will future terms be comparable to prior terms due to the impact of the change. The change, however, has eliminated a source of errant marking in the evaluations and is beneficial over the long run.

The number of student responses by term is seen above. This summer saw more submissions than summer 2023. Canvas can share the link to the evaluation in Canvas as an announcement, something that has been done each term. Online students are almost certain to see the announcement. Students who are in residential sections that may not be using Canvas regularly might be less likely to see the link. Ensuring that faculty in all courses remind students to complete the course evaluation could only be beneficial.

The impact of the loss of the straight ones can be seen in the above chart. The overall mean increased to a 4.47, the highest mean seen in six terms of data. This is not a reflection of improvement, just a result of resolving the spurious ones.

Instructor evaluations

Unlike the prior five terms, the first prompt did not strongly underperform the mean for the first section. At 4.46 the first prompt came in almost at the section mean of 4.47. The means for the prompts in this section were tightly clustered around the section mean. Differences were small.

By calculating a t-statistic, the small differences can still be used as a first approximation to areas of relative strength and relative weakness. Prompts with a t-statistic between -2 and 2 can be considered to be functionally equivalent to the mean. Three areas of relative strength were:

The instructor presented the course content clearly
The instructor emphasized the major points and concept
The instructor made sure that the students were aware of the Student Learning Outcomes (SLOs) for the course

The two areas of relative weakness were the same two areas of weakness seen in the spring 2024 evaluations:

The instructor initiated regular contact with the student through discussions, review sessions, feedback on assignments, and/or emails
The instructor presented data and information fairly and objectively, distinguishing between personal beliefs and professionally accepted views

68% of the 447 responses came from online courses suggesting that the weakness in regular contact may be in the online courses.

When the regular contact prompt mean is split by the course type, the online courses are seen as the area driving down this mean value. This is an area in which there may be room for improvement. This author has a 4.89 average for regular contact in online courses for the fall 2023 and spring 2024 terms combined. The author uses the Canvas Teacher app with notifications enabled. During the term a mobile data package is used to supplement Wi-Fi connectivity. The author's target is to respond within 24 hours, if not less. If other matters prevent a full response, a reply is sent that the author is otherwise occupied or is mobile and will respond more completely as soon as is practically possible.

The other area of weakness, "The instructor presented data and information fairly and objectively, distinguishing between personal beliefs and professionally accepted views," is also an area of repeat relative weakness.

Course evaluations

The section mean for the course evaluations was 4.51.

The two areas of relative strength were the prompts:

The course syllabus was clear and complete.
The student learning outcomes were clear.

Relative weakness in the means was seen for the following prompts;

The student learning outcomes helped me focus in this course.
The testing and evaluation procedures were fair.
Expectations were clearly stated.

How students interpret the underperforming prompt, "The student learning outcomes helped me focus in this course." is not clear to this author. From the student's point of view their focus is on getting assignments done, successfully answering questions on a tests, and completing other work as assigned in a course. How the student learning outcomes would help a student focus on these specific tasks seems unclear. This metric has consistently underperformed term after term, including this summer, which may be an indication that the prompt is asking something that does not necessarily make sense from a student's point of view as to what they need to focus upon.

The other two prompts that strongly underperformed the course section mean are also areas open to interpretation. Bear in mind that the averages are between "Strongly agree" and "Agree" Only the mean value for these two metrics relative to the section mean is low. Students are not saying tests are unfair and there isn't enough time to finish assignments. They are simply rating these two metrics lower than the other metrics. The difficulty is that students tend to answer all prompts with a 4 or 5. Distinguishing meaning is challenging when the range is that small.

Course materials evaluations

The section mean for the course materials evaluations was the lowest of the three sections at 4.42.

The prompt, "Course materials were relevant and useful," had a t-statistic of 2.04, just slightly above the two standard errors for possible significance.

The strongest positive rating in the section was for the Instructure Canvas learning management system platform. Canvas has consistently been well regarded by the students, an area of institutional strength as noted in a recent Canvas review report.

The single area of relative weakness was the accessibility of the textbook. This area is being driven down more strongly by the online courses. Given the dominance of online courses during the summer session, this is an area of concern. Online courses should be supported by online texts that the students are able to access.

The optimal choice for a textbook for our students would be an open educational resource, an area that the leadership at the college is encouraging faculty to explore.

For some courses open educational resources are not an option. In some instances the bookstores may be able to provide access to online resources with the students using financial aid to pay for the resource. I

There are situations in which the bookstore will not be able to assist. This past spring an online textbook in one course cost $11.99 as an ebook available both from the publisher and via Amazon for the Kindle platform. No student in the course could acquire the textbook. The issue was not the cost, the issue was that no student had a debit card with which to make the purchase. The bookstore could not assist because the ebook is not transferable from the account that downloaded the book.

While the state campus students in Kosrae, Chuuk, and Yap, often need bank accounts into which their financial aid is direct deposited, the students on Pohnpei are not required to have bank accounts. Moving to direct deposit of financial aid/work study and encouraging students to not empty and close their accounts every term, would potentially enable students to acquire online materials.

While the change in the rating response order lifted all means, ease of access to the textbook remained the lowest mean for the course materials and has had the lowest mean each term for six terms to date.

Canvas also provides direct support for finding and integrating open textbooks into Canvas courses. Setting the Canvas Commons search filters to Undergraduate and Open Textbooks returns 4,746 results. These resources include texts and courses with links to open education resources built into the course.

Canvas Commons Open Textbooks include the author's own Introduction to Statistics Using Google Sheets. Canvas Commons is the largest repository of academic materials and reflects the global adoption of the platform by institutions of higher education. The Canvas Commons is a good place for faculty to check for open textbook support in their field.

Other metrics

Student evaluation responses have come in from all five campuses that participated in the student evaluations.

To provide context to responses from students by course delivery mode, the above chart shows the distribution of student seats by campus this summer. According to Canvas analytics, there are 1312 students enrolled in courses at the six campuses above. This number will include double counts for students who are in both residential and online courses this summer. Just over half of the student seats are in online courses. The number of student seats in online courses increased from 43.4% spring 2024 to 53.9% summer 2024.

The dominance of the Online courses as a campus can be seen in the above data from Canvas for summer 2024. For purposes of scale, the number of students has been divided down by ten. The online campus is the largest campus in the system. According to Canvas, there are 78 active courses to be overseen. That is a significant load.

Spring 2024 course delivery mode

During the spring 2024 term just over one third of the student evaluations came in from students in online courses.

Summer 2024 course delivery mode

The percentage of evaluations coming in from online courses doubled term-on-term for online courses. This exceeds the share of students in online courses which suggests under-reporting from students in residential courses and/or over-reporting from students in online courses.

Given that online courses underperform residential courses (75.04% average for summer 2024 to date for online sections versus 79.27% for residential sections systemwide with residential sections at the national campus at 81.92%), increasing the share of online courses may not be beneficial for the students.

Although 68% of the responses come from students in online courses, half of the evaluations that come in express a preference for in-class lecture. This summer online presentations edged out online videos by an insignificant difference of one evaluation. Textbooks took fourth while online synchronous videos were a preference of only 9.1% of the respondents.

When broken down by course type, the preference by students in online courses for online presentations more strongly outperformed online videos. That said, students in online courses still dominantly prefer in class lectures. The patterns seen above have been stable for multiple terms, leading this author to supplement the course textbook and videos with presentations containing the same material. Courses benefit from providing more than one mode of course material. Optimally an online course offers an online textbook, presentations, and videos.

When students indicate that they prefer presentations, this is not a reference to synchronous video presentations. Setting aside the students in online courses who prefer to be in a residential course, those students who prefer their online course favor asynchronous over synchronous by a factor of 3.6 to one. Roughly speaking, four out five students in online courses who want to be in an online course prefer asynchronous online courses.

Students who are in hybrid or residential courses dominantly prefer to be in their residential courses. That the students who are in residential courses favor residential courses, and that 27% of the students in online courses prefer to be in a residential course, there is the suggestion that the college ought to be offering more residential courses - that demand for residential courses is not being met.

That said, as of the writing of this the MS 150/O1 Statistics online section is full and closed while the MS 150/1 Statistics residential section with the same instructor is only two-thirds full. This has happened every term: the online section fills up before the residential section. What students report that they want and what they choose to take stand in opposition to each other.

Any errors above are solely those of the author, any opinions expressed are also solely those of the author. The intent is get accurate information shared as quickly as possible. The author owes a debt of gratitude to the college for providing the tools, Google Workspace for Education and Instructure Canvas, that make assembling such a report possible.

Additional information added 24 July 2024

Subsequent to the above work, an analysis was made of the number of student evaluations submitted for each instructor. When the number of evaluations submitted is less than five, then the results are not considered statistically stable. A single strong or weak evaluation can significantly push the averages for the instructor.

For 83 instructors, 12 had no student evaluations submitted. 38 instructors had four or fewer student evaluations submitted. Only 33 instructors, 40%, had five or more student evaluations.

An analysis was also made as to whether the mean current score for an instructor was correlated with their student evaluation mean. The correlation was low to moderate (r = 0.37, p = 0.036). There is a correlation, but the correlation is not predictive. Students value instructors who respond, grade, and communicate in a timely manner, are perceived to be unbiased and fair, and are consistent and organized in their courses.

Instructors should not fear student evaluations nor should they be dismissive of their evaluations. The students more often blame themselves when they do not succeed than they blame the instructor. Improving the student evaluation submission rate is worthwhile for every instructor, not just for the institution.

Comment summaries added 27 July 2024

To further anonymize the student comments to permit sharing, I chose to let Google Gemini AI summarize the comments. The prompt was to summarize the comments in three paragraphs. There were four prompts at the end of the survey that sought open answer comments. The first three were in a section titled "Changes Needed."

Please write down specific suggestions you would like to make for this instructor.
Please write down specific suggestions you would like to make for this course.
Please write down specific suggestions you would like to make for the course materials.

The fourth open answer prompt was, "Other Comments: If you have any other comments you would like to share about your course, distance learning, your instructors, or your experiences at [the college], please write freely here."

Gemini AI summary of instructional comments

This document appears to be a collection of student feedback on instructors and courses. The feedback varies greatly, with some students praising their instructors' teaching methods and helpfulness, while others express dissatisfaction with aspects such as unclear instructions, lack of responsiveness, and unfair grading.

Some students request more specific and clear instructions, additional examples in assignments, and more timely feedback on their work. Others desire more opportunities for interaction with instructors, such as through Zoom meetings or office hours. There are also suggestions for improvements in course structure, including better time management, clearer communication about course materials, and more engaging assignments.

Additionally, some students express a preference for face-to-face classes, citing difficulties with online learning and a desire for more direct interaction with instructors and peers. Overall, the feedback highlights a range of student experiences and expectations, with a common desire for clear communication, helpful guidance, and fair assessment.

Gemini AI summary of course comments

The document contains feedback for a course, likely collated from student surveys. A primary theme is the desire for more instructional videos explaining assignments, with students feeling that written instructions are sometimes insufficient or confusing. There are also several requests for more face-to-face interaction, with some students finding online-only or AI-led sessions ineffective.*

Another theme is the pacing and workload of the course. Some students suggest that there are too many assignments and not enough time to complete them, particularly in the summer term. Others, however, request more assignments and hands-on activities. There are also differing opinions on the format of assessments, with some students suggesting that summaries should be replaced with demonstrations or activities, and that presentations should be included.

Additional comments include requests for clearer explanations from the instructor, more time for quizzes and tests, and changes to the timing of conferences. Some students also note the need for a bigger space and improved facilities. Overall, the feedback indicates a desire for more instructional support, clearer communication, and potentially some adjustments to the course structure and format.

* According to the underlying comments, one of the external tools used by a course apparently included an AI chat bot.

Gemini AI summary of course materials comments

The feedback for this course reveals that students have a variety of suggestions for improvement. A primary area of concern is the course materials. Many students request that textbooks be provided for free or uploaded to Canvas, and that instructional videos be created to demonstrate how to complete assignments. Some students find the written instructions confusing and would benefit from more visual aids. Others are satisfied with the current materials and find them useful.

Another area for improvement is the course structure and delivery. Some students desire more face-to-face interaction and opportunities to network with professionals. Others find the online format convenient but suggest that the instructor provide clearer explanations and more timely feedback. There are also differing opinions on the pacing and workload, with some students finding the number of assignments overwhelming, while others request more hands-on activities.

Additional comments include requests for changes to the timing of conferences, improvements to the classroom facilities, and more variety in the types of assignments. Overall, the feedback suggests that students would benefit from clearer communication, more instructional support, and some adjustments to the course structure and format.

Gemini AI summary of other comments

Overall, student feedback is positive, with many students expressing appreciation for their instructors and the flexibility of online classes. Some students even report that they prefer online classes to in-person instruction. However, the feedback also reveals some common concerns. For example, several students mention technical issues, such as assignments opening after the due date.* Others note that the online format can make it difficult to understand course material and interact with instructors.

Students offer a variety of suggestions to improve the distance learning experience. These include providing clearer instructions, offering more opportunities for interaction, and ensuring that course materials are accessible and engaging. Some students also suggest that the college should consider offering more face-to-face classes, or at least providing opportunities for students to meet with instructors in person.

In addition to feedback about distance learning, the document also includes comments about other aspects of the college experience, such as registration, facilities, and student services. Students express a desire for more transparency in the registration process and better maintenance of campus facilities. They also suggest that the college should provide more support for working students, such as flexible deadlines and on-campus food options.**

* Appears to be a reference in part to an instructor consistently setting due dates to 12:00 PM lunchtime.

** A student at a state campus wanted on campus meal options at their campus for students who had to stay on campus until 5:00 PM.

Search This Blog

danaleeling