Thoughts on faculty observation and evaluation in asynchronous learning environments at the college

 “And no one puts new wine into old wineskins. For the wine would burst the wineskins, and the wine and the skins would both be lost. New wine calls for new wineskins.” - Mark 2:22

Since the summer term I have been and I remain concerned about finding ways to provide for administrative and supervisory oversight and insight into online asynchronously delivered classes. One of my core concerns with Schoology is the lack of oversight capabilities, the lack of supervisory and assessment dashboards to assist administrators in their supervisory duties. 

l gather there is interest in having faculty schedule synchronous teleconference sessions with students during spring 2021 to provide an opportunity for supervisors to observe the class and complete the required faculty observation. With all due respect to my esteemed colleagues, observing a teleconference session for a class that is primarily operating asynchronous is more likely to generate what in English we term a "dog and pony show" for the supervisor. A staged and planned session that does not reveal whether the instructor is effective in their asynchronously delivered course.

To try to use teleconferencing to evaluate an asynchronously delivered course is an attempt to use the old classroom observation approach to evaluate the new learning environment. Online asynchronous education is new wine and will need new approaches to evaluation. Just as videotaping oneself lecturing in front of a whiteboard is a suboptimal way to move a course online, new technologies demand new approaches to delivering material and documenting learning. New technologies change what one can do, what one should do, change the best practices. 

What can a supervisor look at to evaluate faculty? I do not know what supervisors can "see" in an asynchronous course. I will share some of what I have access to and comment on options that supervisors might need to explore. As always, some of this is sensitive especially when working with faculty. Supervisors will have to work diplomatically with faculty who are struggling. 

This evaluation would be best done sometime after midterm when the course should already be well populated both with assignments and student work done.

Calendar and student engagement

The calendar shows work due in. The calendar should be robust, showing an active faculty member providing multiple and regular engagement opportunities. Now there are a lot of caveats. If the faculty member is running their work outside of Schoology then that work will not show up in Schoology. I would argue, however, that faculty teaching asynchronously should not be operating outside of Schoology. I have assignments that are submitted to iNaturalist and YouTube in my ethnobotany class, but those still show up in Schoology. I simply set the assignment to not accept submissions in Schoology, but the assignment still exists in Schoology and calendars. A blank calendar should set off alarm bells. A thinly populated calendar should raise concerns.

Materials

The materials section of the course should also be robust. There is no one right way to organize a course, and even my own courses use a couple approaches. There should be a diversity of materials in use. Not just one type. Not just a long list of PDFs that have been uploaded. Links, discussions, pages, assignments, tests, as appropriate to the course. 

Gradebook

The gradebook should also be robust and well populated. Not all gradebooks will look like this. For a gradebook like mine one thing to look for would be blue assignment submitted icons that do not also have a number in that cell. Those would be unmarked assignments and could be a sign that the faculty member was not getting their work marked in a timely fashion. "Could be a sign" is a key phrase: there may be a reason the faculty member is waiting to mark assignments, perhaps they want all turned in before they start marking. I think that is inappropriate in an asynchronous course, but that is just my opinion. And certainly columns of unmarked submissions should cause concern. Also a paucity of assignments would be a concern. The gradebook can tell one a lot about what is happening in the course. Including the grade distribution. 

Analytics

I do not know if you can see the analytics, but this can show you whether students are engaging, when they last logged into the course and how much time they are spending in the course. If all of the last course material access dates are a month ago, that is a problem. The faculty member has lost their students a month ago. This is why this has to be done later in the term, perhaps as late as 12th week or thereabouts. The above screenshot is from statistics and is from today. So you can see a wide variety of engagement times for my students. 

Again, this will vary with the class. My ESS class submits via a Facebook group, students only log in to take weekly quizzes so the total time spent in the course is much less than my statistics students. One top student who clearly studies for the quizzes and finishes them quickly has only 17 total minutes of course time. But then this student has also only missed a single day of walking since the term start.

Messages

Unfortunately my guess is that supervisors cannot see this all important area. This is where almost all of my non-assignment interactions occur. Assignment interactions occur through the Google Docs and Sheets comments facility, that is an area I have no idea how to reveal to a supervisor. But Schoology messages are another key area and dominate my daily workload. You will see that almost all of the above messages have my face as the icon: that means I have answered that message. The one message I did not respond to was a student thanking me for clarifying an assignment for her. If this screen, and subsequent message screens, were all icons of other people, that would be a sign that I was not responding to student messages on Schoology. Now perhaps some are using email, some of my students prefer to reach out via email, some via Facebook. But most of my students do use Schoology to contact me. 

As I said, I do not think supervisors can get at these messages, and faculty might not want that level of intrusion into their course. Still, this is an important area in an asynchronous class. Is the instructor responding in a timely, supportive fashion? These messages are at the heart of the student-instructor relationship in an asynchronous course. 

I know this would be highly controversial, but perhaps supervisors would have to reach out to students with surveys on how supportive their instructor is and how responsive the instructor has been. 

For me, messaging is my primary communication tool for working directly with my students on a one-to-one basis. I know some might prefer to use teleconferencing, but only 9% of my students prefer teleconferencing. 84% want material delivered via presentations or videos. Only a textbook is less popular than teleconferencing options such as Zoom or Conferences. So messaging is the key way I address individual learning needs. I receive and send messages seven days a week from when I first wake up until I fall asleep in the evening. Messages pop notifications on my phone, so I am always aware when a student needs help. How to help supervisors see how a faculty member handles these messages is a challenge I do not have an answer for. 

I would not recommend that a fully asynchronous course have a one off Zoom session for the sole purpose of evaluating the faculty member's performance: remember that assessment must align with the target to be assessed. To use a Zoom session to assess a course that runs without Zoom sessions is misaligned and will not provide the assessment desired.

Notifications

I am not done in the evening until this panel is clear. Too many times I help a faculty member and this panel is lit up with yellow notification icons. That is always a flag for me. There is a proviso: these icons do not automatically clear when the assignment is marked. And getting to assignments to mark them does not have to run through this panel. That said, a pair of yellow icons with large numbers in this panel is a potential sign of problems - of a non-responsive faculty member who is not keeping up with student messages, submissions. 

For each area above there are caveats and exceptions, one has to look at other explanations, but taken together these areas can tell a supervisor some of what is happening in the asynchronous classroom. Is there the appropriate diversity of assignments? Appropriate number of student deliverables in the course (submissions of all kinds)? Does the mix of materials appear appropriate to the subject area? To the learning outcomes expected? 

A one size fits all checklist is probably not going to work well in evaluating faculty delivering asynchronous online courses. These evaluations will likely be more qualitative and less quantitative. 

I am still unclear on how much of the above supervisors can see. I think the process might have to involve student surveys - an area that the college has not ventured into: student feedback having an impact on faculty performance evaluations. I know that student evaluation of faculty is sensitive terrain. Asynchronous courses are also wholly new terrain and may require new approaches. I work at home, out of sight. This is not the world of the residential classroom. You cannot walk by my room and see that things are going well. Or not. I work in a semi-hidden world largely invisible. Developing necessary oversight and supervision capabilities will be important going forward.