New management tooling for Instructure Canvas outcomes

The following report includes information on a new feature set for outcomes from a Canvas webinar and current practices on the platform at the college.

Three areas will be covered in the webinar.

1. What are outcomes?
2. Account settings and feature options
3. Teacher/student/observer view

Outcomes need to have some sort of scale. They must be measurable. Vocational often uses a binary rating scale on outcomes: 0 not yet competent. 1 Competent.

Five point scales are also common. The college is presently using a modified five point scale.

The logic behind this choice of scale can be read about at: http://danaleeling.blogspot.com/2021/03/learning-outcomes-assessment-ratings.html

A poll was run to determine the level at which attendees are currently using outcomes. I answered C: We have provided some outcomes training but teachers are not required to use outcomes in Canvas.

Eleven out of 81 courses on Canvas are reporting course learning outcomes (14%) that were pulled from the institutional bank of course learning outcomes. Not all sections of those courses are reporting outcomes, only 16 of 130 sections on Canvas (12%) are reporting outcomes.

Bringing outcomes into Canvas can be done a number of different ways. The college uses the CSV file import method.

In the new outcomes feature, outcome groups are displayed as a tree structure seen above. This is more navigable than the previous sidescrolling structure.

The host then began a more detailed dive into the fields in the Outcomes import file. This file appears to be essentially unchanged from the current format, as far as I could tell the fields and structure are identical. The new tree view is only a graphical user interface change. Vendor guid is a SIS ID for outcomes. They must be unique. The CSV being displayed in the webinar uses the highest method for calculations. Many of the webinar attendees noted the use of highest or latest as the calculation method. The college currently uses a 65% decaying average where 65% of the weight is on the most recent measurement and all prior measurements are equally weighted in the remaining 35% of the average. This gives some heft to earlier measurements but puts the heaviest lifting on the most recent measurement.

A straight average calculation option for outcomes calculations will be landing soon in the Canvas platform.

Only root level account outcomes allow for the use of the Outcome Results report from the Admin console.

How to switch on the new interface with the improved outcomes management? At this time the customer success manager has to enable the feature. There will come a time in the future when this feature will be enabled by default. All Canvas instances will have to eventually migrate to the new outcomes management.

Note that in the Outcomes screen are two new tabs: Mastery and Calculation method. The institution will be able to specify mastery levels and calculation methods for all outcomes from these new tabs. Those were previously only set on each individual learning outcome - a more global setting of these was not possible. This new structure allows for applying an institutional standard. One of the challenges institutions have had is that mastery and calculation methods were able to vary from one outcome to the next. When that happens, aggregating the results across all outcomes at the institution is problematic.

Because of the potential impact on existing assessment, the change must be done between terms. The new settings may conflict with existing settings in the outcomes.

The college should be in a good place to move to the new feature set. The use of a CSV import file has meant that all outcomes uploaded into the institutional banks are using the same rating scale, the same mastery level, and the same decaying average calculation setting. As long as those are the settings used in the new feature set, there should be little to no negative impact.

The new outcomes feature set includes a new mastery rating scale tab in the admin console.

The new calculation method screen permits highest score, decaying average, n number of times, most recent score, and highest score methods.

There are additional account role permissions once the new outcomes feature is enabled.

Second poll. 50/50 split. The college currently has a mastery level of four (sufficient) and a decaying average method of calculating the outcomes. Note that none of these changes constrain faculty from entering course learning outcomes into their course at the course level. Faculty can still do this. The above applies only to the institutionally banked learning outcomes that report to the Outcome Results report. Faculty who wish to use a different scale would have to enter outcomes at the course level. This would mean they might have to mark a course learning outcome twice, once for the institution and once for their own benefit - perhaps they want to use a binary yes/no scale for their own in course assessment purposes. The above is really just a mechanism to provide consistency at the institutional level. The course level learning mastery screen will report on any and all outcomes being evaluated by rubrics or quiz question banks in a course.

Learning mastery gradebook shows outcome results except in the case of New Quizzes. At present outcomes aligned to questions in New Quizzes are not reported to the learning mastery gradebook. That is on the roadmap for New Quizzes and will land before the mandatory move to new Quizzes in 2024.

Although the learning mastery gradebook is not enabled by default for instructors, at the college that permission is enabled for faculty at present and will remain enabled for faculty. The current outcomes system does not have a student mastery gradebook, so there is no permission to be enabled. In the new outcomes feature set the student learning mastery gradebook is off by default and will probably be left off as students receive learning outcomes feedback directly from rubrics used in association with assignments.

When faculty add outcomes to rubrics they will still apparently see a horizontal scrolling interface, at least based on what I saw in the webinar. The tree structure may only be visible in the admin consoles.

Mastery scales can be set at the root or subaccount level. If one is making settings at the subaccount level, then the outcome results would have to be exported from the subaccount, not the root account. Best practice is to set outcomes at the highest level possible, typically the root account. A question was asked, "Is there only one mastery scale per subaccount?" Yes, only one.

Some of the items on the outcome features roadmap ahead.

The new outcomes features includes the ability to drill down to the individual student and determine what outcomes an individual student has attained as seen above. Details of what assignments are producing the "0 of 5 mastered" are apparently available by clicking on the arrow and opening up a learning outcome.

Other features are yet to land including an outcome alignment summary. Which ones are being used, what they are attached to, and which ones are not being attached to anything. This will act as a form of curriculum mapping capability.

Another new feature on the road ahead is the option to map non-assessed items such as modules to learning outcomes. This is slated to land in the back half of this year.

The new outcomes features may be high impact for some institutions, currently only customer success managers can enable the new feature set. In the future there will be a cut over, automatic, although a date for this is not yet set.