Thoughts on the taxonomy of assessment

 I was working my way through the excellent video series on how Kansas State tracks, retrieves, and acts upon assessment data gathered in Canvas when the assessment office shared the accompanying diagram when I saw the following diagram.

K-State assessment explained that assessment flowed up from the course level to the program level and then to the college and finally the institutional level. 

Seeing the diagram above caused my mind to spin back to the evening of 28 December 2005. Tangerine's Dream's Exit was playing as I put together a presentation on flowing assessment up a hierarchy for the 2006 assessment summit. Bottom to top alignment. 

At the January 2006 assessment summit was a newly hired director of institutional research and planning in his first week with the college. He would smile, nod, and ignore my presentation and suggestions. From there on forward the college would attempt to build not from the bottom to the top, nor the top to the bottom, but would generally take an approach where each level was worked on independently. Thus the general education program learning outcomes were decided upon sui generis. Without reference to either extant course learning outcomes nor to nascent institutional learning outcomes. The institutional learning outcomes were built without reference to program learning outcomes but rather as an independent list of outcomes assembled essentially by a committee. Naturally this stack did not flow either up or down, there were mismatches. 

In 2017 an assessment summit facilitator, Linda Susskie, suggested that the institutional learning outcomes should be the same as the general education program learning outcomes. The logic was that all students complete the general education outcomes, thus in doing so they will also all accomplish the institutional learning outcomes. I happened to standing nearby during that conversation and understood that the process I used in 2005 to built institutional learning outcomes essentially generated this same result. Because I had worked systematically from the bottom up, every course outcome flowed up to a program learning outcome and every program learning outcome flowed up to an institutional learning outcome. 

Kansas State had this very design, and inside Canvas every outcome lived in a set of nested folders that ensured assessment could flow up from the lowest levels to the highest levels. Each outcome sits in one single folder, and reports to a higher level outcome.

Note that even the numbering system suggests a hierarchy. In the wake of the 2017 summit I began exploring what other colleges were doing with respect to mappings between institutional and general education outcomes. I also engaged in a learning outcomes flow gap analysis to see there outcomes at lower levels had nowhere to flow up to. This analysis revealed both orphaned general education learning outcomes - outcomes with no institutional learning outcome parent- and childless institutional learning outcomes that had no general education program learning outcomes. In the world of taxonomy the outcomes without a parent might be considered to be taxons which are "ungrafted" to the taxonomy backbone. This work would lead to a proposed working alignment of general education and institutional learning outcomes. I had come back around to my 2005 thinking that the institutional learning outcomes should not be identical to the program learning outcomes, but rather higher level aggregated outcomes of which there would be fewer, more general learning outcomes. This would lead to a summary proposal addressing the 2017 conversation by harmonizing the general education learning outcomes and the institutional learning outcomes.

The structure at Kansas State was the data flow structure I saw as being logical and able to aggregate success rates up level by level and provide reporting at each level. I knew, however, that the outline format had been a multiyear curriculum committee process with, for a time, the format changing every year. After years of changing the format, but ultimately not the content, the committee settled on a format that I knew violated the above hierarchy. 

Each author was to write outlines in the above format with each specific learning outcome (an outcome level below the course level, think a "suboutcome") mapping to a course learning outcome, a program learning outcome, and an institutional learning outcome. Thus specific learning outcome 1.1 on the above outline maps to course learning outcome number one, general education program learning outcome 3.1, and institutional learning outcomes 1, 4, and 8. The asterisk means that this specific "suboutcome" is to be used to report on institutional learning outcome eight.

When mapped one gets something closer to spaghetti than a hierarchy. 

One way to think of this would be as if students reported to instructor, instructional coordinate, deans, even cabinet vice presidents. Directly. That is not going to be a functional organization. And in the data world, mapping a many-to-many multi-level system is either chaos or, perhaps, the black box of a neural network. 

Outlines should list nothing above the PSLOs served by the CSLOs.

Specific learning outcomes should only be listed under course level outcomes, no mapping to a program or institutional outcomes. Each course outcome should map to only a single program learning outcome. Of course, other course learning outcomes can map to the same program learning outcome.The course outcomes map to program learning outcomes which map to institutional outcomes. Data is aggregated up the hierarchy.

For course outlines, this would mean deleting the center two columns. Or simply ignoring them, which is what will functionally happen when someone tries to encode the flow of assessment up the hierarchy. No one is likely to try to build an aggregating function that pulls from multiple lower levels - all kinds of statistical bias and problems would arise. 

A separate document should specify how program learning outcomes map to institutional outcomes. This does not belong on any course level outline. This belongs on a program outline, if such were to exist.

The general education program learning outcomes under the proposed "harmonized" general education and institutional learning outcomes would aggregate such that all children outcomes mapped to a single parent outcome and that no parent outcome was without children. While the mapping is only for general education program learning outcomes, these institutional learning outcomes should be broad enough to include all program learning outcomes. Ensuring that this is the case would require building a database of all program learning outcomes and working out the mapping. This was done in 2006 (large html file) to ensure that the institutional learning outcomes being proposed handled all of the possible program learning outcomes.

The result would be a hierarchical assessment system that could be captured in packages such as Nuventive Improve/TracDat or Instructure Canvas.