This post is a part of series of posts informally documenting some of the decisions being made in the setting up of a student learning outcomes hierarchy in an Instructure Canvas instance. This post is preceded by a presentation the taxonomy of assessment which was informed by earlier work on institutional assessment and by videos from a workshop at Kansas State University on their assessment system using Instructure Canvas. Prior posts covered hierarchical assessment, outcome rating scales decisions, and implementing a hierarchy of outcomes in Canvas.
The decisions are being made against a background of having a commercial assessment database which was installed circa 2012. At that time faculty were invited to load course learning outcomes into the database. For some programs, program learning outcomes were also entered. Only later would a serial series of assessment coordinators be hired to tackle the task of entering the rest of the program learning outcomes and the institutional learning outcomes. Over a roughly six year period three assessment coordinators would each, without any formal training per se or background in the database package, work on improving the performance of the package. The unintended result is essentially effectively a committee making decisions without necessarily comprehending the decisions and choices that had already been made. By 2019 one faculty member who looked at the situation thought that the only solution would be to start over from a blank slate.
One of the complications has been that the institutional learning outcomes were developed essentially from the top down while the general education learning outcomes were developed over time from the bottom up. Like two different teams building towards a middle point without communicating, there were places the pieces did not fit together. In August 2017, at a college Vision Summit, consultant Linda Suskie would suggest that the general education outcomes should probably be the institutional learning outcomes on the grounds that the general education outcomes were the only outcomes that every single graduate is certified to have accomplished.
My own work after the summit suggested that while some institutions do use their general education outcomes as their institutional outcomes, a more common approach seemed to be to use a smaller set of aggregated general education outcomes as the institutional outcomes. I then set about building a set of institutional learning outcomes that would encompass all of the existing institutional learning outcomes while reflecting the existing general education program learning outcomes.
Again, a key concept is that the only program all students have in common is the general education program. So the institutional learning outcomes, those outcomes that every single student accomplishes, cannot include outcomes not found in the general education program learning outcomes. Any outcome that was unique to any one specific program would not be accomplished by students in other programs. Hence the tight linkage of general education program learning outcomes to the institutional learning outcomes. Designed properly, these would also be able to include all of the other program learning outcomes.
The following attempts to document the decisions I was making to generate a viable aggregating stack of learning outcomes.
Institutional learning outcome one and two
| Proposed iSLOs. | Current iSLOs | Gen Ed pSLOs | Gen ed courses in area |
| 1. Effective oral and written communication: Demonstrate effective oral and written communication | 1. Effective oral communication: capacity to deliver prepared, purposeful presentations designed to increase knowledge, to foster understanding, or to promote change in the listeners’ attitudes, values, beliefs, or behaviors. | 1.2 Make a clear, well-organized verbal presentation. | EN 120a, EN 120b |
| 2. Effective written communication: development and expression of ideas in writing through work in many genres and styles, utilizing different writing technologies, and mixing texts, data, and images through iterative experiences across the curriculum. | 1.1 Write a clear, well-organized paper using documentation and quantitative tools when appropriate. | EN/CO 205 |
Institutional learning outcomes one and two were combined into a single Effective oral and written communication outcome. These map directly, although invertedly, to general education learning outcomes 1.2 and 1.1 respectively. Here the guiding logic was that the institutional learning outcome should be aggregating of the general education learning outcomes. Note that for all of these mapping examples only a few courses are listed. These proposed institutional learning outcomes will also be served by many other program learning outcomes.
Institutional learning outcome three, four, and six
| Proposed iSLOs. | Current iSLOs | Gen Ed pSLOs | Gen ed courses in area |
2. Critical thinking and problem solving: Demonstrate independent thought, critical thinking, problem solving, and information literacy. | 3. Critical thinking: a habit of mind characterized by the comprehensive exploration of issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion. | 2.1 Demonstrate the ability for independent thought and expression. | EN 110, EN 120b |
| 4. Problem solving: capacity to design, evaluate, and implement a strategy to answer an open-ended question or achieve a desired goal. | [No directly corresponding pSLO. pSLO 3.1 primarily serves the math courses and is under quantitative reasoning.] | |
| 6. Information literacy: the ability to know when there is a need for information, to be able to identify, locate, evaluate, and effectively and responsibly use and share that information for the problem at hand. | 2.2 Demonstrate understanding of the modes of inquiry by identifying an appropriate method of accessing credible information and data resources; applying the selected method; and organizing results. | CA 100 |
When looking at other institutions I came to understand that where critical thinking is being done, problems are usually being solved. These two are part of larger skill set involving demonstrating independent thought, understanding modes of inquiry, evaluating sources of information, implementing a strategy to solve a problem. In this age sources of information are queried on line, formulating and sharing an opinion or conclusion is done using software such as a word processor or presentation slides. Being able to identify, locate, evaluate, and responsibly use information is a reflection of critical thinking skills. This line of thinking led me to combine these into a single institutional learning outcome.
Institutional learning outcome eight
| Proposed iSLOs. | Current iSLOs | Gen Ed pSLOs | Gen ed courses in area |
3. Qualitative and quantitative reasoning: Demonstrate the ability to explain concepts and theories, engage in inquiry, reason and solve qualitative and quantitative problems. | 8. Quantitative Reasoning: ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations; comprehends and can create sophisticated arguments supported by quantitative evidence and can clearly communicate those arguments in a variety of formats. | 3.1 Demonstrate understanding and apply mathematical concepts in problem solving and in day to day activities. | MS 100, MS 101 |
| 3.2 Present and interpret numeric information in graphic forms | MS 100, MS 150, SC 130 |
| 3.3 Communicate thoughts and ideas effectively using proper mathematical terms. | MS 100, MS 101, (MS 150) |
| No directly corresponding iSLO | 3.4 Define and explain scientific concepts, principles, and theories of a field of science. | SC 101, SC 111, SC 112, SC/SS 115, SC 220 |
| No directly corresponding iSLO | 3.5 Perform experiments that use scientific methods as part of the inquiry process. | AG 101, MS 120, MR 240, SC 117, SC 120, SC 130, SC 230 |
Here the numbering of the general education learning outcomes came into play. One of the things that emerged rather naturally from the reduction in institutional learning outcomes was that the institutional outcomes could be numbered as the integer for the existing general education program learning outcomes. This use of numeric matching was a practice I was seeing at other institutions in my research. The integer value for the general education learning outcome informs one which is the corresponding institutional learning outcome.
Having a logical numeric structure will be important for faculty when they attempt to search for a specific course learning outcome in the college wide outcome collection. A faculty member will have to drill down to reach a course learning outcome. With the current institutional learning outcomes there is no indication to a user that they should drill down under outcome number eight to reach general education program learning outcome three. The same is true in the earlier section where one would have to drill down through institutional outcome three to get to general education outcome 2.1, and through six to get to 2.2. Again, the institutional outcomes were designed without direct reference to the general education outcomes, hence there is no overarching design logic to the structure.
In this area is the first of the gaps. The non-laboratory science elective courses in the general education core under 3.4 are not necessarily quantitative reasoning courses. They are content courses with content knowledge such as geology. Institutional learning outcome eight is clearly quite sharply focused only on quantitative reasoning, not knowledge such as identifying rock types, plant diversity, or parts of a cell. Identifying the part of a cell is not a quantitative activity.
The same gap applies for 3.5. The proposed institutional learning outcome may still not be sufficiently comprehensive, but provides cover for 3.4 and 3.5 while remaining flexible enough to be used by programs outside of mathematics and sciences.
Institutional learning outcome five
| Proposed iSLOs. | Current iSLOs | Gen Ed pSLOs | Gen ed courses in area |
4. Intercultural knowledge and competence: Demonstrate a set of cognitive, affective, and behavioral skills and characteristics that support effective and appropriate interaction in a variety of cultural contexts supported by knowledge of people, places, and their cultures | No directly corresponding iSLO | 4.1 Demonstrate a fundamental knowledge of world geography. | [Although SS 120 Geography covers this learning outcome, SS 120 is not a specific requirement of the general education program. Not all students will complete this course. Although other courses could conceivably cover geography, there is no guarantee that each and every student in the general education program achieves this program learning outcome. |
5. Intercultural knowledge and competence: a set of cognitive, affective, and behavioral skills and characteristics that support effective and appropriate interaction in a variety of cultural contexts. | 4.2 Demonstrate knowledge of civic and cultural background of a person’s own culture, including its origins and development, assumptions, and predispositions | [This particular outcome appears to be designed to be met by SS 111 Cultural Anthropology which a student might take, but would not have to take] |
4.3 Demonstrate knowledge of major historical and contemporary events affecting one’s culture and other cultures as well
as their own diverse positions on selected democratic values and practices. | SS 150 |
| No directly corresponding iSLO | 4.4 Participate in a community project which identifies contemporary and global issues encountered and personal insights
gained from this experience and which identifies an economic, environmental, or public health challenge. | No course specifically identified, although this outcome might also be met by activities outside of courses such as via student organizations. |
| No directly corresponding iSLO | 4.5 Demonstrate an understanding of major ethical concerns. | No specific course addresses ethical concerns, this outcome may also be met via other units at the college. |
Institutional learning outcome five was perhaps intended to encompass those general education learning outcomes which primarily serve the social sciences. This another area where there are mapping issues. Social science course SS 150 History of Micronesia is the single required general education course, whether that one course can serve all five general education outcomes is beyond the scope of this report. Here my intent was to design one encompassing institutional learning outcome.
Although only a single course in the general education requirements specifically references this area, other courses can have course learning outcomes that are listed under one of the general education learning outcomes. Ethnobotany includes a course learning outcome that reports through general education outcome 4.2. While not a required course per se, the point is that other non-required courses may fill out the blanks in the fourth column of the table above.
General education program learning outcomes 5. 1 and 5.2
| Proposed iSLOs. | Current iSLOs | Gen Ed pSLOs | Gen ed courses in area |
5: Engage in Healthy Behaviors: engage in healthy physically fit lifestyles, and in physical activities that promote health and wellness of individuals, teams, and their communities. | No directly corresponding iSLO | 5.1 Determine healthy lifestyles by describing the value of physical activity to a healthful lifestyle and participating in regular physical activity for at least one semester. | ESS |
| No directly corresponding iSLO | 5.2 Demonstrate professionalism, interpersonal skills, teamwork, leadership and decision making skills. | ESS |
These two general education program learning outcomes arguably have no institutional learning outcome to which to report. This is an area often seen elsewhere in other institutions as an institutional level outcome, but the eight current institutional learning outcomes are silent on physical fitness, activity, and healthy lifestyles. Teamwork, leadership, and decision making skills might be inferred from some of the existing institutional outcomes, but none of the eight extant institutional learning outcomes contain that specific language.
Thus the five institutional outcomes include a new healthy behaviors institutional outcome. Without this, the mapping in Canvas cannot proceed. This area alone made the set of eight simply not viable - program outcomes must live in a group, a folder, above them at the institutional level. This one area essentially forced the use of the proposed institutional learning outcomes from a technical perspective. That said, these five should not be thought of as being new institutional learning outcomes. They are merely a reformatting of the existing learning outcomes. With the possible exception of seven.
Institutional learning outcome seven
| Proposed iSLOs. | Current iSLOs | Gen Ed pSLOs | Gen ed courses in area |
| 7. Foundations and skills for life-long learning: purposeful learning activity, undertaken on an ongoing basis with the aim of improving knowledge, skills, and competence. | [No directly corresponding pSLO.] | [?] |
Retired president of the Accrediting Commission for Community and Junior Colleges and former senior vice president of the Western Association of Schools and Colleges Dr. Richard Winn in a keynote address to California Southern University said, "I've seen many schools in my work with WASC that have put in their catalog 'We will produce lifelong learners,' and I've found that in most cases they haven't a clue what that means." To measure whether a student is a lifelong learner would require measuring the student's knowledge, skills, and competence across a lifetime.
If one wishes, one can argue that the foundations and skills for lifelong learning are communication, critical thinking and problem solving, qualitative and quantitative reasoning, intercultural knowledge and competence, and a healthy body to support a lifetime of learning. Those are the foundations. Those can be measured while the student is in attendance. Whether they continue to use that knowledge and those skills on an ongoing basis is not something the college can guarantee as the student walks across the graduation stage. This institutional outcome is encompassed in the evaluation of the five outcomes above.
Mapping
With the exception of seven, the above provides for mapping existing program learning outcomes to the proposed program learning outcomes. Those few programs, if any, reporting to institutional learning outcome seven can usually be logically placed under one of the five above.
In Canvas the institutional learning outcomes and some of the program learning outcomes can be seen above. Canvas refers to folders as "Groups"
In the above image the program learning outcome groups can be seen to hold course level learning outcomes as per the structure seen below:
Note that a course cannot be the unit that maps to a program learning outcome as courses have course level outcomes that map to different program learning outcomes. One complication at present is that outlines do not specify to which program learning outcome a course level outcome maps. The current outlines map specific learning outcomes under each course level outcome directly to a program learning outcome. This would require putting specific learning outcomes into program learning outcome folders. There would be hundreds of outcomes in each program folder, navigation would be challenging at best and maintenance - keeping the specific learning outcomes up to date - would be a nightmare.
Elsewhere I have written about the problems this brings to a well designed aggregating hierarchy. I have also
previously addressed the decision not to attempt to include specific learning outcomes in the institution wide outcomes bank.
At the course level a faculty member might see something such as the above. To find an institutionally installed outcome the faculty member would click on Find.
This opens up a Find Outcomes drill down dialog box.
Clicking on Account standards will show two folders in this case. Sandbox is the course I am in at present and would be my own outcomes, perhaps specific learning outcomes. The college folder contains the institutionally installed outcomes.
This is the drill down process that was mentioned earlier. A faculty member would have to know where to go to find their course learning outcome. Outlines should map only to program learning outcomes, a separate document ought to map from the program to the institutional level.
Provided that the faculty member can find their way to the desired program learning outcome...
The faculty member will find their course learning outcome in that folder or group.
Selecting the outcome will provide a description and the rating scale.
If the outcome is the desired outcome, then the faculty member imports that to their course.
The outcomes can be included in rubrics for marking assignments. Rubric can include a mix of criteria specified by the instructor and course level outcomes provided by the institutional outcomes bank.
One key concept to keep in mind: Only outcomes installed at the institutional level can be aggregated across all courses. Faculty will have to be trained to use the institutional outcomes on rubrics to provide data for assessment. Outcomes that are created by individual faculty in their own courses cannot be aggregated. As Kansas State University has noted, they only bring a program into Canvas once that program already has all of their program and course learning outcomes set up in the institutional level outcomes bank. This ensures the university will capture the necessary assessment data.
There are some other caveats. At present outcomes are primarily assessed through rubrics. There is a mechanism for assessing outcomes using classic quizzes, but that requires setting up question banks and linking the banks to outcomes. Classic quizzes deprecates in late 2022. New quizzes are able to directly attach outcomes to the quiz or to individual questions. Mastery results are reported only for that quiz, at present these do not flow up into the mastery reporting system for the course. This is expected to be remedied either later this year or early next year. The short answer is the rubrics drive assessment data at present.
This work was not done completely in isolation. A working group under the assessment team worked on the institutional outcomes with respect to the general education program learning outcomes back in October 2019. The working group provided valuable input at that time on this work and I owe a debt of thanks to them. I also owe a debt of thanks to many others who have continued to support my work, to engage in vigorous discussions around learning, and who have encouraged me.
Comments
Post a Comment