General education program learning assessment workshop discussion postscript
This article is really just a set of breadcrumbs for future reference to documents related to discussions I happened to have today in a workshop. Everything in this article is solely my own opinion at this point. This is really a waypoint on a journey that began in 1998 and which evolved into a vision in 2006 for aggregating learning outcome performance. In 2006 the software to make that vision a reality was not known to me. Roughly six years later Nuventive TracDat would provide a way to potentially implement the vision of 2006. The journey of aligning the outcomes stack, which was misaligned even in 2006, continues.
The general education program assessment plan is to continue to report course student learning outcome results in Nuventive and to aggregate those results in Nuventive to report on achievement of all general education program learning outcomes.
A guiding principle is that the general education program learning outcomes are the only outcomes each and every student can be demonstrated to have met when the student walks across the graduation stage. These are the only outcomes in common among all students. Quoting myself from an earlier communication on 05 July 2024:
While the document purports to map courses to program learning outcomes, courses do not map to program learning outcomes, course learning outcomes map to program learning outcomes as seen in the following diagram from a presentation:
The group in the workshop today noted that the ESS course learning outcomes do not map to the current institutional learning outcomes. I explained that a working group in 2019 produced a gap analysis and recommendation of new institutional learning outcomes in a document.
Course learning outcomes can be evaluated in Instructure Canvas using either rubrics or question banks with course learning outcomes. A video explains how to do this. An article explains how to extract the course learning outcomes from Canvas and convert them using a spreadsheet into a format that can be entered into Nuventive.
Provided Nuventive is mapping the course learning outcome results to the program learning outcomes, then the general education learning outcomes are being assessed each and every term. Which they must be in order to support the position that each and every graduate crossing the graduation stage every December and May has mastered the general education outcomes as well as the proposed five institutional learning outcomes.
In regards to the current eight institutional learning outcomes, I do not believe that the college can demonstrate institutional learning outcome seven because the measurement must happen over a lifetime. As Dr. Richard Winn, former Senior vice President of WASC and later the ACCJC president, noted, "I have seen many schools in my work with WASC put in their catalog 'we will produce lifelong learners' and I found that in most cases they have no clue what that means." (2015 Keynote Speaker - Dr. Richard Winn)
The proposed five institutional learning outcomes being based in the general education outcomes was also the result of a comparison between general education and institutional outcomes at other institutions.
A look at the general education program learning outcomes in Nuventive, however, produces a puzzle.
I have reported course learning outcomes GE_PSLO_3.4 Define and explain scientific concepts, principles, and theories of a field of science and GE_PSLO_3.5 Perform experiments that use scientific methods as part of the inquiry process every term since the earliest days of TracDat usage. Yet the table above, which is set to display values from all terms, shows no students assessed.
A drop down list of courses suggests that perhaps not all courses are being mapped. Which is especially puzzling as there are matrices configured for the courses.
Yet the matrix is also puzzling. SC130_CSLO_1 Explore physical science systems through experimentally based laboratories using scientific methodologies should map only to GE_PSLO_3.5 Perform experiments that use scientific methods as part of the inquiry process and not to the other outcomes shown in the mapping. The same is true for the other three outcomes. The above matrix appears to have each and every course learning outcome mapped to each and every one of four program learning outcomes. Unless I misunderstand the way the mapping matrix operates.
If the matrices are set up correctly, and the results for all general education courses are included in the dashboard results, then data should be available to report on the general education program learning outcomes each and every term.
Nuventive was designed and intended for the aggregation of course level data up to the program and then institutional learning outcomes through these mapping matrices. This was a feature of Nuventive nee TracDat that impressed me and led to my recommending to the assessment committee that the college adopt TracDat back in 2012 or thereabouts. I could see that faculty would only need to measure course learning outcomes, and the program learning outcome results would flow from data values aggregated up through the matrices.
On a separate but related topic, I understand the concern that course level learning outcome assessments might not be aligned with program level learning outcome assessments. Such as a program learning outcome that demands demonstration of oral skills being assessed at the course level by a quiz or test. For me the solution is two fold.
One, the assessment at the course level is what should be fixed. The fix should not be doing a separate activity at the program level to separately assess the program learning outcome. The course should generate the appropriate assessment data. This is a course level issue to be resolved at the course level, not by additional assessments at the program level. When someone says that they are assessing some program outcomes during some terms and some program learning outcomes other terms, a part of me cringes. Every program learning outcome can be assessed every term if data is aggregated from the course level. At least as far I comprehend the capabilities of Nuventive.
Two, the CSLO Assessment types specified now at the course level do not make sense to me given the capabilities of Canvas. Canvas can report assessment data from rubrics or tests, with assessment data being pulled from multiple assignments and/or quizzes and tests. Assignment that might include laboratories, classwork, group work, journals. Assessments that cross-cut many categories.
Here is a performance chart for SC 130 for the four course learning outcomes. All four are measured by rubrics attached to assignments. Course level outcomes two and three are also measured by questions on quizzes.
Underneath the chart are 1,309 measurements of student learning outcomes. These cannot be summarized under a single assessment type other than to say that they derive from a mix of rubrics used to evaluate assignments and quiz questions. All one can say is that demonstration of learning was done via multiple elements directly related to the specified course learning outcome.
In today's workshop I also mentioned that the assessment committee had a track record of meeting for a period of time punctuated by periods of not meeting that I first documented in 2003. The eventually proposed solution was to integrate the assessing authority into the deploying authority: to integrate assessment into curriculum. That proved to be too much work for the curriculum committee. The assessment committee would be reconstituted in August 2016 and would continue to function into 2021.
Also mentioned in the workshop today was institutional memory. Assessment as an institutional focus at the college began in a 1998 journey to a workshop in Guam. The above picture was taken using the college's very first digital camera.
Comments
Post a Comment