Assessing Learning in Physical Science
SC 130 Physical Science proposes to serve two institutional learning outcomes (ILO) through four general education program learning outcomes (GE PLO) addressed by four course level student learning outcomes (CLO). This report assesses learning under the course level learning outcomes which in turn support program and institutional learning outcomes.
Note that this course has a focus on "doing" science, on science as a process, a way of understanding the natural physical world and the mathematics that underlies many physical systems. The course does not focus on memorized facts. The course is centered on science as being that which can be measured, observed, evidenced. The course is intended as a counter to memorized science. Once one shifts to memorized facts as the basis of a science, then any set of memorized facts can be seen by the learner as equally valid. Somewhere down at the bottom of that slope are those who are convinced the earth is flat, climate change is not happening, evolution does not explain the diversity of life, and no one actually walked on the moon. By doing simple experiments that seek to measure physical properties and quantities, by gathering and analyzing data, the intent is that students come to see science as a way of thinking about and analyzing the world around them.
The course directly supports institutional learning outcome eight and three general education outcomes.
ILO 8. Quantitative Reasoning: ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations; comprehends and can create sophisticated arguments supported by quantitative evidence and can clearly communicate those arguments in a variety of formats.
ILO 2. Effective written communication: development and expression of ideas in writing through work in many genres and styles, utilizing different writing technologies, and mixing texts, data, and images through iterative experiences across the curriculum.
Laboratory fourteen was assessed to determine whether students properly recorded data in a table, generated xy scatter graphs, made a decision on the appropriate mathematical model and, if deemed to be a linear relationship, added linear trend lines to the graph, reported the slopes in their analysis, and discussed the results.
Note that while the course had 28 students enrolled at term end, one student had stopped attending by midterm. A second student was in hospital giving birth to a baby girl. Two other students were absent on the day of laboratory 14 and did not submit a report. Of the remaining 24 students, 18 submitted a report.
Although laboratories one to thirteen can be submitted late, laboratory fourteen is at the end of the term and provides only a single week to get a report submitted. This tends to contribute to a lower submission rate for this laboratory.
In the past an item analysis of the final examination has provided information on the extent to which students are able to define and explain concepts, theories, and laws in physical science. The adoption Schoology Institutional provides an alternate insight into achievement of this outcome. This outcome was assessed 14 times during the term. Not all students were present for all 14 assessments.
The mastery settings are 70% to meet learning expectations, 80% to exceed learning expectations on that assignment. At some deeper level this is still a form of assessing learning outcomes based ultimately on percentage achievement on a particular assignment, rather than a purely binary "the student can/cannot demonstrate the required skill/knowledge/value.
Whether learning outcomes are purely binary is a matter of discussion. On an outcome, "The student will be able to construct a cabinet" in theory the student can either construct a cabinet or cannot. The complication comes in that not all cabinets will necessarily be of equal quality. At some point one is left determining what the minimum acceptable quality is for a cabinet. Quality opens up gray areas between the binary options.
In the mastery settings the students are deemed to have crossed that "binary" threshold by exceeding 70% on five or more assignments. Schoology denotes this with a green star. Noting that the class consists of two sections, one has the following results.
Note that seventeen of the twenty-eight students (61%) exceeded 70% on five or more assessments of course learning outcome two.
The focus in physical science is on science as a process, as a system of experimentally generated and verified knowledge, not as a collection of memorized facts. The course is not content free, but the heavier emphasis is on data gathering, fitting mathematical models, and writing up results in reports. Once science becomes a collection of memorized and regurgitated factoids, then all collections of memorized and regurgitated factoids are equally valid. At that point one is left choosing among factoids to "believe" in and the result are those who "do not believe in science" whether that science is climate change, evolution, or any other area of science. Thus the course focuses on knowledge generated by the students and is guided in part by the concepts of non-overlapping magisteria and a constructivist epistemology.
Noting that the focus of the course has been on the laboratories, on "doing" science as a process and not on specific facts nor memorization, the final examination had been a vehicle only for assessing course learning outcome two. The mastery information provides a alternate and perhaps more authentic way to assess course learning outcome two. As a result, the final examination was redesigned and re-purposed to act more as a reflective wrapping up of the course than as a vehicle for comprehensive assessment.
While the laboratory fourteen assessment above provides some data on this course learning outcome, this outcome supports the general education program learning outcome "3.2 Present and interpret numeric information in graphic forms." With this focus in mind, a pre-assessment and post-assessment was included in the course. In the past the post-assessment was embedded in the final examination, as seen in the final examination of fall 2017. With the decision to redesign and re-purpose the final examination, the post assessment was integrated into a test at the end of the week fourteen.
A earlier report covered improvement on this learning outcome from the pre-assessment to the post-assessment.
Course level learning outcome four focuses on communication, specifically writing. In the late 1990's assessment data at the college suggested some students were graduating with limited writing communication skills. As noted by the languages and literature division at that time, two college level writing courses in the general education core cannot by themselves produce collegiate level writers. Writing must occur across the curriculum, across disciplines. In 2007 SC 130 Physical Science at the national campus was redesigned to put an emphasis on writing. A "fill-in the blank" cook book style laboratory manual was replaced by laboratories which led to laboratory reports constructed using spreadsheet and word processing software.
This term the adoption of Schoology Institutional provided access to the Google Drive Assignments app. An earlier article covered the use of Google Drive Assignments in the course.
Course level student learning outcome four was directly evaluated up to eight times on rubrics used to mark laboratory reports.
Of the 28 students in the course, 25 students (89%) met or exceeded expectations on this outcome. Of the three students who did not meet this outcome, one student had not attended the course after midterm. A second student was attending only irregularly and not submitting assignments. The third of the three was a pregnant student with a difficult pregnancy that also led to absences, low submission rates, and work that needed improvement.
The achievement chart is effectively an average performance and as an average is, in effect, related more to grades than to a "binary" demonstration of the learning outcome having been accomplished or not.
Nineteen students (68%) demonstrated student learning outcome four five or more times. Although demonstrating the outcome is set at a 70% performance on the underlying outcome, the demonstration of this level of performance five or more times separates this statistic from the achievement average above. This is closer to the binary assertion that "students will be able to demonstrate basic communication skills by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.
At the end of the term the students were asked to respond to the following questions:
1. Before I took this class my attitude towards science was:
☐ Positive: I liked science
☐ Neutral
☐ Negative: I did not like science
2. During the semester:
☐ I enjoyed physical science class
☐ Neutral
☐ I did not enjoy physical science class
3. During the semester:
☐ I was glad I signed up for physical science class
☐ Neutral
☐ I was not glad I signed up for physical science class
4. After taking this class my attitude towards science is:
☐ Positive: I like science
☐ Neutral
☐ Negative: I do not like science
Prior to the term 70% of the students self-reported attitudes towards science that were neutral or negative.
During the course 83% of the students surveyed responded that they enjoyed the course, 91% responded that they were glad they signed up for physical science. At term end 87% of the students had a positive attitude towards science.
As an instructor who has a passion for science, I would hope to encourage my students to also enjoy that passion. New scientists are not minted from students who dislike science and come away from the course with a negative attitude towards science. That the students enjoyed the course does not mean the course was easy, not with lab reports to be done each and every week. A demanding course can also be a positive experience, perhaps even enjoyable, perhaps fun.
A favorite laboratory was one that was fun and enjoyable, presented new information that the student did not previously know, was useful to the student, or was the first time for them to engage in a particular activity.
Students disliked a least favorite laboratory when the laboratory was confusing, difficult to understand, or perceived to be hard. Students also, curiously enough, disliked laboratories for which they were absent. This appears to be more of an expression of disliking that they missed the laboratory than a dislike of the content of the laboratory experience.
One student responded that they liked all of the laboratories and disliked none. No reasons were given.
Another student put that they liked none of the laboratories and disliked all of the laboratories. Their reason for this was a single word, "Reports." The student did not explain further, but their single word answers to other portions of the survey, answers that did not always make sense, suggest written communication challenges for the student. There is also the suggestion in some answers that the student might be tech adverse. This would make completing the laboratory reports particularly challenging.
Perhaps there was a time when one could hope to engage in a career without having to utilize software applications, technology. Yet even three decades ago technology was pervasive in the workplace. At this point being technology adverse is effectively disqualification from all but the lowest paying service industry positions. The use of multiple technologies in an integrative manner is the modern working world, and physical science actively prepares a student for this environment. As one student noted in response to what they liked about Google Docs, "Sharing." This is a new capability in a fully modern cloud based productivity software suite. This is a student who is prepared for the working world of the next thirty years, not the one that vanished forty years ago.
Overall, students who successfully completed the course are able to explore physical science systems through experimentally based laboratories using scientific methodologies; define and explain concepts, theories, and laws in physical science; generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science; and demonstrate basic written communication skills.
Note that this course has a focus on "doing" science, on science as a process, a way of understanding the natural physical world and the mathematics that underlies many physical systems. The course does not focus on memorized facts. The course is centered on science as being that which can be measured, observed, evidenced. The course is intended as a counter to memorized science. Once one shifts to memorized facts as the basis of a science, then any set of memorized facts can be seen by the learner as equally valid. Somewhere down at the bottom of that slope are those who are convinced the earth is flat, climate change is not happening, evolution does not explain the diversity of life, and no one actually walked on the moon. By doing simple experiments that seek to measure physical properties and quantities, by gathering and analyzing data, the intent is that students come to see science as a way of thinking about and analyzing the world around them.
The course directly supports institutional learning outcome eight and three general education outcomes.
ILO 8. Quantitative Reasoning: ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations; comprehends and can create sophisticated arguments supported by quantitative evidence and can clearly communicate those arguments in a variety of formats.
| GE PLO | SC 130 CLO |
|---|---|
| 3.5 Perform experiments that use scientific methods as part of the inquiry process. | 1. Explore physical science systems through experimentally based laboratories using scientific methodologies |
| 3.4 Define and explain scientific concepts, principles, and theories of a field of science. | 2. Define and explain concepts, theories, and laws in physical science. |
| 3.2 Present and interpret numeric information in graphic forms. | 3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science. |
Merenda tests for acids and bases
The course also supports institutional learning outcome two.
| GE PLO | SC 130 CLO |
|---|---|
| 1.1 Write a clear, well-organized paper using documentation and quantitative tools when appropriate. | 4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software. |
CLO 1
Explore physical science systems through experimentally based laboratories using scientific methodologies
For the purposes of evaluating this course learning outcome, laboratory fourteen was evaluated. Laboratory fourteen functioned as a laboratory practical examination. The students were given a system to investigate. The students were tasked with analyzing the system, determining whether the variables under investigation were related, and discussing their findings. Where previous laboratories often provided a clear goal, a specific quantity to be measured, laboratory fourteen provided less scaffolding. Previous laboratories provided section headers via Google Drive Assignments within Schoology for the required sections of a complete laboratory report, laboratory fourteen provided only a blank Google Docs document.
Some of the objects investigated in lab 14. The G2 Air would prove to be a radar stealth object
Laboratory fourteen was assessed to determine whether students properly recorded data in a table, generated xy scatter graphs, made a decision on the appropriate mathematical model and, if deemed to be a linear relationship, added linear trend lines to the graph, reported the slopes in their analysis, and discussed the results.
Note that while the course had 28 students enrolled at term end, one student had stopped attending by midterm. A second student was in hospital giving birth to a baby girl. Two other students were absent on the day of laboratory 14 and did not submit a report. Of the remaining 24 students, 18 submitted a report.
Although laboratories one to thirteen can be submitted late, laboratory fourteen is at the end of the term and provides only a single week to get a report submitted. This tends to contribute to a lower submission rate for this laboratory.
Analysis of laboratory fourteen fall 2018
Of 24 students, six students did not turn in laboratory fourteen. Eighteen of the 24 students produced laboratory reports with data recorded in a properly formatted table. Seventeen students also generated a correct xy scattergraph. Nine students ran a linear regression analysis using Desmos and of those nine, eight generated a reasonably complete discussion of the meaning of the slope and the results of the laboratory.
Performance on laboratories across multiple terms
Statistics are given context by knowing the history of the value being measured. Although fall 2016 saw an unusually high turn-in rate, since then the success rates for each metric have trended in a consistent rank order. Inter-term variability is high, with some suggestion that the fall terms might be performing more strongly than the spring terms. While eighteen laboratory reports submitted for laboratory fourteen out of 28 registered students might seem low, the submission rate is in line with historic values for laboratory fourteen.
Laboratory submission rates fall 2018 versus long term average rate since fall 2014
Laboratory report completion rates for each laboratory fall 2018 were slightly lower than historic average completion rates. Laboratories four, five, and six were anomalous in their low turn-in rate this term. Each term sees some slippage in the turn-in rate from lab-to-lab during the term. Some of this reflects students who opt to no longer attend class but have not officially withdrawn from the course. Some of this is students who are still actively attending but not completing and submitting heir laboratory assignments.
Laboratory four this term included no explanatory scaffolding and was done as an open ended discovery learning laboratory. The amount of explanatory scaffolding varies from term-to-term, this term I opted to provide only the most minimal of structure and did not try to steer any of the work in later part of the laboratory. I wanted to see where the students would go with the system.
Laboratory five proved more confusing for the students this term than in past terms. I remain unclear as to why this occurred.
Laboratory six demonstrated Newton's law of cooling, but remains a laboratory that could be developed further. The cooling curves of different liquids might be attempted in future terms.
The rest of the laboratories performed generally on par with historic submission rates.
CLO 2
2. Define and explain concepts, theories, and laws in physical science.In the past an item analysis of the final examination has provided information on the extent to which students are able to define and explain concepts, theories, and laws in physical science. The adoption Schoology Institutional provides an alternate insight into achievement of this outcome. This outcome was assessed 14 times during the term. Not all students were present for all 14 assessments.
Mastery settings in Schoology
Whether learning outcomes are purely binary is a matter of discussion. On an outcome, "The student will be able to construct a cabinet" in theory the student can either construct a cabinet or cannot. The complication comes in that not all cabinets will necessarily be of equal quality. At some point one is left determining what the minimum acceptable quality is for a cabinet. Quality opens up gray areas between the binary options.
In the mastery settings the students are deemed to have crossed that "binary" threshold by exceeding 70% on five or more assignments. Schoology denotes this with a green star. Noting that the class consists of two sections, one has the following results.
Section one and two learning outcome mastery
Note that seventeen of the twenty-eight students (61%) exceeded 70% on five or more assessments of course learning outcome two.
The focus in physical science is on science as a process, as a system of experimentally generated and verified knowledge, not as a collection of memorized facts. The course is not content free, but the heavier emphasis is on data gathering, fitting mathematical models, and writing up results in reports. Once science becomes a collection of memorized and regurgitated factoids, then all collections of memorized and regurgitated factoids are equally valid. At that point one is left choosing among factoids to "believe" in and the result are those who "do not believe in science" whether that science is climate change, evolution, or any other area of science. Thus the course focuses on knowledge generated by the students and is guided in part by the concepts of non-overlapping magisteria and a constructivist epistemology.
Noting that the focus of the course has been on the laboratories, on "doing" science as a process and not on specific facts nor memorization, the final examination had been a vehicle only for assessing course learning outcome two. The mastery information provides a alternate and perhaps more authentic way to assess course learning outcome two. As a result, the final examination was redesigned and re-purposed to act more as a reflective wrapping up of the course than as a vehicle for comprehensive assessment.
CLO 3
3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science.While the laboratory fourteen assessment above provides some data on this course learning outcome, this outcome supports the general education program learning outcome "3.2 Present and interpret numeric information in graphic forms." With this focus in mind, a pre-assessment and post-assessment was included in the course. In the past the post-assessment was embedded in the final examination, as seen in the final examination of fall 2017. With the decision to redesign and re-purpose the final examination, the post assessment was integrated into a test at the end of the week fourteen.
A earlier report covered improvement on this learning outcome from the pre-assessment to the post-assessment.
CLO 4
4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.Course level learning outcome four focuses on communication, specifically writing. In the late 1990's assessment data at the college suggested some students were graduating with limited writing communication skills. As noted by the languages and literature division at that time, two college level writing courses in the general education core cannot by themselves produce collegiate level writers. Writing must occur across the curriculum, across disciplines. In 2007 SC 130 Physical Science at the national campus was redesigned to put an emphasis on writing. A "fill-in the blank" cook book style laboratory manual was replaced by laboratories which led to laboratory reports constructed using spreadsheet and word processing software.
Typical head end of a laboratory report in Google Docs
This term the adoption of Schoology Institutional provided access to the Google Drive Assignments app. An earlier article covered the use of Google Drive Assignments in the course.
Course level student learning outcome four was directly evaluated up to eight times on rubrics used to mark laboratory reports.
Achievement on course student learning outcome four
The achievement chart is effectively an average performance and as an average is, in effect, related more to grades than to a "binary" demonstration of the learning outcome having been accomplished or not.
Number of students who demonstrated outcome five or more times
Affective Domain Assessment
SC 130 Physical Science functions as a science with laboratory requirement for the general education core at the college. The only major required to take the course is an associate of applied science in telecommunications technology two year program. This term no students in the telecommunications technology program were in the course. Of the 28 students in the course, eight were in agriculture and natural resources management, three were in marine sciences. The course is not a requirement for either major. Seventeen students were in non-science majors. One of the intents of the course is to reach out to students who may not enjoy science as a subject and show them science in a new light.At the end of the term the students were asked to respond to the following questions:
1. Before I took this class my attitude towards science was:
☐ Positive: I liked science
☐ Neutral
☐ Negative: I did not like science
2. During the semester:
☐ I enjoyed physical science class
☐ Neutral
☐ I did not enjoy physical science class
3. During the semester:
☐ I was glad I signed up for physical science class
☐ Neutral
☐ I was not glad I signed up for physical science class
4. After taking this class my attitude towards science is:
☐ Positive: I like science
☐ Neutral
☐ Negative: I do not like science
Student attitudes towards science, 23 students completed the survey
As an instructor who has a passion for science, I would hope to encourage my students to also enjoy that passion. New scientists are not minted from students who dislike science and come away from the course with a negative attitude towards science. That the students enjoyed the course does not mean the course was easy, not with lab reports to be done each and every week. A demanding course can also be a positive experience, perhaps even enjoyable, perhaps fun.
Affective Domain Assessment of Apps Used During Course
The physical science course utilized three core applications during the term. Desmos calculator was used to handle data tables, graphs, and analysis. Google Docs was used for laboratory reports, And Schoology was used by students for receiving and submitting assignments. For each application the students were asked open ended questions as to what they liked and disliked about the application. Answers were then grouped into categories determined by the answers themselves.
Overall the students favored the use of the applications and found the applications to be easy to use and helpful. The students especially liked the east of use of Desmos, the automatic saving nature of Google Docs, and the ability to stay up to date on grades and assignments in Schoology.
Favored and Least favored laboratories
Some insight into the impact of the laboratories on learning can be gleaned from the reaction students had to the laboratories. At term end, the students were asked to provide feedback on a favorite laboratory and a least favorite laboratory. The feedback fell naturally into a few categories.
One student responded that they liked all of the laboratories and disliked none. No reasons were given.
Another student put that they liked none of the laboratories and disliked all of the laboratories. Their reason for this was a single word, "Reports." The student did not explain further, but their single word answers to other portions of the survey, answers that did not always make sense, suggest written communication challenges for the student. There is also the suggestion in some answers that the student might be tech adverse. This would make completing the laboratory reports particularly challenging.
Perhaps there was a time when one could hope to engage in a career without having to utilize software applications, technology. Yet even three decades ago technology was pervasive in the workplace. At this point being technology adverse is effectively disqualification from all but the lowest paying service industry positions. The use of multiple technologies in an integrative manner is the modern working world, and physical science actively prepares a student for this environment. As one student noted in response to what they liked about Google Docs, "Sharing." This is a new capability in a fully modern cloud based productivity software suite. This is a student who is prepared for the working world of the next thirty years, not the one that vanished forty years ago.
Overall, students who successfully completed the course are able to explore physical science systems through experimentally based laboratories using scientific methodologies; define and explain concepts, theories, and laws in physical science; generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science; and demonstrate basic written communication skills.
Comments
Post a Comment