Assessing Learning in Physical Science
SC 130 Physical Science proposes to serve two institutional learning outcomes (ILO) through four general education program learning outcomes (GE PLO) addressed by four course level student learning outcomes (CLO). Not listed are proposed specific student learning outcomes that in turn serve the course level learning outcomes. This report assesses learning under the proposed course level learning outcomes which in turn supports program and institutional learning outcomes.
ILO 8. Quantitative Reasoning: ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations; comprehends and can create sophisticated arguments supported by quantitative evidence and can clearly communicate those arguments in a variety of formats.
Explore physical science systems through experimentally based laboratories using scientific methodologies
Laboratory fourteen has been used in previous terms to provide a vehicle for assessing this course level outcome. In that laboratory students were given a system that was unfamiliar to them and asked to determine the underlying mathematical models for the objects in the system. Due to a collision of holidays with Thursdays this term, laboratory fourteen was omitted. For the purposes of evaluating this course learning outcome, laboratory nine was evaluated. Laboratory nine focused on measuring the speed of sound by timing the delay between seeing boards clapped together and arrival of the sound of the boards clapping.
The laboratory reports were assessed to determine whether students properly recorded data in labelled tables, generated xy scatter graphs, made a decision on linearity and if deemed to be a linear relationship added linear trend lines to the graph, reported the slopes in their analysis, and discussed the results. Twenty-five of twenty-six students submitted this laboratory.
2. Define and explain concepts, theories, and laws in physical science.
The final examination asked students to plot data, interpret graphs, make calculations, and define concepts (facts) in physical science. This term the course generated 2043 points, with 1228 of those points in laboratory reports.
The focus in physical science is on science as a process, as a system of experimentally generated and verified knowledge, not as a collection of memorized facts. The course is not content free, but the heavier emphasis is on data gathering, fitting mathematical models, and writing up results in reports. This reflects my own biases. Once science becomes a collection of memorized and regurgitated factoids, then all collections of memorized and regurgitated factoids are equally valid. At that point one is left choosing among factoids to "believe" in and the result are those who "do not believe in science" whether that science is climate change, evolution, or any other area of science. Thus the course focuses on knowledge generated by the students and is guided in part by the concepts of non-overlapping magisteria and a constructivist epistemology.
Given the above, the final examination was not weighted and would not be weighted. At only 63 points, the final examination was only 3% of a student's overall mark. By the time of the final examination the work throughout the term, especially the laboratory work, has assessed the students. The students were aware of the low impact of the final examination and understood that the test would not have a large impact. High stakes testing is rarely a reliable measure of longer term knowledge retained. Performance levels may be lowered by these factors, but the hope is that the results more closely represent retained knowledge than memorized, regurgitated, and then forgotten knowledge.
On the final exam six male students outperformed sixteen female students. Note that one male student did not sit the final examination, that student had stopped attending class in early April. The difference in performance is most likely to be a result of the small sample sizes involved, especially for the male students. The course averages are a statistical tie. As a result of the stronger performance by the males, the item analysis shows higher averages for the males on an item-by-item basis.
Average success rates based on an item analysis of the final examination were aggregated by three skill areas: analysis and interpretations of graphs, citing of facts, and calculations. Aggregate average success rates based on an item analysis of the final examination were generally low but in line with pre-existing values.
Graphing skills were the strongest area for the students, calculations, recollection of facts, and inferences performed more weakly.
The overall final examination average for the 22 students who took the examination, based on an item analysis, was a 46% success rate. While the previous two terms of had seen an improvement from the spring 2016 success rate of 51%, this spring term the success rate fell to the second lowest since 2010. The students, however, were aware that the final examination was not intended to be a "make or break" high stakes grade that would impact the work they had already done. The students knew that the emphasis and value were placed on a "doing" science in the laboratory.
While the laboratory nine assessment above provides some data on this course learning outcome, this outcome supports the general education program learning outcome "3.2 Present and interpret numeric information in graphic forms." With this focus in mind, a pre-assessment and post-assessment was included in the course. The post-assessment was embedded in the final examination.
SC 130 Physical Science includes a focus on the mathematical models that underlie physical science systems. Laboratories one, two, five, seven, nine, eleven, and twelve have linear relationships. A number of assignments in the course also have linear relationships. The students also encounter a quadratic relationship in laboratory three, now modeled as a quadratic using Desmos' ability to regress to any arbitrary function. By the end of the course students have repeatedly worked with linear relations. One relationship, one equation, at a time, not "problems one to thirty, even problems only." Every equation is built from data that the students have gathered. From the concrete to the abstract, repeated throughout the term, providing cognitive hooks on which to "hang" their mathematical learning.
A performance boost from a success rate of 47% on the pre-assessment to 77% on the post-assessment is documented in another article in this blog.
The post-assessment does not answer whether there will be long term student retention of the ability to present and interpret numeric information in graphic forms. The course has had, at least in the short run, a positive impact on the students' ability to work with numeric information in graphical forms.
Course level learning outcome four focuses on communication, specifically writing. In the late 1990's assessment data at the college suggested some students were graduating with limited writing communication skills. As noted by the languages and literature division at that time, two college level writing courses in the general education core cannot by themselves produce collegiate level writers. Writing must occur across the curriculum, across disciplines. In 2007 SC 130 Physical Science at the national campus was redesigned to put an emphasis on writing. A "fill-in the blank" cook book style laboratory manual was replaced by laboratories which led to laboratory reports constructed using spreadsheet and word processing software.
This term the laboratory report marking rubrics added student learning outcomes from the proposed* course outline:
CLO 4.0 Communication skills: Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.
SLO 4.1 Produce laboratory reports: Produce laboratory reports with integrated tables, charts, and graphs
SLO 4.2 Use technology to produce laboratory reports: Use spreadsheets and word processing software
SLO 4.3 Effective communication: Communicate in writing using proper syntax and correctly used physical science vocabulary.
These were marked on a four point scale:
4 Exceeds expectations
3 Meets expectations
2 Does not meet expectations: needs improvement
1 Severe does not meet expectations: no evidence of outcome being met
Laboratory reports for laboratory nine were marked against the above metrics with the following results:
By the end of the term these 13 students could produce a laboratory report with tables and charts integrated from a Desmos. In the past students have used spreadsheets to lay out their tables and graphs. This term the ability of Desmos to regress to arbitrary functions using arbitrary variables drove the use of Desmos throught the course. Where a spreadsheet graphically shows a regression to y = mx +b, Desmos can be told to regress to d = vt using the notation d1~vt1, producing a value for v.
The result is that the student obtains results such as v = 353.37 which is in meters per second. This made sense to the students and all of the students shifted to using Desmos in lieu of a spreadsheet. As Desmos does not currently interact with software such as Google Docs, screenshots were used to put tables and graphs from Desmos into the laboratory reports.
Note that were the author to have access to Schoology Institutional version a more thorough analysis would be possible. Schoology Institutional provides access to a screen that reports on mastery against standards, learning outcomes, that is not available in the free version. Analysis in the free version is a manual tabulation process and difficult to extend beyond the analysis of a single laboratory report.
Although this spring term showed some weaknesses in learning against prior terms, performance on the four learning outcomes was still within a range that might be attributed to a return to the mean. The impact of two multi-day holidays late in the term may also have negatively impacted learning among the students. And while the shift to Desmos early in the term appeared to be beneficial to the students, the term did not start with only Desmos being used. During the early part of the term the instructor was also learning some of the capabilities of Desmos. The use of Desmos from day one should prove beneficial to student performance and comprehension.
* The current course outline actually dates to 2007. Up to 2007 an out-of-date 88 outcome list of science facts constituted the SC 130 Physical Science outline. In 2007 that outline was essentially compressed into four broad subject areas as a first step in revising the outline and running the course with a writing focus and a new textbook. As the course was being taught in 2007 to 2010, work continued on revamping the outline. In 2011 the college undertook reformatting all outlines at the college, but instructors were instructed to not change content at that time. That meant that the outline approved in 2012 was still the outline designed in 2007. Each year since 2012 a revised outline has been prepared and submitted. The proposed outline has yet to be considered by the appropriate bodies. The course covers the material specified on both outlines, there has not been a loss of coverage of the 2007 specified materials. This assessment report, however, is organized around the proposed outline.
ILO 8. Quantitative Reasoning: ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations; comprehends and can create sophisticated arguments supported by quantitative evidence and can clearly communicate those arguments in a variety of formats.
| GE PLO | SC 130 CLO |
|---|---|
| 3.5 Perform experiments that use scientific methods as part of the inquiry process. | 1. Explore physical science systems through experimentally based laboratories using scientific methodologies |
| 3.4 Define and explain scientific concepts, principles, and theories of a field of science. | 2. Define and explain concepts, theories, and laws in physical science. |
| 3.2 Present and interpret numeric information in graphic forms. | 3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science. |
Anjannet, Regina, and Mandylae experiment with a circuit
ILO 2. Effective written communication: development and expression of ideas in writing through work in many genres and styles, utilizing different writing technologies, and mixing texts, data, and images through iterative experiences across the curriculum.
| GE PLO | SC 130 CLO |
|---|---|
| 1.1 Write a clear, well-organized paper using documentation and quantitative tools when appropriate. | 4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software. |
CLO 1
Explore physical science systems through experimentally based laboratories using scientific methodologies
Timers
Distance measuring, Anjannet
The laboratory reports were assessed to determine whether students properly recorded data in labelled tables, generated xy scatter graphs, made a decision on linearity and if deemed to be a linear relationship added linear trend lines to the graph, reported the slopes in their analysis, and discussed the results. Twenty-five of twenty-six students submitted this laboratory.
Analysis of laboratory nine
Of 23 students, eight students did not turn in laboratory nine. Submission rates for laboratory reports fell as the term progressed during the spring term. 15 of the 23 students produced laboratory reports with data recorded in a properly formatted table. Those 15 students also generated a correct xy scattergraph. While all 15 would then also run a regression using Desmos, the students were also expected to explain the results of the regression analysis.
The result of the 8:00 section data can be seen above, with a sound speed of 363 m/s. Given the temperature that day, the sound speed was up around 348 to 350 m/s. The fifteen students knew to use Desmos to generate the above image. The students should then have progressed to reporting in an analysis that the relationship is linear and reported the speed of sound based on the slope of the relationship. Eleven students of the 15 went on to describe the nature of the relationship, only seven of those then went on to produce a reasonable discussion of the results.
The view of the clappers from the timer's position
The number of students completing laboratory nine was 83%, on par with the current average submission rate of 81%. Still, at 83% the number of laboratories submitted is the third lowest by percentage.
Laboratory nine was selected in part because later laboratories saw an even lower submission rate. This tendency of students to back off on their level of effort may have been exacerbated by dual sets of multiple day holidays that landed two weeks apart in the last week of March and the second week of April. Laboratories 11 and 12 would land just prior to and between these holidays and would see a 70% and 78% submission rate respectively, suggesting that the holidays were involved in negatively impacting submission of laboratory reports. At least one local school has dropped spring break, taking only the Friday of that week off from classes. Spring break was related to northern hemisphere temperate farming practices. Perhaps spring break is an anachronism from pre-independence Micronesia and could be removed from the college calendar.
CLO 2
2. Define and explain concepts, theories, and laws in physical science.
The final examination asked students to plot data, interpret graphs, make calculations, and define concepts (facts) in physical science. This term the course generated 2043 points, with 1228 of those points in laboratory reports.
The focus in physical science is on science as a process, as a system of experimentally generated and verified knowledge, not as a collection of memorized facts. The course is not content free, but the heavier emphasis is on data gathering, fitting mathematical models, and writing up results in reports. This reflects my own biases. Once science becomes a collection of memorized and regurgitated factoids, then all collections of memorized and regurgitated factoids are equally valid. At that point one is left choosing among factoids to "believe" in and the result are those who "do not believe in science" whether that science is climate change, evolution, or any other area of science. Thus the course focuses on knowledge generated by the students and is guided in part by the concepts of non-overlapping magisteria and a constructivist epistemology.
Given the above, the final examination was not weighted and would not be weighted. At only 63 points, the final examination was only 3% of a student's overall mark. By the time of the final examination the work throughout the term, especially the laboratory work, has assessed the students. The students were aware of the low impact of the final examination and understood that the test would not have a large impact. High stakes testing is rarely a reliable measure of longer term knowledge retained. Performance levels may be lowered by these factors, but the hope is that the results more closely represent retained knowledge than memorized, regurgitated, and then forgotten knowledge.
On the final exam six male students outperformed sixteen female students. Note that one male student did not sit the final examination, that student had stopped attending class in early April. The difference in performance is most likely to be a result of the small sample sizes involved, especially for the male students. The course averages are a statistical tie. As a result of the stronger performance by the males, the item analysis shows higher averages for the males on an item-by-item basis.
Average success rates based on an item analysis of the final examination were aggregated by three skill areas: analysis and interpretations of graphs, citing of facts, and calculations. Aggregate average success rates based on an item analysis of the final examination were generally low but in line with pre-existing values.
Aggregate average success rates by skill area on the final
Graphing skills were the strongest area for the students, calculations, recollection of facts, and inferences performed more weakly.
Multi-term final examination averages based on item analysis aggregate average
The overall final examination average for the 22 students who took the examination, based on an item analysis, was a 46% success rate. While the previous two terms of had seen an improvement from the spring 2016 success rate of 51%, this spring term the success rate fell to the second lowest since 2010. The students, however, were aware that the final examination was not intended to be a "make or break" high stakes grade that would impact the work they had already done. The students knew that the emphasis and value were placed on a "doing" science in the laboratory.
CLO 3
3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science.While the laboratory nine assessment above provides some data on this course learning outcome, this outcome supports the general education program learning outcome "3.2 Present and interpret numeric information in graphic forms." With this focus in mind, a pre-assessment and post-assessment was included in the course. The post-assessment was embedded in the final examination.
SC 130 Physical Science includes a focus on the mathematical models that underlie physical science systems. Laboratories one, two, five, seven, nine, eleven, and twelve have linear relationships. A number of assignments in the course also have linear relationships. The students also encounter a quadratic relationship in laboratory three, now modeled as a quadratic using Desmos' ability to regress to any arbitrary function. By the end of the course students have repeatedly worked with linear relations. One relationship, one equation, at a time, not "problems one to thirty, even problems only." Every equation is built from data that the students have gathered. From the concrete to the abstract, repeated throughout the term, providing cognitive hooks on which to "hang" their mathematical learning.
A performance boost from a success rate of 47% on the pre-assessment to 77% on the post-assessment is documented in another article in this blog.
The post-assessment does not answer whether there will be long term student retention of the ability to present and interpret numeric information in graphic forms. The course has had, at least in the short run, a positive impact on the students' ability to work with numeric information in graphical forms.
CLO 4
4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.Course level learning outcome four focuses on communication, specifically writing. In the late 1990's assessment data at the college suggested some students were graduating with limited writing communication skills. As noted by the languages and literature division at that time, two college level writing courses in the general education core cannot by themselves produce collegiate level writers. Writing must occur across the curriculum, across disciplines. In 2007 SC 130 Physical Science at the national campus was redesigned to put an emphasis on writing. A "fill-in the blank" cook book style laboratory manual was replaced by laboratories which led to laboratory reports constructed using spreadsheet and word processing software.
This term the laboratory report marking rubrics added student learning outcomes from the proposed* course outline:
CLO 4.0 Communication skills: Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.
SLO 4.1 Produce laboratory reports: Produce laboratory reports with integrated tables, charts, and graphs
SLO 4.2 Use technology to produce laboratory reports: Use spreadsheets and word processing software
SLO 4.3 Effective communication: Communicate in writing using proper syntax and correctly used physical science vocabulary.
These were marked on a four point scale:
4 Exceeds expectations
3 Meets expectations
2 Does not meet expectations: needs improvement
1 Severe does not meet expectations: no evidence of outcome being met
Laboratory reports for laboratory nine were marked against the above metrics with the following results:
Based on sixteen reports submitted for laboratory nine, thirteen students met course level learning outcome number four, seven females and four males.
By the end of the term these 13 students could produce a laboratory report with tables and charts integrated from a Desmos. In the past students have used spreadsheets to lay out their tables and graphs. This term the ability of Desmos to regress to arbitrary functions using arbitrary variables drove the use of Desmos throught the course. Where a spreadsheet graphically shows a regression to y = mx +b, Desmos can be told to regress to d = vt using the notation d1~vt1, producing a value for v.
The result is that the student obtains results such as v = 353.37 which is in meters per second. This made sense to the students and all of the students shifted to using Desmos in lieu of a spreadsheet. As Desmos does not currently interact with software such as Google Docs, screenshots were used to put tables and graphs from Desmos into the laboratory reports.
Although this spring term showed some weaknesses in learning against prior terms, performance on the four learning outcomes was still within a range that might be attributed to a return to the mean. The impact of two multi-day holidays late in the term may also have negatively impacted learning among the students. And while the shift to Desmos early in the term appeared to be beneficial to the students, the term did not start with only Desmos being used. During the early part of the term the instructor was also learning some of the capabilities of Desmos. The use of Desmos from day one should prove beneficial to student performance and comprehension.
Sasha
Comments
Post a Comment