Assessing learning in physical science
SC 130 Physical Science proposes to serve two institutional learning outcomes (ILO) through four general education program learning outcomes (GE PLO) addressed by four course level student learning outcomes (CLO). Not listed are proposed specific student learning outcomes that in turn serve the course level learning outcomes. This report assesses learning under the proposed course level learning outcomes which in turn supports program and institutional learning outcomes.
ILO 8. Quantitative Reasoning: ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations; comprehends and can create sophisticated arguments supported by quantitative evidence and can clearly communicate those arguments in a variety of formats.
ILO 2. Effective written communication: development and expression of ideas in writing through work in many genres and styles, utilizing different writing technologies, and mixing texts, data, and images through iterative experiences across the curriculum.
CLO 1
Explore physical science systems through experimentally based laboratories using scientific methodologies
Laboratory fourteen in the penultimate week of the term provided a vehicle for assessing this course level outcome. The students were given a system that was unfamiliar to them and asked to determine the underlying mathematical models for the objects in the system. The specific system was the launch velocity versus the flight distance for a variety of flying objects.
Data was gathered and then reported to the board. Students were to produce tables and xy scatter graphs for each object type.
The laboratory reports were assessed to determine whether students properly recorded data in labelled tables, generated xy scatter graphs, made a decision on linearity and if deemed to be a linear relationship added linear trend lines to the graph, reported the slopes in their analysis, and discussed the results. Twelve of the thirteen students submitted this laboratory.
For the twelve laboratory reports submitted, students recorded their data in properly labelled tables, generated labelled xy scatter graphs, opted for a linear models, added a linear trend lines and trend line equations. Nine students went on to discuss their results in a reasonably meaningful manner.
CLO 2
2. Define and explain concepts, theories, and laws in physical science.
The 58 question final examination asked students to define and explain concepts, theories, and laws in physical science. Some questions consisted of two to seven subparts. Counting subparts there were 108 questions to be answered. The item analysis was done at the question level and a question was considered correct if a strong majority of subparts were substantively correct. The final had students make calculations, use formulas, and interpret graphical data. Average success rates based on an item analysis of the 58 item final examination were aggregated by topic and by skill. Aggregate average success rates based on an item analysis of the final examination were generally low.
The overall average for 13 students on these 58 items based on the item analysis was 64%, a considerable improvement from the spring 2016 success rate of 51% and a return to the 66% success rate seen on the fall 2015 final examination.
CLO 3
3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science.
While the laboratory fourteen assessment above provides some data on this course learning outcome, this outcome supports the general education program learning outcome "3.2 Present and interpret numeric information in graphic forms." With this focus in mind, a pre-assessment and post-assessment was included in the course. The post-assessment was embedded in the final examination.
SC 130 Physical Science includes a focus on the mathematical models that underlie physical science systems. Laboratories one, two, three, five, seven, nine, eleven, and twelve have linear relationships. A number of assignments in the course also have linear relationships. The students also encounter a quadratic relationship in laboratory three. A plot of height versus velocity generates a power relationship, specifically a square root relationship. By the end of the course students have repeatedly worked with linear relations. One relationship at a time, not "problems one to thirty, even problems only." Every equation is built from data that the students have gathered. From the concrete to the abstract, repeated throughout the term, providing cognitive hooks on which to "hang" their mathematical learning.
The first thirteen questions on the final examination were identical to the questions on the pre-assessment. The following bar chart depicts the percentage of students answering correctly on the pre-assessment on the right end of the turquoise bar, the percentage of students answering correctly on the post-assessment on the left end of the turquoise bar. Thus the chart shows the improvement from the pre-assessment to the post-assessment.
Of note is that on the very first question, a calculation of slope for a line that has a non-zero y-intercept, performance on the final examination was worse than on the pre-assessment. On the pre-assessment eight students answered correctly, on the final examination only five students answered correctly. This represented a drop of 28% which is not displayed on the chart above. On all other questions performance improved.
The average score for the students increased from 4.85 out of 13 to 9.77 out of 13. This increase was significant with a large effect size (1.40).
The post-assessment does not answer whether there will be long term student retention of the ability to present and interpret numeric information in graphic forms. The course has had, at least in the short run, a positive impact on the students' ability to work with numeric information in graphical forms.
CLO 4
4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.
Course level learning outcome four focuses on communication, specifically writing. In the late 1990s assessment data suggested some students were graduating with limited writing communication skills. As noted by the languages and literature division at that time, two college level writing courses in the general education core cannot by themselves produce collegiate level writers. Writing must occur across the curriculum, across disciplines. In 2007 SC 130 Physical Science at the national campus was redesigned to put an emphasis on writing. A "fill-in the blank" cook book style laboratory manual was replaced by laboratories which led to laboratory reports constructed using spreadsheet and word processing software.
By the end of the term students could produce a laboratory report with tables and charts integrated from a spreadsheet package. The students could produce reports that included the use of quantitative tools.
As reported above, student ability to include thoughtful discussion and interpretation of data supported by their quantitative evidence was demonstrated by nine students as measured by laboratory fourteen.
Inherent in supporting institutional learning outcome two, which course learning outcome four serves, is proper mechanics. Physical Science laboratory report marking rubrics at the national campus include evaluation of four broad metrics: syntax (grammar), vocabulary and spelling, organization, cohesion and coherence. Each of these four metrics is measured on a five point scale yielding a total possible of twenty points. In general, students enter the course with writing skills. Errors of tense and agreement tend to mirror areas in students' first language that do not have similar tense or agreement structures. All students in the class are working in English as a second language.
The brief six week duration of the summer session meant that up to three laboratory reports were due per week. The students in the course this summer generally started with fairly strong writing abilities and those abilities remained with them throughout the summer session. Thus there was little room for improvement against the existing rubrics. Coupled with the small sample size, significant improvement in writing mechanics could not be shown this summer.
Learning has occurred for all course level outcomes, the program learning outcomes served by those course level outcomes, and the institutional learning outcomes served in turn by the program learning outcomes.
ILO 8. Quantitative Reasoning: ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations; comprehends and can create sophisticated arguments supported by quantitative evidence and can clearly communicate those arguments in a variety of formats.
| GE PLO | SC 130 CLO |
|---|---|
| 3.5 Perform experiments that use scientific methods as part of the inquiry process. | 1. Explore physical science systems through experimentally based laboratories using scientific methodologies |
| 3.4 Define and explain scientific concepts, principles, and theories of a field of science. | 2. Define and explain concepts, theories, and laws in physical science. |
| 3.2 Present and interpret numeric information in graphic forms. | 3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science. |
ILO 2. Effective written communication: development and expression of ideas in writing through work in many genres and styles, utilizing different writing technologies, and mixing texts, data, and images through iterative experiences across the curriculum.
| GE PLO | SC 130 CLO |
|---|---|
| 1.1 Write a clear, well-organized paper using documentation and quantitative tools when appropriate. | 4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software. |
CLO 1
Explore physical science systems through experimentally based laboratories using scientific methodologies
Laboratory fourteen in the penultimate week of the term provided a vehicle for assessing this course level outcome. The students were given a system that was unfamiliar to them and asked to determine the underlying mathematical models for the objects in the system. The specific system was the launch velocity versus the flight distance for a variety of flying objects.
Data was gathered and then reported to the board. Students were to produce tables and xy scatter graphs for each object type.
The laboratory reports were assessed to determine whether students properly recorded data in labelled tables, generated xy scatter graphs, made a decision on linearity and if deemed to be a linear relationship added linear trend lines to the graph, reported the slopes in their analysis, and discussed the results. Twelve of the thirteen students submitted this laboratory.
Analysis of laboratory fourteen
For the twelve laboratory reports submitted, students recorded their data in properly labelled tables, generated labelled xy scatter graphs, opted for a linear models, added a linear trend lines and trend line equations. Nine students went on to discuss their results in a reasonably meaningful manner.
CLO 2
2. Define and explain concepts, theories, and laws in physical science.
The 58 question final examination asked students to define and explain concepts, theories, and laws in physical science. Some questions consisted of two to seven subparts. Counting subparts there were 108 questions to be answered. The item analysis was done at the question level and a question was considered correct if a strong majority of subparts were substantively correct. The final had students make calculations, use formulas, and interpret graphical data. Average success rates based on an item analysis of the 58 item final examination were aggregated by topic and by skill. Aggregate average success rates based on an item analysis of the final examination were generally low.
Aggregate average success rates by topic and skill
The overall average for 13 students on these 58 items based on the item analysis was 64%, a considerable improvement from the spring 2016 success rate of 51% and a return to the 66% success rate seen on the fall 2015 final examination.
Multi-term final examination averages based on item analysis aggregate average
CLO 3
3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science.
While the laboratory fourteen assessment above provides some data on this course learning outcome, this outcome supports the general education program learning outcome "3.2 Present and interpret numeric information in graphic forms." With this focus in mind, a pre-assessment and post-assessment was included in the course. The post-assessment was embedded in the final examination.
SC 130 Physical Science includes a focus on the mathematical models that underlie physical science systems. Laboratories one, two, three, five, seven, nine, eleven, and twelve have linear relationships. A number of assignments in the course also have linear relationships. The students also encounter a quadratic relationship in laboratory three. A plot of height versus velocity generates a power relationship, specifically a square root relationship. By the end of the course students have repeatedly worked with linear relations. One relationship at a time, not "problems one to thirty, even problems only." Every equation is built from data that the students have gathered. From the concrete to the abstract, repeated throughout the term, providing cognitive hooks on which to "hang" their mathematical learning.
The first thirteen questions on the final examination were identical to the questions on the pre-assessment. The following bar chart depicts the percentage of students answering correctly on the pre-assessment on the right end of the turquoise bar, the percentage of students answering correctly on the post-assessment on the left end of the turquoise bar. Thus the chart shows the improvement from the pre-assessment to the post-assessment.
Of note is that on the very first question, a calculation of slope for a line that has a non-zero y-intercept, performance on the final examination was worse than on the pre-assessment. On the pre-assessment eight students answered correctly, on the final examination only five students answered correctly. This represented a drop of 28% which is not displayed on the chart above. On all other questions performance improved.
The average score for the students increased from 4.85 out of 13 to 9.77 out of 13. This increase was significant with a large effect size (1.40).
The post-assessment does not answer whether there will be long term student retention of the ability to present and interpret numeric information in graphic forms. The course has had, at least in the short run, a positive impact on the students' ability to work with numeric information in graphical forms.
CLO 4
4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.
Course level learning outcome four focuses on communication, specifically writing. In the late 1990s assessment data suggested some students were graduating with limited writing communication skills. As noted by the languages and literature division at that time, two college level writing courses in the general education core cannot by themselves produce collegiate level writers. Writing must occur across the curriculum, across disciplines. In 2007 SC 130 Physical Science at the national campus was redesigned to put an emphasis on writing. A "fill-in the blank" cook book style laboratory manual was replaced by laboratories which led to laboratory reports constructed using spreadsheet and word processing software.
By the end of the term students could produce a laboratory report with tables and charts integrated from a spreadsheet package. The students could produce reports that included the use of quantitative tools.
As reported above, student ability to include thoughtful discussion and interpretation of data supported by their quantitative evidence was demonstrated by nine students as measured by laboratory fourteen.
Inherent in supporting institutional learning outcome two, which course learning outcome four serves, is proper mechanics. Physical Science laboratory report marking rubrics at the national campus include evaluation of four broad metrics: syntax (grammar), vocabulary and spelling, organization, cohesion and coherence. Each of these four metrics is measured on a five point scale yielding a total possible of twenty points. In general, students enter the course with writing skills. Errors of tense and agreement tend to mirror areas in students' first language that do not have similar tense or agreement structures. All students in the class are working in English as a second language.
The brief six week duration of the summer session meant that up to three laboratory reports were due per week. The students in the course this summer generally started with fairly strong writing abilities and those abilities remained with them throughout the summer session. Thus there was little room for improvement against the existing rubrics. Coupled with the small sample size, significant improvement in writing mechanics could not be shown this summer.
Learning has occurred for all course level outcomes, the program learning outcomes served by those course level outcomes, and the institutional learning outcomes served in turn by the program learning outcomes.
Comments
Post a Comment