Assessing Learning in Physical Science

SC 130 Physical Science proposes to serve two institutional learning outcomes (ILO) through four general education program learning outcomes (GE PLO) addressed by four course level student learning outcomes (CLO). This report assesses learning under the course level learning outcomes which in turn support program and institutional learning outcomes.

This report is informed by a new stream of information not previously available. The adoption of Schoology Institutional with the start of 2018 provided access to the learning outcomes mastery screens within Schoology Institutional.

Note that this course has a focus on "doing" science, on science as a process, a way of understanding the natural physical world and the mathematics that underlies many physical systems. The course does not focus on memorized facts. The course is centered on science as being that which can be measured, observed, evidenced. The course is intended as a counter to memorized science. Once one shifts to memorized facts as the basis of a science, then any set of memorized facts can be seen by the learner as equally valid. Somewhere down at the bottom of that slope are those who are convinced the earth is flat and no one actually walked on the moon. By doing simple experiments that seek to measure physical properties and quantities, by gathering and analyzing data, the intent is that students come to see science as a way of thinking about and analyzing the world around them.

The course directly supports institutional learning outcome eight and three general education outcomes.

ILO 8. Quantitative Reasoning: ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations; comprehends and can create sophisticated arguments supported by quantitative evidence and can clearly communicate those arguments in a variety of formats.

GE PLO SC 130 CLO
3.5 Perform experiments that use scientific methods as part of the inquiry process. 1. Explore physical science systems through experimentally based laboratories using scientific methodologies
3.4 Define and explain scientific concepts, principles, and theories of a field of science. 2. Define and explain concepts, theories, and laws in physical science.
3.2 Present and interpret numeric information in graphic forms. 3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science.

Yummy tests for acids and bases while visting local elementary school students watch

The course also supports institutional learning outcome two.

ILO 2. Effective written communication: development and expression of ideas in writing through work in many genres and styles, utilizing different writing technologies, and mixing texts, data, and images through iterative experiences across the curriculum.

GE PLO SC 130 CLO
1.1 Write a clear, well-organized paper using documentation and quantitative tools when appropriate. 4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.

CLO 1

Explore physical science systems through experimentally based laboratories using scientific methodologies

For the purposes of evaluating this course learning outcome, laboratory fourteen was evaluated. Laboratory fourteen functioned as a laboratory practical examination. The students were given a system to investigate. The students were tasked with analyzing the system, determining whether the variables under investigation were related, and discussing their findings. Where previous laboratories often provided a clear goal, a specific quantity to be measured, laboratory fourteen provided less scaffolding.

Justin prepares to throw a flying ring

The laboratory reports were assessed to determine whether students properly recorded data in labelled tables, generated xy scatter graphs, made a decision on linearity and, if deemed to be a linear relationship, added linear trend lines to the graph, reported the slopes in their analysis, and discussed the results. 18 of the 24 students submitted this capstone laboratory.

Claire measures launch velocity with a radar gun

Flora and Venister report the distance flown



Analysis of laboratory fourteen spring 2018

Of 24 students, seven students did not turn in laboratory fourteen. 17 of the 24 students produced laboratory reports with data recorded in a properly formatted table. Those 17 students also generated a correct xy scattergraph. Eight students ran a regression using Desmos and of those eight, five generated a reasonably complete discussion of the meaning of the slope and the results of the laboratory.

Desmos with data plotted for flying rings, disks, and swerve balls

Laboratory report completion rates overall for spring 2018 were down slightly at 66% against the a long term spring average of 74%. Spring always underperforms fall term, fall term has an 83% long term laboratory submission rate. Faculty sometimes note that academic performance slips each spring, the reduced submission rate seen in SC 130 in the spring is one piece of evidence in support of this. 
Laboratory submission (completion) rates for terms since fall 2014

This spring saw low initial turn in rates and a steep fall in submissions through laboratory six. While the students reacted positively to the use of Desmos and Google Docs, submission rates suggest that some students may have been experiencing challenges in the first six weeks. 

In prior terms the laboratory was due a week after the experiment and submissions were locked off in Schoology two weeks after the experiment. This practice developed back when the reports were being hand marked on hard copy as a way of managing the marking load. The use of Schoology, however, has made marking easier and more efficient. Report submissions for the first half of the term were left unlocked until midterm. Report submissions for the last half of the term remained unlocked until the Friday of the penultimate week of class.  

Submission rates did climb after the sixth week and remained stable at 70%. This statistic is confounded in part by three students who effectively stopped attending the course after midterm. If those three students are omitted, submission rates return to 80% which is also the long term overall submission rate. 

CLO 2

2. Define and explain concepts, theories, and laws in physical science.

In the past an item analysis of the final examination has provided information on the extent to which students are able to define and explain concepts, theories, and laws in physical science. The adoption Schoology Institutional provides an alternate insight into achievement of this outcome. This outcome was assessed 18 times during the term.

Individual student report on achievement by assignment

Some students either did not complete one of the eighteen assignments or were absent on the day of the test or quiz.

Individual student report on achievement by assignment

The percentage is the percentage success rate on that particular assignment, quiz, or test. The colors are determined by the mastery settings which are set at 70% to meet, 80% to exceed expectations on that assignment. At some deeper level this is still a form of assessing learning outcomes based ultimately on percentage achievement on a particular assignment.

Mastery settings in Schoology

One could make a philosophic argument that student learning outcomes are binary, either the students can or cannot demonstrate the learning outcome. On an outcome, "The student will be able to construct a cabinet" either the student can construct a cabinet or cannot. The complication comes in that not all cabinets will necessarily be of equal quality. At some point one is left determining what the minimum acceptable quality is for a cabinet. And quality leads down a rabbit hole best left to those who have read Zen and Art of Motorcycle Maintenance.

In the mastery settings the students are deemed to have crossed that "binary" threshold by exceeding 70% on five or more assignments. Noting that the class consists of two sections, one has the following results.

Section one

Note that nine of the twelve students in section one have exceeded 70% on five or more assessments of course learning outcome two, even though some of the nine students have overall averages less than 70%.

Section two

In section two, ten of twelve students have exceeded 70% on five or more assessments of course learning outcome two.

Section one

Section two

For summary reporting, Schoology's mastery summary does not tally the "green star" counts but rather the overall average.

The focus in physical science is on science as a process, as a system of experimentally generated and verified knowledge, not as a collection of memorized facts. The course is not content free, but the heavier emphasis is on data gathering, fitting mathematical models, and writing up results in reports. Once science becomes a collection of memorized and regurgitated factoids, then all collections of memorized and regurgitated factoids are equally valid. At that point one is left choosing among factoids to "believe" in and the result are those who "do not believe in science" whether that science is climate change, evolution, or any other area of science. Thus the course focuses on knowledge generated by the students and is guided in part by the concepts of non-overlapping magisteria and a constructivist epistemology.

Noting that the focus of the course has been on the laboratories, on "doing" science as a process and not on specific facts nor memorization, the final examination has been a vehicle only for assessing course learning outcome two. The mastery information provides a alternate and perhaps more authentic way to assess course learning outcome two. As a result, the final examination was redesigned and repurposed to act more as a reflective wrapping up of the course than as a vehicle for assessment.

CLO 3

3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science.

While the laboratory fourteen assessment above provides some data on this course learning outcome, this outcome supports the general education program learning outcome "3.2 Present and interpret numeric information in graphic forms." With this focus in mind, a pre-assessment and post-assessment was included in the course.  In the past the post-assessment was embedded in the final examination, as seen in the final examination of fall 2017. With the decision to redesign and repurpose the final examination, the post assessment was integrated into a quiz at the end of the week fourteen.

While students study and prepare for the final examination, there is less - if any - preparation for a weekly quiz. In addition, the students had no knowledge that the quiz would include the pre-assessment. The intent was to measure not what the students can accomplish by studying, but what they know without preparation. In this sense the intent was to mimic the pre-assessment which was administered on the first of class. The hope is to measure what the students know without studying - to potentially get at longer term retention.

SC 130 Physical Science includes a focus on the mathematical models that underlie physical science systems. Laboratories one, two, four, five, seven, nine, eleven, and twelve have linear relationships. A number of assignments in the course also have linear relationships. The students also encounter a quadratic relationship in laboratory three, now modeled as a quadratic using Desmos' ability to regress to any arbitrary function. By the end of the course students have repeatedly worked with linear relations. One relationship, one equation, at a time, not "problems one to thirty, even problems only." Every equation is built from data that the students have gathered. From the concrete to the abstract, repeated throughout the term, providing cognitive hooks on which to "hang" their mathematical learning.

In the fall of 2017 average performance increased from a success rate of 46% on the pre-assessment to 87% on the post-assessment. Students showed improvement on every item. This is typical of the performance seen each term in the past.

QuestionPrePost
Calculate slope from line on graph0.580.96
Determine y-intercept from line on graph0.501.00
Write out the equation of line from graph0.420.88
Make an inference based on slope0.170.56
Plot xy data given a table labeled x, y0.921.00
Calculate slope from data0.170.84
Determine y-intercept from data0.420.92
Calculate y given x0.580.84
Calculate x given y0.460.76
Determine slope from slope-intercept form0.460.88
Determine y-intercept from slope-intercept form0.380.96
Means0.4580.873
Fall 2017 results on the pre-assessment and post-assessment

With the unannounced shift of this material to quiz fourteen, performance would be expected to slip versus performance on the final examination.

QuestionPrePost
Calculate slope from line on graph0.450.52
Determine y-intercept from line on graph0.150.29
Write out the equation of line from graph0.050.19
Make an inference based on slope0.350.81
Plot xy data given a table labeled x, y0.950.90
Calculate slope from data0.300.67
Determine y-intercept from data0.150.33
Calculate y given x0.300.67
Calculate x given y0.350.52
Determine slope from slope-intercept form0.400.48
Determine y-intercept from slope-intercept form0.400.48
Means0.3500.532
Spring 2018 results on the pre-assessment and post-assessment

While performance did indeed fall on the post-assessment, improvement was seen on each item except the plotting of points. The drop here was not a reduction in the number answering correctly. Twenty students sat for the pre-assessment of whom 19 were able to plot data points. Twenty-one students tood quiz fourteen, of whom 19 were able to plot data points. The denominator changed, not the numerator.

An analysis suggested that the strongest improvements were seen on the ability to calculate the slope from data and to calculate a y-value from a linear relationship when given the x value. Knowledge of the density of soap experiment contributed to the large rise in the score on making an inference based on the slope.

While the post-assessment does not answer whether there will be long term student retention of the ability to present and interpret numeric information in graphic forms, the shift of the post-assessment to the week fourteen quiz should provide a better measurement of what the students can do without specifically studying. This may provide better insight into longer term retention.

CLO 4

4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.

Course level learning outcome four focuses on communication, specifically writing. In the late 1990's assessment data at the college suggested some students were graduating with limited writing communication skills. As noted by the languages and literature division at that time, two college level writing courses in the general education core cannot by themselves produce collegiate level writers. Writing must occur across the curriculum, across disciplines. In 2007 SC 130 Physical Science at the national campus was redesigned to put an emphasis on writing. A "fill-in the blank" cook book style laboratory manual was replaced by laboratories which led to laboratory reports constructed using spreadsheet and word processing software.

Typical head end of a laboratory report in Google Docs

This term the adoption of Schoology Institutional provided access to the Google Drive Assignments app. An earlier article covered the use of Google Drive Assignments in the course.

Course level student learning outcome four was directly evaluated up to nine times on rubrics used to mark laboratory reports.
Achievement on course student learning outcome four

Of the 24 students in the course, 21 students met or exceeded expectations on this outcome.  Of the three students who did not meet this outcome, two were no longer attending class by the end of the term. The third student was attending only irregularly, attending class once a week or less.

The achievement chart is effectively an average performance and as an average is, in effect, related more to grades than to a "binary" demonstration of the learning outcome having been accomplished or not.

Number of students who demonstrated outcome five or more times

Seventeen students demonstrated student learning outcome four five or more times. Although demonstrating the outcome is set at a 70% performance on the underlying outcome, the demonstration of this level of performance five or more times separates this statistic from the achievement average above. This is closer to the binary assertion that "students will be abel to demonstrate basic communication skills by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.

Affective Domain Assessment

SC 130 Physical Science functions as a science with laboratory requirement for the general education core at the college. The only major required to take the course is an associate of applied science in telecommunications technology two year program. This term no students in the telecommunications technology program were in the course.  Only three of the 24 students were in a science major, those three were in the Agriculture and Natural Resources program. One of the intents of the course is to reach out to students who may not enjoy science as a subject and show them science in a new light.

At the end of the term the students were asked to respond to the following questions:

1. Before I took this class my attitude towards science was:
☐ Positive: I liked science
☐ Neutral
☐ Negative: I did not like science

2. During the semester:
☐ I enjoyed physical science class
☐ Neutral
☐ I did not enjoy physical science class

3. During the semester:
☐ I was glad I signed up for physical science class
☐ Neutral
☐ I was not glad I signed up for physical science class

4. After taking this class my attitude towards science is:
☐ Positive: I like science
☐ Neutral
☐ Negative: I do not like science

Prior to the term student self-reported attitudes towards science were generally neutral to negative.
Student attitudes towards science

After the course student attitudes towards science were primarily positive with a few students remaining neutral towards science.
Students on whether they enjoyed the course during the term

During the semester students generally reported that they enjoyed the course.

Students on whether they were glad that they signed up for the course


Physical science is a one of a choice of science with laboratories that a student could choose. Thus there is potentially a difference between a student enjoying the course and being glad that they chose to take physical science. Although individual students did not respond identically on the two questions, the resulting percentages were the same. No students responded negatively to either question.

Overall the course can claim to have improved student attitudes towards science as a subject.

Favored and Least favored laboratories

Some insight into the impact of the laboratories on learning can be gleaned from the reaction students had to the laboratories. At term end, the students were asked to provide feedback on a favorite laboratory and a least favorite laboratory.

Why a student chose that laboratory as a favorite...

All of them were my favorite.
[01] Density of soap. I learned about what object float and what object sink.
[01] Because this is the first time to find out that soap can float if the density is less than one.
[04] Conservation of marble momentum. Playing with marbles is fun.
[07] GPS and meters per arc minute. We used the GPS to find where the instructor hide.
[07] It is fun how we try and locate the teacher using the GPS. We had been to many places but all were wrong.
[07] I enjoyed doing.
[07] Using a GPS to determine meters per minute because it is something new for me and I like the navigating part.
[08] Cloud drawing. Because it is fun knowing the different kinds of clouds and knowing how they look like and draw them.
[08] I learned a lot about clouds.
[09] Speed of sound. It is fun to walk on the road and see or heard the clappers are clap the woods and we time the sound.
[10] I enjoyed the colors we mixed on the computer.
[10] Its related to my field of study.
[12] Batteries and bulbs, conductors, and Ohm's law. It could help me to know which objects are conductors or insulators, and even how to work on electricity.
[12] It helps me know that not only by electricity that can conducted light but also from things like metal and gold and even batteries.
[13] Floral litmus solutions. I learned that colors can be changed into different color when I add some acids or base to my red color flower.
[13] I enjoyed doing.
[13] It is the only lab that I got a good grade for. I understand most of the bases and acids.
[13] Everything in our house can be also called and used for something else. Experience/experiment on household items.
[13] I like flowers and the things we did to the flowers. I enjoyed the colors that we mixed for the flowers.
[14] Flying disks and mathematical models. It was fun.

Why a student chose that laboratory as a least favorite...

[01] Density of soap. I didn't have the chance to submit.
[02] Linear velocity. I lost track on the rolling ball and I mixed or messed up my datas upong the things I am working on.
[03] Acceleration of gravity. I did not understand much of acceleration.
[07] GPS and meters per arc minute. Going through those grass makes me itch all over.
[07] I am so lazy to walk.
[07] Because I do not know how to follow the GPS. It just because my own problem of not learning how to follow it.
[08] Cloud drawing. I's not good at drawing, it is just not my thing.
[09] Speed of sound. We go under the sun which make so hot.
[09] It is tiring walking a long way but, but it's fun though.
[12] Batteries and bulbs, conductors, and Ohm's law. I didn't learn something.
[12] I don't understand much about Ohm's law.
[13] Floral litmus solutions. In my opinion, to me, it is like dangerous not always safe for health. [Note that the lab uses household compounds]
[14] Flying disks and mathematical models. I don't understand it.
[14] It is too hot when we are going out to the field and find the velocity and meter of the flying disk.

Final examination

In the past the final examination was a comprehensive test designed to assist in measuring course learning outcomes through an item analysis of the exam. This term Schoology Institutional and the Mastery screen provided more direct information on learning. The course was always focused on doing science, on the laboratory reports, and not on tests and quizzes. This term the term ended with 1378 points - the final examination was not going to have a significant impact and was no longer needed to inform student learning outcome achievement.

As a result the final exam was recast to focus on an area of concern that arose when marking laboratory fourteen: the students were still having difficulty in their analysis, discussion, and conclusion with articulating the intent of the laboratory and reporting that to the reader. Thus the final examination was redesigned to be 15 questions which focused on the intent of each laboratory. The students were informed of the redesign.

  1. In laboratory one on a graph of volume versus mass for soap, what physical quantity did the slope represent?
  2. In laboratory two on a graph of time versus distance for a rolling ball, what physical quantity did the slope represent?
  3. In laboratory three on a graph of time versus distance for a falling ball, what physical quantity was calculated from a quadratically increasing slope?
  4. In laboratory four a graph of the number of marbles in versus marbles out out a collision provided support for what law in physical science?
  5. In laboratory five on a graph of weight versus force to pull an object across sandpaper, what physical quantity did the slope represent?
  6. In laboratory six a graph of time versus temperature for a cooling cup hot water, provided support for what law in physical science?
  7. In laboratory seven on a graph of arcminutes versus meters as measured by GPS receivers and a surveyor's wheel, what physical quantity did the slope represent?
  8. In laboratory eight, what was the importance in science of drawings such as drawings of clouds?
  9. In laboratory nine, on a graph of time from seeing a clap to hearing a clap versus distance from the clap, what physical quantity did the slope represent?
  10. For laboratory ten, list the seven colors of the Newtonian rainbow in order, the primary colors of light, and the secondary colors of light. 
  11. In laboratory eleven on a graph of apparent depth underwater versus actual depth underwater for a penny, what physical quantity did the slope represent?
  12. In laboratory twelve on a graph of current versus voltage in an electrical circuit, what physical quantity did the slope represent?
  13. For laboratory twelve, list one acid, one base, and one neutral substance.
  14. In laboratory fourteen on a graph of velocity versus distance, which has the higher slope and thus flies farther for the same launch velocity: a flying disk or a swerve ball?
  15. In laboratory twelve site swap mathematical models describes patterns for what system?
The solutions were:
  1. Density of soap
  2. Velocity of the ball
  3. Acceleration of gravity
  4. Conservation of momentum
  5. Coefficient of friction
  6. Newton's Law of Cooling
  7. Conversion factor from arcminutes to meters
  8. Good, accurate drawings can assist in developing classification systems in science and provide insight into the relationships between objects being studied.
  9. The speed of sound
  10. Red Orange Yellow Green Blue Indigo Violet. Red Green Blue. Yellow Cyan Magenta
  11. Index of refraction for water
  12. The resistance in the circuit
  13. [answers will vary but can be lime juice, ammonia, and water]
  14. Flying disk
  15. Juggling
The intent was to focus on the meaning of the mathematics at the core of each experiment, but the structure of the questions was new for the students and proved far more challenging and difficult than anticipated. Scored at one point each, correct or incorrect, the minimum score was also the mode at two correct, the median was 4.5, the average was 5.25, with a maximum of 12. No student obtained a score of 15.

This format could perhaps be used in the future, but each of these questions should appear on a weekly quiz during the term.

Comments

Popular posts from this blog

Box and whisker plots in Google Sheets

Setting up a boxplot chart in Google Sheets with multiple boxplots on a single chart

Creating histograms with Google Sheets