Assessing Learning in Physical Science

SC 130 Physical Science proposes to serve two institutional learning outcomes (ILO) through four general education program learning outcomes (GE PLO) addressed by four course level student learning outcomes (CLO). This report assesses learning under the course level learning outcomes which in turn support program and institutional learning outcomes.

ILO 8. Quantitative Reasoning: ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations; comprehends and can create sophisticated arguments supported by quantitative evidence and can clearly communicate those arguments in a variety of formats.

3.5 Perform experiments that use scientific methods as part of the inquiry process. 1. Explore physical science systems through experimentally based laboratories using scientific methodologies
3.4 Define and explain scientific concepts, principles, and theories of a field of science. 2. Define and explain concepts, theories, and laws in physical science.
3.2 Present and interpret numeric information in graphic forms. 3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science.

Amy-lang with floral litmus solutions

ILO 2. Effective written communication: development and expression of ideas in writing through work in many genres and styles, utilizing different writing technologies, and mixing texts, data, and images through iterative experiences across the curriculum.

1.1 Write a clear, well-organized paper using documentation and quantitative tools when appropriate. 4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.


Explore physical science systems through experimentally based laboratories using scientific methodologies

For the purposes of evaluating this course learning outcome, laboratory nine was evaluated. Laboratory nine focused on measuring the speed of sound by timing the delay between seeing boards clapped together and arrival of the sound of the boards clapping.

Students measuring the time delay at 400 meters between seeing boards clap and hearing them

The laboratory reports were assessed to determine whether students properly recorded data in labelled tables, generated xy scatter graphs, made a decision on linearity and if deemed to be a linear relationship added linear trend lines to the graph, reported the slopes in their analysis, and discussed the results. Twenty-five of twenty-six students submitted this laboratory.

Analysis of laboratory nine

Of 26 students, seven students did not turn in laboratory nine. 19 of the 26 students produced laboratory reports with data recorded in a properly formatted table. Those 19 students also generated a correct xy scattergraph. Eighteen students ran a regression using Desmos and of those fifteen generated a reasonably complete discussion of the meaning of the slope and the results of the laboratory.

Desmos with sound data plotted and fit to a linear regression

As occurs each term, submission rates for laboratory reports fell as the term progressed during the fall term. The number of students completing laboratory nine was 73%, on par with submission rates at that point in the fall term 2017. 
Laboratory submission (completion) rates for terms since fall 2014

The 73% submission rate for laboratory nine was below the longer term average submission rate of 80% for laboratory nine.

Long term laboratory submission rates per laboratory for those laboratories which have a submitted report

Note that the weakness seen in laboratory seven submission rates may be due to laboratory seven occurring during midterm, a busy time in the term for many students. Laboratory 14 has the lowest submission rates at 67% over the long term, although this term the submission rate was 77%. 
Long term laboratory submission rates fall 2017 minus the long term average submission rate

In prior terms the laboratory was due a week after the experiment and submissions were locked off in Schoology two weeks after the experiment. This practice developed back when the reports were being hand marked on hard copy as a way of managing the marking load. The use of Schoology, however, has made marking easier and more efficient. Experimentally, report submissions for the first half of the term were left unlocked until midterm. Report submissions for the last half of the term remained unlocked until the Friday of the penultimate week of class.  In class the students were only told that reports were due a week after the experiment. The change in the locking practice was not announced in class. As seen above, this change had no significant positive nor negative effect on submission rates over the long term submission rates since fall 2014. The average submission rate for all prior terms was 81%, this term the average was 80%. Submission rates per lab varied in a narrow band of plus and minus ten percent, not significant given the sample sizes involved.


2. Define and explain concepts, theories, and laws in physical science.

The final examination asked students to plot data, interpret graphs, make calculations, and define concepts (facts) in physical science.

The focus in physical science is on science as a process, as a system of experimentally generated and verified knowledge, not as a collection of memorized facts. The course is not content free, but the heavier emphasis is on data gathering, fitting mathematical models, and writing up results in reports. This reflects my own biases. Once science becomes a collection of memorized and regurgitated factoids, then all collections of memorized and regurgitated factoids are equally valid. At that point one is left choosing among factoids to "believe" in and the result are those who "do not believe in science" whether that science is climate change, evolution, or any other area of science. Thus the course focuses on knowledge generated by the students and is guided in part by the concepts of non-overlapping magisteria and a constructivist epistemology.

Given the above, the final examination was not weighted and would not be weighted. The final examination was only 3.4% of a student's overall mark. By the time of the final examination the work throughout the term, especially the laboratory work, has assessed the students. The students were aware of the low impact of the final examination and understood that the test would not have a large impact. High stakes testing is rarely a reliable measure of longer term knowledge retained. Performance levels may be lowered by these factors, but the hope is that the results more closely represent retained knowledge than memorized, regurgitated, and then forgotten knowledge.
Average performance on final examination and overall course averages by gender

No significant difference in performance by gender was seen on the final exam nor the overall course averages. 

Average success rates based on an item analysis of the final examination were aggregated by three skill areas: calculations, citing of facts, and graphing skills.
Aggregate average success rates by skill area on the final

Although in spring 2017 saw graphing skills as the strongest area for the students, this fall calculational skills edged out factual recall and graphing skills.

Multi-term final examination averages based on item analysis aggregate average

The overall final examination average for the 25 students who took the examination was a 65% success rate against the total possible. The low performance of spring 2017 can now be seen as part of longer term variability in this average which returns towards a long term mean of 58%.


3. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science.

While the laboratory nine assessment above provides some data on this course learning outcome, this outcome supports the general education program learning outcome "3.2 Present and interpret numeric information in graphic forms." With this focus in mind, a pre-assessment and post-assessment was included in the course. The post-assessment was embedded in the final examination.

SC 130 Physical Science includes a focus on the mathematical models that underlie physical science systems. Laboratories one, two, five, seven, nine, eleven, and twelve have linear relationships. A number of assignments in the course also have linear relationships. The students also encounter a quadratic relationship in laboratory three, now modeled as a quadratic using Desmos' ability to regress to any arbitrary function. By the end of the course students have repeatedly worked with linear relations. One relationship, one equation, at a time, not "problems one to thirty, even problems only." Every equation is built from data that the students have gathered. From the concrete to the abstract, repeated throughout the term, providing cognitive hooks on which to "hang" their mathematical learning.

Average performance increased from a success rate of 46% on the pre-assessment to 87% on the post-assessment. Students showed improvement on every item.

Calculate slope from line on graph0.580.96
Determine y-intercept from line on graph0.501.00
Write out the equation of line from graph0.420.88
Make an inference based on slope0.170.56
Plot xy data given a table labeled x, y0.921.00
Calculate slope from data0.170.84
Determine y-intercept from data0.420.92
Calculate y given x0.580.84
Calculate x given y0.460.76
Determine slope from slope-intercept form0.460.88
Determine y-intercept from slope-intercept form0.380.96

The post-assessment does not answer whether there will be long term student retention of the ability to present and interpret numeric information in graphic forms. The course has had, at least in the short run, a strong positive impact on the students' ability to work with numeric information in graphical forms.


4. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.

Course level learning outcome four focuses on communication, specifically writing. In the late 1990's assessment data at the college suggested some students were graduating with limited writing communication skills. As noted by the languages and literature division at that time, two college level writing courses in the general education core cannot by themselves produce collegiate level writers. Writing must occur across the curriculum, across disciplines. In 2007 SC 130 Physical Science at the national campus was redesigned to put an emphasis on writing. A "fill-in the blank" cook book style laboratory manual was replaced by laboratories which led to laboratory reports constructed using spreadsheet and word processing software.

Typical head end of a laboratory report

As a proxy for demonstrating basic communication skills, twenty-six students submitted 252 laboratory reports of 312 total possible submissions (81%). Eleven of the 26 students (42%) submitted all twelve of laboratory reports, half of the students submitted 11 or more reports, and 75% of the students submitted nine or more reports. A quarter of the class submitted fewer than nine reports.
Distribution of the number of lab reports out of 12 submitted by each student

Three students submitted an unusually low number of laboratory reports. Although these students only completed a fraction of the reports assigned, these reports demonstrated attainment of basic written communication skills.

Each term there are a few students who turn in an unusually low number of laboratory reports. Perhaps there might be a section level requirement of some specified minimum number of reports required.

Some insight into the impact of the laboratories on learning can be gleaned from the reaction students had to the laboratories. At term end, the students were asked to provide feedback on a favorite laboratory, a least favorite laboratory, and changes they would make in a laboratory.

Students had the following reactions to their most favored laboratory

All of them were my favorite.
[01] Density of soap. I didn't know how to tell what soap will and will not float.
[01] I enjoyed getting the bar of soap into smaller pieces and measuring. I just found out that Ivory soap can float but the others not.
[05] Force of friction. I got learn how to do we measure the force and that there were different kinds, like smooth and rough which one has the most force of friction.
[05] Lab five is new to me too and exciting to know.
[07] GPS and meters per arc minute. It's interesting to learn about latitude and longitude, how it works, and how to spot a place with a GPS.
[07] It was my first time experience to actually use a GPS and look for a person.
[07] It gives me an idea to know where I am and where I'm going and also I'm panning to join the military.
[07] It helps me learning about to use a GPS device, and more importantly it enable to have idea how to get home such as in the forest. Then I can use my cell phone to track the GPS all the way home.
[07] It was more fun than the other labs. And also because we learned more about how the GPS work.
[07] We used the GPS to search for Dana. When we are looking for him there were some people outside agriculture building told us that he was on the container. Then we search for him there, he was not there.
[07] Can help me to find someone who is lost.
[07] I learned how to GPS, and I can use it to find a missing person.
[07] It was my first time using a GPS to track the precise location of things and it was so hilarious because I failed because of misusing it.
[07] I learned to how to navigate using GPS.
[07] We did many of the experiment outdoors which is very interesting.
[07] It would be really useful to know in the future and I am also very interested in GPS systems.
[08] Easy.
[12] Batteries and bulbs, conductors, and Ohm's law. Very interesting and amazing that I learned that water conduct electricity on that day.
[12] Since electricity is everywhere, I'm really interested in learning these things so I can applied it at home and wherever is needed.
[13] Floral litmus solutions. Creative. Doesn't know we can do experiment on flowers too. Thought only for decoration.
[13] We dealt with a lot of dangerous chemicals and I got to see visual results. [Note that the lab uses household compounds]
[16] Learning more about black holes and time is fun, kind of confusing but fun to learn.
[16] This lab surprised me when I watched the movie called Interstellar. Interstellar means between the stars. Once you enter into those dimensions, times getting slower. Wormhole leads to another galaxy and it slows the time.
[16] Interstellar. I love this lab because I get to understand some of the terms and concepts of what we've learned about.

Students had the following reactions to their least favored laboratory.

None. I enjoyed all of the labs.
None. I like all of them.
None. All of them were very interesting.
None. I like all the labs that we did because I learned many things that I do not know from this lab.
None. I enjoyed every lab because I do to do hands-on experiments and helped me work well with my colleagues.
[04] Marble momentum because I don't fully understand it. I have little knowledge on how to perform or make a laboratory based on the experiment we did on it.
[04] I don't understand what to put in the tables because we were just focusing on how many marbles goes in and how many goes out and I also don't know what formula to use to make the graph of my data.
[04] I was asked to make my own table and I did not because I did not know what to do a table from.
[06] It was slightly boring waiting for the water to cool down.
[07] Using GPS to determine meters per arc minute. The part where I did a lot of walking is the what I dislike about the lab.
[07] Sweat and we end up in another direction.
[07] I never got to find the location and my shape was awful.
[08] I don't know how to draw a nice cloud. I draw like a kid.
[08] I really don't know how to draw.
[09] Speed of sound. I think trying to time the speed of sound was really hard for me. I find it difficult to understand more about it.
[09] Because I was clapping away and under the sun or it was hot I couldn't be on the road that long. Also it was kind of like hard kind too.
[11] It was so hard.
[11] I don't feel that I could explain it.
[12] I didn't understand and I was kind of lost and confused (complicated).
[15] Site swap notation. I have no idea what I am learning.
[16] I didn't really learned.
[16] Black holes, time, and interstellar, I really don't understand the what the subject is.
[16] Those things are so clear to me.

Students made the following recommendations on a laboratory they would change.

None. The teaching skills on each lab are considered very learning resources for everyone around the world.
Nothing, all of them are effective for this course.
I think none of them need changes.
I don't want to change anything from these laboratories. They were all good, helpful, and also challenging. But please, this class was the best!
There is no laboratory that I want to change because I think all of the lab were explained well by the instructor but it was I who didn't pay attention to the explanations and experiments.
I'll just say that those labs cannot be changed but need to be explained more about them because some are confusing and hard to make a report.
There is no laboratory I want to change because I think all of those listed are good laboratories.
I don't think there is any laboratory that you would change. All of the laboratories were fun and exciting.
To be honest, none. All of them are good and very interesting.
The labs are perfect for as it is of now. If will change some of it, there maybe a mistake that I might make. It should stay the same. All of the labs are useful.
[04] In lab four time should be measured in order the fill out the table correctly and some formula can be used to make the graph.
[06] We don't have much time for counting the time on each cooling of the water.
[07] Tried to know how to use a compass and know the exact location.
[09] I want to change the place and time. Our instructor should at least change it if he or she can.
[09] I would like to do this lab in a highway where there will be no cars running.
[13] I will give the students the name of the flowers that will not work so everyone will use different flowers that may work.

Other insights into learning

At term end a separate survey asked students to pick one of seven prompts and to respond to that prompt. These responses also provide some insight as to student learning, some insight "behind the numbers" that often illuminate paths to course improvement. There were seven prompts, each student was asked to respond to one.

1. My learning of the content was most helped when… … because… 
...we do our laboratory reports because they give me most of the points to pass the class and I learned more important information on doing the experiments during the laboratory.
...we did the laboratory because I am an active learner.
...I saw things happen in class. It help me a lot because when I see things happen I easily learn much.
...I get to be doing labs and doing measurements which I say that its good for me. I learn a lot but also it helps me to improve my bad scores and hard problem solving. It even made me learn new math problems and short cuts even a new app like Desmos. It really did help a lot.
... we have activities or experiments. From my point of view experiments and hands on exercises improved my knowledge and understanding.
...we did lab because the labs were very useful that I learned many things that I will always remember and can do it now like measuring speed of sounds, because before I didn't know that sounds can be measured and I can measure it now that I know how to do it.
...I faced serious conditions that's going to be appearing along the way because the learnings I have encounter will prepare me to face it.
...doing labs because laboratory times were good and I learned things from it.
...we have hands on lab because it helps to understand what the topic is about

2. The activity that contributed the most to my learning was… … because… 
...the labs time because I see thing, touch thing, and do things on my own. So it helped me a lot.
...outdoor activities because it applied to our learning. Better to go out and learn at the same time. Plus, learning is learning is not boring. This class was awesome and it motivated me. Thank you.
...when were doing the labs and also making the lab reports. I have seen some of the topics we did on the labs but this does help me learn more and the reality of science and also now I can already observe things and make a full report of what I see.
... the laboratories. I learned a lot when doing hands-on activities.
... doing the lab reports. During the lab we explored and demonstrated on what we learned and because of that I learn so many thing from doing the lab reports.
...laboratories contributed the most to my learning because I have the idea of what the topic is about.
...clouds, because I do not know any of the clouds names until I took this class and also many things that we covered really opened my eyes through their own perspectives.

3. The biggest obstacle for me in my learning the material was… … because…
...procrastination because later is never really later with my busy schedule.

4. I was most willing to take risks with learning new material when… … because… comes to exploring something new because I want to learn what I do not know.

5. During the first day, I remember thinking… … because… 
...what if I fail this class. There was a lot that came to my mind. I remember thinking what if I don't understand what to do. I always think of these thoughts because I know this class deal with math and that is my weakness. I have hard time solving math problems.
...this class will be fun because I saw during summer that this class spends most of its time outdoors exploring.
...nothing really, because I thought there is nothing about the world or around the world we need to know.

6. What I think I will remember five years from now is… … because… 
...the labs because all of them are effective in this course and are essential also.
...this course because it improved my learning skills and knowledge. to navigate using GPS. I believe so because I have been planning to join the military force. So when I will be there I will already knew how to use the GPS. to use the GPS because it is more fun to locate the other places.
...using the GPS to locate something, someone, or a target because I want to be a soldier and I guess using it or knowing how to use it might be required. to detect if something is acidic or base because of lab 13. It was really simple to do and I know how to detect the differences.
...flying disks because I like to play outdoor.
...time because I learn that it never slower or faster but runs at a constant speed. to conserve energy on my own because of the skills I have learned, I can apply it myself.

7. What is something covered in this course material that you can do now that you could not do or did not fully understand at the beginning of the term? … because…
...using GPS and for now I am 100% sure that the next time I will use GPS I will never fail again instead I will find what I am looking for because you taught me now to use a GPS.
...lab 13 helped me realize that there are chemicals on this island that can be found at everyone's house or store we can use to see changing a material or organism.

Overall the laboratories, designed and intended to be central to the course, are perceived that way by the students. The laboratory experiences and outdoor experiences are both self-reported to strongly contribute to learning. Learning to use the GPS is a highly regarded skill by many students.

Normally the term has only 15 laboratory sessions. Due to holidays and the structure of the calendar this fall, there were 16 laboratories. For the 16th week the students first watched four hours of videos on fundamental particles, the Higgs boson, mass, black holes, and the nature of time. Then during the lab day the student watch the three hour movie Interstellar and wrote up a reaction paper to the science of the movie. I had not anticipated that this non-laboratory laboratory would be named as a favorite laboratory. This surprised me. Also of interest was an equal number of students who were confused by Interstellar.

The data and the comments provide evidence that the course is accomplishing the learning outcomes of exploring physical science systems through experimentally based laboratories, defining and explaining concepts, theories, and laws in physical science, generating mathematical models for physical science systems, and demonstrating basic communication skills through writing up the result of experiments.


Popular posts from this blog

Box and whisker plots in Google Sheets

Areca catechu leaf sheaf petiole plates

Setting up a boxplot chart in Google Sheets with multiple boxplots on a single chart