Friday, December 12, 2014

Assessing learning in introductory statistics

MS 150 Statistics has for the past three years utilized a modified curriculum based on a proposed outline. The three course level student learning outcomes for MS 150 Statistics are:

  1. Perform basic statistical calculations for a single variable up to and including graphical analysis, confidence intervals, hypothesis testing against an expected value, and testing two samples for a difference of means.
  2. Perform basic statistical calculations for paired correlated variables.
  3. Engage in data exploration and analysis using appropriate statistical techniques including numeric calculations, graphical approaches, and tests.

The first two outcomes involve basic calculation capabilities of the students and are assessed via an item analysis of the final examination. 76 students in three sections took the final examination.

Measuring flight distance towards calculating a 95% confidence interval for the flight distances. Students build and throw their own aircraft, then the calculation shows that the confidence interval captures a previously published population mean distance.
Twenty-one questions on the final examination required the students to perform basic statistical calculations on a small sample. Based on the item analysis, 80.2% of the items were answered correctly by the students. In general basic single variable statistical calcuations are an area of strength for the students.

Performance on the second student learning outcome was measured by eight questions on the final examination. Student performance on this section was lower at 68.2%. This section has historically been weaker than the basic single variable statistics section.

The third student learning outcome, open data analysis, was assessed using a simple rubric that looked at whether a student made an appropriate analysis with correct answers to questions posed in the problem and the level of statistical support for those answers. The results for the 76 students are reported in the following table and graphically in the chart.

n RF Performance
10 0.132 An appropriate analysis with optimal statistical support for that analysis
20 0.263 An appropriate analysis with reasonable statistical support for that analysis
9 0.118 An appropriate analysis with minimal statistical support for that analysis
18 0.237 Specific questions are answered correctly but without statistical support
17 0.224 An inappropriate analysis with incorrect answers to posed questions
2 0.026 A statistical analysis that should have led to correct answers to posed questions, but those questions were left unanswered.

51% of the students answered correctly with varying levels of appropriate statistical support. Another 24% were able to obtain correct answers but did not cite supporting statistical evidence. Just under a quarter of the students, 22%, answered incorrectly. Some of the incorrect answers included the appropriate statistical analysis, but the wrong conclusion was drawn from the results. Other incorrect answers were when the student left the section blank. Observations during the examination did not indicate that students "ran out of time" to work this section, but rather simply did not know what to do with the open data exploration.

Overall performance on this section on a point basis was weak with a 36.5% average on this material - only optimal answers received full credit in points.

Data rarely comes wrapped up with nice neat specific statistical calculation questions such as "What is the mean of this data?" Data comes with general questions and the data analyst has to choose the appropriate tools for the analysis. The open data exploration seeks to probe the students' ability to handle data in the "wild."

The three sections corresponding to the three student learning outcomes have been measured since the fall of 2012. The following chart provides some historical perspective on these values.

The chart shows student success rate performance against the three proposed outcomes for the past five terms. The uppermost, yellow, circles are performance levels on single variable basic statistical calculations - course learning outcome number one. The middle, blue, circles are performance levels on two variable linear regression calculations - course learning outcome number two. The lowermost, orange, circles are the performance on the open data exploration. Note that the variation in performance on the open data exploration is due in part to differences in the marking schemes term-on-term. To some extent the open data exploration is not comparable across terms due to differences in the marking schemes. The marking scheme used fall 2014 is similar to that used spring 2014 and those performances can be compared. 

Overall students prove quite capable, by term end, of making specific statistical calculations when told what calculation to make. When given data and questions that do not explain what statistics should be calculated, student performance is weaker. This addition of open data exploration to the course was stimulated by the American Statistical Association's Guidelines for Assessment and Instruction in Statistics Education. In the full report the ASA recommends to "give students plenty of practice with choosing appropriate questions and techniques, rather than telling them which technique
to use and merely having them implement it." The ASA also calls on statistics instructors to move from using naked or realistic data to using real data. My data remains in the realistic realm more than in the real realm, the exercise does require the student to choose the appropriate techniques. MS 150 Statistics continues to seek to implement best practices in the field of statistical education. 

Assessing learning in physical science

proposed outline for SC 130 Physical Science includes the following three course level student learning outcomes:
  1. Explore physical science systems using scientific methodologies
  2. Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science.
  3. Demonstrate basic communication skills by working in groups on laboratory experiments and by writing up the result of experiments, including thoughtful discussion and interpretation of data, in a formal format using spreadsheet and word processing software.

The first learning outcome requires that the students be able to explore physical science systems using scientific methodologies. For SC 130 Physical Science this exploration would be framed by the theme of mathematical models that underlie physical science systems. This, in turn, serves the general education program learning outcome, "3.5 Perform experiments that use scientific methods as part of the inquiry process."

Paulino times and drops, Alex holds meter sticks, Mailynda records data in laboratory three.

Laboratory nine involved the measuring of the speed of sound by timing the echoes off of claps of a pair of boards. Of twenty-six students in the course, twenty-two turned in laboratory nine. An analysis of the laboratories provides an assessment of the ability of the students to explore a physical science system using scientific methodologies.

The students were accustomed to the basic procedure of making a table, an xy scattergraph, adding a linear trend line, and using that result to analyze and discuss the laboratory results. Student success rates on different portions of this process are shown in the chart.

100% of the laboratory reports submitted included a properly formatted data table, all but one laboratory report had a correctly done xy scattergraph. The most common error students make is to use a line chart in lieu of an xy scattergraph. 17 students (77% of the 22) added a linear trend line to their graph. Four students did not add any trend line to their xy scattergraph. 16 students analyzed the nature of the relationship with all but two noting that the relationship was probably linear. Two students expressed the opinion that the relationship was non-linear. Only fifteen students were then able to correctly quote the slope in their analysis section of their laboratory report. The slope was the speed of sound, and measuring that speed was the intent of the laboratory.

The closing section of the laboratory report format in SC 130 Physical Science is a discussion of the results. Only five students had a discussion that showed clear comprehension of the intent of the laboratory. The remaining students clearly did not understand that the slope was indeed the speed of sound and that measuring this value had been the point of the laboratory.

Thus students are capable at performing the mechanics of analyzing data - making tables, graphs, adding trend lines, and incorporating these into a word processing document. The students are, in general, unable to understand what their data means. At higher levels of Bloom's taxonomy the students are less capable - analysis, synthesis, and evaluation of their results.

The second learning outcome, "Generate mathematical models for physical science systems and use appropriate mathematical techniques and concepts to obtain quantitative solutions to problems in physical science," serves, in part, the general education program learning outcome, "3.2 Present and interpret numeric information in graphic forms."This, in turn, serves an institutional learning outcome for quantitative reasoning: "Students will be able to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations; comprehend and can create sophisticated arguments supported by quantitative evidence and can clearly communicate those arguments in a variety of formats."

Underneath the focus on physical systems, SC 130 Physical Science is built on a foundation of connecting physical systems to their mathematical models and communicating the results in writing. Laboratory exercises lead to the writing of a full laboratory report that is marked for content, syntax, grammar, vocabulary, organization, and cohesion.

The majority of the laboratories investigate systems that involve a linear mathematical relationship. Reports include xy scatter graphs, best fit linear trend lines, slope, and y-intercept analysis.

Twenty-six students in physical science fall 2014 were given eleven questions which focused on interpreting and generating numeric information in graphic forms. Specifically, the pre-assessment focused on xy scatter graphs and linear trend lines. The pre-assessment was done on the first day of class and included 26 of the students in the course. Performance on the pre-assessment was weak at best.

SC 130 Physical Science is designed to address these mathematical weaknesses. The course has as one of its intents the placing of the mathematics into less abstract contexts. The concept is that the laboratory systems and data might provide cognitive hooks on which the students can attach a stronger comprehension of linear mathematical models.

Laboratories one, two, three, four, five, seven, nine, eleven, twelve, and fourteen involve linear relationships between the variables being studied. Non-linear relationships are also generated by some activities in the course. Although the students use spreadsheets to obtain the best fit trend line, the students were still working with concrete systems with variables that are related linearly.

Performance improved significantly on the same questions when those questions were included in the final examination.

The chart above depicts the number of students answering a question correctly on the pre-assessment on the left end of the line, the number of students answering a question correctly on the post-assessment on the right end of the line. Improvement is seen on all questions except the one question where students are asked to calculate an independent x value when given a dependent y value. This operation requires solving the equation for the independent x variable. Student performance on this question was actually relatively high on the pre-assessment and did not improve significantly on the post-assessment.

Improvement in student performance can also be seen when the distribution of the student scores is examined. Marking each question as being worth one point, the high score on the pre-assessment was ten of eleven correct and this was an outlier at over three inter-quartile ranges above the third quartile. The median correct on the pre-assessment was two. On the post-assessment the median correct was eight and a perfect score was not an outlier. On the pre-assessment there were students who scored zero correct, no student scored an absolute zero on the post-assessment. That said, two students answered only three questions correctly on the post-assessment.

The third learning outcome serves, in part, the general education program learning outcome, "1.1 Write a clear, well-organized paper using documentation and quantitative tools when appropriate."

When I took over and redesigned SC 130 Physical Science in 2007 I had two focuses. The dual focuses were to put mathematics and writing into the core of the course. By building laboratories around mathematical models and having students write up the results of those laboratories in reports marked for content, grammar, vocabulary, organization, and cohesion, both goals were simultaneously achieved.

The redesigned course is intended to include support for general education program student learning outcome 1.1. The course also now serves the second institutional learning outcome, "Effective written communication: development and expression of ideas in writing through work in many genres and styles, utilizing different writing technologies, and mixing texts, data, and images through iterative experiences across the curriculum." The laboratory reports include tables and charts prepared in a spreadsheet and then inserted into the final report using word processing software. Effective written communication also requires command and control of grammar, vocabulary, organization, and cohesion. This article reports on these writing metrics in a non-language and literature course.

The course includes 15 laboratories. Odd numbered laboratories include a full write-up with grammar, vocabulary, organization, and cohesion being marked. The exception is that laboratory 15 is not turned-in, so laboratory 14 is done as a full write-up laboratory. In the regular term eight full laboratory reports are done during 16 weeks.

Grammar (G), vocabulary (V), organization (O), and cohesion (C) are scored using a rubric with a total possible of 20 points. Each of the four metrics are scored on a 0 to 5 point scale. The rubric was reported in an earlier blog article. Laboratories one (1 in the chart) and nine (2 in the chart) were analyzed.

Performance on the metrics was not weak at term start on laboratory one. The rubric is based on an adaptation of the one used by the college to mark entrance test essays. With exceptions, most of the students fall 2014 had an ability to write at the start of the course. There is no writing pre-requisite for the course and the above data would argue that there does not need to be a writing pre-requisite to the course. The students most commonly make errors tense (tense shifts), singular/plural verb agreement, and spelling errors that often relate to sounds not present (or used differently) in their L1 languages (p and b swaps, t and d).

There was no significant improvement on the metrics from lab one to lab nine in part because the students wrote fairly well on laboratory one.

The median score lifted from 17 to 20, a difference that was not significant. There is only the suggestion that students may have focused more on their writing in laboratory nine than in laboratory one.

During the analysis and assessment of the course I noted that submission rates fell after a triple holiday in November (November 3, 7 and 11). The string of three day weekends, three and four day weeks, appears to derail student effort as measured by laboratory submission rates. In the chart below, these holidays hit after laboratory nine. Hence the use of laboratory nine in an analysis earlier in this assessment article.

The chart provides anecdotal evidence that the sheer number of holidays and their concentration in an eight day period, is detrimental to learning and may ultimately affect course completion rates. 

In a separate note, a proposal to put only course level student learning outcomes on outlines at the college is an excellent concept. The reduction from the number of specific student learning outcomes is exactly what allows for an analysis such as the one above - an analysis that I believe generates real and actionable information on learning. Course outlines should move to including only course level student learning outcomes, permitting instructors to provide more thoughtful analysis of those outcomes.

Thursday, December 4, 2014

Site Swap Notation in Physical Science

Daniel Kahneman in Thinking, Fast and Slow noted that the remembering mind rates experiences using a peak-end rule. Although I had not known this particular fact when I designed laboratory 15 five years ago, I had always shared George M. Cohan's belief that one should "always leave them laughing when you say goodbye."
Laslyn Siden

A class that ends on a pleasant activity makes for better memories looking back on the course. SC 130 Physical Science Laboratory 15 is both fun and yet is also a chance to introduce a mathematical model that is very different.
Judy Andon, three balls

Laboratory fifteen in physical science sought to push the boundaries on the mathematical box for the students. In laboratory one a quote from Freeman Dyson was used to start a journey through the mathematical models that explain physical systems. Dyson calculated how an electron ought to behave. Later someone went into a laboratory and the electron behaved as predicted by the mathematical model.
Judy Andon with three balls

In laboratory two a linear model predicted the location of a rolling ball. In laboratory three a falling ball obeyed a quadratic mathematical relationship. The behavior of a marble rolling off of a banana leaf obeyed a square root relationship. And in laboratory four the marbles knew what to do in order to mathematically conserve momentum. Sound, the relative depth of an image, and Ohm's law all exhibited linear relationships.
Shari Crystal Pablo

There are other mathematical relationships that govern physical systems. There are systems that are modeled by exponential and logarithmic functions. The path of a RipStik formed a sine wave on a sheet of paper. There are exotic functions such as the hyperbolic sine and hyperbolic cosine. Some systems are best described by complex variables that include a real and an imaginary component. Many of these systems are beyond the mathematical scope of this course.
Lodonna Osawa

The relationships described above are algebraic mathematical models. Much of the mathematics curriculum is centered on algebra in part because algebra is important to describing the physical world. There are, however, other mathematical models, non-algebraic models. This laboratory seeks to broaden the students mathematical horizons by introducing a mathematical model and notation that is not algebraic. In laboratory fifteen the students were introduced to the mathematics of site swaps.
Mailynda Maycry

In an attempt to connect site swap theory back to the language of algebraic equations, after introducing site swap notation I referred to sequences such as 33342333 as site swap equations. The sequence is a mathematical statement that can be true (can be juggled) or false (cannot be juggled).
Judy Andon, four ball multiplexing

This laboratory continues to provide a fun way to wrap up a term of exploring the mathematics at the core of physical science while expanding the students thinking with a mathematics system like nothing they have ever seen before.

Judy Andon, four ball with multiplex catches
The laboratory is also an end of term enjoyable experience. As the course is not required by any major, the students are primarily from majors other than those in the natural sciences. For many of these students science is a requirement, possibly even a dreaded requirement.
Judy Andon, four ball start

I want the students to have the chance to do science, engage in exploring systems, grapple with the mathematical language underneath physical science. I can only hope that the students catch a glimpse of the beauty of science - of even pure science for science's sake.
Judy Andon, three balls aloft in a four ball juggle. Impressive.

Friday, November 28, 2014

Cultural ceremony with Piper methysticum

On Thursday 28 November the SC/SS 115 Ethnobotany class visited soumas en kousapw Nan Madap, Usepio Hadley.

The drive up onto the ridge line was scenic and all uphill.

At the nahs the wie sak were already in place waiting for the class. Two peitehl were put into action each with four pounders, shirtless as is the tradition, coated in coconut oil.

The second peitehl would later have four pounders in place.

The students arrive

Sakau (Piper methysticum) enters the nahs replete with stems.

The stems are cut.

The class was seated in the place of honor on the high platrom. Sandra You, Kevina Berngun, Katielyne Nianugmwar, Melody Tulenkun in the front row. An oaurir for the wife of the soumas on the left.

Elson Elias, Judy Andon, Dwayne Hadley, Joemar Wasan, and Maylani Clarence

Soumas en kousapw with his oaurir.

Elson takes notes.

Pounding the sakau.

There is a musical quality to the pounding. Note that some pounders stop pounding the sakau and play a something akin to a double time beat on the edge of the peitehl (the large stone on which the sakau is pounded).This is a performance, a celebration, a happy tune.

The menindei confers with soumas

Women of the kousapw bring drinks and food to the gathering. The class has not seen such a welcome nor such a ceremony elsewhere. Two stones ringing together, full formal kamadipw style celebration. The kousapw showed great respect and love for their soumas, and made the students feel welcome as members of the family.

Soumas addresses the gathering, welcoming the students and noting the need for education to include the culture and traditions.

Afterwards a few students stayed back, here Kanio Torres takes over wungwung demonstrating his ability in the traditional arts.
Trickson Ladore, Marvin Louis

Rockson Salihk, Marvin, Trickson

Forming the Hibiscus tiliaceus phloem wrap around the Piper Methysticum

Kanio makes the power

The class owes a debt of gratitude to Soumas en kousapw Nan Madap and to Francisco Mendiola who made the arrangements for the class to visit. Kalahngan!

Tuesday, November 18, 2014

Banana patch cleaning and some tentative banana identifications

On the 18th of November the SC/SS Ethnobotany class cleaned up the banana patch, which continues to be overrun by Clidemia hirta, and I worked with the students on identifying those with fruit or flowers.

This banana at N 6° 54.663' and E 158° 9.329' provoked the most discussion. The location best matches a rhizome planted by Roxann Moya and thought by her to be karat. Clearly this is not karat. The initial determination was kerenis, although another thought it might be uhten wai, which would be a AAA Cavendish. A third student thought it was simply a young uhten ruhk. Utin kerenis, which may be a reference to Kapinga, is an AA; Pisang Raja banana by my sources. Utin Ruhk is an ABB Saba banana.

The same banana at N 6° 54.663' and E 158° 9.329'.

Another view of the banana at N 6° 54.663' and E 158° 9.329'.

The above banana at N 6° 54.670' and E 158° 9.328'well matches the location of an uhten rais planted by Karmi Soar at N 6° 54.670' and E 158° 9.330' . The student concurred in the uhten rais apellation.

Uhten rais at N 6° 54.670' and E 158° 9.328'

Uhten rais at N 6° 54.670' and E 158° 9.328'

Uhten rais at N 6° 54.670' and E 158° 9.328'

Although not fruiting, the banana at N 6° 54.671' and E 158° 9.330' appears to roughly match the location of an uhten menihle planted by Joey Seiola. Some students said this was an uhten kuam which they claim is the same as an uhten menihle. My sources say both are AAB; Silk but finger is longer in utin kuam (source spelling), menihle has a smaller finger. That kind of distinction could come down to soil differences.

 Virginia Sartilug.

 Judy Ligohr
 Judy Andon, Shirley-ann Rudolph

 Katielyne Nianugmwar

 Rockson Salihk, Katielyne

 Melody Tulenkun

 Jake Manuel


 Dwayne Hadley

 Gary Totong


 Joemar Wasan, Dwayne

 Judy Ligohr

 Marvin Louis


 Jennnifer Panuelo with a knife, Kanio idle in the background

 Arnold Panuelo, observer only

 Katielyne and Andrea Ewarmai
 Judy Ligohr, left, and Judy Andon, still working when most were now resting including Maylani on the left in the background. Dwayne, Marvin, and Joemar were also still at work.
 Dwayne still working
 Judy Andon also still working

 Jake cleaning up around a Daiwang in the background, Ronda up front.

 Fence damage.

 Daiwang, AAB; Pisang kelat bananas

 Jake Manuel and Dwayne Hadley cleaned up around the Daiwang banana planted at at N 6° 54.673' and E 158° 9.332'. Rockyner Hadley claimed to haveplanted a Daiwang at that same location in the corner, coordinates then of N 6° 54.674' and E 158° 9.333' Dwayne identified the bananas as Daiwang before knowing that Rockyner had planted a claimed-to-be Daiwang, so that seems to be confirmatory for me.

There are more to be identified, but without fruit this is difficult. The Clidemia hirta may be allelopathic and slowing the growth of some bananas.