General Education Science Assessment Report 2009 – 2012
General Education Science Assessment
Report 2009 – 2012
The general education science
assessment has been performed for the past three academic years. Each
fall a sample of laboratory reports was analyzed against a rubric.
Performance factor score | |||
4 | 3 | 2 | 1 |
Metric: Scientific Procedures and reasoning | |||
Accurately and efficiently used all appropriate tools and technologies to gather and analyze data | Effectively used some appropriate tools and technologies to gather and analyze data with only minor errors | Attempted to use appropriate tools and technologies but information inaccurate or incomplete | Inappropriate use of tools or technology to gather data |
Metric: Strategies | |||
Used a sophisticated strategy and revised strategy where appropriate to complete the task; employed refined and complex reasoning and demonstrated understanding of cause and effect; applied scientific method accurately | Used a strategy that led to completion while recording all data; used effective scientific reasoning; framed or used testable questions, conducted experiment, and supported results with data | Used a strategy that led to partial completion of the task/ investigation; some evidence of scientific reasoning used; attempted but could not completely carry out testing, recording all data and stating conclusions | No evidence of procedure or scientific reasoning used; so many errors, task could not be completed |
Metric: Scientific communication/using data | |||
Provided clear, effective explanation detailing how the task was carried out; precisely and appropriately used multiple scientific representations and notations to organize and display information; interpretation of data supported conclusions and raised new questions or was applied to new contexts; disagreements with data resolved when appropriate | Presented a clear explanation; effectively used scientific representations and notations to organize and display information; appropriately used data to support conclusions | Incomplete explanation; attempted to use appropriate scientific representations and notations, but were incomplete; conclusions not supported or were only partly supported by data | Explanation could not be understood; inappropriate use of scientific notation; conclusion unstated or data unrecorded |
Metric: Scientific concepts and related content | |||
Precisely and appropriately used scientific terminology; provided evidence of in-depth, sophisticated understanding of relevant scientific concepts, principles or theories; revised prior misconceptions when appropriate; observable characteristics and properties of objects, organisms, and/or materials used; went beyond the task investigation to make other connections or extend thinking | Appropriately used scientific terminology; provided evidence of understanding of relevant scientific concepts, principles or theories; evidence of understanding observable characteristics and properties of objects, organisms, and/or materials used | Used some relevant scientific terminology; minimal reference to relevant scientific concepts, principles or theories; evidence of understanding observable characteristics and properties of objects, organisms, and/or materials used | Inappropriate use of scientific terminology; inappropriate references to scientific concepts, principles or theories |
The rubric generates four scores on a
scale from one to four. The maximum is sixteen. Two assessors
evaluated each laboratory report. The scores for both assessors are
combined yielding a total possible sum of 32 per laboratory report.
Each of the four subsections has a maximum possible of eight (four
plus four from each grader) per laboratory report.
Over the course of three academic
years, 2009-2010, 2010-2011, and 2011-2012, three different teams of
assessors evaluated the reports. Three different authors then wrote
up reports which are referred to as Assessment Report Worksheet #3
at the College of Micronesia-FSM. Laying the three different reports
side-by-side, one is left unable to draw comparative conclusions
across the three academic years. Setting aside the issue of different
teams of assessors for each of the three years, each report chooses
different data, different data presentations, and different analyses.
After the completion of this report,
the original data was found for 2010-2011. Prior to access to this
data, Assessment Report Worksheet #3
for 2010-2011 was used to reverse engineer the table values. The
discovery of the data revealed that only one assessor was involved in
marking each laboratory report. In order to generate comparable data
after-the-fact, these values were doubled to produce scores out of
32.
The
raw data for 2009-2010 generates results which do not match the
results reported Assessment Report Worksheet #3
for 2009-2010. Unable to generate matching data, this report chooses
to use values from a re-analysis of the 2009-2010 data.
The key results are
summarized by the averages in the following table. The total possible
is noted in the first column.
Category
|
2009
|
2010
|
2011
|
Chuuk /32
|
11.57
|
|
10.28
|
Kosrae /32
|
9.75
|
18
|
13.79
|
National /32
|
13.77
|
20.41
|
13.91
|
Pohnpei /32
|
7.5
|
21.64
|
12.2
|
Yap /32
|
11.5
|
20.55
|
|
SC117 /32
|
9.69
|
22.36
|
10.04
|
SC120 /32
|
12.58
|
18.97
|
13.69
|
SC130 /32
|
10.6
|
21.29
|
14.83
|
SC255 /32
|
13.57
|
|
|
Sci Processes /8
|
3.18
|
5.41
|
3.74
|
Strategies /8
|
3.16
|
5.30
|
3.28
|
Communication/8
|
2.73
|
4.90
|
3.01
|
Concepts/8
|
2.86
|
4.90
|
2.74
|
Overall averages /32
|
11.92
|
21.9
|
12.77
|
The strong rise in
scores for 2010-2011 is not due to any know intervention strategy and
is considered to be an artifact of the changes in the assessors with
each academic year. N/A refers to data that is either unavailable or
which was not gathered. SC255 was dropped from the study after the
first year, SC117 has been dropped for the 2012-2013 academic year.
Data analysis was not reported for Chuuk in 2010, nor Yap in 2011.
The above tabular
data is presented graphically below.
In the following
discussion academic year 2010-2011 is generally set aside as being a
marking standards issue. Thus there may be small improvements
2009-2010 to 2011-2012 for Pohnpei and Yap sites. The National site
held at a constant level and first rank in 2009-2010 and 2011-2012.
The small sample sizes, reported in the table below, for this study
mean that small differences in average performance are not
statistically significant.
Sample n
|
2009
|
2010
|
2011
|
Chuuk
|
7
|
|
32
|
Kosrae
|
8
|
9
|
19
|
National
|
26
|
29
|
65
|
Pohnpei
|
6
|
22
|
25
|
Yap
|
4
|
11
|
|
Note too that
sample sizes are not consistent across the years.
The results as
reported by course, setting aside 2010-2011, suggests some
improvement for SC 130 Physical Science and SC 120 Biology. An
instructor for SC 117 noted in an August 2012 conference that the SC
117 does not produce laboratory reports which are aligned to the
particular rubric being used. The instructor felt that SC 117 should
be removed from the general education science program assessment in
the future.
The following table
provides the sample size for each course in each academic year.
Sample n
|
2009
|
2010
|
2011
|
SC117
|
13
|
11
|
47
|
SC120
|
19
|
29
|
58
|
SC130
|
5
|
31
|
36
|
SC255
|
14
|
|
|
Performance by the
specific subcategory is out of eight possible rather than the 32
possible when all four sections are combined. Thus the above is the
average score per laboratory out of eight possible. Excluding the
2010-2011, no subcategory has shown a strong movement system-wide.
There may be a small up tick in performance on scientific processes
and reasoning skills.
The above chart is
“rotated” below to match a chart presented in the 2010-2011
report.
This
chart appears different from that seen in the 2010-2011 Assessment
Report Worksheet #3 primarily
due to the inability of this author to generate the numbers seen in
the report from 2009-2010.
Overall there may
be a small statistical improvement from 11.92 in 2009-2010 to 12.77
in 2011-2012.
For the 2012-2013
academic year the assessment team has recommended resampling and
remarking laboratory reports from across all four academic fall terms
(2009, 2010, 2011, 2012). This effort is scheduled to occur early in
2013. The hope is that this will provide a set of comparable scores.
A summary of the past assessment reports is available in a WordPress blog.
Comments
Post a Comment