What's in a Report Card?
by MikeGeneAccording to PZ Myers, the "The Fordham Foundation report is stirring some people up." They've handed out their report card on science standards, causing someone at the New York Times to follow up with some hand-wringing:
The report also appears to support concerns raised by a growing number of university officials and corporate executives, who say that the failure to produce students well-prepared in science is undermining the country's production of scientists and engineers and putting the nation's economic future in jeopardy.
As you can see here, the lead author of the Fordham Foundation report is Paul Gross. While I appreciate all the effort that goes into grading the science standards in various states, I'm left wondering what the grades actually mean in the real world. One could appeal to common sense and argue that excellent science standards in an educational system will produce students with a better understanding of science, but is there any evidence for this belief? So I decided to put Gross et al.'s report card to a simple test.
Thanks to the people at the Panda's Thumb, we have the Science Grades and the Evolution Scores for the various states. What if we compared these to the average science ACT scores from the same states? The ACT test is a standardized test that is given to many students in all states and colleges/universities use the scores as a criterion for admission, given they serve as a fairly good predictor of success in higher education.
Figure 1 shows a state's science standards (as graded by Gross et al.) compared to the same state's average science ACT scores.
Figure 1. 1=F; 2=D; 3=C;4=B; and 5=A.
I don't see any correlation. The A states don't produce students who do any better on the science section of the ACT than the B states, the C states, the D states, or the F states. In fact, the spread of ACT scores are much the same regardless of the score that Gross et al. handed out.
Figure 2 compares the ACT scores to the Evolution score and again we see no correlation and the same basic spread.
Figure 2
Of course, one could always argue that the ACT science scores used by colleges and universities don't truly reflect the student's understanding of science. Or, one might argue that having standards and implementing them are different things. Nevertheless, I'm still left wondering if standards and Gross et al.'s grades have anything to do with producing "students well-prepared in science."
The NYT article cited by Myers states,
"Many states are not yet serious about teaching science," said Michael Petrilli, vice president for national programs and policy of the institute, a group that supports education reform. "The first step is to set higher expectations, and too many states have low or a lack of expectations to respond to the new global competitiveness."
But where is the evidence that these standards have any effect on producing students to "respond to a new global competiveness." Where is the evidence that Gross et al.'s grades actually measure the ability to expand our "talent pool in science to stay ahead of countries like China and India that put vast resources into science education?"
I'm certainly not arguing science standards are bad. But you would think people who are preaching about the importance of science would approach the problem like scientists.
December 9th, 2005 at 12:06 pm
WhatÂ’s in a Report Card?
MikeGene has attempted a simple test of the hypothesis that "excellent science standards in an educational system will produce students with a better understanding of science" by comparing state ACT scores to state ratings from the Fordham Foundation…
Trackback by Damion's Journal — December 9, 2005 @ 12:06 pm
December 9th, 2005 at 12:51 pm
Great analysis. I was going to ask does A=1, B=2 or are they in "GPA" order–but then I realized with the lack of correlation, it doesn't really matter.
Comment by David Heddle — December 9, 2005 @ 12:51 pm
December 9th, 2005 at 2:20 pm
Well, having gone on over to read what 'Damion' had to say about Mike's observations, there's one extremely funny conclusion in that blog which is even more silly than the denial (apparently a river in Egypt) that public grandstanding by people like Gross using manipulated statistics doesn't mean anything. It goes like this:
According to census figures (available at census.org) over the last 5 decades or so, all the scientists, engineers and technicians in the US (not including physicians and technicians in health care) have consistently numbered right around 2 - 2.5% of the overall population. And this includes the 1960 figures, when the good ol' Cold War led to mass recruiting in sciences so we could "beat the Russians." If 'Damian' is correct about science claiming only the top scorers on ACTs, we can go ahead and figure his designation ("top scorers") to represent the top ~10%. Most of whom go into medicine or law or business management or advertising or some other career.
Which means science isn't increasing its attractiveness to the top 10% of high school graduates over the percentage it attracted in 1960. When public science education didn't seem to have that much trouble teaching kids how to read, write, do math, understand whatever science was telling them this year, etc., etc. Part of this is no doubt the fact that science isn't well known to be a particularly lucrative career choice. American commercial, consumerist culture and its worship of wealth. Another part of this is one you won't see people like 'Damion' or Peez admitting to: what the public is taught about science this year is known to be likely to change next year (we went from Steady State to Big Bang in only one year once, per what public schoolchildren were taught, merely because the textbooks changed), so it's not appealing for its "truth value." It doesn't mean a whole lot to the majority of people, who need never once use what they were taught about evolution in high school over the course of their entire lives, while still managing to live a productive, successful and meaningful life. That's a plain fact.
As the overall population increases, the number of scientists should also increase per their standard 2.5% rate. Of course, in order for the real numbers in that 2.5% figure to actually rise along with the number of citizens in the country (increasing daily), there would have to be actual paid work available for those increasing numbers of scientists. Questions about societal funding of 'big' science projects are a whole different issue from teaching RM-NS as Absolute, Unassailable Truth in public high schools.
My father lost his job along with thousands of other scientists and engineers when funding for the space program went away. He had to take a job at a factory for less than half his previous salary, despite the fact that he was a recognized genius - not just top 10%, but well into the upper portions of the top 1%. If he was so necessary to science, and scientific hegemony was so necessary to America, why did he and most other scientists and engineers he knew get diverted to grunt industry? Too many of 'em, right?
If American science doesn't have the positions available for American science graduates, it's not going to maintain even its meager 2.5% draw. If science wants to make itself worthy of more than 2.5% of the population and its current portion of the wealth, it should do something interesting and worthwhile, stop insulting the taxpayers and labor pool.
Comment by Joy — December 9, 2005 @ 2:20 pm
December 9th, 2005 at 3:12 pm
Far be it from me to say that the report can be dismissed as a political stunt just because Paul Gross led it. However, if his science standards are anything like his factual standards in Creationism's Trojan Horse, I wouldn't find it entirely surprising if they didn't map neatly to an objective reality.
Dave:
Well, keeping in mind that this report is by Paul Gross, and noting that it just so happens that no state got above a 3 according to his Evolution Standard, I think we can make an educated guess
Comment by Deuce — December 9, 2005 @ 3:12 pm
December 9th, 2005 at 3:47 pm
It's just Fun With Statistics as political stunt ala Gross. That's what 'Damion's' argument about the "high scorers" on ACTs was all about - made total mincemeat of himself on that.
The top 10% of scorers is still the top 10% no matter how many people happen to be in the population sampled. Science has for half a century managed to provide jobs for about 2.5% of those (along with quite a few who only earned 'Cs' in high school and college). With periodic economic and policy fluctuations such as the massive lay-offs that my father's generation suffered. Before that it was the "Great Depression," which had thousands of scientists who came up in the first decades of the 20th century seeking grunt work for CCC and such. And science's percentage of the overall population in those days was less than 2%.
So, one might reasonably ask, what does belief-in RM-NS neodarwinism among 100% of American high school students have to do with the top 10% of ACT scorerers or the 2.5% of those that science has been able to recruit over the last half century? Answer: nothing.
Duh…
Comment by Joy — December 9, 2005 @ 3:47 pm
December 9th, 2005 at 4:24 pm
Well, note that he has tried to change the standard of success from the actual performance and number of kids who get high scores, which is what Mike used, to the "conception of science" - a vague, unmeasurable quantity - that the kids who perform well possess. Of course, the "conception of science" that they may have is only a problem in this context inasmuch as it affects their performance and the number of them that there are, so the argument collapses in on itself.
Comment by Deuce — December 9, 2005 @ 4:24 pm
December 9th, 2005 at 5:42 pm
So, what would it mean if fundamentalist Christian students, schooled in fundieism, happened to score, on average, better on the ACT science portion than students trained in the factuality of Evolution? One could apply this to increasingly high ranges of scores, as well.
Comment by Douglas — December 9, 2005 @ 5:42 pm
December 9th, 2005 at 5:46 pm
I hear you, Deuce. I didn't want to be accused of doing what Gross did by attacking an unwritten implication rather than what was there. 'Damion' argues that the demonstrated non-existence of a correlation between Gross' scorekeeping and ACT scores is irrelevant because it's what the top scorers know about science that counts. Which I can't imagine would present any sort of problem, if those are the only ones who count. I was going with "knowledge" - as related to teaching/learning - rather than with "conception," which is an idea conceived in mind ABOUT what is known.
Teachers these days teach to the tests. Kids are all very familiar with them - know what they are and how they work. If high school students have to answer "RM-NS" when asked what the mechanisms of evolution are, the teacher will certainly let them know that's the expected answer. It doesn't matter one bit whether any student (or teacher) believes RM-NS is the whole story of evolution - or even believes evolution ever happened at all - so long as they know the expected answer. ACT doesn't test beliefs, it tests knowledge. About the questions it asks.
Of course this does harken back to the "hidden agenda" that's not really hidden. The true concerns of passionate DarwinDefenders are about what people believe, not what they're able to correctly answer on a test question. I took biology I and II, chemistry I and II, physics I and II and all prerequisites in high school. I studied chemistry, crystallography and physics in college. I worked in physics long enough to get disgusted. I support the scientific inference of teleological design in biology and evolution anyway. Learning the consensus view doesn't necessitate belief-in the consensus view. Never has, never will.
Comment by Joy — December 9, 2005 @ 5:46 pm
December 9th, 2005 at 6:57 pm
David,
LOL. I just added it.
Comment by MikeGene — December 9, 2005 @ 6:57 pm
December 9th, 2005 at 11:58 pm
Damion says:
I don't deny the intuitive appeal of this point. But where is the scientific evidence to back up this claim? It would seem to me that kids who remained awake, alert, and attentive in science classes would learn about the need for evidence. After all, Damion notes that he aced the ACT science test and went on to obtain advanced degrees in science/engineering despite the glaring omissions of his home state. In fact, as far as we know, his state didn't even have any standards when he aced his ACT as a teenager.
Damion misses the point of the Gross et al. survey. They spell it out quite clearly (p. 8 ):
Notice it's not the "state's brightest children," it's just the "state's children." And Gross et al. provide no evidence to support their claim. After all, they have just shown that state's with flabby, vague, or otherwise useless standards seem to deliver a science education that is no worse than the states they praise.
Comment by MikeGene — December 9, 2005 @ 11:58 pm
December 10th, 2005 at 12:09 am
Another baseless study on evolution in schools? What a surprise…
Kansas is 16th on that list and most students take the ACT; I wonder if the Darwinists expect that number to roll off a cliff. The facts say otherwise, but since when have they ever relied on the FACTS?
Comment by jasonng — December 10, 2005 @ 12:09 am
December 10th, 2005 at 7:25 am
jasonng's comment remainds me, ACT scores have some systematic bias state-to-state. For regions where the SAT is emphasized over the ACT, it will be mainly the students applying to out-of-state schools who take the ACT, and they probably have higher-than-average scores.
Comment by ragesoss — December 10, 2005 @ 7:25 am
December 10th, 2005 at 7:34 am
Actually, upon perusing the state ACT scores compared to the percent of graduates tested, no overwhelming trend jumps out to support the fewer test-takers/higher scores hypothesis. A lot of the high averages are in states with high percentages tested. But some correlation may be there.
Comment by ragesoss — December 10, 2005 @ 7:34 am
December 11th, 2005 at 5:23 am
Yes that's what I would expect as well but for it doesn't seem to show up that much. I guess out of state applicants aren't necessarily better academically since there's so many anyway. That said, the ACT site does stress that since not everyone in every state takes the ACT the state average scores should be taken with caution.
Comment by jasonng — December 11, 2005 @ 5:23 am