Back in 1997, California made a bold statement for providing all children with strong, comprehensive college-preparatory curricula when it mandated that all districts and schools should provide seventh- and eighth-grade students with Algebra 1, the key course for helping children master advanced mathematics. But between the time the state legislature mandated the coursework and the move this year by the state board of education to drop the mandate, traditionalists — including the  lobbying group representing traditional districts, the California School Boards Association, and the Association of California School Administrators (which represents school leaders) — did all they could to hold off full implementation. It took seven years for the state to develop the actual standards for the state’s Algebra 1 curricula. Then it took another four years before the state mandated that its battery of standardized exams would be the only test used to determine whether middle-schoolers reached any level of proficiency. Even after the state mandated the test, CSBA and ACSA sued to block the use of the test because they wanted they wanted kids to take a less-challenging exam that would have not forced districts to actually follow through with the state’s Algebra 1 mandate.

wpid10020-wpid-this_is_dropout_nation_logo2.pngAs a result, only 281,890 California seventh- and eighth-graders — or 30 percent of all middle-schoolers in those grades — took Algebra 1 by 2009-2010, according to Dropout Nation‘s analysis of data from the U.S. Department of Education’s Office for Civil Rights. That is certainly far fewer than all middle-schoolers taking the course. The lack of full implementation of the Algebra 1 standard — and the fact that so few children were provided that curriculum — is likely one reason why California reduced eighth-grade math illiteracy by four percentage points between 2005 and 2011 — just keeping pace with the decline in innumeracy across the nation in that same period — according to the National Assessment of Educational Progress, and why the state’s three percentage point increase in number of middle-schoolers scoring at Proficient and Advanced levels trailed the four percent gain for the nation as a whole. With so much resistance from districts and other traditionalists — and with the change in the reform climate from the strong efforts undertaken by governors Pete Wilson and Arnold Schwarzenegger to the weaker approach of once-and-future governor Jerry Brown — it’s little wonder why the state education board members, under the guise of enacting Common Core math and reading standards, took the opportunity this year to ditch the Algebra 1 push.

California isn’t the only state in which efforts to provide kids with the college preparatory curricula don’t always lead to traditional districts actually doing so. There are numerous obstacles — from obstructionism by districts as well as affiliates of the National Education Association and American Federation of Teachers, to the low quality of teaching in mathematics, to the Simply legislating that all children should be provided college preparatory math isn’t enough to make it happen — a lesson reformers who work on the ground (as well as reporters who cover such efforts) understand all too well. Education researchers should also consider such context and incorporate it into evaluating the results of a reform.

This lesson comes courtesy of Tom Loveless of the Brookings Institution, who recently raised questions about the efficacy of providing Algebra 1 to middle-school students in the latest edition of the think tank’s annual report on the state of education policy and practice. Based on Brookings’ analysis of NAEP data from 2005 to 2011, Loveless concludes that there was no evidence that increasing enrollment in Algebra 1 and other advanced math courses led to improvements in student achievement. Loveless admits that NAEP’s scale scores may “assess mathematics too broadly” to figure out what role efforts to provide Algebra 1 to middle school students is having on results, and concedes that the effort in California has so far shown positives in the form of more kids (especially poor and minority kids often denied such college preparatory work) participating in the courses and passing the exams (as well as thriving on the challenge). But Loveless still hypothesizes that the effort may not be having much of a positive effect on student achievement — especially when one compares states that have required Algebra 1 versus those who have not done so — for a number of reasons, including the possibility that Algebra 1 courses are being “watered down”. As a result, media outlets and others have run wild with Loveless’ analysis without giving it a closer look.

But Loveless’ analysis is incomplete. Why? Because he failed to consider the most-likely reason for why Algebra 1 (as well as other advanced math courses) hasn’t helped reduce innumeracy or increased the number of kids becoming proficient in math: The fact that few middle-school students are actually being taught introductory algebra in the first place. Dropout Nation has made this clear in its collection of briefs on how districts such as Philadelphia, Minneapolis, and Fairfax County, Va., fail to provide all kids with college preparatory curricula. Some of this oversight is understandable; Loveless was relying on responses to the NAEP questionnaire, which indicates that 47 percent of students taking the exam had taken algebra or some other form of advanced math. But even Loveless admits that students may have mistakenly believed they were taking such courses when that wasn’t so. That alone should have led Loveless to look at another U.S. Department of Education database that would have provided additional context: The Office for Civil Rights data collection, which garners its data from administrators who would have better data on the classes students are taking.

This publication, on the other hand, did look at the OCR data  — particularly at four states Loveless singled out for increasing enrollment in Algebra 1 and other advanced math, as well as for requiring districts to provide Algebra 1 instruction — and did some simple math: Dividing the number of students taking Algebra 1 in seventh- and eighth-grade in states that have mandated such courses to be taken from the number of kids in those grades overall. Sure, it isn’t multivariate analysis. But this time around, none is needed. Now, because OCR’s database combines middle-school enrollment with high school numbers, getting the latter data can be tricky. But thanks to that other federal database called the Common Core of Data, one can get accurate data on middle school enrollment in each state. And the results were revealing.

dn_chart_algebra_2009_2010In Minnesota, one of the states Loveless cited, just 19 percent of seventh- and eighth-graders took the course in 2009-2010. Four-fifths of kids in those grades were not provided the coursework.  One reason lies with the fact that Minnesota didn’t require districts to provide Algebra 1 to middle-schoolers until it released its standards in 2007. Considering that it can take time for districts to implement the standards — Minneapolis’ traditional district, for example, expected to provide Algebra 1 to all students sometime around 2008-2009 and 2009-2010 — it is unlikely that the policy would have immediately helped kids improve their numeracy. This, by the way, is also true for Washington State, where Algebra 1 only became required learning for middle-schoolers in 2008; just 14 percent of Evergreen State seventh- and eighth-graders took Algebra 1 in 2009-2010. [Washington State, by the way, is still requiring middle-schoolers to take introductory algebra even as it is implementing Common Core standards.]

In Virginia, just 23 percent of seventh- and eighth-graders were taking Algebra 1; the other 77 percent of kids in those grades were not provided that college preparatory course. This shouldn’t be all that shocking given that Virginia didn’t mandate eighth-graders to learn introductory algebra until 2009, when the Old Dominion released the most-recent version of its Standards of Learning regimen. As a result, it is unlikely that districts would have fully implemented Algebra 1 by the start of the new school year. Even if all middle-schoolers were taking Algebra 1, it wouldn’t have meant the kids would have learned anything of high quality. The Thomas B. Fordham Institute gave a “C” rating to SOL’s math standards because it wasn’t comprehensive or strong enough.

Then there is Pennsylvania, which has mandated that middle-schoolers are provided Algebra 1 since 2003. Just 23 percent of seventh- and eighth-graders took Algebra 1 in 2009-2010, while the other 77 percent of their schoolmates didn’t take the course. This isn’t shocking; as Dropout Nation reported last year, just 45 percent of Latino students, 46 percent of white students, and 51 percent of Asian students were provided introductory algebra; Philly, oddly enough, is doing better at helping middle schools access this course than most of its peers in the rest of the Keystone State. Because of Pennsylvania’s decision to scale back that mandate as part of its move to implement Common Core — which the state (along with California) could have still required anyway — it is likely that even fewer middle-schoolers are taking Algebra 1 now than three years ago. [By the Way: Later this month, Dropout Nation will look at how many minority kids in each state are being provided access to Algebra 1.]

These numbers aren’t unusual. Just 19 percent of seventh- and eighth-graders in Florida — a state that Loveless didn’t cite in his analysis — took Algebra in 2009-2010. In fact, the U.S. Department of Education noted last year in its sampling of civil rights data that seventh- and eighth-grade students represented just a quarter of all students taking Algebra 1. The reality is that there are far too few middle-schoolers taking Algebra 1 (or any advanced math course) to actually have much of an effect on either NAEP scores or on any other macro measure of impact of systemic reform. [There’s also the fact that Loveless doesn’t consider that the benefits of Algebra 1 may not initially show up in eighth grade, but over time as kids progress into high school ready to take on the challenges of higher level math, or the need to improve math instruction and curricula in the early grades, but those are different discussions.]

Certainly there may not been much improvement in scores even if every middle in each of these states were taking Algebra 1. After all, only three of the five states covered in this analysis — California, Washington State, and Minnesota — had math standards rated A or B by Fordham in its most-recent analysis, while none of the states on the list ranked as having eighth-grade math standards equal to the top seven nations in math, according to the American Institutes for Research’s 2010 report on curricula standards. But Loveless doesn’t offer any evidence that providing Algebra 1 to all students doesn’t help kids succeed. If anything, based on the California evidence Loveless cited (as well as on the work of Education Sector’s Peter A. Cookson Jr. and Constance Clark, as well as Allan W. Gottfried of California State University, Fullerton), more-challenging curricula (along with high-quality teaching) actually works.

This isn’t to say that Loveless’ analysis doesn’t raise important questions. The fact that so many young adults in the middle grades are not getting opportunities to take advanced math — even in states where there are strong efforts to require schools to provide it — is one more example of how American public education cheats so many kids out of brighter futures. It is also another reminder that reformers have to work harder on the ground to make sure efforts come to fruition. But Loveless’ analysis doesn’t include the context needed to understand why efforts to provide middle school students with Algebra 1 are not working as well as expected. And the need for such context is one all education researchers should incorporate into their analysis.