Your editor didn’t initially pay much attention to the report released last week by the Broader Bolder Approach gang criticizing systemic reform-minded regimes in New York City, Chicago and Washington, D.C.. For one, the 17-page piece is supposedly the very long executive statement for a full report to be released sometime this week. This is certainly not the practice of respectable organizations in the education space, who release their studies in full. The fact that Broader Bolder is associated with the Economic Policy Institute, the outfit whose education studies always curiously dovetail nicely with the positions of two of its biggest backers, the National Education Association and American Federation of Teachers, also makes any report coming from it rather suspect. Then there’s the fact that the report seems to be a clip job, drawing its conclusions from news clips and and other pieces from the likes of Diane Ravitch acolyte Leonie Haimson instead of from original research. Add in the fact that the report strategically leaves out other reform-oriented districts, especially Boston and Houston (largely because there’s no political gain to be made by traditionalists there) and you can understand why I thought the report wasn’t worth my time.
But then I decided to pay the Broader Bolder report a little more mind. After all, the group is trying to revive itself after fizzling out since its debut on the scene five years ago. While other traditionalists have largely moved beyond Broader Bolder onto their own efforts (including once-respectable education historian Diane Ravitch’s Network for Public Education, as well as the NEA’s and AFT’s efforts to co-opt progressives in the Democratic Party ranks), Broader Bolder still retains as its co-chair Pedro Noguera, the New York University professor whose occasional dissension from traditionalist thinking shows his timidity in challenging the thinking of his fellow-travelers (as well as his own). So I read it. The conclusion: The summary, authored by Broader Bolder staffer Elaine Weiss and Don Long (who serves as a consultant for both the organization and EPI) have put together one of the more-stunningly shoddy reports ever published by a policy or advocacy outlet. It does little to make the case for Broader Bolder’s traditionalist agenda, or make a strong case against systemic reform. In fact, one can say that the report actually embarrasses Broader Bolder and all those who back it.
In the Broader Bolder report, Weiss and Long cite the notorious 2009 report released by Stanford’s Center for Research on Education Options to proclaim that charters are no better than traditional public schools. Of course they would. What they fail to note is that the CREDO study is flawed because it matches individual charter schools to groups of traditional public school students; they also fail to admit that the study is really a series of reports that show the wide differences in charter school oversight in 15 states and the District of Columbia and not necessarily a measure of the long-term success or failure of charters. Meanwhile Weiss and Long studiously ignore CREDO’s follow-up studies of charters which have shown how well charters have performed against traditional districts over time. In New York City, for example, CREDO noted that 75 percent of charter school students either outperformed or stayed apace of traditional district counterparts in literacy (and 86 percent of charter school students either outperformed or stayed apace of traditional district peers in math). The fact that Broader Bolder chose to cite CREDO’s outdated research over more-recent long-term data makes the Broader Bolder report more of a propaganda piece than anything to be considered empirical.
The Broader Bolder report plays fast and loose in its approach to using data to make its case. For example, Weiss and Long cited National Assessment of Educational Progress data on D.C.’s traditional district to criticize former chancellor Michelle Rhee’s declaration during her tenure that students improved their reading and math performance on the District of Columbia’s battery of standardized tests; from where they sat, D.C.’s standardized tests were too “arbitrary” to be used by Rhee as evidence of success. [The fact that Weiss and Long includes NAEP data from 2005 — or two years before Rhee left teacher quality reform outfit TNTP — to make part of its case is particularly sloppy.] But then, Weiss and Long used data from New York State’s standardized tests to supposedly fact-check New York City Mayor Michael Bloomberg’s claim that the city significantly reduced achievement gaps; by the way, cut scores for Empire State tests (along with the underlying content) has been revised within the past decade. The attempt to have it both ways becomes especially ridiculous when you keep in mind that proficiency levels for tests in both states are lower than those set by NAEP for Proficient and Advanced levels.
This craptastic approach becomes even more apparent when it admits that its critique of New York City’s school reform efforts are based not on actual NAEP scale scores, but on adding together fourth- and eighth-grade NAEP reading and math scores together and then averaging them out. This approach to NAEP data is one that the U.S. Department of Education doesn’t do at all in its reporting, and not something researchers with any level of credibility attempt at all. [By the way, Broader Bolder seems to have gotten the idea for doing so from both Haimson’s own effort and that of Aaron Pallas, the resident traditionalists over at Columbia University’s Teachers College.] The better approach would have been to break out fourth- and eighth-grade reading and science scores, then compare score increases in each category to that of other large districts as well as overall national results. Since Broader Bolder has not published a full report, it is hard to know whether it did so. But based on how Weiss and Long have presented the data, it’s hard to imagine that they even tried.
Certainly Broader Bolder could have taken a more-systematic, long-term, and data-driven approach to analyzing the success or failure of reforms in at least New York and Chicago. [D.C. would be left out because it is now just in its sixth year of its reform efforts, although one could look at the success of its IMPACT teacher evaluation system.] This starts with asking a few questions such as how well have the districts done in improving literacy, or what are the percentage increases in graduation rates overall as well as among poor and minority kids, or what percentage of graduates are completing higher education. For the first question, you can look at the percentage of fourth-graders who are no longer functionally illiterate as measured by NAEP, as well as the percentage of kids in the early grades who are reading at Proficient and Advanced levels, then compare to the national average. Objective researchers working in universities wouldn’t use NAEP to draw conclusions about the impact of policies because it doesn’t cannot measure the wide range of impacts of any policy or practice decision. But an advocacy group can use NAEP data to consider what those impacts could be. In any case, the analysis wouldn’t have necessarily worked in Broader Bolder’s favor.
Let’s take New York City: Big Apple fourth-graders reading Below Basic declined by 8 percentage points (from 47 percent to 39 percent) between 2003 and 2011, a greater decline than the six percent national average, while the eight percentage point increase in fourth-graders reading at Proficient and Advanced levels (from 21 percent to 29 percent in that period) also outpaces the two percentage point national gain. The 10 percentage point decline (from 51 percent to 41 percent) in the number of Big Apple free- and reduced-priced lunch eligible fourth-graders reading Below Basic is greater than the seven percent decline for the nation as a whole, while the seven percentage point increase in poor fourth-graders reading at Proficient and advanced levels outpaces that three percentage point gain nationwide. There’s also the fact that the percentage of black fourth-graders in New York City reading Below Basic declined by 10 percentage points in that period, better than the nine percent decline for peers nationwide, while the 10 percentage point increase in black fourth-graders reading at Proficient and Advanced also outpaced the three percentage point gain in the nation. New York City didn’t do so well in improving achievement for Latino fourth-graders with its five percentage point decline in those reading Below Basic (and percentage of kids scoring at Proficient and Advanced levels) either trailing or merely keeping pace with the national average.
Or consider Chicago: The percentage of fourth-graders reading Below Basic declined by eight percentage points (from 60 percent to 52 percent) between 2003 and 2011 while the percentage of fourth-graders performing at Proficient and Advanced levels increased by four percentage points in that period; both outpaced the national average. The Second City also experienced an eight percentage point decline in the number of fourth-graders on free- and reduced-priced lunch reading Below Basic, while the percentage of fourth-graders reading at Proficient and Advanced levels increased by four percentage points; both also outpaced the national average. Meanwhile the percentage of black fourth-graders in Chicago reading Below Basic declined by seven percentage points, while the number of peers reading at Proficient and Advanced levels increased by a mere two points; both are lower than the national average. The eight percentage point decline in the number of Second City Latino fourth-graders scoring Below Basic did slightly outpace the seven percentage point decline nationwide, while the four percentage point increase in the number of Latino fourth-graders reading at Proficient and Advanced levels equaled the percentage increase nationwide for similar peers.
Based on the data, one can surmise that both New York and Chicago have reduced illiteracy at a faster pace than the nation as a whole. This doesn’t necessarily mean systemic reform in either district has done unquestionably well. As your editor made earlier this month in criticizing the districts chosen by the Eli & Edythe Broad Foundation for its eponymous annual award for top districts, reform-minded school leaders must move beyond simply helping kids achieve basic levels of literacy and numeracy to providing the comprehensive college-preparatory education they need for success in an increasingly knowledge-based economy. The unacceptably high levels of New York City graduates stuck forced to take remedial reading and math courses while in college (along with the well-documented struggles of Chicago grads progressing into higher education) show that there is still much to do in providing all kids with high-quality education.
That is a nuanced argument that Broader Bolder could have attempted to make in its executive summary (and should make in its full report if it actually gets released this week). But it didn’t. What it has offered up so far instead is useless sloppiness. And chances are the full report will be no worthier of consumption.