ReinventED Solutions

While we are free to choose our actions, we are not free to choose the consequences of our actions."

 - Stephen R. Covey 

Let me start out by saying that my blog is about reinventing public education in the United States and the world.  I typically apply the framework of disruptive innovation theory, because that is the lens that one should look through in order to understand what conditions are required in order to successfully innovate our learning methods for the digitally-driven 21st century work environment.   But I also possess more than 20 years of media industry experience, and some of those lessons are quite relevant to the following post which I felt compelled to write after the chain of events that have transpired since March 25, 2012.

Last weekend, I picked up the Sunday Atlanta Journal-Constitution ("AJC") and was completely mortified by the cover story, which was titled, "Cheating Our Children: Suspicious School Test Scores across Nation."  It seems that my local newspaper has made an aggressive, pre-emptive strike against standardized test scores around the nation.  A few years ago, the newspaper analyzed test scores in Atlanta Public Schools, and the anomalies identified in the results triggered the investigation leading to the largest cheating scandal in the history of American K-12 public education.   Their objective was to extend this work, analyze test scores in hundreds of districts around the United States, and ascertain whether similar cheating activities occurred on a national scale.   They claim that there are "suspicious" test scores in roughly 200 districts, including many of our nation's largest urban centers, and these "irregularities" resemble those that entangled Atlanta Public Schools.   The story was picked up by all of the major news outlets.  In addition, several public figures added their verbal "fuel" to the findings, even before verifying its credibility.   These included Education Secretary Duncan, U.S. Senator Johnny Isakson, NEA Head Randi Weingarten, and Diane Ravitch, the latter who is perhaps the most visible protector of the status quo.  It has sought to un-nerve the system of administering standardized tests in public education.

Let me be clear.  These are SERIOUS allegations leveled by a local newspaper.   Instead of having direct evidence of cheating, they are saying, "Here's some data we compiled - it's not conclusive, but we think you should look into it."  And instead of reviewing this privately, they took this directly to the public to sway public opinion before the facts could truly be corroborated.    The report has sent shock waves through the education system and caused politicians and other education figures to comment prematurely.  No one wants to look like they are 'soft" on security around testing, and Georgia's politicians were not going to throw the AJC "under the bus."   It has raised the rhetoric around testing, distracted the nation from the real systemic problems with our public education system, and will undoubtedly force educators to take excessive time to review this study.   There is a domino effect here, and the question is:   has there been a rush to judgment, and was this research thorough, conclusive, and valid?

Let me again reiterate that my opinion is that testing is given FAR too much weight in this country, or any country for that matter.  Tests are supposed to be an indicator, only an indicator, of student achievement.  This author would NOT have been admitted to an Ivy League university if the overriding emphasis was on one day's test performance over the body of work spanning a student's entire academic career.  Fortunately, the school chose to look at other factors, such as GPA, class rank, leadership activities, and other intangibles that provide a far more comprehensive picture of a student's ability than the performance on one test, whose integrity gets challenged and ridiculed on a regular basis.   Many students just don't test well.

Before I comment further, let me provide an important analogy.    Back in the late 1990s when I was working at Turner Broadcasting, the research department embarked on an explosive, breakthrough project titled "Media at the Millennium."  This was the period in the history of cable television where cable network distribution reached "critical mass" in terms of household penetration and was nearing parity with broadcast television.  However, advertisers were still purchasing cable as a low-cost, high frequency medium and were not getting their fair share of advertising dollars even though some networks were starting to achieve parity in terms of not only reach, but also certain daypart ratings. Turner's research department developed a comprehensive data model that proved unequivocally that an advertiser could purchase cable WITHOUT sacrificing reach.   Turner's researchers looked at every single angle of the analysis, anticipated all questions it would receive from ad agencies, media buyers and competitors.  It checked its methodology several times and ensured it was "bulletproof."   Once they completed the extensive, robust quality control process, they released the presentation and underlying methodology to the general public.  It was a MAJOR success, and resulted in an estimated $1 Billion of ad dollars shifting from broadcast television to cable television.  This analysis helped transform the media value chain and turned cable and broadcast into a "one television world."  If they had erred in their quality control, Turner would have not only ostracized itself from the advertising industry, but it would have materially damaged its brand because it could not be trusted to deliver accurate data analysis to its stakeholders.  These were some of the brightest minds I have ever worked with in my professional career, and they were wise to take the time to do all necessary quality checks before putting their reputations on the line and releasing such an explosive research study.

So what does this have to do with the AJC study?  The AJC came out with its findings, paraded out several independent researchers who reviewed and endorsed the validity of its methodology, and created "sensationalist" headlines while burying disclaimers such as "this does not prove that cheating occurred."   It then hired a marketing firm to tout its investigative story on testing data from several large school systems despite being informed of numerous flaws in its methodology and analysis.  Let us summarize what has taken place since the newspaper unveiled its story:

• The best synopsis I have seen can be found here.  It gives the complete inside story of the back and forth between Dr. Miron (see below) and the AJC.  It is QUITE revealing, to say the least.
• To no one's surprise, several of these districts came out with statements that appeared somewhat "caught off guard" and "defensive," including Houston's school district.   Typically one would surmise that when you act "defensively", you are likely guilty of some transgression.   It is likely that certain states may quietly start to conduct more detailed investigations and look further into the data to see if any of it is indicative of unethical behavior (i.e., cheating).
• Gary Miron, a respected professor at Western Michigan University who is well regarded for his work on education policy and reforms, had reviewed the study prior to its release and was commented in the Washington Post about his serious concerns about its methodology, including the fact that the AJC did not use student level data and erasure data as had been done in a study conducted by USA Today a year earlier.  Miron was heavily involved in that study, and the newspaper went back and got the extra data needed to conduct a more thorough analysis.   This data would have allowed student mobility to be factored into the equation, because you wouldn't know if you were looking at the same students or not.
  • Dr. Miron aired his concerns, and the AJC was quick to respond to them.   Their response barely touched the surface, and was a superficial attempt to not only discredit Dr. Miron, but also respond only to the mobility issue, and not the other issues raised by Dr. Miron.
  • As quoted by AJC Editor Kevin Riley:  "Had we waited for absolute proof before publishing news about alarming scores in Atlanta, the investigation would have likely never happened."  
  • The AJC was forced to remove all references to Nashville's Two Rivers Middle School from the original story.   Metropolitan Nashville Public Schools did not only defend their data, but levied a detailed, persuasive rebuttal that took all of the air out of the AJC methodology, and showed that the AJC released data with no understanding of the Nashville district and its demographics.  You can read the full analysis here.  Their assessment is validated by notable researchers:  Dr. Dale Ballou, associate professor of Public Policy and Education with Vanderbilt University’s Peabody College of Education; and Dr. Brian Jacob,  Walter H. Annenberg Professor of Education Policy, Gerald R. Ford School of Public Policy, and University of Michigan.  A few key highlights:
    • The AJC study does not take into account student mobility, and in Nashville, these rates are 35-40%
    • Obvious errors in data provided where children who were absent from testing were assigned a “zero” rather than being excluded from the analysis - resulting in average scores that were below the minimum score possible
    • The AJC methodology will automatically identify 5% of the cases/classes analyzed statewide, but schools with changing populations (see previous bullet) or higher than usual numbers of highly effective or highly ineffective teachers (research supports that teacher effectiveness make the greatest difference in test score gains) are likely to have higher percentages
    • Analysis of test gains of MNPS middle school math teachers over three years (2006-07 through 2008-09) by Dr. Brian Jacob of the University of Chicago, on behalf of Vanderbilt University, did not show any unusual or suspicious patterns.
    • And many other factors you can read at the link above.
  • From a blogger who clearly has a background in statistical research:  "I’m at least equally curious about why the AJC analysts used simple linear regression at a statewide level. This implies that the issues traditionally considered barriers to learning, such as free/ reduced participation, limited English proficiency, mobility, special education, ethnicity and race are homogenous at a statewide level. Extremely unlikely. And it’s fairly easy to show that such differences between school districts, schools, and even classes within a school have a statistically significant, even substantial influence on predicting the mean scaled score on a benchmark exam."  He goes further to say,
    • "Some states also use criterion referenced tests for measuring growth. This means that the 3rd grade test focuses on 3rd grade curriculum standards. 5th grade tests focus on 5th grade standards. While not an excuse for settings in learning, it does counter the journalist’s statement that learning rarely disappears. That may be true, but learning multiplication tables by rote doesn’t always adequately prepare a student who has limited English skills to solve word problems or do algebra. We hope — and even expect — that a strong early performance in literacy or math translates into sustained strong learning. But I’ve seen too much student data to believe that’s true across the board, especially when the standards (necessarily) shift to expecting higher level thinking skills in 6th or 7th grade."
  • From the Texas Education Agency: "The story raises questions, but we do have a concern about their methodology. The newspaper tracked test scores by school, not by student, which we found can "have up to a 20 percent variance."  They take testing security seriously, but they have determined that the analysis was not sufficient enough to warrant any further investigation.  Based on their own review of the analysis, one Texas Superintendent called the report "reckless and irresponsible at best" because they correctly noted that the Atlanta paper's examination flagged classes for large gains and large losses, which wouldn't be associated with cheating. And the report examines scores for groups of students rather than individual students.  With many urban districts in Texas and elsewhere having a very high mobility rate, you simply cannot ascertain from their methodology whether you're looking at the same students or not.
    
    Read more here: blogs.star-telegram.com/extra_credit/2012/03/weatherford-school-officials-respond-to-cheating-report.html#storylink=cpy
  • A spokesman for the Indiana Department of Education told the Washington Post that "a study of erasures on Indiana’s ISTEP-Plus exam, performed last year by the testing contractor at the state’s request, turned up “very few areas of concern statewide.”

These references are merely a few that have risen to the surface over the past week.  I have no doubt that there will be more.   A few of my own questions are below:

• Shouldn't the AJC have been prepared for these questions about its methodology? 
• Why didn't they consult with the US Dept. of Education first, rather than try and besmirch the reputations of several out-of-state school districts?  Why does the AJC get the right to claim that other states may have potentially similar issues as Atlanta Public Schools without conclusive, irrefutable evidence? 
•  How should Secretary Duncan have responded?  Did he have any other choice but to say, “These findings are concerning.  States, districts, schools and testing companies should have sensible safeguards in place to ensure tests accurately reflect student learning.”
• Can a reasonable person surmise that the AJC was not only doing this to sell newspapers, but also doing it, in part, to exonerate the Atlanta Public Schools by claiming that rampant cheating is happening in other cities across the nation?
• If you read the op-ed published by AJC Editor Kevin Riley, where he defends and celebrates the "journalism" of his reporters, shouldn't we be concerned about their true intentions?
• Note that the Atlanta Public Schools have been conspicuously silent here.  I'm sure they would rather not have this report made public because they are trying to rebuild the public trust, only to have this story create lingering sentiment and focus on the APS Cheating Scandal, which was the most pervasive cheating incident in the history of K-12 public education.  APS had nothing to gain (or lose) here, and if they were asked to speak on the record, they would have probably said that "this doesn't pertain to APS and you should speak to the various districts identified in the AJC analysis."
• Why does Riley feel so strongly that if they waited, there would likely have never been an investigation?  Why the rush, and why now?  Is it because the CRCT exams are taking place this month?

Let us look at what the consequences were of releasing a study that was far from bulletproof.  The reference links above talk about the fact that more than 300 news outlets around the country picked up the story, including the Huffington Post, ABC News, CNN, the Washington Post, NBC Nightly News and MSNBC  This is one instance where media outlets exacerbated the problem by airing the story without making CLEAR disclaimers about the credibility of the underlying content.   This was an explosive report, with serious consequences if proven true, and the media felt comfortable with promoting a story that had serious methodology concerns.  This article from the American Journalism Review  is certainly well-timed, given the circumstances we are dealing with here.  I may not agree with all of its contents, but it certainly makes me VERY skeptical about how the media is covering public education in the United States.  Like just about every other issue our country is dealing with, I come at it from the perspective that it is never as good, or as bad, as one is led to believe.

As an American citizen who cares about education, as well as a resident of Atlanta, I felt that it was my duty to provide a "fuller account" of the activities of the past week related to this news story.   When you release research, even a hint of a methodology flaw could materially damage its credibility and should make one consider delaying its release.  In my opinion, from what I have reviewed to date, it is clear that my local newspaper has wrecked havoc and instilled great fear in our nation's public schools, without conclusive evidence.  They have used the media to sway public opinion and shift the focus back to the overemphasis on high stakes testing. 

Cheating on tests is a serious issue, and this author believes that our schools are taking appropriate measures to ensure the accuracy of its test results.   To paraphrase another reporter’s assessment:  “The analysis may not be entirely invalid, but the irregularities uncovered require the newspaper to do some SERIOUS explaining.”  If the newspaper is able to successfully address every material concern about the methodology, I will be the first person to update my post and acknowledge my error in judgment.   HOWEVER, that scenario seems far from likely.

 NOTE: Special thanks go out to education blogger Audrey Watters who contributed reference material to support this story.

The views expressed in this post are those solely of Al Meyers, and do not represent the views of any organizations affiliated with Mr. Meyers.

It's no coincidence about the title of this blog post.    Over the past several weeks, I have been reading, and participating, in the discussion around an Atlanta Journal-Constitution Blog titled, "Get Schooled."  And after letting my blood pressure drop a bit, I decided to reflect and author this post.  

I had to let my blood pressure drop because I have never been so aggressively attacked personally, as I quickly found that there is almost no moderation conducted on the blog.   The tone is hostile, and most of the readers are teachers or former teachers who are regrettably "prisoners of the system."  If you have a contrarian point of view, you are ridiculed, labeled as a Republican (why the labels?), or someone who immediately wants to let for-profit organizations into the public schools!   By the way, is that really the coming of the apocalypse?  I'd say the textbook publishers already see it as "for-profit" - more like "for LOTS OF PROFIT."

The blog's author, Maureen Downey, may be a nice person.  I do not know her and would not make any personal attacks on her character.   But what I can comment on is the fact that her views are 100% reflective of the "digital immigrant" mentality that tries to protect the status quo.  She has an obvious agenda against charter schools (not that all charter schools are good, but new models need to be given support nonetheless), and one of recent posts was titled, "Should Every Georgia Student Take An Online Course?  Why?"  Ms Downey is a "laggard," meaning, she wants to see data o