Fix the Failure instead of Calling it a Myth: Breaking Down the Conversation about CREDO’s Online Learning Study

This was a big, albeit controversial week for K-12 online learning. There was much buzz over the CREDO report that assessed online learning outcomes for K-12 students across the country. The results reflect terribly on online schools throughout the United States. EdWeek reporter Benjamin Herold explains these results in a recent article and effectively distilled the results so that a large audience can understand what they mean. The Washington Post picked up this study and even used the word “damning.”

I read the study, the results, and the counter arguments. I should here add that I have taught online classes, study K-12 online learning practices, and believe that in some instances online learning is an appropriate tool for students in K-12 schools, but not always. With these caveats in mind, my opinion is that the advocates of online learning are wrong in saying this study is inadequate. Yes, there are bits and pieces that we need to consider, but the study shows me that we need drastic improvement in the online learning sector. The models used in the United States are not working and they need to change. I am not saying we need to give up online learning in K-12 always and forever, but the results were so alarmingly bad that they are impossible to ignore. They are so bad that they outweigh even the greatest minutiae-based, nit-picking in the reports that the pro-online learning/charter organizations have released.

My conclusion about this conversation, as stated at the end of the article: I suggest looking for the few schools that do promote quality learning outcomes in an online setting and/or try different models of online learning (such as blended schools) in order to talk about how we can fix this failure as opposed to saying this failure is a myth.

I am going to frame my argument about this around the counter points I have seen to the CREDO study. These come from two main sources: A story in gettingsmart.com and a press release sent from the Center for Education Reform (a massively pro-charter school group). For this email, see Appendix A.

Let’s start with the CER email and break down their points:

A new report released today by the Center for Research on Education Outcomes (CREDO) attempts to answer long-held questions and introduce a more substantial discussion of online charter schools into the great education debate. To some degree, they accomplish this through the sheer amount of information and data they present (with additional online learning statistics and survey results contributed by Mathematica and the Center for Reinventing Public Education). However, it should be noted that the report does attempt to make sweeping conclusions and generalities about online charter schools, even though the sample size is just 158 schools across 17 States and Washington, D.C. Unfortunately, their findings echo much of what is already known and identify obstacles to success that developers and providers of full-time online learning have shared publicly as they continue to innovate.

This struck me as a major red flag, especially since this was their leading argument. Sample size? There are only 26 states with fulltime online schools and they looked at 17 of these states. There were about 315,000 fully online students in 2013 and the CREDO study matched 55,000 + of them. One would only need about 15,000 students in a randomly sample from a pool of 315,000 to get a confidence interval of 1 and and confidence level of 99% (a extremely precise interval). Now, the researchers did not randomly sample their data because they couldn’t (they matched, which is a different strategy) but the size of the sample here, in my opinion, is the study’s strength, not its weakness! 

• Many parents choose online options for their children based on exceptional circumstances and situations, ranging from safety and bullying concerns, to academic issues, to social and emotional issues, medical reasons, and more. For some families, an online school is the only public option available aside from the assigned traditional public school that is not working for their child.

Ok, this point really doesn’t say much. The study was about the outcomes based on this choice. This argument is akin to saying “a terrible option is better than no option at all!” Also, interestingly, the CREDO study followed mobility and noted most students choose to leave. For an advocacy that prides itself on determining quality based on parental choices, the overwhelming choice to leave should tell them something. Perhaps, for example, the students left because their short-term issue abated (pregnancy, illness, etc.); however, it is highly likely that not all these choosers were in this situation. I agree we can (and probably should) give students options, but we can still be critical and make sure these options are high quality.

• Data indicates that a majority of students who enroll in online schools do so after the beginning of the school year. This is an important factor this study left out, as the length of time a student is enrolled in a school impacts performance and the ability of the school to improve a student’s academic outcomes.

See same arguments above.

• Another concern is the (continued) use of a contested methodology throughout the report. The “virtual twin” methodology used, over which CER and other researchers have voiced concerns before, fails to take into account factors such as reasons for enrolling or mobility, dangerously assuming online charter school students face similar circumstances to traditional public school students, when the reality is they are very different.

So there are issues with the twins strategy, of course. But in the absence of experimental design, it is the best option these researchers have. Also, do not forget, the outcomes are so consistently terrible and the magnitude of the direction of the results is so negative that it is impossible to ignore the results. If I took 55,000 kids out of the 315,000 for, say, a study about if cigarettes cause cancer and simply used descriptives that showed that the vast amount of kids were inflicted with cancer, the burden of proof would then be on the cigarette company. Yes, the CREDO study perhaps could be stronger in minor ways, but the negative implications are so severe that one can simply not just look the other way. I read the methods and the results, and I am convinced that they are robust enough for us to seek changes in the online learning sector. The debate should be why these results are so bad rather than if they are real or not.

Now, in terms of the gettingsmart.com — They lead their article by saying the results are old and contrived. They concede that some of the results “remain accurate” but “miss the mark on other fronts.”

Like traditional schools, online schools take all comers even if they enroll late and are way behind. In most online schools, half of the students show up late and most are way behind. It’s not surprising that many don’t achieve grade level proficiency at the end of the year. However, many make good academic strides during the year and that’s not well captured by current measures — or this study.

Well, the study uses a learning growth model. It captures that relative to peers in other settings there is very little growth. I am not sure what else these authors want here.

The Need for Better Measures. Since NCLB states have relied primarily on end of year tests to measure grade level proficiency. These testing and accountability systems don’t provide accurate measures of individual academic growth.  

I agree this is a good point across all of the schooling studies. However, I find it interesting that typically these measures are the same that those advocating for choice will point to. In other words, they look to “failing schools traditional schools” as a need for change. These indicators of failing schools are largely drawn from end of the year exams. I agree the use of end of the year testing is not the sole indicator of progress. However, again, the results are so alarmingly bad that we need to stop and pause. If we are going to come to knee-jerk defenses of online schools because tests “aren’t a good measure” then shouldn’t we use the same argument for traditional public schools? You can’t complain about the cake and eat it too.

Raising the bar for quality. The CREDO report reviewed 2012 data. Since then, K12’s has published three comprehensive Annual Academic Reports (see the 2015 Report) along with school-by-school outcomes and a series ofacademic white papers highlighting results of individual schools and K12’s instructional programs.

Recent data also shows broad progress in schools supported by Connections Education. Both K12 and Connections use adaptive pre- and post-assessments that provide a more accurate picture of individual student progress than the synthetic twin study method.

I read through the reports. They are longer and with newer data, but are they better? No. Not at all. They aren’t analytic (they use descriptive data) and even these don’t really show many impressive findings. My takeaway from these examples of better articles is that even in the major cyber charter schools’ own reports they present results that are unimpressive.

Online learning obviously works well for some and less well for others. Rather than limiting access as a result of a bad comparison, public systems should focus on supporting good choices. State laws prohibit virtual charters from having a conversation with a student and family about fit. State laws are often a disincentive for rolling enrollments, mastery-based progress, and educating underserved students.  

Well, to me this is a doozy. Yes of course we want to make better choices! Of course, laws should prevent cyber charters for contacting and trying to convince families! The nature of the cyber charter school recruitment often means convincing disadvantaged and underserved populations to try something new. The public should act as a regulatory agency that takes the bad options off the table so families aren’t tricked into choosing them. This is common sense!

I suggest perhaps, just perhaps, the reason these organizations are scrambling and producing responses like these that come across as weak, sour-grapes arguments is because they have a vested interest in ensuring that a charter school model of online schools continues to recruit (disadvantaged populations of) students. I agree choice can be an effective policy tool if used wisely. I also agree online learning offers awesome potential in becoming a powerful educational tool. But the models we have right now are not getting it done.

I suggest looking for the few schools out there that do promote quality learning outcomes and talking about how we can fix the failure of online schools (if possible) as opposed to the advocates who say the failure is a myth.

Together we can smartly implement online learning where needed and make sure we do not have programs with questionable practices. We can do better and we need data like the CREDO report to help push us in the right direction.

Appendix A: Email from CER

CER Responds to Online Charter School Report

An area of education generally, and school choice specifically, that has suffered from a lack of clarity in the K-12 policy space has been online learning. It seems that this can be traced to a common “we don’t really understand how that works” mentality. Unfortunately, this includes many vital stakeholders necessary to the creation of strong school choices for America’s students, including policymakers, authorizers, and school boards. When coupled with the politically contentious issue of charter schools — the school model through which a good number of full-time online learning programs have been established — one sees camps of education reformers dig in on one side or the other when it comes to supporting the ability of online schools to achieve student growth and success.

A new report released today by the Center for Research on Education Outcomes (CREDO) attempts to answer long-held questions and introduce a more substantial discussion of online charter schools into the great education debate. To some degree, they accomplish this through the sheer amount of information and data they present (with additional online learning statistics and survey results contributed by Mathematica and the Center for Reinventing Public Education). However, it should be noted that the report does attempt to make sweeping conclusions and generalities about online charter schools, even though the sample size is just 158 schools across 17 States and Washington, D.C. Unfortunately, their findings echo much of what is already known and identify obstacles to success that developers and providers of full-time online learning have shared publicly as they continue to innovate.

While we appreciate the desire to learn more about how online charter schools are impacting student outcomes, we have concerns with CREDO’s Online Charter School Study.

• Many parents choose online options for their children based on exceptional circumstances and situations, ranging from safety and bullying concerns, to academic issues, to social and emotional issues, medical reasons, and more. For some families, an online school is the only public option available aside from the assigned traditional public school that is not working for their child.

• Data indicates that a majority of students who enroll in online schools do so after the beginning of the school year. This is an important factor this study left out, as the length of time a student is enrolled in a school impacts performance and the ability of the school to improve a student’s academic outcomes.

• Another concern is the (continued) use of a contested methodology throughout the report. The “virtual twin” methodology used, over which CER and other researchers have voiced concerns before, fails to take into account factors such as reasons for enrolling or mobility, dangerously assuming online charter school students face similar circumstances to traditional public school students, when the reality is they are very different.

Online charter schools provide a much-needed option within a larger portfolio of public school programs that offer students the opportunity to identify a learning environment that is right for them. All schools should be held accountable for outcomes, regardless of how or where education is delivered and learning assessed. And in fact, all charter schools are by their very nature held accountable for results in exchange for some operational freedoms, while this is not true of traditional programs. Accountability is achieved through clear policy expectations, professional authorization and unbiased oversight, intentional and research-based practice, and an unrelenting focus on providing students with schools that fit their unique needs.

Policies must also be in place that allow parents choices so that when traditional schools aren’t meeting students’ needs, they can seek out an education option that will meet their child’s unique set of circumstances. Online charter schools are an important part of that equation, and fill a unique void in education in the United States. We must not forget that while all schools must be held accountable for student outcomes, students’ learning needs are unique and varied and require similarly varied modalities to support their success.

School Choice, Racial Segregation, and Poverty Concentration: Evidence From Pennsylvania Charter School Transfers

Our article has been published for Educational Policy and is now available online:

Abstract: This article examines how student movements between traditional public schools (TPSs) and charters—both brick and mortar and cyber—may be associated with both racial isolation and poverty concentration. Using student-level data from the universe of Pennsylvania public schools, this study builds upon previous research by specifically examining student transfers into charter schools, disaggregating findings by geography. We find that, on average, the transfers of African American and Latino students from TPSs to charter schools were segregative. White students transferring within urban areas transferred to more racially segregated schools. Students from all three racial groups attended urban charters with lower poverty concentration.

http://epx.sagepub.com/content/early/2015/09/25/0895904815604112.abstract

A quote about cyber charters (p.13)

“In general, the racial distribution of students entering and exiting cyber charters closely resembled the student demographics of the state, but is substantially different from students who transfer to and from B&M charter schools. For instance, African Americans, who constitute around 15% of the state student population, make up about 14% of all students transferring to cybers, but more than 62% of all students transferring to B&M charter schools. In terms of geography, a higher proportion of rural than urban and suburban students transferred to and from cyber schools. Notably, a greater share of urban students moved from a TPS to a cyber (23%) than moved back from a cyber to TPS (16%).”