Blind Data: On Inputs, Outcomes And Whether Parents Care

By Jennifer Wagner

Hi there.

Do you like research?

We have lots of research. Like, an entire library full of it that I bet five bucks you couldn’t read all the way through if you had a week of unstructured free time.

Research has been the backbone of our organization for 25 years — something that sets us apart in the school choice movement — so we pay close attention every time someone publishes something looking at the effects of school choice on students, schools, families or anything else, for that matter.

An Indiana University doctoral candidate recently published a study stating that the competitive effect within public schools caused by the state’s decade-old voucher program may be limited to short-term gains. (You can read the study in its entirety here.)

Indiana has one of the largest voucher programs in the country that, during this year’s legislative session, just became one of the most expansive, with almost 80 percent of families qualifying to receive 90 percent of the state funding allocated to educate a student in a traditional public school.

So, yeah, we’re interested in this study because policymakers and advocates — on all sides of the K-12 conversation — want to know what happens when the landscape shifts.

I’m going to spend a little time sharing our research team’s take on it, but not until I use this space to make a very different point.

Despite editing lots of research and working with our incredibly smart researchers to tease out the biggest takeaways from their incredibly smart work, I am not, in fact, a researcher.

This regression analysis?

Greek to me. Literally.

What I do know, however, is what it’s like to be a parent looking for the right school for my kids. (Bonus: I happen to be a mom in the great state of Indiana, where EdChoice is located, and I choose to send my kids to private school despite a positive experience with the district magnet school my daughter attended from kindergarten through second grade.)

Brace yourself for some truth, research community: Most families are not looking at studies on the competitive effects of a school choice program when they select a school for their child.

And by most, I mean pretty much none. In fact, when we surveyed Indiana school choice families a few years ago to find out what sources they trusted to learn about schools of choice, here’s how they responded:

(Do you like how I totally used research to back up my post that I said wasn’t really about research? That’s just how much research we have laying around our office.)

To be clear, I’m not saying policymakers and advocates don’t or shouldn’t care how school choice programs affect other schooling types. But those factors shouldn’t be analyzed in a vacuum or given outsized importance in the public debate about educational opportunity. After all, isn’t our K-12 system designed to serve families?

I’ve long said that decades ago, school choice advocates sold these programs the wrong way. Instead of focusing on what parents needed, we talked about free markets and competition and the proverbial rising tide that lifts all boats.

In many cases, we used standardized test scores and school letter grades to determine which students would qualify for choice programs. We advocated for income-eligibility standards based on the federal government’s free and reduced-price lunch qualifications. We made big, bold promises about vouchers and other school choice programs that, looking back, were statistically impossible to deliver short-term.

We didn’t talk at all about social justice, the systemic racism baked into our K-12 system or the basic concept that a whole lot of kiddos across our country are stuck in schools that were never designed to meet their individual needs — and unless they have the means to move or pay tuition, those families historically haven’t been able to access other options.

Shorter: I’ve never talked to another mom about K-12 competition; I’ve talked with oodles of parents about their kids’ schools and whether they’re delivering what those parents and their students want.

Now, if you’re still here to find out why this most recent Indiana competitive effects study doesn’t match up with prior findings on the issue, both in Indiana and elsewhere, prepare yourself for the nerdy part of the post.

I’m told that perhaps the most important difference between the IU study and other competitive effects studies appears to be that the IU researcher was only able to use publicly available school-level data to compare results. We use student-level data, which allows for more accurate comparisons.

For comparison, imagine two studies are being conducted on the efficacy of dieting. One researcher carefully follows thousands of individual people, tracking what they eat and their health markers. Another researcher just looks at the average diet of a state or country and averages of health markers. They can both reach some conclusion, but their levels of specificity are totally different, and the second researcher couldn’t reasonably state his findings with the same certainty as the first.

I don’t know whether this researcher had access to student-level data that could have made his analysis stronger. We’re able to get that information because we sign legal agreements with the Indiana Department of Education to ensure privacy, wait months and months to receive the data, and store all that data on a secured computer in our office. Again, not having access to student-level data doesn’t mean the researcher’s findings are wrong, but they’re much blurrier than they would be if he had obtained that data.

According to our research team, the biggest issue in the IU study, outside of using school-level proficiency rates, is the possible inclusion of all private schools in a federal dataset from just 2009–10 to measure competition from the voucher program in each year of analysis. That means they most likely included non-voucher private schools (potentially even some Amish schools — we emailed the author for confirmation and are eagerly awaiting a response) to measure competition from the voucher-participating schools for all years and likely omitted some of the voucher-participating schools in many of the years.

The paper didn’t precisely document the samples of public and private schools used. The analysis included 1,268 public schools while there are roughly 1,900 public schools in Indiana. This may also account for outcomes that were different from what we found in our original research.

When we published our competitive effects study of the Indiana voucher program last year, we found “little evidence that the average student in traditional public schools has been affected — either positively or negatively — by the enactment and growth of the Indiana Choice Scholarship Program. We do find, however, consistent evidence of small positive effects for low-income children.”

That finding is consistent with other competitive effects studies from other states, and our researchers noted in their conclusion that “[i]n states where stronger positive competitive effects have been observed, it may be the case that those programs have design features that are more likely to yield competitive effects than the ICSP or it may be an errors-in-variables issue that biases their estimates of competition on student outcomes toward zero because the researchers are mis-measuring competition.”

In other words, how a program is designed could play a role in its effects on public schools. It’s worth noting that in Florida, Ohio and Milwaukee, researchers have found positive long-term competitive effects from private school choice programs.

Given that Indiana has opened its voucher program up to more and more families using multiple pathways to entry beyond income and poor public school performance, it’s not out of the realm of possibility that the competitive effects might wane over time, though that’s not what our research to date has shown.

But all of that brings me back to my original point: Parents don’t give a hoot. They want to find a school where a child can get an education that meets his, her or their needs. That school could be a traditional model within walking distance, a charter school halfway across town or a private school that would be financially out of reach without some sort of assistance. It could be none of those options.

Research is critically important to our work and how we objectively measure change over time, but our focus as advocates should be leveling the playing field so that all families — not just the privileged ones — can get in where they fit in.

Jennifer Wagner is a mom, a recovering political hack and the Vice President of Communications for EdChoice, a national nonprofit that supports and promotes universal school choice.