« Change.edu, by Andy Rosen, charts the higher ed landscape and the course forward | Main | The Roots of STEM challenges »



Feed You can follow this conversation by subscribing to the comment feed for this post.

John Pane

Hi Bror,

Interesting comments! I wonder if we're still too early in the process -- if maybe once there is a body of evidence from large-scale studies the buyers will gravitate towards the evidence-based products. This, of course, should increase the incentive among other developers to produce yet more evidence to make their products competitive.

I'm hearing that developers are often asked by administrators and school boards what is the evidence behind their products. This is a good sign, but it is not clear whether all of these stakeholders are able to distinguish rigorous research from that which is not rigorous; or that they are clear about the differences between evidence of promise, evidence of efficacy, and evidence of effectiveness. Moreover, even if a developer's hand is weak in terms of rigorous evidence of effectiveness, it might not be so damaging if all their competitors' hands are weak too.

As you may know, my organization conducts efficacy and scale-up effectiveness trials and is looking to do more of them. We have been approached by developers to conduct such studies, so it seems many of them do believe there would be value in establishing evidence for their products. The #1 impediment to moving forward has been to identify school districts willing to participate in randomized controlled trial studies. Many are reluctant to withhold treatment from the half of schools assigned to the control group. Many others are put off by the long delay between when a grant proposal is written and when the resulting study could start. They might not want to wait that long to start implementing (and consequently make their control group schools wait even longer).

They may feel that their own needs for evaluation of a new intervention can be met in a less obtrusive way. Thus, participating in a study might be more beneficial to the community at large than to the participating districts. While we can structure the study to provide a variety of benefits to the districts, these might not be sufficient to offset their perceived inconvenience and loss of control over how things will be implemented for several years.

Other than direct benefits from the study, it would be nice if there were other incentives for districts to participate in research for the sake of the field. But I'm not sure what those might be.


Bror Saxberg

Thanks for your comments, John. If you are the John Pane that I and a couple of colleagues from Kaplan University met with at RAND, you are indeed in an organization that knows what it's doing with studies.

Your point about buyers is likely right. If buyers at scale are not familiar with evaluating the quality of studies to ensure they'll get the benefits at scale they hope for, then providers of interventions don't have much commercial incentive to do these studies. A few good firms might still do them, but in the competitive forge which is a marketplace, doing a lot of things that buyers are not apparently interested in may not get you long term success.

Fundamentally, the problem may be how to link long term student outcomes to buying behaviors, instead of other pressures.

Since buyers in states or large districts are tied back to their school systems, your point about schools not being interested in participating links to the buyer problem. Larger systems, in theory, should have reason to gear themselves up to do good studies quickly and well - but if buyers do not see value in focusing on buying things that significantly and measurably lift learning performance, it's tough for the schools to gear up to participate.

It would be great if there were, indeed, a network of schools established for this kind of work, with some professional staff wrap-around funded to streamline the running of pilots, so that the schools could become used to running well-concieved pilots. Right now, both for researchers and school systems, each careful trial is a hassle - much better to have systems and processes set up to make it as scalable and "easy" as possible. Perhaps some incentives to participate in such a network would help?

Not easy, no question!

Best -


The comments to this entry are closed.

My Photo