I am just back from helping review science and math grant applications for the Institute of Education Sciences within the Department of Education. It’s making me think: this seems a great process, with great potential – what's in the way of big impacts?
The process really is careful, with terrific ideas in the pipeline, and gives all of us guidance in how to pursue scaling of innovation in education. I think we may be expecting too much from the ecosystem of faculty researchers to be the only pull at the end of the pipeline – to expect them to be the drivers of the most complex scaled up research that nails interventions down at scale.
Where are the buyers in all this?
- Exploration grants. These hunt through large existing data sets about learning. Whether the hunt is for what helped, or factors that hindered or helped learning, the focus is on generating hypotheses for what to do next – e.g., a development grant.
- Development grants. Contrary to some views, most of the applications (and I believe grants) the IES has are not for randomized controlled trials (RCTs). They are for development projects, which fund the building of a plausible intervention, and getting some evidence (not RCT) of promise for learning. (Indeed, in the latest round, the pilot itself must be less than 30% of the funding request.). The range of ideas is amazing: games, teacher training, collaborative learning, hardware for learning, almost anything you can think of – as long as it has some plausible connection to what the plentiful existing evidence seems to show works for learning.
- Efficacy grants. For projects that already show promise of effect (e.g., successful development grants), the goal is to show evidence that the learning impacts are likely to happen at scale. This is where RCTs come in: An RCT gives strong evidence about the uncertainties of the impact, protecting you from differences in groups you don't know about, to help you gauge the value of scaling up from here. These projects also look for what else makes a difference for learning: Characteristics of schools? Students? Families? Teachers? Amounts of time? Other factors?
- Scale-up grants. These are large, multi-center trials: Will what seems a promising idea really work at very large scale, in the hands of people other than the original developers? Arguably, these are the most important studies of all for national impact.
- Finally, there are assessment grants. These grants develop new measures for learning, or new systems for gathering learning data to help tease apart learning problems. These are critical enabling tools, both for research folks who need trustworthy measures, but many of these projects enable teachers with better and more targeted information about how their students are learning, to help guide what to do next.
There are remarkably good insights about learning buried even with the requests for application for these grants (RFAs):
- Objective data should guide our sense of whether we are making a difference. Not always an RCT (especially at the start, before scale), but some form of believable learning measurement applied to learners who had and did not have the intervention, to look for changes, before you start to scale up.
- Innovation testing needs to be phased. Start with small groups, look for promise, and then try larger groups, with better predictive methods. Things fail for different reasons, at different scales.
- Before plunging ahead, be clear about the “active ingredient” for an intervention. Bet first on things that tie to evidence of what works for learning, not random shots.
- Deal with the fact that implementation will be hard: How will you train and support teachers/faculty for the new idea? How will you know (observations, video, artifacts, etc.) that students had the intended experience?
- As scale increases, look for characteristics of students, families, or schools that make interventions work better or worse. This may lead to new ideas.
All of this sounds good, but why are there so few (handfuls) of the largest trials even proposed, which would give us all the best direction for scale?
This could be because the pipeline of “promising enough” studies is not that big (and like large health care trials, you do expect weeding out from the smaller scales), but I don’t think that’s it.
I think it may have more to do with the academic research ecosystem. To execute very large trials is a giant management challenge – large numbers of schools (often not very experienced with measured practices), many people to manage, collaborations with others. You may get a few more papers than a development project, a few more graduate students out the door – but you are likely working far from your comfort zone. Not an easy reward, compared with smaller studies.
Consider differences (and similarities) with health care. Large clinical trials are very complex things, very hard to do. However, there are often organizations and groups focused on this challenging work – as with anything, if you do it repeatedly and well, you develop habits and practices that make it faster and more reliable.
Why are the specialized players in health care focused on getting such trials done? Perhaps because at the other end of the pipe are large buyers, anxious for (and benefitting from) the results, pulling (and funding) the most impactful results through the clinical trials pipeline, ready to deliver them to anxiously waiting physicians for their patients’ use and benefit.
So where are the buyers in all this education work? Are the check-writers anxiously pummeling the publishers and other vendors for when they will package up the latest results from the evidence-based pipeline? Are publishers hovering around learning science researchers, trying to smooth the way, find ways to support the best results? Are large school systems and universities familiar (and knowledgeable) presences in the offices of our best learning science researchers, trying to get the latest successful work tested and made available for their systems first?
Houston, we have a problem.
As I’ve written about in my blogs, there is a ton of existing evidence-based learning science research already published that seems rarely to be applied at scale. Of course this is difficult to do, just as it is in health care – but the problems in learning are critically important for our future as well. We likely need more buyers at scale demanding evidence of success to pull projects through the demanding pipelines needed.
With all this great work already out there (and, judging by the applications I’ve seen, even more good work coming), it makes sense at least to us at Kaplan to get going and apply these evidence-based principles within our own learning environments. We can then see if we can improve our student experiences and outcomes out here in the real world.
More to come on this – we’re starting to see some intriguing results!
Hi Bror,
Interesting comments! I wonder if we're still too early in the process -- if maybe once there is a body of evidence from large-scale studies the buyers will gravitate towards the evidence-based products. This, of course, should increase the incentive among other developers to produce yet more evidence to make their products competitive.
I'm hearing that developers are often asked by administrators and school boards what is the evidence behind their products. This is a good sign, but it is not clear whether all of these stakeholders are able to distinguish rigorous research from that which is not rigorous; or that they are clear about the differences between evidence of promise, evidence of efficacy, and evidence of effectiveness. Moreover, even if a developer's hand is weak in terms of rigorous evidence of effectiveness, it might not be so damaging if all their competitors' hands are weak too.
As you may know, my organization conducts efficacy and scale-up effectiveness trials and is looking to do more of them. We have been approached by developers to conduct such studies, so it seems many of them do believe there would be value in establishing evidence for their products. The #1 impediment to moving forward has been to identify school districts willing to participate in randomized controlled trial studies. Many are reluctant to withhold treatment from the half of schools assigned to the control group. Many others are put off by the long delay between when a grant proposal is written and when the resulting study could start. They might not want to wait that long to start implementing (and consequently make their control group schools wait even longer).
They may feel that their own needs for evaluation of a new intervention can be met in a less obtrusive way. Thus, participating in a study might be more beneficial to the community at large than to the participating districts. While we can structure the study to provide a variety of benefits to the districts, these might not be sufficient to offset their perceived inconvenience and loss of control over how things will be implemented for several years.
Other than direct benefits from the study, it would be nice if there were other incentives for districts to participate in research for the sake of the field. But I'm not sure what those might be.
John
Posted by: John Pane | 11/09/2011 at 09:21 AM
Thanks for your comments, John. If you are the John Pane that I and a couple of colleagues from Kaplan University met with at RAND, you are indeed in an organization that knows what it's doing with studies.
Your point about buyers is likely right. If buyers at scale are not familiar with evaluating the quality of studies to ensure they'll get the benefits at scale they hope for, then providers of interventions don't have much commercial incentive to do these studies. A few good firms might still do them, but in the competitive forge which is a marketplace, doing a lot of things that buyers are not apparently interested in may not get you long term success.
Fundamentally, the problem may be how to link long term student outcomes to buying behaviors, instead of other pressures.
Since buyers in states or large districts are tied back to their school systems, your point about schools not being interested in participating links to the buyer problem. Larger systems, in theory, should have reason to gear themselves up to do good studies quickly and well - but if buyers do not see value in focusing on buying things that significantly and measurably lift learning performance, it's tough for the schools to gear up to participate.
It would be great if there were, indeed, a network of schools established for this kind of work, with some professional staff wrap-around funded to streamline the running of pilots, so that the schools could become used to running well-concieved pilots. Right now, both for researchers and school systems, each careful trial is a hassle - much better to have systems and processes set up to make it as scalable and "easy" as possible. Perhaps some incentives to participate in such a network would help?
Not easy, no question!
Best -
Bror
Posted by: Bror Saxberg | 11/11/2011 at 02:03 PM