I am just back from helping review science and math grant applications for the Institute of Education Sciences within the Department of Education. It’s making me think: this seems a great process, with great potential – what's in the way of big impacts?
The process really is careful, with terrific ideas in the pipeline, and gives all of us guidance in how to pursue scaling of innovation in education. I think we may be expecting too much from the ecosystem of faculty researchers to be the only pull at the end of the pipeline – to expect them to be the drivers of the most complex scaled up research that nails interventions down at scale.
Where are the buyers in all this?
- Exploration grants. These hunt through large existing data sets about learning. Whether the hunt is for what helped, or factors that hindered or helped learning, the focus is on generating hypotheses for what to do next – e.g., a development grant.
- Development grants. Contrary to some views, most of the applications (and I believe grants) the IES has are not for randomized controlled trials (RCTs). They are for development projects, which fund the building of a plausible intervention, and getting some evidence (not RCT) of promise for learning. (Indeed, in the latest round, the pilot itself must be less than 30% of the funding request.). The range of ideas is amazing: games, teacher training, collaborative learning, hardware for learning, almost anything you can think of – as long as it has some plausible connection to what the plentiful existing evidence seems to show works for learning.
- Efficacy grants. For projects that already show promise of effect (e.g., successful development grants), the goal is to show evidence that the learning impacts are likely to happen at scale. This is where RCTs come in: An RCT gives strong evidence about the uncertainties of the impact, protecting you from differences in groups you don't know about, to help you gauge the value of scaling up from here. These projects also look for what else makes a difference for learning: Characteristics of schools? Students? Families? Teachers? Amounts of time? Other factors?
- Scale-up grants. These are large, multi-center trials: Will what seems a promising idea really work at very large scale, in the hands of people other than the original developers? Arguably, these are the most important studies of all for national impact.
- Finally, there are assessment grants. These grants develop new measures for learning, or new systems for gathering learning data to help tease apart learning problems. These are critical enabling tools, both for research folks who need trustworthy measures, but many of these projects enable teachers with better and more targeted information about how their students are learning, to help guide what to do next.
There are remarkably good insights about learning buried even with the requests for application for these grants (RFAs):
- Objective data should guide our sense of whether we are making a difference. Not always an RCT (especially at the start, before scale), but some form of believable learning measurement applied to learners who had and did not have the intervention, to look for changes, before you start to scale up.
- Innovation testing needs to be phased. Start with small groups, look for promise, and then try larger groups, with better predictive methods. Things fail for different reasons, at different scales.
- Before plunging ahead, be clear about the “active ingredient” for an intervention. Bet first on things that tie to evidence of what works for learning, not random shots.
- Deal with the fact that implementation will be hard: How will you train and support teachers/faculty for the new idea? How will you know (observations, video, artifacts, etc.) that students had the intended experience?
- As scale increases, look for characteristics of students, families, or schools that make interventions work better or worse. This may lead to new ideas.
All of this sounds good, but why are there so few (handfuls) of the largest trials even proposed, which would give us all the best direction for scale?
This could be because the pipeline of “promising enough” studies is not that big (and like large health care trials, you do expect weeding out from the smaller scales), but I don’t think that’s it.
I think it may have more to do with the academic research ecosystem. To execute very large trials is a giant management challenge – large numbers of schools (often not very experienced with measured practices), many people to manage, collaborations with others. You may get a few more papers than a development project, a few more graduate students out the door – but you are likely working far from your comfort zone. Not an easy reward, compared with smaller studies.
Consider differences (and similarities) with health care. Large clinical trials are very complex things, very hard to do. However, there are often organizations and groups focused on this challenging work – as with anything, if you do it repeatedly and well, you develop habits and practices that make it faster and more reliable.
Why are the specialized players in health care focused on getting such trials done? Perhaps because at the other end of the pipe are large buyers, anxious for (and benefitting from) the results, pulling (and funding) the most impactful results through the clinical trials pipeline, ready to deliver them to anxiously waiting physicians for their patients’ use and benefit.
So where are the buyers in all this education work? Are the check-writers anxiously pummeling the publishers and other vendors for when they will package up the latest results from the evidence-based pipeline? Are publishers hovering around learning science researchers, trying to smooth the way, find ways to support the best results? Are large school systems and universities familiar (and knowledgeable) presences in the offices of our best learning science researchers, trying to get the latest successful work tested and made available for their systems first?
Houston, we have a problem.
As I’ve written about in my blogs, there is a ton of existing evidence-based learning science research already published that seems rarely to be applied at scale. Of course this is difficult to do, just as it is in health care – but the problems in learning are critically important for our future as well. We likely need more buyers at scale demanding evidence of success to pull projects through the demanding pipelines needed.
With all this great work already out there (and, judging by the applications I’ve seen, even more good work coming), it makes sense at least to us at Kaplan to get going and apply these evidence-based principles within our own learning environments. We can then see if we can improve our student experiences and outcomes out here in the real world.
More to come on this – we’re starting to see some intriguing results!