How to Follow the Evidence
Borrowing a page from NASA, the PERL framework helps leaders assess the strength of evidence behind their decisions
A new product feature wins rave reviews from beta testers. A regional marketing campaign seems to boost sales. A sustainability initiative earns praise from customers and stakeholders. A new management practice has been shown to increase employee motivation. From these scraps of evidence, it’s tempting to declare “It works!” and adopt the practice broadly.
But uncomfortable questions can get lost in the enthusiasm: How much does the evidence actually tell you about how well the new initiative will work? How well does it scale across contexts and operations?
An initiative that succeeds in one setting may fail in another. The customers may be different, the employees less receptive or the conditions that enabled the original success absent. Even a carefully conducted study demonstrating a genuine effect does not necessarily establish that the same results will apply elsewhere.
“When we see a robust study conducted at a different firm, a different location, or with different customers, it’s tempting to conclude there’s robust evidence that it will work for us, our location, our customers,” says Christopher Cotton, an economist at Queen’s University who studies evidence-based policy and decision making. “When really all that the evidence shows is that it has worked somewhere for someone before.”
The danger, Cotton says, is “when we use the evidence to dismiss the risks” and conclude “it will work for us rather than it could work for us.”
Cotton points to a cautionary tale from the policy-making world. Microcredit was once promoted as a powerful way to lift people out of poverty, with promising early examples helping fuel rapid expansion by donor agencies, non-governmental organizations and foundations. But later, systematic reviews found no consistent evidence of transformative effects on poverty. The lesson, Cotton says, is how compelling stories and the momentum behind an idea can drown out cautionary voices.
Compare that with the rollout of insecticide-treated bed nets to prevent malaria. The nets moved from early trials through replication to systematic reviews supporting broad guidance, with implementation expanding step by step alongside the evidence.
Organizational leaders will recognize the pattern from their own industries. Cotton points to the corporate equivalents, from customer-relationship systems to fashionable management practices. A company might adopt a sales incentive system after reading a rigorous study showing it improved performance at another firm. Differences in compensation structures, workforce composition or organizational culture, however, could produce a different outcome.
He sees the problem as especially acute in sustainability and corporate social responsibility, where “organizations can implement and publicize initiatives without directly measuring whether they work as intended.”
PERLs of wisdom
In a paper published in the journal Science, Cotton proposed a framework called Policy Evidence Readiness Levels (PERL) that distinguishes among the various levels of evidence. Although developed for policy-makers, the framework offers organizational leaders a staged process to assess new programs, practices and investments by matching the size of a commitment to the strength of the evidence behind it.
The idea comes from NASA’s Technology Readiness Levels, which tell engineers how well a new technology is tested and whether it is mission-ready. PERL applies the same logic to evidence, sorting interventions into four levels — from untested theory to well-replicated findings — according to how much confidence the research warrants, and recommending a different course of action for each.
At PERL 1, an intervention rests on a plausible theory or preliminary observations but lacks rigorous empirical evidence that it produces the intended results. A new approach to employee engagement might have an appealing rationale, for example, but no reliable evidence of its effectiveness. The appropriate response is to develop and test the idea, not assume it works.
PERL 2 represents preliminary insights. An intervention has shown promise in a controlled trial or other research, but has not been convincingly evaluated under real-world conditions. The prudent response is to fund tests under realistic conditions and describe results as promising, not conclusive.
At PERL 3, rigorous studies show that an intervention can work under typical real-world conditions. But it remains unclear whether the results will hold across different markets, populations or organizational settings. Leaders can expand in contexts similar to those already studied, while testing the approach incrementally in less familiar settings.
PERL 4 is the strongest level: a body of research, systematically assessed, supports guidance across different contexts. Multiple studies and methods help establish not only whether an intervention works, but for whom, under what conditions and with what limitations. Even then, the assessment is not permanent. Cotton calls it “a living standard,” because effectiveness can change with new circumstances.
Surfacing the uncertainty
The framework’s real value lies in pacing: each level sets a ceiling on how much an organization should commit until better evidence arrives.
Consider a company contemplating a new employee-training program. Early research might suggest that the approach improves learning, giving leaders a reason to explore it. Instead of immediately deploying it across the organization, they could run a pilot in representative departments, establish clear measures of success and assess whether the benefits justify the costs.
If results are encouraging, the company could expand the program to similar teams while testing whether it works in different roles or locations. Each stage would generate information to help determine whether the next commitment is warranted.
Staging alone, however, does not eliminate confirmation bias. Cotton recommends deciding in advance what results would justify expanding the program, and what results would prompt leaders to pause, revise or abandon it. Setting those thresholds early makes it harder to reinterpret ambiguous findings as support for a decision already made.
PERL is not a demand to wait for perfect evidence. Some decisions cannot wait, while others — such as infrastructure investments, regulatory strategy and macroeconomic bets — may never be testable in ways that produce a comprehensive body of evidence. Even so, the framework makes uncertainty explicit and encourages leaders to draw on observational, historical and comparative evidence where appropriate.
“If we recognize that its success is not guaranteed, then we can move forward, but cautiously,” Cotton says. That means gathering data to verify the initiative works and being prepared to change course before making a major investment, rolling out an enterprise-wide system or making public pledges.
PERL may offer an additional benefit: a shared language for productive skepticism. Employees who question a proposed initiative can be dismissed as risk averse or obstructive. A common scale gives them something more concrete to point to. It frames caution, Cotton says, “not as differences in preferences or priors, but as grounded judgment that action was outpacing the evidence.” It also puts the onus on advocates of rapid expansion to show that the evidence warrants the support.
As Cotton points out, “The big questions running through all this are: What does the evidence actually allow us to conclude, and how should the remaining uncertainty affect what we do next?”