Pre-specify, or explain yourself later

Adigens Health helps pharmaceutical and rare disease companies design real-world evidence strategies built around the target trial framework: from early development planning through HTA submission. Get in touch at info@adigenshealth.com.

Two FDA advisory committees, convened on successive days in late July, expose two distinct pre-specification failures in two unrelated programmes. One needed no external comparator for its pivotal evidence, and failed instead to lock its analysis plan before the data were unblinded. The other never properly specified the external control arm its single-arm trial needed.

On July 29th, the FDA's Cellular, Tissue, and Gene Therapies Advisory Committee meets to consider deramiocel, Capricor's cell therapy for cardiomyopathy in Duchenne muscular dystrophy. The following day, the same committee reconvenes to consider vusolimogene oderparepvec (RP1), Replimune's oncolytic virus therapy for melanoma that has progressed on checkpoint inhibitors. Briefing documents for each meeting converge on the same underlying issue: pre-specification has to be applied to key elements evidence package: the population, the treatment strategies compared, the comparator or assignment mechanism, the outcome and how it is measured, the time-zero, and the analysis plan that estimates the effect. This approach reinforces the key tenents of a target trial framework.  Capricor's programme shows what happens when that discipline is applied rigorously to one element and abandoned for another, within the same trial. Replimune's programme shows what happens when it is never applied to the element carrying the most causal weight of all: the comparator itself.

Capricor: an analysis plan finalised too late?

The pivotal evidence for deramiocel comes from HOPE-3, a randomised, double-blind, placebo-controlled Phase 3 trial. The FDA's briefing document raises real problems with HOPE-3; the process of finalising the statistical analysis plan (SAP) is where Capricor's problem lies. It went through at least five versions spanning a Type B meeting, a Complete Response Letter, and a Type A meeting called to resolve it. The primary endpoint moved: from upper-limb function, to cardiac ejection fraction after the CRL, back to upper-limb function following the FDA's own request. The final version is dated one day before database lock and unblinding and, per the FDA, was never submitted for agency review beforehand.

Because those decisions were made after unblinding, the FDA writes that, “given the post-hoc nature of the SAP changes, FDA considers all analyses implemented following study completion,” treating what was designed as a confirmatory trial as, in effect, exploratory.

A large imbalance in hypersensitivity reactions between arms compounds the problem, raising the possibility that blinding had functionally eroded before those late decisions were made.

The contrast within Capricor's own programme is the useful part for this discussion. An external comparator features elsewhere in Capricor's programme, in a supportive HOPE-2-OLE, the open-label extension of the earlier Phase 2 trial, which used a propensity-matched external comparator once every patient in the extension had moved onto treatment. That comparator was reviewed with FDA well before the SAP dispute, and the FDA's briefing document raises no issue with it. 

Replimune: the comparator that was never specified?

IGNYTE, the trial supporting Replimune's BLA for RP1, began as an exploratory, multi-cohort Phase 1/2 trial spanning several tumour types, and was only later amended into a single-arm cohort of anti-PD-1-refractory melanoma patients. Once the single-arm response data existed, Replimune built its case for effect from what was available afterwards: published literature on checkpoint-inhibitor rechallenge, and a within-patient comparison against each subject's own response to their prior anti-PD-1 therapy.

The FDA's document explains why this retrofit does not hold up as a valid comparison. Cross-trial response rates depend on matching patient populations against the interventional cohort on the factors that drive response (prior therapy type, treatment setting, disease extent) and none of that matching was specified before the historical sources were chosen. No eligibility criteria were harmonised with IGNYTE's; no time-zero was aligned; no prospective statistical plan governed how the historical rate would be selected or adjusted. The FDA concludes that the comparator cannot support a reliable estimate of the treatment effect, and adds two further concerns: that RECIST v1.1, a criterion built for systemic therapy, may not validly capture response to a tumour-injected agent, and that survival data with no concurrent control offers no basis for inferring a treatment effect at all.

A comparator assembled from whatever historical or literature data best fits the interventional result carries the answer it was chosen to support.

Capricor's failure sits inside an otherwise well-specified randomised trial. Replimune's sits at the foundation: the single-arm design made the comparator the entire causal argument, and that argument was built after the data that needed it already existed.  It is exactly what the target trial framework intends to prevent.

The target trial framework's rule, and why it extends beyond observational data.

Read in this order, the two cases make the same point. Every element of a comparison — eligibility criteria matched between groups, a time-zero aligned so both groups' clocks start at the same point in their disease course, an outcome defined and measured identically, and a pre-specified plan for adjusting for confounding — should be fixed before the data are examined. Capricor shows that this requirement is genuinely separable: a company can satisfy it for the comparator and violate it for the analysis plan, within the same programme, and each failure is judged on its own terms. Replimune shows what happens when the requirement is never satisfied for the comparator at all, in a design where the comparator was the focus of the causal claim. 

Once the interventional data are unblinded, every choice about which historical dataset to use, which patients within it to include, or which statistical model to apply carries the risk of being shaped, even unconsciously, by knowledge of the answer it needs to produce. The practical implication is a sequencing one: the design of the external control should start much earlier and follow the principles of the target trial framework.  Taking this further: whichever element of a trial's design carries the causal weight (e.g., the comparator, or the analysis plan) has to be locked at the same time the interventional protocol is written.

What this means for programmes considering an ECA

Capricor's case is the more encouraging of the two. It shows the discipline working as intended in one place: an ECA specified and reviewed with FDA well ahead of the data that would eventually depend on it. It is true that the ECA was supportive in nature; however, pre-specification allowed it to sail through the review.  Replimune's case provides a clear warning for any single-arm programme. It is possible to argue that IGNYTE-3, a randomised confirmatory trial now underway, exists because the company failed to pre-specify its external comparator.

For companies weighing an external control arm for a rare disease or a refractory-population oncology programme, the lesson extends beyond the comparator itself: pre-specification has to be applied to whichever element of the design is doing the causal work in a given trial, and a correct decision elsewhere in the protocol does not compensate for one left open. Building that specification (matching criteria, time-zero, outcome definitions, and analytic methods) at the same time as the interventional protocol, with every element agreed before any data exist, lets the evidence do the work asked of it.