A Poisson approach to the validation of failure time surrogate endpoints in individual patient data meta-analyses

The advanced GASTRIC meta-analysis³⁸ included 4069 patients from 20 trials. It has been previously reported¹⁷ that the rank correlation coefficient between PFS and OS was 0.85 (95% CI: 0.85–0.85) as estimated by a Plackett copula, corresponding to a Kendall’s τ of 0.68 (0.68–0.68). The R² between the treatment effects on PFS and on OS, adjusted for measurement error, was 0.61 (95% CI: 0.04–1.00) so that the validity of PFS as surrogate endpoint for OS could not be confirmed for chemotherapy trials of patients with advanced gastric cancer. Supplementary Figure S2 shows the estimated treatment effects on OS versus the estimated treatment effects on PFS.

The adjuvant GASTRIC meta-analysis³⁹ included 3288 patients from 14 trials. The rank correlation coefficient between DFS and OS reported by Oba et al.¹⁶ was 0.97 (95% CI: 0.97–0.98) as estimated by a Plackett copula, corresponding to τ = 0.88 (0.88–0.89). Because of a high amount of estimation error, the between-trial R² estimate was highly numerically unstable and, hence, unreliable. The unadjusted R² was 0.96 (95% CI: 0.93–1.00) so that DFS was deemed valid as surrogate endpoint for OS in adjuvant trials of gastric cancer patients. However, due to such high association, the estimated R² was close to the upper limit 1 and the obtained numerical results needed to be interpreted with caution.

Burzykowski et al.⁸ extended the meta-analytic approach of Buyse et al.⁷ to the case of failure time endpoints. Their estimation approach consists in two steps, one for the individual level and one for the trial level.

At the first step, the marginal distributions of the two endpoints are modeled using the following bivariate proportional hazard model:

The dependence parameter of the copula model θ can be reparametrized into the Kendall’s τ as measure of individual-level surrogacy. We considered Weibull marginal hazards and three copula functions. The Clayton⁴¹ copula is defined as $C_{\theta }(u,v)=(u^{-\theta }+v^{-\theta }-1{)}^{-1/\theta }$ , with $\theta >0$ and Kendall’s $\tau =\theta /(\theta +2)$ . The Plackett⁴² copula is defined as $C_{\theta }(u,v)=[Q-R^{1/2}]/[2(\theta -1)]$ , with $\theta >0$ , $Q=1+(\theta -1)(u+v)$ , and $R=Q^2-4\theta (\theta -1)\mathit{uv}$ ; the Kendall’s τ is computed numerically for this family as no analytical expression is available. The Hougaard⁴³ copula is $C_{\theta }(u,v)=\exp (-[(-\ln u{)}^{1/\theta }+(-\ln v{)}^{1/\theta }{]}^{\theta })$ , with $\theta \in (0,1)$ and the Kendall’s $\tau \text{is}1-\theta$ .

At the second step of the estimation algorithm, the treatment effect estimates obtained at the first step are assumed to follow the mixed model

The trial-level surrogacy is measured in terms of $R_{\text{trial}}^2={\rho }_{\text{trial}}^2$ . This is computed in a linear regression of the β_i’s over the α_i’s with measurement error adjustment, by fixing the ${\Omega }_i$ ’s at their estimates from the first step.^10,44 In practice, this adjusted (for measurement error) $R_{\text{trial}}^2$ can be difficult to compute. Whenever this error-in-variables linear regression cannot be reliably fitted, the unadjusted $R_{\text{trial}}^2$ is often obtained using a linear regression—equivalent to fixing all the elements of ${\Omega }_i$ equal to 0—by weighting the observations $({\alpha }_i,{\beta }_i)\text{'}$ by the trial size, in order to account somehow indirectly and approximately for measurement error. The number of events could also be used for weighting, but using the trial size is what is done in practice in clinical applications.

Let us consider the semiparametric Cox model, $h(t)=h_0(t)\exp \{\beta Z\}$ . By dividing the time scale into intervals $k=1,\dots ,K$ corresponding to the intervals between the observed event times, the so-called auxiliary Poisson model for the event indicator ${\delta }^{(k)}\sim \mathit{Poi}({\mu }^{(k)})$ in the k-th interval is $\log ({\mu }^{(k)})={\mu }_0^{(k)}+\beta Z+\log (y^{(k)})$ , where $y^{(k)}$ is the time spent at risk during the period k.^32,37 In such a model, the parameters $\{{\mu }_0^{(k)}{\}}_{k=1,\dots ,K}$ are the logarithms of the baseline risk rates during each interval k, under the assumption that the risk is constant within the interval. In practice, different time intervals can be employed, leading to an approximation of the estimators of the Cox model parameters.

In meta-analyses, each subject j is recruited within a trial i. To account for both within-trial and within-subject dependence, the joint model for the two endpoints can be expressed as

Table 2 reports the results of simulations for the adjusted copula models and the Poisson models when data were generated using the bivariate mixed proportional hazard model.

Table 2. Results with data simulated with mixed proportional hazard models.

Table 2. Results with data simulated with mixed proportional hazard models.

View larger version

The trial-level R² (see also Figure 2) was often well estimated by the Clayton and Plackett copula models when the R² was moderate (0.4) and underestimated by 10%–23% when it was high (0.8). The Hougaard model overestimated moderate R² by 13%–32%, but had lower bias when $R^2=0.8$ (7%–9%, downwards). Of note, the Hougaard copula model is probably the most different model from the data generating process used to generate the data, which can explain at least part of this bias. The Poisson TI model was often the model with the smallest bias and MSE when the R² was high, with bias $\le 8$ % and MSE as low as 0.46 times the Clayton copula (used as reference). Nevertheless, this model showed 11%–28% of bias (upwards) when the R² was low. The Poisson TIa model provided the least biased estimates in the scenarios with high R² and high τ, but its results were mixed in other cases. The MSE of all methods for the estimation of the R² were similar to each other across different scenarios.

Download

Open in new tab

Download in PowerPoint

Figure 2. Relative bias (with respect to the true value) of the estimate of R² with data generated with mixed proportional hazard models. N: number of trials. n_i: number of patients per trial.

The estimates of the Kendall’s τ (Table 2, supplementary Figure S3) were not or little biased for all methods except the Hougaard copula model, which tended to overestimate (+16% – 17%) the τ when it was moderate (0.4) and to underestimate it (−42%) when it was high (0.6). Also, its MSE was extremely high as compared to the Clayton copula model (up to $7\times {10}^2$ -fold higher when $\tau =0.6$ ).

Unadjusted copula models (supplementary Table S1) often underestimated the R² (bias as high as 49%), and in general, the bias was higher than the associated adjusted counterparts. On the other hand, such increased bias for unadjusted copula models came with excellent convergence rates (see supplementary Table S2). Among models with measurement-error adjustment, the Clayton and Plackett copula models showed the highest convergence rates (see also supplementary Figure S4); the Hougaard was the copula model with the worst convergence rates, in particular with several small trials. Within the Poisson family of models, the Poisson TI often had the highest convergence rates, in particular, when $\tau =0.4$ .

The results with variable trial sizes (scenarios 3s and 4s, Table S3) were very similar to those in meta-analyses of fixed-size trials with the same total number of trials and patients (scenarios 3 and 4).

In summary, in this first part of simulations, the adjusted Clayton model showed the best results when $R^2=0.4$ , while the Poisson TI model was the most accurate and precise when $R^2=0.8$ .

When data were generated from the Clayton copula model (Table 3 and Tables S4 and S5; Figure 3), the true model recovered the R² with no or small bias in general, showing the worst results with few trials (10), high τ (0.6), and low R² (0.4). The Clayton model recovered the Kendall’s τ without bias in all the scenarios (Figure S5). The Plackett and Hougaard models showed serious convergence issues in this case (supplementary Table S6 and Figure S6). The former model had convergence rate <10% in all the scenarios, whereas the latter converged to a reliable result in ≥10% of the replicates only when $\tau =0.6$ . Of note, also in this case, the Plackett and Hougaard copulas were misspecified models with respect to the Clayton copula used to generate the data.

Table 3. Results with data simulated with Clayton copula.

Table 3. Results with data simulated with Clayton copula.

View larger version

Download

Open in new tab

Download in PowerPoint

Figure 3. Relative bias (with respect to the true value) of the estimate of R² with data generated with Clayton copulas. The estimated value is not plotted whenever the convergence rate was lower than 10%. N: number of trials. n_i: number of patients per trial.

Models Poisson TI and TIa, which often converged in at least 80% of the replicates, performed similarly to the Clayton copula, with absolute bias ≤15% and very often ≤5%, notably with higher values of N. On the other hand, the Poisson models estimated the Kendall’s τ with (downwards) bias of 10%–21% and high MSE as compared to the Clayton model (relative mean squared error as high as $3\times {10}^2$ ).

In the sensitivity analyses mimicking the association observed in the adjuvant GASTRIC meta-analysis, the results (Table S7) confirmed convergence issues when dependence is high. Such issues can heavily depend on the data generating process. The convergence rates of the adjusted Plackett model were 30% and 0% with data generated by mixed proportional hazard models and Clayton copulas, respectively, while for the Poisson models, they were 0% and 63%–73%. The R² was estimated accurately in general, except for the Clayton copula when the data were generated by mixed proportional hazard models (18% downwards). The Hougaard copula strongly underestimated the Kendall’s τ (−73% and −64%).

Table 4 shows the estimates of the trial-level R² and of the individual-level Kendall’s τ obtained with the models described in the paper. The Kendall’s τ for the previously published results was recovered based on the published Spearman’s ρ, according to the Plackett’s copula function employed therein (see Section 2 and Figure S1).

Table 4. Surrogacy results. Advanced GASTRIC meta-analysis.

Table 4. Surrogacy results. Advanced GASTRIC meta-analysis.

View larger version

The estimates of trial-level surrogacy from adjusted copula models (0.41 [95% CI: 0.15–1.00], 0.40 [0.04–1.00], and 0.38 [0.01–1.00] for Clayton, Plackett, and Hougaard, respectively) were lower than those from their unadjusted equivalents (0.45 [0.28–0.74] for all the three). The estimate from model Poisson TI was $R^2=0.63$ (95% CI: 0.32–1.00). The model Poisson TIa, which had the best convergence metrics of all models, provided the highest estimate of R²= 0.83 (95% CI: 0.24–1.00).

Individual-level surrogacy was the highest for the Clayton ( $\tau =0.61$ , 95% CI: 0.59–0.62) and the Plackett ( $\tau =0.62$ , 95% CI: 0.60–0.63) copulas. The Hougaard copula provided the lowest estimate (0.32, 95% CI: 0.32–0.33), whereas the Poisson models estimated the τ at 0.51 (95% CI: 0.50–0.52). The confidence intervals of all the models were very narrow.

The convergence metrics for the advanced case are provided in Table S8. In this context, the copula models had high maximum gradient ( $\ge 1.5$ ), whereas this was much smaller ( $\le 5\times {10}^{-5}$ ) for the Poisson models. The smallest eigenvalue of the covariance matrix of the random treatment effects was positive for all the adjusted models.

The results above are from auxiliary Poisson models estimated over eight time intervals. Supplementary Figure S7 shows that, even though the estimates of $R^2\text{ and }\tau$ are instable for a small number of intervals, they are quite robust between 8 and 24 time intervals, whereas models failed to converge when 32 intervals were used.

When analyzing the adjuvant gastric meta-analysis data, all the models had serious convergence problems (Table S9), with high maximum gradient ( $\ge 660$ for the copula models, $\ge 5.1\times {10}^{-5}$ for the Poisson models) and low minimum eigenvalue of the covariance matrix of the random treatment effects ( $\le 3.5\times {10}^{-11}$ ). Serious convergence problems were also reported in the original publication by Oba et al.¹⁶ As shown in supplementary Figure S8, using a different number of time intervals for the Poisson approach was not superior with regard to this aspect.

The results obtained, which have to be interpreted with high caution, are shown in Table 5. The R²s estimated by copula models with adjusted linear regression ranged from 0.94 (0.08–1.00, Hougaard model) to 1.00 (0.69–1.00, Plackett model). With the Poisson models, the R² was estimated to be 0.54 (0.02–1.00, model TI) and 1.00 (0.08–1.00, model TIa). The estimates of the Kendall’s τ ranged from 0.74 (95% CI: 0.73–0.76) to 0.82 (0.81–0.83), except for the Hougaard copula (0.18 [0.17–0.19]).

Table 5. Surrogacy results. Adjuvant GASTRIC meta-analysis.

Table 5. Surrogacy results. Adjuvant GASTRIC meta-analysis.

View larger version

Of note, the convergence diagnostics (Table S9) suggest to be very careful in interpreting the point estimates of all models. Furthermore, due to the width of the confidence intervals, it is difficult to make conclusive comparisons between these estimates.