An Efficient Method for Computing Expected Value of Sample Information for Survival Data from an Ongoing Trial

Background Decisions about new health technologies are increasingly being made while trials are still in an early stage, which may result in substantial uncertainty around key decision drivers such as estimates of life expectancy and time to disease progression. Additional data collection can reduce uncertainty, and its value can be quantified by computing the expected value of sample information (EVSI), which has typically been described in the context of designing a future trial. In this article, we develop new methods for computing the EVSI of extending an existing trial’s follow-up, first for an assumed survival model and then extending to capture uncertainty about the true survival model. Methods We developed a nested Markov Chain Monte Carlo procedure and a nonparametric regression-based method. We compared the methods by computing single-model and model-averaged EVSI for collecting additional follow-up data in 2 synthetic case studies. Results There was good agreement between the 2 methods. The regression-based method was fast and straightforward to implement, and scales easily to include any number of candidate survival models in the model uncertainty case. The nested Monte Carlo procedure, on the other hand, was extremely computationally demanding when we included model uncertainty. Conclusions We present a straightforward regression-based method for computing the EVSI of extending an existing trial’s follow-up, both where a single known survival model is assumed and where we are uncertain about the true survival model. EVSI for ongoing trials can help decision makers determine whether early patient access to a new technology can be justified on the basis of the current evidence or whether more mature evidence is needed. Highlights Decisions about new health technologies are increasingly being made while trials are still in an early stage, which may result in substantial uncertainty around key decision drivers such as estimates of life-expectancy and time to disease progression. Additional data collection can reduce uncertainty, and its value can be quantified by computing the expected value of sample information (EVSI), which has typically been described in the context of designing a future trial. In this article, we have developed new methods for computing the EVSI of extending a trial’s follow-up, both where a single known survival model is assumed and where we are uncertain about the true survival model. We extend a previously described nonparametric regression-based method for computing EVSI, which we demonstrate in synthetic case studies is fast, straightforward to implement, and scales easily to include any number of candidate survival models in the EVSI calculations. The EVSI methods that we present in this article can quantify the need for collecting additional follow-up data before making an adoption decision given any decision-making context.


Introduction
The Expected Value of Sample Information (EVSI) quantifies the expected value to the decision maker of reducing uncertainty through the collection of additional data, 1,2 for example a future randomised controlled trial.Although a few studies have considered the use of EVSI methods at interim analyses of adaptive trials, 3 overall little research has been done on EVSI for trials that are ongoing at the point of decision making.

Decision problem and model definition
We assume a decision problem with d = 1, . . ., D decision options.The net benefit of option d is NB(d, θ), and we have a cost-effectiveness model that predicts this quantity, given a vector of p possibly correlated model input parameters, θ = {θ 1 , . . ., θ p }. Our current judgements about the vector θ is represented by the joint probability distribution p(θ).Our goal is to choose the decision option with the greatest net benefit.

EVSI for further follow-up in an ongoing study
The EVSI for a new study that will provide (as yet uncollected) data, x, is defined as: where the first term is the expected value of a decision based on our beliefs about θ given the new data, p(θ|x), and the second term is the expected value of a decision based on our beliefs about θ given current information alone, p(θ). 12We now imagine that data x have been collected during a given follow-up period for this study, which we denote time t 1 .This could be an interim analysis, or the end of the study follow period.
The value of extending follow-up from current time t 1 to some future point t 2 is given by where the first term is the expected value of a decision based on our beliefs about θ given both new data, x, collected between t 1 and t 2 , and data, x, collected between time zero and t 1 .The second term is the expected value of a decision based on our beliefs about θ given only the information collected up until t 1 .See Appendix A for a fuller explanation.

Specifying current beliefs about model parameters for an ongoing study
The distribution for the cost-effectiveness model parameters given knowledge at t 1 p(θ|x) can be defined either in a fully Bayesian manner, by updating (possibly vague) prior information about θ with data x, or by fitting a standard frequentist statistical model to x and obtaining the maximum likelihood estimate for θ along with some expression of uncertainty, and treating this as a Bayesian posterior.In the absence of strong prior information about θ, the two methods will produce very similar distributions for p(θ|x), even with relatively little data. 13

Specifying the likelihood for ongoing time-to-event data and left-truncation
To compute EVSI we must define the data-generating distribution for the follow-up data between t 1 and t 2 , p(x|θ).We first consider the structure of the data we will observe.We assume our study has two arms: new treatment and standard care, and that N participants are recruited into each arm.Data, x, collected from time zero to t 1 take the form of a vector of times-to-death, -end of follow-up or -loss to follow-up, whichever is soonest.Survival times for those alive at t 1 are censored.If we continue to collect data x from t 1 to t 2 we may observe times-to-death for the participants whose observations were censored at t 1 .Survival times for those alive at t 2 or lost to follow-up are now the only observations censored.Table 1 illustrates the structure of the data for one arm of a study with follow-up at 12 and 24 months.
Table 1: The structure of the data for one arm of a study with follow-up at 12 and 24 months.Five participants are shown.Data are denoted x = { (9.3, 12, 12, 6.7, 12), (1, 0, 0, 0, 0)} for observations up until t 1 = 12 months, and x = {(13.4,24, 15.9), (1, 0, 0)} for observations between t 1 and t 2 = 24 months.Survival times are usually assumed to arise from a data generating process that can be described using a parametric model, the form of which must be chosen by the analyst. 14Censoring is common when collecting time-to-event data, since the follow-up time may not be long enough to observe the endpoint of interest for all individuals in the trial, and some individuals may be lost to follow-up. 15The likelihood function for survival data, x, obtained up until t 1 for a model with hazard function h(•) and survivor function where i indexes the n 1 = N study participants at risk at time zero, where the censoring indicator δ i = 1 when x i is an observed event, δ i = 0 when x i is a censored observation, and where θ are the parameters of the survival distribution.The observed dataset at time point t 1 consists of the n 1 survival times and censoring indicators, x = {x 1 , . . ., x n1 , δ 1 , . . ., δ n1 }.
The data collected between time points t 1 and t 2 is denoted x = {x 1 , . . ., xn2 , δ1 , . . ., δn2 }, where n 2 is the number of study participants at risk at t 1 .The likelihood function for x is left-truncated at t 1 to reflect that events beyond t 1 are conditional on not having occurred prior to t 1 . 16Unlike censoring, which contributes to the likelihood by plugging in a survival factor for censored observations as well as observed survival times, truncation does not add any data points to the likelihood.This distinction is important, since we want to avoid double counting the observed data x when we compute the likelihood for the ongoing study data x.
The left-truncated likelihood has an additional term in the denominator that re-normalises the truncated distribution so that it integrates to 1, i.e.

Left-truncated likelihood p
Once we have derived the posterior distribution for the model parameters given data at t 1 , p(θ|x), and the likelihood for the ongoing follow-up data, p LT (x|θ), we require a method for actually computing expression (2).In almost all realistic applications this will require numerical methods.Nested Monte Carlo can be used, but this is computationally expensive.A regression-based approach is much quicker, 10 and this is described along with the Monte Carlo approach in Appendix B.
We are now in a position to describe methods for computing EVSI that account for uncertainty about the choice of survival model.

Survival model uncertainty and model averaging
In this section, "model" refers to the survival model for the time-to-event data p(x|θ), not the cost-effectiveness model, NB(d, θ).In many real applications we will be uncertain about which survival model is most appropriate and should be used to extrapolate the data beyond the observed follow-up period t 1 , though we may be comfortable with proposing a candidate set of models, M = M r , r = 1, . . ., R, that covers plausible approximations of the data generating process, i.e. the set is M − open in the terminology used by Bernardo  and Smith (1994). 17In these circumstances, we may account for model uncertainty using predictive model averaging, and average over model predictions using model weights based on each model's predictive ability. 18,19fter observing data x at time t 1 , we place probability weight P (M r |x) on the r th model producing the best predictions, with and the optimal choice at time point t 1 is the decision d that maximises this expectation.

EVSI for an ongoing study accounting for model uncertainty
Additional follow-up data x will not only update our judgements about parameters, p(θ r |x, x, M r ), but will also update our judgements about the relative plausibility of each model, P (M r |x, x), for each model r = 1, . . ., R.
The EVSI for an ongoing study, where we average over models, is given by which is identical to (2), except that expectations are now taken over models as well as parameters (see Appendix C for a derivation).
To compute (6) we will need a method for generating plausible datasets x from p(x|x), the distribution of the follow-up data given the observed data, which takes account of the fact that we now consider plausible a number of different data generating models.We will also need to define model probabilities given observed data, P (M r |x), and then find a method for computing posterior model probabilities P (M r |x, x), given each sampled future plausible dataset x.We address the issue of defining model probabilities given observed data first.

Deriving model probabilities given observed data up until t 1
We assume that before we see the observed data x, that we are indifferent about the 'correct' model, so P (M r ) = 1 /R for all r.After we observe data x, we use the Akaike's Information Criterion (AIC) 20 to derive posterior model probabilities giving greater weight to models with better predictive ability (according to Kullback-Leibler divergence), as described by Jackson, Thompson, and Sharples (2009) 18 .We set where AIC r (x) = −2 log{p(x| θr )} + 2u r .
The term θr is the maximum likelihood estimate for the parameters of model M r , and u r is the number of parameters in model M r .
Generating plausible ongoing follow-up datasets, x, that we may observe between t 1 and t 2 Plausible datasets from the distribution p(x|x) are generated as follows.Firstly, we sample a model M (k) r with probability P (M r |x) given by Equation (7).Next, we draw a sample θ r .We can repeat this process k = 1, . . ., K times to generate an arbitrary number of datasets.

Updating model probabilities given ongoing follow-up data from t 1 to t 2
We can derive our posterior model probabilities at time point t 2 , for dataset x(k) , via Bayes theorem: where p(x (k) |M r , x) is the marginal likelihood ('marginal' because we have integrated out the model parameters): 2][23][24] The key notion behind bridge sampling is that the marginal likelihood can be written as the ratio of two expectations, each of which can be estimated via importance sampling.The name 'bridge' reflects the incorporation in the estimator of a density function that 'bridges' (i.e. has good overlap with) the two densities from which samples are drawn.A detailed tutorial on the bridge sampling method is given in the article by Gronau et al. (2017) 23 , and the method is straightforward to implement in the R package bridgesampling 25 .Given the bridge sampling estimates of p(x (k) |M r , x) for each model, posterior model probabilities are trivial to compute via expression (8).
As with single-model EVSI, computing model-averaged EVSI (expression 6) will require numerical methods.
Nested Monte Carlo and a regression-based approach are described in Appendix D. In the next section, we will apply these methods in a synthetic case study.

Synthetic case study
We will model survival with and without accounting for survival model uncertainty.

Decision problem and model definition
Our decision problem is to determine which of two treatment options has the longest mean survival; a new treatment (d = 1), or standard care (d = 2).
In the single-model case, survival is assumed to follow a Weibull distribution, and the net benefit of each treatment option is assumed to equal the restricted mean survival time, given an overall time horizon of t h = 240 months (i.e. the area under the survival curve from 0 to 240 months).So the net benefit function is: where the model parameters are the log-transformed Weibull shape and scale parameters, θ d = (θ kd , θ λd ).
Computing restricted mean survival for distributions other than the exponential requires numerical integration, but easy-to-use functions are available in the R package flexsurv. 26 the model-averaged case, the decision problem is as above, but we assume we are uncertain about the choice of survival model, M r , to extrapolate the observed data beyond the current follow-up period t 1 .We assume that our set of plausible models M contains the following four parametric distributions: Weibull (r = 1), Gamma (r = 2), Lognormal (r = 3), and Log-logistic (r = 4).

Generating synthetic case study datasets, x, collected up to t 1 = 12 months
We generated two synthetic case study datasets: one in which the hazard of death is monotonically increasing, and the other in which it is monotonically decreasing.For each case study we generated a dataset with 200 participants per trial arm with a maximum follow-up of t 1 = 12 months.We denote the datasets x 1 for new treatment and x 2 for standard care.
To explore the performance of the method when the survival model was mis-specified we generated survival times evenly spaced from either a Weibull or a Gamma distribution, using the 0.005 th , 0.015 th , . . ., 0.985 th , 0.995 th quantiles from each distribution (i.e. 100 evenly spaced quantiles that avoid 0 and 1).We could have randomly generated survival times, but this would have just added additional Monte Carlo error when assessing the methods for computing EVSI.The parameters of the Weibull and Gamma distributions that we used to generate the synthetic case study datasets are shown in Table 2.We enrolled all patients in the trial at t 0 = 0, and right-censored the datasets at t 1 = 12 months.We assumed no loss to follow-up and did not apply any other censoring.Figure 1 shows Kaplan-Meier plots for the two synthetic case study datasets.

Initial trial analysis at t 1 = 12 months
For each synthetic case study, we analysed the two trial arms separately.We fitted all four models to the data from each arm and estimated the model parameters using maximum likelihood (as implemented in the flexsurvreg function). 26We assumed that our judgements about the log-transformed parameters for each survival model conditional on the observed data up to t 1 , p(θ r |x), are represented by a bivariate Normal distribution with the mean vector and covariance matrix derived from the maximum likelihood estimation.We computed the AIC for each model fit and derived model probability weights via Equation (7).
Net benefits, AICs and model probabilities are shown in Table 3, and means and covariances for each model are reported in Appendix G.

Generating plausible ongoing follow-up datasets, x, for the EVSI computation
Both the nested Monte Carlo and regression-based EVSI methods require a set of sampled ongoing follow-up datasets for each trial arm, denoted x1 and x2 .We generated k = 1, . . ., K datasets with K = 6,000 for each trial arm, where the k th dataset was generated as follows.
In the single-model case, we first sampled log-shape and log-scale values, (θ for new treatment and θ (k) 2 for standard care), from the bivariate Normal distributions in Appendix G.We computed the net benefit for each decision option, given the sampled parameters, NB(d, θ (k) d ) and stored this (these values are required for the regression-based approximation).For each arm, we then sampled n survival times from a truncated Weibull distribution (see Appendix E) with the sampled shape and scale values where n was the number of patients who were still alive in the trial arm at t 1 = 12 months.Finally, survival times were censored at the proposed endpoint for the ongoing data collection, t 2 .
In the model-averaged case, we first chose a model M

Computing EVSI for ongoing follow-up via nested Monte Carlo
To sample from the posterior distributions, p(θ d |x d , x(k) d ), we used Hamiltonian Monte Carlo (HMC) as implemented in the package rstan 27 .HMC is a Metropolis-Hastings MCMC algorithm with a particularly efficient sampling scheme that reduces Monte Carlo sampling error, therefore requiring fewer posterior samples for any inference.The package rstan is an R interface to the Stan language. 28An alternative option would have been to use OpenBUGS. 29 the single-model case, for each outer loop sampled dataset, k = 1, . . ., 6,000.00,we averaged the net benefit functions over J = 2,000.00inner loop posterior samples of the model parameters, and stored the maximum net benefit of the two treatment options.We then averaged these maximised net benefits and subtracted the expected value of a decision based on current information to obtain the EVSI following expression (14) in Appendix B.
In the model-averaged case, for each outer loop dataset, we generated the J posterior samples of the model parameters for each of the r = 1, . . ., 4 models (we needed to identify the truncated likelihood function for each model as we did for the Weibull example above, but this is straightforward.See Appendix E).We weighted the parameter averaged net benefits NB k r (d) by the posterior model probabilities P (M r |x (k) ) to give the posterior model-averaged expected net benefit, and identified the treatment d that maximized this for iteration k = 1, . . ., 6,000.00.We then subtracted the expected value of a decision based on current information to obtain the EVSI following expression (23) in Appendix D.

Computing EVSI for ongoing follow-up via regression
The GAM approach to computing EVSI for extending the follow-up until time t 2 for the hypothetical example is as follows.
For each trial arm, we computed a low dimensional summary statistic for each dataset.A convenient choice here is the number of observed events e as independent variables.We allowed a smooth, arbitrary, non-linear relationship between the independent and dependent variables, plus arbitrary interaction between the independent variables, by specifying a 'tensor product' cubic regression spline basis for the independent variables.This has the simple syntax gam(nb_d ~te(e_d, y_d)) in the mgcv 30 package in R. We extracted the GAM model fitted values ĝ(k) d from each regression model fit, and estimated the EVSI using Equation (18) in Appendix B.
The GAM-based approximation method for model-averaged EVSI is identical to that used in the single-model case.

EVSI values for the Weibull ongoing data
The nested Monte Carlo-and GAM-based EVSI estimates for additional follow-up times of 12, 24, 36, and  As expected, the EVSI reflects diminishing marginal returns for increasing follow-up duration and converges towards the EVPI.The EVSI varies depending on the underlying hazard pattern, even when point estimates of mean incremental survival benefit are similar (6.95 months for the increasing hazard dataset and 6.97 months for the decreasing hazard dataset).The increasing hazard dataset has lower numbers of prior observed events and higher expected numbers of future events for the additional follow-up time than the decreasing hazard dataset, which -all else equal -is expected to result in greater EVSI values.This upwards effect on EVSI is however canceled out by the downwards effect of lower estimates of mean survival, resulting in greater EVSI values for the decreasing hazard dataset than for the increasing hazard dataset.
The GAM method agrees well with the MCMC method, with the benefit of a greatly reduced computational cost.The MCMC inner loop for the Monte Carlo method used parallel processing, but even with this additional efficiency, the regression method was approximately 700 times faster than the nested Monte Carlo method.We used a machine running Windows 10 with an Intel ® Core TM i9 CPU with 15 threads running on 8 cores at 2.40GHz, and with 32 GB RAM.
Of note is that the standard errors for the nested Monte Carlo estimator slightly increase with increasing follow-up duration, while the opposite is true for the GAM estimator.This is due to different mechanisms through which the effective sample size of the generated data x affects the standard errors of the nested Monte Carlo and GAM estimators, which is further explained in Appendix F.

Model-averaged EVSI values
The nested Monte Carlo-and GAM-based model-averaged EVSI estimates for additional follow-up times of 12, 24, 36, and 48 months (i.e.t 2 = 24, 36, 48, 60 months) are shown in Table 5.As expected, the EVSI converges towards the EVPI as follow-up time increases, and there is good agreement between the two methods.The model-averaged EVSI values for additional follow-up are greater than the Weibull model EVSI (Table 4), which reflects the additional value in reducing model as well as parameter uncertainty.The GAM method is approximately 8,000 times faster than the nested Monte Carlo method.

Expected Net Benefit of Sampling
The net value of additional data collection can be quantified by computing the Expected Net Benefit of Sampling (ENBS). 32In the context of an ongoing study, the ENBS is the difference between the EVSI for collecting additional data between t 1 and t 2 and the expected cost of continuing the study and potential health benefits foregone if approval is withheld.When the ENBS is positive, it is worthwhile to continue the study and collect more data before making an adoption decision.If the adoption decision is reversible, approval can be granted while additional data is being collected.This is referred to as "approval with research" (AWR). 6If the adoption decision is irreversible, approval should be withheld until the additional data has been collected, which is referred to as "only in research" (OIR).
Figure 2 illustrates that when approval is reversible and AWR can be recommended, the marginal benefit in terms of model-averaged EVSI equals the marginal cost of continuing the trial at 47 and 50 months of additional follow-up for the increasing and decreasing hazard datasets, respectively.These are the time points at which the ENBS is at a maximum.When approval is irreversible and OIR is recommended, the ENBS is at a maximum when the marginal benefit of delaying the decision until more data has been collected equals the marginal cost of continuing the trial and withholding approval, which is at 20 and 24 months of additional follow-up for the increasing and decreasing hazard datasets, respectively.

Discussion
EVSI is useful not only for informing the design of a future trial, but also for deciding whether an ongoing study should continue in order to collect additional data before making an adoption decision.This article is the first to set out generic EVSI algorithms for survival data from an ongoing trial with or without accounting for survival model uncertainty.The EVSI algorithms generalise to any decision context in which structural uncertainty is present, provided that the analyst is able to derive probability weights for the competing scenarios.

Strengths and limitations
The nonparametric regression-based method is fast and straightforward to implement, even when we include consideration of model uncertainty.In fact, extending the method to include model uncertainty does not increase the complexity or computation time.The nested Monte Carlo procedure, on the other hand, is extremely computationally demanding when we include model uncertainty.
When a large part of the relevant time horizon is unobserved, the clinical plausibility of the survival extrapolations is often of greater importance than the mathematical fit to the observed data. 14Deriving prior model probabilities from purely statistical measures such as AIC may therefore not always be appropriate when data are immature, since these measures do not reflect the plausibility of the extrapolations. 8This became evident in the hypothetical case studies, as the AIC-based prior model probabilities of the lognormal and log-logistic models were similar to those of the Weibull and Gamma models for the increasing hazard dataset, despite the fact that the former two models do not allow for monotonically increasing hazards and therefore cannot capture the true underlying hazard pattern.
An alternative approach to dealing with model uncertainty could be to consider a single very flexible model that includes all the models the analyst believes plausible.For example, the Generalized F distribution includes most commonly used parametric survival distributions as special cases. 33It is however more common to view model uncertainty as structural uncertainty in choosing between competing survival models. 9,34urthermore, the use of a very flexible model requires the specification of a prior that appropriately reflects uncertainty in choosing between alternative functional forms within the flexible model, which may be not be straightforward.Flexible models such as the Generalized F distribution are also prone to overfitting and may not always provide reliable predictions of mean survival, particularly when data is immature. 9 Although we did not consider flexible parametric models such as Royston-Parmar spline-based models 35 or mixture cure models 36 in our case studies, the principles outlined in this article apply to any parametric survival model.
In the synthetic case studies, we assumed all patients had the same follow-up at t 1 .In clinical trials, patients are usually recruited over a period of time, which means the individual follow-up times will vary at t 1 .In these circumstances, additional follow-up will not only provide more information about the tail of the survival curve (from patients that were enrolled early), but also about the central part (from patients that were enrolled later).
We did not consider sequential trial designs 37 , which require EVSI to be recalculated after each observation and to account for all the possible ways in which future patients may be allocated to the trial arms or when to stop the trial. 38This can give rise to a large number of subproblems that may have to be solved using dynamic programming methods, which can be computationally very demanding.

Policy implications
Immature evidence leads to a high level of decision uncertainty, which may result in the uptake of technologies that reduce net health benefit.The decision making context in which trials are ongoing and evidence is immature is particularly pronounced for new oncology drugs.The purpose of the Cancer Drug Fund (CDF) in the UK, for example, is to enable early patient access to promising new cancer drugs while allowing evidential uncertainty to be reduced through ongoing data collection.In the period between 2017 and July 2018, the National Institute for Health and Care Excellence (NICE) recommended over half of the appraised cancer drugs through the CDF, typically due to concerns about immature survival data. 39e EVSI algorithms in this article can help decision makers determine whether early patient access to a new technology can be justified on the basis of the current evidence or whether more mature evidence is needed.The option to enroll more patients into an ongoing trial should also be considered if it has a positive net value.Unlike most of the existing work on EVSI that primarily targets commissioners and funders of research, EVSI for ongoing trials also addresses the policy context of decision makers who do not have the remit to commission additional research.

Appendices Appendix A -Deriving an expression for the EVSI for an ongoing study assuming no model uncertainty
Before the study starts, we have only prior knowledge about model parameters, which we represent via the distribution p(θ).In many cases we will not have strong prior information, and p(θ) will therefore be minimally informative (typically flat on some scale).
We collect data x during an initial period of follow-up that extends up until time t 1 .At t 1 we update our judgements about θ, conditional on x to give the posterior distribution p(θ|x).The question is, should we continue the study to collect more data before making an adoption decision?
The optimal decision option given observed data up to time t 1 has expected value Further data collection up until time point t 2 will give us additional data x, which we can use to update judgements about θ to give p(θ|x, x).The optimum decision option will have expected value, At time point t 1 data x are as yet uncollected, however we can take the expectation of expression (11) with respect to the distribution of the ongoing follow-up data x conditional on the observed follow-up data x, p(x|x), giving The EVSI for continuing the study from t 1 to t 2 is the difference between the expected value of a decision made after collecting data up to t 2 , expression (12), and the expected value of a decision based on observed data collected up to t 1 , expression (10),

Nested Monte Carlo method for computing EVSI for an ongoing study
Calculating EVSI for an ongoing study requires evaluation of the expectation of a maximised conditional expectation, E x|x [max d E θ|x,x {NB(d, θ)}].This will rarely have an analytic solution.A nested expectation can be evaluated using a nested 'double-loop' Monte Carlo scheme, which leads us to the following estimator for EVSI, NB(d, θ (j,k) ).(14)   In this scheme, we generate samples from p(x|x) in the 'outer loop.'We do this by first sampling θ (k) , k = 1, . . ., K from p(θ|x), and then sampling x(k) from the truncated likelihood p LT (x|θ (k) ).For each sample x(k) , we then sample values θ (j,k) , j = 1, . . ., J from the posterior distribution p(θ|x, x(k) ) in the 'inner loop'.Unless p( LT x|θ) and p(θ|x) are conjugate, which will be rare in practice, then sampling from p(θ|x, x(k) ) will require Markov Chain Monte Carlo (MCMC) or a similar scheme.The total number of samples required for each d is J × K.
Note that the second term in expression (14)

Regression-based method for computing EVSI for an ongoing study
Strong and others (2015) 10 developed a fast, non-parametric regression-based method that greatly reduces the computational burden of the nested Monte Carlo procedure to EVSI.Their approach relies on estimating the functional relationship between the posterior expected net benefits and the generated datasets, thereby avoiding the inner loop and markedly increasing efficiency over the nested Monte Carlo method.
In the regression approach, we first generate a random parameter vector θ (k) from the distribution of model parameters p(θ|x) at time point t 1 , and a random data sample x(k) from the truncated likelihood p LT (x|θ (k) ), where k indicates the k th sample.The net benefit is evaluated at the same k th sample of the model parameters, NB(d, θ (k) ).We then express the observed net benefit NB(d, θ (k) ) as a sum of the conditional expectation of the net benefit given the data, E θ|x,x (k) {NB(d, θ)}, which we wish to estimate to evaluate the EVSI (Equation ( 2)), and a mean-zero error term, ε (k) , As explained by Strong and others (2015) 10 , we can think of the conditional expectation E θ|x,x (k) {NB(d, θ)} as an unknown function of x(k) .We denote this function g(d, x(k) ) and substitute this into Equation (15), giving Since x is a vector of (possibly censored) time-to-event data, and therefore high-dimensional, we write the the function g in terms of a low-dimensional summary statistic of the data T (x), We then use a generalized additive model (GAM), which is a flexible non-parametric regression model, to estimate the target function g.This means that we fit a GAM model to each decision option d and extract the regression model fitted values to estimate posterior net benefit.We denote the GAM model fitted values as ĝ(k) d .The GAM-based EVSI estimate is given by EVSI

Appendix C -Deriving an expression for the EVSI for an ongoing study accounting for model uncertainty
In the model averaging setting, additional follow-up data x will update our judgements about both parameters and the relative of each model.
The net benefit function for decision option d given model M r and parameters θ r is denoted NB(d, θ r , M r ).At time point t 1 after observing data x, the expected net benefit, averaging over both parameters and models is and the optimal choice at time point t 1 is the decision d that maximises this expectation.
The net benefit after observing ongoing follow-up data x between t 1 and t 2 is and the optimal choice at time point t 2 is the decision d that maximises this expectation.Follow-up data x are not available at t 1 , but we can compute the expected value of the maximised net benefit based on our beliefs from the data collected by t 1 , The EVSI for an ongoing study, where we average over models, is then the difference between (21) and the maximised value of (19), where xi and δi are the survival time and censoring indicator for patient i, where censoring is at the proposed new follow-up time of t 2 .The expressions above are similarly defined for standard care (d = 2) with θ k2 , θ λ2 replacing θ k1 , θ λ1 , and x2 replacing x1 .
Let i index the n 1 = N study participants at risk at time zero, where the censoring indicator δ i = 1 when x i is an observed event, δ i = 0 when x i is a censored observation, and where θ are the parameters of the survival distribution.

Truncated likelihood function for the Gamma distribution
The Gamma density function given log-shape θ α , log-rate θ β and survival time x is .

Method for sampling from a truncated distribution
We can sample values from a truncated survival distribution that lie in the interval (t 1 , ∞) as follows.We denote the cumulative density function evaluated at time t with parameters θ as F (t, θ).We first compute the value of the cumulative density function at t 1 , p = F (t 1 , θ), (i.e. the probability that a survival time will exceed t 1 ).We then sample n values from a uniform distribution on the interval [p, 1], and plug these into the corresponding inverse cumulative density function F −1 (•, θ).This results in n survival times greater than t 1 that follow the required truncated survival distribution.

R r=1 P
(M r |x) = 1.The net benefit function for decision option d given model M r and parameters θ r is denoted NB(d, θ r , M r ).Taking the expectation over both parameters and models after observing data x up to time point t 1 gives us Model-averaged NB d |x = R r=1 E θr|x,Mr NB(d, θ r , M r )P (M r |x) of the parameters of our chosen model p(θ r |x, M (k) r ).Finally, we generate a dataset x(k) from the distribution of the data p(x|θ

Figure 1 :
Figure 1: Kaplan-Meier plots for the increasing hazard dataset (left) and decreasing hazard dataset (right) (k) r with probability P (M r |x), before sampling θ (k) r from the bivariate Normal distribution p(θ r |x) for the chosen model M (k) r and generating the n survival times for each arm.The remainder of the data generation step is as above.

d
} for d = 1, 2. Then, for each of the two decision options, we fitted a GAM regression model with the stored net benefits NB(d, θ (k) d ) as the dependent variable, and the two summary statistics, e

Figure 2 :
Figure 2: Marginal benefit (MB EVSI ), marginal cost of 'approval with research' (MC AWR ) and marginal cost of 'only in research' (MC OIR ) given different durations of additional follow-up.Estimates are based on the model-averaged EVSI analyses for the increasing hazard dataset (left) and decreasing hazard dataset (right), trial costs of 5 life months per month, 5 new patients receiving treatment each month and a decision time horizon of 10 years.

Table 2 :
Weibull and Gamma distribution parameters for the synthetic case study datasets +

Table 3 :
Mean survival, Akaike's Information Criterion and prior model probabilities P (M r |x) for the two hypothetical datasets The expected net benefits (mean survival times) assuming a single Weibull model computed via Equation (9) are 50.96versus44.01 months (incremental = 6.95 months) for the increasing hazard dataset, and 84.81 versus 77.85 months (incremental = 6.97 months) for the decreasing hazard dataset.The Expected Value of Perfect Information (EVPI) values, computed via Monte Carlo simulation with a sample size of 10 5 , are 4.93 and 6.33 months for the increasing and decreasing hazard dataset, respectively.The model-averaged net benefits, weighted by model probabilities, are 72.93 versus 62.36 months (incremental = 10.57months) for the increasing hazard dataset, and 93.31 versus 85.85 months (incremental = 7.46 months) for the decreasing hazard dataset.The model-averaged EVPI values are 10.32 and 9.97 months for the respective datasets.

Table 4 :
48 months (i.e.t 2 = 24, 36, 48, 60 months) are shown in Table 4.The methods used to estimate the standard errors of the nested Monte Carlo and GAM estimators are described in an appendix of the article by Strong, Oakley, and Brennan (2014). 31EVSI (SE) values for additional follow-up time for the two hypothetical datasets given a Weibull distribution for the survival times

Table 5 :
EVSI (SE) values for additional follow-up time for the two hypothetical datasets given a Weibull distribution for the survival times times for the analyses in the table are 289,211 seconds (Nested Monte Carlo) and 37 seconds (GAM).
10s a nested double loop structure, even though the target estimand is the single maximised expectation max d E θ|x {NB(d, θ)}.We reuse the same samples for both terms in the EVSI expression in order to reduce Monte Carlo error, noting that maxd E x|x [E θ|x,x {NB(d, θ)}] = max d E θ|x {NB(d, θ)} by the law of total expectation.10

Table G3 :
Bivariate Normal distribution hyperparameters for the Lognormal model parameters given data collected up to t 1 = 12 months Parameter Mean, µ Covariance matrix, Σ

Table G4 :
Bivariate Normal distribution hyperparameters for the Log-logistic model parameters given data collected up to t 1 = 12 months Parameter Mean, µ Covariance matrix, Σ