Mode and Frame Matter: Assessing the Impact of Survey Mode and Sample Frame in Choice Experiments

Background. Choice experiments (CE) are applied in health economics to elicit public preferences and willingness to pay (WTP). CEs are frequently administered as Internet-based surveys. Internet surveys have recognized advantages, but concerns exist about the representativeness of Internet samples, data quality, and the impact on elicited values. Aim. We conducted the first study in health comparing an Internet-based CE survey with the more traditional general population mail survey. We also compared the Internet-based and mail CE surveys with computer-assisted personal interviews (CAPIs), which are commonly used to elicit health state valuations. Methods. Two separate samples were drawn from 2 United Kingdom (UK) volunteer Internet panels (IPs), CAPIs were undertaken with respondents sampled from UK Census Output Areas, and mail surveys were sent to UK households drawn from the postcode address file (PAF). Each mode received more than 1000 respondents. We compared modes and frames using objective measures (response rate, sample representativeness of the UK population, elicited values, theoretical validity, and cost per response) and subjective/self-reported measures (time taken to complete the study, perceived study consequentiality, and stated attribute nonattendance). This study intentionally confounded the survey modes and sample frame by choosing sample frames that are typically used by researchers for each mode. Results. Estimated WTP differs across mode-frame pairs. On most measures, CAPIs dominated. They are more expensive, however. On all measures, except response rates, Internet surveys dominated the mail survey. They were also cheaper. Conclusion. Researchers using IPs should pay attention to response rates and be aware that the quality of IPs differs. Given the importance of perceived consequentiality and attribute attendance in CEs, future research should address their impact across modes and frames.

however, influence who is asked to respond to a survey (sample frame), who does or does not respond to a survey (nonresponse bias), how respondents answer the survey questions, and the respondents' ability to provide accurate responses (measurement error).
Literature on the effects of survey mode in health economics is limited in the number of studies and the modes compared. The literature also provides mixed evidence of survey mode effects. Norman et al. 2 compared a computer-assisted personal interview (CAPI) and online survey to elicit health state valuations using a timetradeoff task. They found the online survey had higher variability and more extreme responses than in-person interview valuations. 2 In contrast, Mulhern et al. 3 compared a CAPI survey and Internet panel (IP) survey to elicit health state valuations using a choice experiment (CE) and found no difference in valuations across modes but found significant differences in respondents' characteristics. They concluded that both modes may be equally valid. Rowen et al. 4 compared online and face-to-face interviews in a pairwise comparison study of social preferences for burden of illness. They found that the mode of administration affected responses and the socioeconomic characteristics of respondents. Determann et al. 5 compared patient preferences for health insurance elicited using a CE across 2 samples drawn from an IP's membership: one sample completed the questionnaire online, and the other received a mail survey. They found no evidence that online surveys yield inferior results compared with paper-based surveys but that they did have a lower price per completed respondent.
In-person interviews have been the mode of choice for health state valuation studies and for studies eliciting social preferences. Therefore, a comparison between in-person and online surveys is particularly relevant to the contexts studied by Mulhearn et al. 3 and Rowen et al. 4 Between 2000 and 2012, about half of all CE studies eliciting preferences for health and care were mail surveys, and 6% were online surveys, 6 but since 2013, more than half have been administered via the Internet. 7 No study has tested whether the results of a CE survey of health care preferences differ when the survey mode is a mail survey of the general population or an IP survey. Although Determann et al. 5 compared an online survey with pen-and-paper completion, both samples were drawn from the same IP.
We present the first study comparing CE results across IP surveys and a mail survey sent to UK households from the postcode address file (PAF) and a CAPI of UK households. Given differences across the operation and composition of IPs, we compared responses for 2 IPs. We compared modes and the sample frames most commonly associated with them using objective measures (response rate, representativeness of the samples [compared with the UK population], elicited values, theoretical validity, cost per response) and subjective/self-reported measures (time taken to complete the survey, perceived study consequentiality, stated attribute nonattendance).

Experimental Design
The CE The CE elicited general population preferences for characteristics of community pharmacies when managing a minor ailment (cold or flu). 8 This is a health care ''good'' relevant to the general population. The attributes and levels ( Table 1) were selected based on a review of quantitative and qualitative studies that elicited preferences for the use of pharmacy services to manage minor ailments and a parallel cohort study investigating how the public used pharmacy services to manage minor ailments. 8 Respondents were asked to complete a series of choice tasks in which they had to choose among 3 options: 2 pharmacy service options and doing nothing. In the minor ailment context, doing nothing is a realistic option. The experimental design, selected using SAS v9.2 and the Mktex macro, 9 resulted in 48 choice tasks. To reduce the burden to each respondent and maximize response rates, we blocked these into 6 blocks of 8 choice sets. An example choice set is presented in Figure 1.
The questionnaire also collected information on respondents' socioeconomic characteristics, health status, attitudes and beliefs (both about health care and in general), and data quality. The questionnaire is available online at the UK data service. 10

Modes of Data Collection and Sample Frames
We compared 3 survey modes: CAPI, mail, and IP. Data collection was concurrent across modes. Given IP surveys are quicker than both CAPI and mail surveys, data collection for this mode was staggered over 3 weeks. We aimed to receive 150 responses per mode in the pilot and 1000 responses per mode in the main study. We used a unimode approach, with the same questionnaire used across all modes to ensure survey differences across modes were minimal. 1 The IP and CAPI surveys were computer screen based, and the mail survey was a penand-paper, self-completed questionnaire.
We applied each method of data collection as it is typically applied by researchers; thus, for each mode, we used the most common UK sample frame to create a modeframe pair:  The mail survey was sent to a random sample of UK residential postal addresses from the PAF, stratified by geographical region. For the pilot study 1000 questionnaires and for the main study 6669 questionnaires were mailed out, based on an expected response rate of 15%. Each envelope was addressed to ''the occupier,'' and the invitation requested that ''the person in the household, of 18 years or over, whose birthday fell next'' should complete the questionnaire. Nonrespondents received a reminder and duplicate questionnaire 3 weeks after the initial mailing. Ipsos-MORI administered the mail survey. For the IP mode, samples were drawn from 2 UK volunteer IPs: Ipsos-MORI (IP-IM) and ResearchNow (IP-RN). These panels differ in member recruitment and reward, and the management of inattentive or dishonest members. 11 Such differences may affect the sample available to survey, the characteristics of respondents, and respondents' task engagement. The IP sample frames were stratified by age, gender, working status and geographical location. Potential participants were selected at random from those eligible within strata. Invitations to complete the questionnaire were emailed to the sample, with 2 reminders to nonrespondents. When individuals responded to the invitation, they were screened against the quotas until the target number of responses had been received. As per standard practice, respondents received a nominal incentive in the form normally used by the survey companies as a reward for completing the survey. The CAPI sample frame was based on UK Census Output Areas (COAs), stratified by region. Trained interviewers recruited participants in their own homes; invitations were issued verbally by interviewers to potential respondents. Approaches within a COA were guided by quotas (age, gender, and working status); interviewers worked to these quotas until the target number of responses had been achieved. The CAPI data were collected by Ipsos-MORI.

Comparison Tests, Hypotheses, and Results
We compared 4 data sets (CAPI, mail, IP-IM, IP-RN) according to response rates, respondent representativeness, values elicited, data quality (theoretical validity, time to complete survey, perceived consequentiality, and attribute nonattendance), and cost per response. Our results are summarized in Table 2.

Response Rates
We used the American Association for Public Opinion Research guidelines for response (participation) rate calculations. 11 Given it is not possible to calculate response rates for volunteer IP panels, we calculated participation rates. The results are shown in Table 3. CAPI had the highest rate and IP-RN the lowest. Participation rates for both IPs were lower than response rates for both offline mode-frame pairs and differed across IPs.

Respondent Representativeness
Survey response rates are linked to nonresponse error, which occurs when individuals who respond to the survey differ from those who do not respond. We compared respondent characteristics with the population to understand if there are observable differences between the samples that result from responses to a mode-frame pair. We note that unobservable differences may remain. We compared each survey mode-frame pair's respondents with the UK general population, based on a broad set of socioeconomic and health characteristics (see Table 4). We supplemented this with 2 additional characteristics-Internet access and voting attitude-as a proxy for social attitudes. All IP respondents had Internet access, but we collected Internet access data in the CAPI and mail surveys to gauge the number of respondents who do not have Internet access and therefore would be restricted in accessing IPs.
We used population data from 4 sources: the 2011 UK census (age, gender, education, employment, selfassessed health, activities of daily living 12 ), the Family Resources Survey (income 13 ), Eurostat community survey on Information and Communication Technologies (ICT) usage in households and by individuals (Internet access 14 ), and British Social Attitudes Survey (voting attitude 15 ). We hypothesized that CAPI respondents would be older and more likely to be retired than the general population 4 and that IP respondents would have higher educational qualifications, 2 have lower health status, be younger, and be less likely to be employed or retired. 4 We also compared respondents across mode-frame pairs based on 3 additional health-related variables for which there is no UK population data: experience of chronic illness, membership of a minor ailments scheme (MAS), and general beliefs about medicines. These characteristics are likely to influence routine use of community pharmacies, which may in turn affect CE responses and subsequent engagement/quality of data. Individuals who experience a chronic illness may be more likely to attend a pharmacy than those with no such illness. We hypothesized that IP respondents would be more likely to have a chronic illness than CAPI respondents. 4 MASs allow eligible National Health Service patients in the UK (e.g., those in receipt of social benefits or exempt from paying prescription charges) to receive free advice and medicines from community pharmacies for specific minor ailments (https://www.gov.scot/publications/nhs-minor-ailment-servi ce-local-pharmacy/); MAS membership signals that respondents already attend the pharmacy for minor ailments. The Beliefs about Medicines Questionnaire (BMQ) is a validated instrument developed to better understand people's percepti ons about medicines. 16 Previous studies have found that an individual's beliefs about medicines can influence health behavior, particularly their medicine use. [17][18][19][20] We used the BMQ-General, a measure of the extent to which respondents believe that medicines in general are overused, harmful, or beneficial (see Supplementary Appendix, Table  A1). We made no a priori hypotheses regarding membershi p of MAS and responses to the BMQ. Respondents' characteristics across survey modeframe pairs are reported in the Supplementary Appendix (Tables A2 and A3). Statistical comparison tests are presented in Table 4. Compared with the UK population: CAPI respondents were older, overrepresenting the population in all age categories from 65 to 84 years, Respondents were unavailable in the CAPI if no one answered the door and in the mail if the questionnaire was undeliverable. Respondents were ineligible in the CAPI if they fell into a quota category in which the quota had already been achieved. Respondents were ineligible in the Internet surveys if they were younger than 18 years or if they fell into a quota category in which the quota had already been achieved. No one was ineligible in the mail mode.
b Nonresponse in the mail survey means that the survey was not returned; in the CAPI, this means that respondents answered the door but refused to take part in the survey, and in the Internet panels, this means that respondents received the e-mail invitation but did not start or complete the survey. less likely to be an employee and more likely to have retired, to be in worse health, and earning less than the UK population. Mail respondents were older (in particular, respondents overrepresented the population in all categories older than 54 years), more likely to be female, better educated, more likely to be retired, in worse health, more likely to have a lower income, less likely to have Internet access, and more likely to state that it was one's duty to vote than the UK population. IP respondents were overrepresented by people aged 30 to 34 years and 65 to 74 years, better educated, more likely to be retired, in worse health, overrepresented by people with low or high incomes, and more likely to have Internet access than the UK population.
The distribution of the 3 additional health-related characteristics are reported in the Supplementary Appendix, Table A4. Table 5 reports comparison tests for these characteristics. Mail and CAPI respondents were more likely to report a chronic illness than IP respondents, and mail respondents were more likely to report a chronic illness than CAPI respondents. CAPI respondents were twice as likely to be MAS members as respondents to other mode-frame pairs. CAPI respondents were more likely than mail and IP-IM respondents to believe medicines are overused, and CAPI and IP-IM respondents were more likely to believe medicines are harmful than were mail or IP-RN respondents.

Elicited Values
We estimated respondents' preferences for pharmacy service attributes using an error components logit model. We assumed that the utility u of individual i for alternative j in choice set t for mode m is based on a systematic component specified as a linear and additive function of the study attributes and levels and an additive random component e ijt . The error-components model allowed us to merge data across all mode-frame pairs, accounting for mode-frame-specific scale differences. We thus estimated where d m is a dummy variable (that takes the value ''1'' if an alternative appears in mode m and ''0'' otherwise), a no and a B are alternative specific constants that control for aspects other than pharmacy characteristics (these explain respondents' choice of using a pharmacy service rather than taking no action (a no ) and any left/right bias in respondents' choice of pharmacy (a B )), x n are the levels of the n = 1 . . . N attributes included in the experimental design, b n are their respective parameter estimates, u represents a deviation from the mean alternative specific constant, and z i is a draw from a normal distribution N(0,1). We estimated separate error components for each mode-frame pair. We specified an additive error term e ijt that is Gumbel distributed. Following estimation of Equation 1, we explored differences in the estimated mean marginal willingness to pay (mWTP) across mode-frame pairs. We calculated the difference in mWTP for pairs of mode-frame pairs and tested if these differences were significant using the delta method. 21 We hypothesized no differences in mWTP across respondents.
We found differences across mode-frame pairs ( Table  6). CAPI and mail, and IP-IM and IP-RN were most similar. Mail respondents' mWTP estimates were significantly different from other mode-frame pairs for all attributes except ''pharmacy location''; they were the highest across all mode-frame pairs and for some attributes they were twice as large. CAPI and both IPs revealed significant differences in mWTP for ''pharmacy location,''''who you speak to,'' whether they are ''friendly,'' and whether you ''better understand symptoms.'' mWTP estimates across the IPs were significantly different for ''who you speak to,'' whether they are ''friendly,'' and whether you ''better understand symptoms.'' mWTP estimates were lowest among IP respondents.

Data Quality
Theoretical validity. Across mode-frame pairs, we compared if the parameters estimated with the errorcomponent model were in line with a priori hypotheses. We hypothesized that respondents would prefer being able to park, shorter waiting times for treatment, being seen by a more qualified person who is friendly and approachable, gaining a better understanding of their symptoms, and lower-cost treatments. We had no a priori hypotheses regarding preferences for location or being asked questions about symptoms and general health. We noted which parameters were not in line with a priori assumptions. We hypothesized that the CAPI would have the highest number of parameters in line with assumptions because the interviewer can assist respondents who find the task difficult.
The regression coefficients, reported in Table 7, are in line with a priori hypotheses for all attributes and modeframe pairs except for the car parking attribute in the IP-IM (where the estimated coefficients are not monotonic across the attribute levels).
Survey completion decision. Social exchange theory applied to surveys assumes that respondents answer a survey when perceived benefits outweigh costs. 1 Response costs are the time and effort required to complete the survey; benefits are the reward and the feeling of importance respondents (may) receive from taking part. We proxied response costs using time taken to complete the survey. Computer administered surveys (CAPI and IPs) collected this information automatically. Mail survey respondents manually recorded the time at the beginning and end of the survey. We compared time taken across pairs of mode-frame pairs using a 2-sample t test with unequal variances. We hypothesized that IP respondents would complete the survey faster than CAPI or mail respondents. 3 One proxy for respondents' benefit of responding is perceived consequentiality of the survey. Carson and Groves 22 argued that when respondents perceive that their responses are consequential, they have an incentive to reveal their true preferences. Vossler and Watson 23 found that survey respondents who perceived their responses to be consequential accurately predict votes in a public referendum. We measured what respondents perceived to be the impact of their responses using questions proposed by Scheufele and Bennett. 24 Respondents were asked whether they thought their answers to the survey would change how services are provided. 24 Responses were compared across mode-frame pairs using a chi-squared test. No previous studies have compared perceived consequentiality across mode-frame pairs. We hypothesized that CAPI respondents would have the highest perceived consequentiality because the presence of a trained interviewer should signal the importance of the survey. Table 8 reports the mean and standard deviation of survey completion times and the perceived consequentiality of responses across mode-frame pairs. We found no statistically significant difference in response time between the CAPI and mail surveys or between the IP-IM and IP-RN. In line with expectations, IP respondents completed the survey faster than CAPI and mail respondents.
Perceived consequentiality was significantly different across mode-frame pairs; CAPI and IP-RN respondents were more likely than mail respondents to agree or strongly agree that their responses would change how services are provided. However, our hypothesis that CAPI respondents would have the highest perceived consequentiality was not supported; perceived consequentiality was highest for IP-RN respondents.
Stated attribute nonattendance. Respondents not engaged with the CE task may use simple rules and shortcuts (heuristics) to make choices (e.g., ignoring attributes). This attribute nonattendance (ANA) behavior has been measured using debriefing questions. 25 Most CEs aim to elicit respondents' WTP; therefore, attendance to the cost attribute is of particular interest. We asked respondents whether they considered all attributes and, if not, which attributes they did not consider. We compared ANA across mode-frame pairs using a binomial test. No previous studies have compared stated ANA across modes or frames. If ANA is due to low engagement in the task, then we would expect the CAPI to have lower reported ANA because interviewers are trained to engage respondents in the task. Table 9 reports the results of 2 comparisons across mode-frame pairs: the proportion of full attribute attendance and, conditional on not attending to all attributes, the proportion of respondents who ignored the cost attribute. The proportions reporting that they attended to all attributes were statistically significantly different across mode-frame pairs: IPs were more likely to report that they considered all attributes compared with CAPI or mail respondents, respectively. Across mode-frame pairs, there was a significant difference in nonattendance to the cost attribute; about 11% of IP respondents reported not attending to the cost attribute compared with 24% of CAPI and 20% of mail respondents. These findings are contrary to our a priori expectations.
Cost per respondent. We compared data collection costs per respondent for each mode-frame pair. We designed a single master survey using word-processing software. A hard copy of that document was posted to the mail sample (with reminders as specified above). The master survey was taken as the basis for scripting the CAPI and IP surveys. The cost per respondent across the mode-frame pairs was £37.50 for the CAPI survey, £18.20 for the mail survey, £8.20 for the IP-IM survey, and £2.50 for IP-RN survey.

Discussion
We investigated combined survey mode and frame effects in a CE eliciting health care preferences for a health care good: pharmacy services. We compared a commonly used CAPI and mailed survey with the increasingly popular Internet survey. We intentionally confounded survey mode and sample frame 26 ; in the CAPI and mail modes, the sample frame was the UK population, whereas for the IP-IM and IP-RN modes, the sample frames were

Constant
No panel members. This confound is inevitable in countries with only volunteer IPs. We used the sample frames researchers normally use. Our results, therefore, represent the (counterfactual) differences researchers might reasonably obtain across modes and the sample frames commonly associated with those modes.

Response Rates
CAPI had the highest response rate. The mail survey response rate was in line with those reported in other studies. 6 The IPs had the lowest participation rates. These results are consistent with studies that find Internet participation rates are lower than mail survey response rates. 27 The low participation rates for the IPs and the finding that about 15% of CAPI and mail respondents do not have Internet access indicate that IP responses may be prone to unobserved sample selection. The response rate to IP-RN was considerably lower than IP-IM. Our pilot study combined an online survey with a PAF sample frame, mailing an invitation to respondents to complete an Internet survey. We received 3 responses to 1000  mailed letters, highlighting the reality that researchers face when considering survey mode and sample frame.
Response rates/participation rates may reflect how good the IPs and PAF are at keeping information up to date. We used resources routinely available to researchers wishing to conduct large-scale surveys of the general public. In the UK, the Royal Mail states that the PAF is ''constantly updated'' by ''making updates to 3,500 records each day'' (https://www.royalmail.com/business/ services/marketing/data-optimisation/paf). Such a database is recommended for the address-based sampling required for mailed surveys. 28 When asked about keeping IP membership data up to date, Ipsos Mori commented, Considerable efforts are taken to maintain engagement of panellists [sic]. This would include managing the frequency of invitations, the design of surveys, appropriate incentivisation. It is important to offer panellists the ability to take part in on any type of device, including smartphones. A panellist is free to unsubscribe at any time. Panels are also actively managed and within Ipsos we would purge the panel twice a year to remove those that have not responded to survey invitations.
The response rate for the mail survey, although typical of response rates achieved in CEs, 6 may have been improved if our contact with the respondents was more personalized. 1 However, in the UK, there is no suitable postal sample frame that includes names and addresses. Although the electoral register includes names and addresses, it may be used for only a limited set of activities (set by law

Respondent Representativeness
Many mode comparison studies compare respondents' characteristics across modes but not with the characteristics of the population of interest. Comparisons of respondent characteristics across modes may help explain preference differences. However, these do not address the more important questions of how representative the sample is of the population of interest. Those mode comparison studies that compare their sample with the population of interest do so using a narrower set of socioeconomic characteristics than we did. Using common sampling practice, none of our mode-frame pairs resulted in a fully representative sample. The CAPI performed best in that respondents were representative of the UK population for 3 characteristics (gender, education, and Internet access). The mail survey performed worst; it was not representative of the UK population on any of the 9 representativeness characteristics considered.
As in Mulhern et al., 3 we found that quota sampling is no guarantee of representativeness. Our CAPI sample, with a quota applied for age and working status, differed from the UK population in both of these. Our IP samples, with a quota for age, also differed from the UK population. Rowen et al. 4 found that UK IP respondents were in poorer health than the UK population. We found that respondents to all mode-frame pairs were in poorer health than the UK population. Previous studies have reported that respondents' perceptions about the personal relevance of a survey can influence response rates. [29][30][31] This may explain why people in poorer health are more likely to respond to a survey about the use of health services. Future stated preference surveys should consider this source of nonresponse bias.
Our results regarding the lack of representativeness raise questions regarding how quota sampling is conducted. Quotas for the IPs were filled at the start of the surveys. For both CAPI and IPs, the survey companies used age bands for their quotas broader than those we used. We opted for 5-year age bands that mirror the UK census (e.g., 18-19, 20-24, 25-29, 30-34, 35-39, . . . 85+ years), whereas the survey companies used broader age bands (i.e., 18-24, 25-34, 35-54, 55+ years). This is likely to have been a significant factor in our finding that respondents were not representative (by age) of the UK population. By using the narrower census-based age bands, we wanted to demonstrate one of the effects that researchers might encounter when following the standard practice of commercial survey companies. Analysis of the data using the same age bands used in quotas finds no statistically significant difference for the CAPI, but statistically significant differences still exist for mail and both IPs. Survey companies could sample using narrower age bands; however, it would cost significantly more and is not standard practice in the industry.

Elicited Values
Our mWTP estimates revealed statistically significant differences across mode-frame pairs. In general, mWTP was highest in the mail survey and lowest for the IPs. Snowball and Willis 32 found that in-person surveys elicit higher mWTP than self-complete surveys. They suggest this is because in-person respondents are not given enough time to think about their responses to complex CE tasks. Our results do not support this explanation; our mail respondents' completion times were similar to CAPI respondents. Further, IP respondents completed the surveys faster than both CAPI and mail respondents but reported lower mWTP. An alternative explanation for higher mWTP among CAPI respondents is social desirability bias 33 ; respondents may exaggerate their WTP to project a favorable self-image during face-toface interviews. This, however, would not explain why WTP is highest among mail respondents. We did not model observable preference heterogeneity. Keane and Wasi 34 and King et al. 35 compared econometric models of observable (or scale) heterogeneity captured by interactions between respondents' characteristics and the CE attributes with mixed logit models of unobservable heterogeneity. They found that respondents' characteristics explain very little of the variability in preferences found in the mixed logit. The error component logit model we estimated is equivalent to a mixed logit without distributions. Our mean mWTP estimates are unchanged if a mixed logit model with normal distributions for each attribute (except cost) is estimated. Consequently, comparisons of mWTP and tests for differences across the mode-frame pairs are unchanged.

Data Quality
Our results combine objective measures (response rates, respondent representativeness, and theoretical validity of elicited preferences) with subjective or self-reported measures of data quality (response times, perceived consequentiality, and ANA). Our objective measures may be more reliable. For example, self-reported beliefs about perceived consequentiality are likely to be measured with error. 36 Perceived consequentiality may be affected by how well respondents remembered, or how much they believed, the information in the study invitation. Mail respondents were significantly less likely to perceive consequentiality; they were also the only group able to refer back to the information sheet when completing the survey (we thank an anonymous reviewer for noting this).
The reliability of respondents' stated ANA has also been questioned. 37 We found evidence of inconsistency between the different subjective quality measures, supporting this notion. For example, if ANA measures respondents' use of simplifying decision heuristics, we would expect respondents attending to fewer attributes to complete the survey faster. However, although CAPI and mail survey respondents were least likely to attend to all attributes, they took longer to complete the surveys. No previous research has investigated how survey mode or frame affects the stated ANA. It is possible that observed differences in the stated ANA were due to the nature of different modes. For example, social desirability may affect the stated ANA if CAPI respondents want to appear altruistic (and therefore not consider cost). Further, IP members received a nominal reward for participation. It might then be argued that IP respondents care more about money than respondents to other modes who give their time freely (we thank an anonymous reviewer for raising these points). Panel membership is also managed by panel providers, and low-quality respondents are removed from panels. This may deter respondents from reporting ANA. Future stated ANA research should consider how mode or frame may affect responses.

Limitations
Our study has several limitations. Given our aim was to identify expected differences when using different survey modes and the frames commonly associated with them, this limits our ability to isolate the effect of just the modes. Our results represent the differences one might expect to observe between different mode-frame pairs, including the possibility that nonresponse may differ for those pairs. Further, our IP respondents are members of commercially managed access panels that use financial incentives to encourage responses. The use of incentives is standard practice among IP companies and is advocated by experts. 1 The incentives offered for completing the survey were of low value (actual values were not revealed). It is possible that people volunteering for such panels differ from the general population in ways that we have not controlled for (including those responding to nonincentivized surveys such as the CAPI or mail survey in this study). The incentives offered to Internet panelists may introduce a degree of bias. We compared mWTP, however, income levels were not comparable across mode-frame pairs. We did not include quotas for income in the CAPI or IP surveys. Such quotas are not typical in general population surveys.

Conclusion
The CAPI is better than, or equivalent to, all other mode-frame pairs on almost all measures, but it is also the most expensive. The mail is the least representative of the population. Our 2 IPs are similar on most measures and are cheaper than other modes. However, we note the extremely low participation rate for IP-RN. While we find many differences, the importance of each of these differences would depend on the context in which the results are used. Researchers need to be aware of these differences when selecting the best survey mode and sampling frame for their study.

Supplemental Material
Supplementary material for this article is available on the Medical Decision Making Web site at http://journals.sagepub .com/home/mdm.