Exploring the external validity of survey data with triangulation: A case study from the Norwegian Offender Mental Health and Addiction (NorMA) Study

Objectives This paper demonstrates how to investigate the external validity of a study sample by triangulating survey and registry data, using data from the Norwegian Offender Mental Health and Addiction (NorMA) Study as a case. Methods We use survey data from the NorMA study (n = 1495), including the NorMA cohort (n = 733), and data from the Norwegian Prison Registry on all people imprisoned on 1 September 2013 (n = 3386). Triangulation was performed by (1) comparing the NorMA cohort to those lost to follow-up (n = 762), using survey data from the NorMA study. Secondly, we compared the NorMA cohort to the one-day population, using data from the Norwegian Prison Registry. We also stratified the one-day sample by possession of a Norwegian personal identification number (PIN). Results We found differences in birthplace, imprisonment and drug use between the NorMA cohort, lost to follow-up and the one-day population. Twenty-three percent of the one-day population did not have a Norwegian PIN. The NorMA cohort was more similar to those with a Norwegian PIN in the one-day population. Our triangulation demonstrates that 56–62% of the Norwegian prison population had an indication of drug use before imprisonment. Conclusions The NorMA cohort was overall representative of the one-day prison population holding a Norwegian PIN and less representative of prisoners without a Norwegian PIN. Using this method provides tangible inputs on the strengths and limitations of a study sample and can be a feasible method to investigate the external validity of survey data.


Background
External validity rarely receives quantitative analysis, perhaps due to lack of comparison data. 1,2 A frequent threat to external validity is selection bias; bias caused by nonrepresentative sampling, which consequently may hamper the study results. A representative study sample is especially important in descriptive studies that wish to describe a specific population at a point in time. 3 In this article, we will explore how external validity and selection bias in survey data can be assessed with data triangulation with alternative data sources. To illustrate the triangulation method, we use the Norwegian Offender Mental Health and Addiction (NorMA) Study and the Norwegian Prison Registry.
Globally, as well as in Norway, epidemiological research on drug use in the prison population has been based on a wide variety of research designs, sampling strategies and measurements of drug use. [4][5][6] However, cross-sectional surveys, which are more sensitive to selection bias, are most common, 4,7 while national registries and databases provide untapped potential for research on drug use and the prison population. 8,9 The Nordic countries have developed advanced national registries and databases based on personal identification numbers (PINs) assigned to all residents. The unique PIN enables the linkage of information on an individual level between several registries for the purpose of official statistics and research. The advantage of these registries is their almost complete inclusion of the population. Registry data can therefore be used to triangulate research data 10,11 in order to investigate whether a study sample is representative of the target population. One important condition for validating with triangulation is that the study sample and the more complete data are drawn from the same target population. Triangulation can therefore be conducted using data from national surveys, insurance data or administrative records as alternatives to national registries, as long as it is possible to identify the study sample by a PIN.
The NorMA study was conducted in 2013-2014 in Norway and aimed to describe substance use and mental health among people in Norwegian prisons. A total of 1495 inmates responded to the survey, and 733 (49%) provided a Norwegian PIN and informed consent for its use in future research, thus constituting the NorMA cohort. Anonymous participation was also encouraged, and 762 (51%) inmates responded without providing a PIN, thereby constituting the lost to follow-up cohort.
In this study, we triangulated data from the NorMA study and the Norwegian Prison Registry to assess the representativeness and external validity of the NorMA cohort. We identified two selection processes potentially introducing bias into the NorMA cohort. Firstly, selection into the NorMA cohort among respondents in the NorMA study; those who provided their PIN's at baseline and consent to link it in future research. Secondly, selection into the NorMA study from the target population; those who participated in the NorMA study among all prisoners in Norway at the time of data collection.
In this article, we will investigate these two selections with triangulation and discuss the representativeness of the NorMA cohort to the general Norwegian prison population. The specific objectives of the study are the following: 1. To compare the individuals in the NorMA cohort to those lost to follow-up using baseline survey data from the NorMA study, and 2. To compare the NorMA cohort to the general prison population using registry data from the Norwegian Prison Registry.

Setting
Norway has one of the lowest prison population rates in the world, with an average of 3850 individuals imprisoned in 2016, equal to a prison population rate of 73 per 100, 000 of national population, compared to a global average of 145 per 100, 000 and 655 per 100, 000 in the United States. 12,13 As 85% of prisoners serve sentences of less than 1 year and 60% serve less than 3 months, the yearly turn-over is high. 14 Women constitute a minority in Norwegian prisons, with an annual proportion of approximately six percent. Almost two-thirds of prisons are high security prisons. All Norwegian prisons are publicly funded, and all inmates have access to universal health care coverage, like all other individuals in Norway. 15 The criminal justice system is characterized by having a rehabilitative focus on incarceration, and some prisons have separate drug treatment units to achieve the goals of rehabilitation. 16 Patients and/or the public were not involved in the design, conduct, reporting or dissemination plans of this research.

Study design and data sources
Triangulation requires two or more sources of data. We triangulated the NorMA cohort with data from the NorMA study (Objective 1) and registry data from the Norwegian Prison Registry (Objective 2), as illustrated in Figure 1.

The NorMA study
The NorMA study includes a cross-sectional study sample with baseline survey data. The survey contains a 116-item questionnaire, including questions on mental health and substance use, as well as demographics and other background information. The questionnaire took 30-60 min to complete and was available in five languages, including Norwegian, English, Russian, French and German. Data was collected by study investigators who visited 57 prison units (of 63 eligible) during the time of data collection from June 2013 until July 2014. Six prisons were not visited due to limited staff capacity and geographical inconvenience. Those prison units had a total capacity of 179 inmates and did not differ from the prisons included in the data collection. The questionnaires were administered by the study investigators and distributed to the prisoners on the day of the visit. The respondents filled out the questionnaire themselves. They could complete the questionnaire in a common room with others or alone in their cell, depending on their prison situation and preferences. All inmates in the prison on the day of data collection were considered eligible to participate in the study. The questionnaire included a consent form that explained the purpose of the study and sought permission for linking the survey responses with registry data. In addition to a signature field, an 11-digit field was provided to capture the PINs. For a more thorough description of the methodology and study design, please see Bukten et al. 16 The Norwegian Prison Registry The Norwegian Prison Registry was established in 1992 to serve a range of administrative and statistical purposes. The registry includes data from all Norwegian prisons on sentences, imprisonment status, security level, participation in programmes and other variables on the activities related to imprisonment. The registry is administered by the Correctional Service of Norway, who oversees execution of all remand and prison sentences in Norway. 16 From the Norwegian Prison Registry, we retrieved data on all prisoners imprisoned on 1 day within the data collection period of the NorMA study. To choose a comparison date, we drew 10 separate days within the data collection period, excluding public holidays. The 10 days had an average prison population of 3995 individuals with deviation of less than ±2.5% from this average. For the methodological purpose of this study, we randomly selected the 1 st of September 2013 among the 10 days, on which 3908 individuals were imprisoned.
Among the 3908, 522 (13%) were also included in the NorMA cohort and therefore excluded from the one-day population. We retrieved data on all imprisonments from 1 January 1992 until 31 December 2019 for all persons in both the NorMA cohort and the one-day population. All the data from the Norwegian Prison Registry, including information referred to as 'lifetime events', therefore refers to imprisonments within the period from 1992 till 2019.

Measures
The two data sources included different variables and we therefore have different sets of measures in objective 1 and objective 2 of our analysis. The measures used in each objective and how they are defined are described in Table  1. The analysis of each data source and their measurements was performed separately, and we did not compare or test somewhat similar measures from both datasets against each other. We chose not to perform this comparison and verification of the survey data, to protect the integrity of the study participants, in accordance with our ethical approval from REK.

Analysis
We analyzed representativeness of the NorMA cohort according to our two objectives: Objective 1: The NorMA cohort compared to lost to follow-up.
For objective 1, we compared the individuals in the NorMA cohort to the lost to follow-up group using baseline survey data from the NorMA study, and focusing on measures related to demographics, imprisonment and drug use.
Objective 2: NorMA cohort compared to the general prison population.
For objective 2, we compared the NorMA cohort to the one-day population (n = 3386), retrieved from the Norwegian Prison Registry. Objective 2 only included measures retrieved from the Norwegian Prison Registry on age, sex and imprisonment. We analyzed the one-day population in total and stratified by possession of a Norwegian PIN. An overview of the objectives, data and study samples is summarized in Table 2.
Medians and the interquartile ranges (Q1-Q3) were reported for skewed data, such as age at first imprisonment, number of imprisonments and days imprisoned. Test of difference between the NorMA cohort, the lost to follow-up group and the one-day population was done using chisquared tests, T-test of means on normally distributed numerical variables and the Mann-Whitney-test for ordinal and categorical variables. We did all statistical analyses in SPSS Statistics version 26.

Missing data
The percentage of missing data ranged from 0 to 28.5% and was more frequent among the self-reported variables used to answer objective 1. Detailed information is described in Table 3. The registry data we used to answer objective 2 had no or very little missing data on all variables (not reported in table).
The standardized measures AUDIT, DUDIT and HSCL-10 were imputed according to standard methods. Missing values were replaced with a mean of the completed answers for individuals with at least five items answered for DUDIT and AUDIT and seven items answered for HSCL-10. 21,22 We performed multiple imputation by chained equations (MICE), with 10 imputations, on all missing values for the remaining variables. We tested the imputations with Chi 2 -test and T-test, but did not influence the statistical significance in difference between the groups. Due to the descriptive scope of this paper, we therefore chose to report only on data analyzed using complete case analysis.
We used the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) cohort checklist when writing our report. 23

Results
Objective 1: NorMA cohort compared to persons lost to follow-up The NorMA cohort had more previous imprisonments compared to the persons lost to follow-up. Sixty-eight percent of the NorMA cohort had a previous sentence, and their median number of imprisonments was three (Table 3).
A larger proportion of the NorMA cohort reported involvement in all types of crimes compared to the lost to follow-up group (except for 'other' with 15% in both groups).
The most common type of conviction in both groups was drug-related crime. Fifty-four percent of the NorMA cohort had this type of crime as one of their convictions or charges, while this was the case for 45% of the lost to follow-up group.
The NorMA cohort reported more drug use and mental distress, compared to the lost to follow-up. This included higher lifetime drug use (76 vs 62%), daily drug use (56 vs 38%) and as measured with a DUDIT score above cutoff (65 vs 48%) compared to the lost to follow-up group (See Table 3). Objective 2: NorMA cohort compared to the general prison population The NorMA cohort was younger when they entered prison the first time, compared to the one-day population (24 vs 27 years) ( Table 4). Both groups had more than 90% male participants. Except for days per imprisonment, the NorMA cohort had more imprisonments (median four vs two) and had served more days in prison (902 vs 792 days) compared to the one-day population (Table 4). Three in four individuals in the NorMA cohort had more than one lifetime imprisonment, compared to approximately half of the individuals in the one-day population. Sixty-two percent of the NorMA cohort had been convicted of 'use and possession' at least once, compared to 54% in the one-day population.
In the one-day population (n = 3386), 907 individuals (27%) did not have a PIN, equivalent to 23% of the total prison population on this day (including the NorMA cohort, who all had PIN). When stratifying the one-day population by PIN/no PIN, those without Norwegian PIN where older and had fewer  imprisonments but with more days per imprisonment compared to the NorMA cohort. Their total days of life-time imprisonments were less than the NorMA cohort's (519 compared to 902 days) and fewer had a conviction for use and possession (46%).

Discussion
Studying the representativeness of a study sample is key in epidemiological health research. This includes considering potential sources of selection bias, related to the condition of the study setting and research design. Assessing this can require new perspectives, including alternative sources of data. Rothman has argued that representativeness of a study sample does not in itself improve external validity; rather, it is the knowledge of specific conditions in the study setting and an understanding of causal mechanism that makes for proper generalization. 24 We argue that triangulation is a feasible method to assess these specific conditions and mechanisms of potential selection, and improves our knowledge of our sample's external validity.
The NorMA cohort was different from the lost to followup group and the one-day population on most variables. However, when we stratified the one-day population on PIN, the one-day population with PIN and the NorMA cohort were more similar. This indicates that some of the differences between the NorMA cohort and the lost to follow-up group could be explained by the presence of participants without PIN in the lost to follow-up group.
The NorMA study had 51% lost to follow-up: Those who participated in the survey without providing a PIN and/or informed consent. We do not have information on why they did not provide a PIN. We do not know, for example, whether it was conscious choice, or if they simply could not remember their PIN, or if they did not have a Norwegian PIN. However, since we have the information provided in the NorMA survey, we are able to describe the lost to follow-up group with a broad range of characteristics. When comparing the NorMA cohort to the lost to follow-up group, we saw a difference in drug use and birth country. More than half of the NorMA cohort had used drugs in the previous 6 months before imprisonment, compared to 38% in the lost to follow-up. The NorMA cohort was mainly born in Norway, while a larger proportion of the lost to follow-up were born elsewhere (84% within NorMA cohort vs. 55% among lost to follow-up).
All Norwegian residents are assigned a PIN. As having a PIN was a criterion for participation in the NorMA cohort, immigrants without a visitor's permit and tourists were not eligible for follow-up in Norwegian registries. The NorMA cohort is vulnerable to selection bias based on country of birth and citizenship, and it was therefore important to stratify the one-day population by PIN possession. One in four of the one-day population from the general prison population did not have a PIN, and this group was different from those with a PIN. The one-day population with a PIN was similar to the NorMA cohort when looking at sex, re-offending and total days in prison. The tendency was the same for all variables. The stratification thus indicated that the NorMA cohort is largely representative of the general prison population in possession of a Norwegian PIN, but not to those without.

Drug use
Fifty-six percent of the NorMA cohort reported daily drug use before imprisonment. From the registry data, we have information on convictions related to 'use and possession', which we use as a proxy for drug use prior to imprisonment. When looking at this proxy, 62% of the NorMA cohort and 57% of the general prison population with a Norwegian PIN had this conviction. Our triangulation can therefore demonstrate that 56-62% of the individuals in the Norwegian prison population had an indication of drug use prior to their imprisonment. However, the analysis also showed that this estimate will probably only be valid for the prison population holding a Norwegian PIN.

Limitations and strengths of the study
Both survey and registry data introduce some general strengths and limitations to our study. The survey data we used from the NorMA study included a very broad set of questions and provided rich baseline data. However, some variables had higher degrees of missing, which was more prevalent in the lost to follow-up group, with most missing values on the variables related to mental health status. On the other hand, the data we retrieved from the Norwegian Prison Registry had complete follow-up, but a limited and pre-defined set of variables chosen for administrational purposes. By combining the two, we gained valuable insight into the representativeness and external validity of the NorMA cohort, which is important for future research based on this cohort. Another strength of the Norwegian Prison Registry is the available data on those who did not have a Norwegian PIN and therefore are lost to followup in other registries. We do not know how many in the lost to follow-up group did not have a PIN, nor why those who had a PIN chose not to provide theirs. Though we cannot follow these individuals in future linkages with registries, the information provided to us in the Prison Registry data provides us with valuable insight into their characteristics, in comparison to the prison population with a PIN.

Implications
Studying selection bias is important in all sample-based research, because the selection itself, and the underlying mechanisms of selection, can affect the research results. By triangulating different data source, as we have demonstrated in this article with survey data and registry data, we can investigate these selections and its effect on our results.
If PINs (or another linkable identification code) and informed consent are obtained, registry data can be used to investigate the representativeness of a study sample. For the triangulation method to be applicable, one should be able to identify the study sample within the alternative data source, which requires a linkable identification code. If possible, pay particular attention to groups that do not hold linkable identification codes, such as foreign citizens, and consider how they would appear in the study sample and how they should be treated in the analysis.
In our case, we linked our cohort to Norwegian registry data by using the Norwegian PIN. However, other potential data sources could be national surveys, insurance data or institutional records from institutions such as hospitals, education systems or prisons. Even with a limited number of variables available, for example, on demography or imprisonment, registry data can be valuable when investigating representativeness and evaluating the external validity of data and results. Including this information can therefore be relevant, even if linkage with registry data is not the main purpose of the study. When designing surveys with the intention to collect PIN, remember to emphasize to potential participants that anonymous participation is also encouraged and valuable to the survey.

Conclusion
Though all studies are at risk of selection bias, the characteristics and implications of these biases are often difficult to assess and measure. As we have shown in this study, triangulation of data sources can shed light on the specific conditions between the study sample and target population. With this method, we can improve our understanding of the mechanisms affecting selection bias and clarify the external validity of the study results.
Ministry of Justice and Public Security, and by the Directorate of the Norwegian Correctional Services, the national prison authorities in Norway. All methods were carried out in accordance with relevant guidelines and regulations. The PINs were provided by the participants following written informed consent. Furthermore, the use of Prison Registry data for comparison, including the identification of the NorMA cohort using PINs, was approved by REK in a supplementary approval in 2020. A precondition of the ethical approval from REK was that we refrain from testing internal validity of the NorMA study, as this could question the credibility and integrity of the participants in the study.

Consent for publication
Not applicable

Availability of data and materials
The datasets generated and/or analyzed during the current study are highly sensitive data and not publicly available due to the limitations of the ethical approval from the Norwegian Committee of Research Ethics. The data can be made available from the corresponding author on reasonable request.