Development and validation of a population-based risk algorithm for premature mortality in Canada: the Premature Mortality Population Risk Tool (PreMPoRT)

Data sources

A retrospective cohort study was used to develop a prediction model for premature mortality using population-based survey data from the Canadian Community Health Survey (CCHS). The CCHS is a cross-sectional survey covering 98% of the Canadian population 12 years and older that collects self-reported data on personal health status, healthcare utilisation and health determinants.24 Populations excluded from the CCHS sampling frame include individuals living in Aboriginal settlements, Canadian Force Bases and some remote regions. A detailed description of the CCHS survey methodology is available elsewhere.24 All respondents were linked to the Canadian Vital Statistics Database (CVSD) to ascertain deaths during the follow-up period. The data were held at the Statistics Canada Research Data Centre. Due to Research Data Centre vetting requirements, all study output is rounded deterministically to the nearest 10.

Participants

Respondents who consented to link their responses to administrative data from the first six cycles 1.1 (2000/2001), 2.1 (2003/2004), 3.1 (2005/2006), 2007/2008, 2009/2010 and 2011/2012 of the CCHS were used to create the study cohort. Respondents were excluded if they were pregnant, resided in the territories (ie, Yukon, Northwest Territories or Nunavut) or were under 18 years or older than 74 years at the date of CCHS interview. For further details on our cohort creation, please see online supplemental figure S1.

Outcome: premature mortality

Respondents were followed longitudinally for the incidence of premature mortality (binary yes/no variable), defined as deaths from all causes between the ages of 18 and 74 as recorded in the CVSD. This definition is based on the cut-off adopted by the Canadian Institute of Health Information,25 which is consistent with the definition used in reporting premature mortality among other industrialised nations.26–29

Predictors

As stated in our protocol, 38 candidate predictors were identified from the survey data in accordance with established associations with premature mortality,20 30–34 subject matter knowledge, user input, our team’s experience with the development and validation of population-level risk algorithms19 35–41 and predictor availability across survey cycles. Predictors included sociodemographic characteristics, self-perceived measures, health behaviours, chronic conditions and area-based measures.

Missing data

Multiple imputation methods were used to impute missing data, specifically, Fully conditional specification (FCS) was used to develop five imputed datasets which previous research has established as sufficient.42 43 Total missingness was low and ranged in the six combined cycles from <1% to 10%.22 All imputation was done using the multivariate imputation by chained equations (mice) algorithm in R.42 First, each cycle was separated into sex-stratified groups and imputation was run on each sex-stratified cycle separately to avoid potential between cycle variations and differences between males and females. A three-step approach was used to impute the missing values. FCS imputation was unable to converge for several chronic conditions that had a low prevalence and low missing data (ie, less than 1% missing). As such, the first step was to assume that a respondent did not have said chronic condition if any of the chronic conditions had less than 1% missing across all sex-stratified cycles. These chronic conditions included Alzheimer’s disease, arthritis, asthma, back problems, bowel disease, cancer, chronic obstructive pulmonary disease (COPD)/emphysema, diabetes, high blood pressure, heart disease, intestinal ulcers, migraines, stroke and urinary incontinence. Afterwards, five imputed datasets were generated using FCS on all sociodemographics, health behaviours and chronic conditions except for anxiety and mood disorders. Anxiety and mood disorders were not asked in the CCHS cycle 1.1, and as such, a third and final step needed to be implemented. Each imputed dataset was divided by sex and by cohort group (ie, derivation cohort and validation cohort). Once separated, FCS was run to impute the missing values for anxiety and mood disorders.

A sensitivity analysis was conducted to examine the variation in model performance measures across the imputed datasets, which would help inform whether we averaged the measures or reported performance measures on the first imputed dataset only. After finding negligible differences in the model performance measures (online supplemental table S1), we opted to present model development and validation measures on the first imputed dataset.

Study design

Models were developed and validated in the Canadian adult provincial population.44 45 There were five steps involved in the development and validation: (1) model derivation in the first three CCHS cycles ((1.1 (2000/2001), 2.1 (2003/2004) and 3.1 (2005/2006)). Two models, one for females and one for males, were developed due to the important biological sex differences in premature mortality34 46 (further outlined in the ‘Model specification’ section). This was followed by (2) the internal validation using a split sample approach where the 70% development model was applied to the remaining 30%. The model was then (3) externally validated in the last three CCHS cycles (2007/2008, 2009/2010 and 2011/2012). Next, (4) the derivation and validation data were combined to estimate the final application of the PreMPoRT model (ie, the final Canadian provincial cohort) using the same model specification as in the original model derivation. Lastly, (5) the predictive performance of the final primary model was assessed among more than 20 programme and policy-relevant subgroups with <20% difference between observed and predicted, representing good calibration.15 19 In the main results, we report observed and predicted risk for three of these subgroups (ie, education, income and immigration status).

Model specification

We began with the prespecified forms of predictors as described in the study protocol (online supplemental table S2). Candidate predictors that did not improve predictive performance were removed. Next, the primary model was selected from the minimal model, using the sequential addition of predictors until the performance resembled that of the full model. Finally, we verified the sequential addition of predictors in each of the three models using the least absolute shrinkage and selection operator as described in our study protocol.22 Following sequential predictor selection, alternative specifications were examined. For example, both BMI and age were respecified from continuous with restricted cubic splines to categories for both sexes. The focus of the results is on the primary model, however, results for the minimal and full models are in online supplemental file 1.

Statistical analysis

The predictive performance was based on measures of overall predictive accuracy (ie, Nagelkerke R², Brier score), discrimination (ie, Harrell’s c-statistic, time-specific discrimination slope) and calibration (ie, time-specific calibration curve, calibration intercept and calibration slope). The definitions of each measure are found in online supplemental table S3. From January 2000 to December 2017, respondents from the CCHS were followed until premature mortality, 5 years of follow-up or study end date. The models were estimated using a Weibull accelerated failure time model with the proportional hazard’s assumption assessed according to the scaled Schoenfeld residuals. The CCHS survey weights were applied to ensure all estimates were representative of the population.47 48 Data cleaning and predictor coding were conducted in SAS V.9.4, with all model development and validation executed in R V.3.6.2.

Patient and public involvement

This study and the published analytical protocol were developed in consultation with decision-makers at three local public health departments in urban and rural Ontario to ensure that the design of the prediction model is optimised for applied public health applications. These decision-makers were engaged at the beginning of this work to provide feedback on the published study protocol which detailed the design, analysis and reporting of the present study.

Study population characteristics

Baseline characteristics of the entire Canadian female provincial cohort (n=267 460), male provincial cohort (n=233 410), and external development and validation cohorts (which are a subset of Canadian provincial cohort) are detailed in table 1. The average number of premature deaths over the study period was 1.40% for females and 2.05% for males. Among the Canadian provincial cohort, female respondents were 44.18 years and 43.43 years for males at survey date. Most individuals had completed post-secondary education, and the most common marital status was domestic partnership (married/common law). The most common BMI category for females was 18.5 to <25.0 kg/m², whereas for males, it was 25.0 to <30.0 kg/m². Most participants rated their self-perceived general health as good or very good. Physical activity levels were lower in females than in males, with 51.08% of females classified as inactive compared with 46.41% of males. Most participants were white, Canadian-born and lived in urban areas. Differences in the population characteristics were greatest in the external development and validation cohort compared with the split set development and validation cohort (online supplemental table S4). For example, there were larger proportions in the lowest income quintile, larger proportions of people with post-secondary education, more never-smokers and greater representation of visible minorities.

Model development

The full female and male model consisted of 16 predictors, the primary female model consisted of 12 predictors and the male model consisted of 13 predictors, the minimal female model consisted of 8 predictors and the male model contained 10 predictors. The model coefficients and HRs for the primary model are outlined in table 2. Age, self-rated health, smoking, COPD/emphysema, heart disease, diabetes, cancer and stroke were the predictors with the largest contributions to model performance and appeared in all versions (ie, minimal, primary and full) of the sex-specific models. We assessed the proportionality assumption (via scaled Schoenfeld residuals) inherent in Weibull accelerated failure time models and found no violations. The model coefficients and HRs for the minimal and full models are found in online supplemental table S5 (females) and online supplemental table S6 (males).

Model external development

The predictive performance of the primary female and male models in different development and validation settings are described in table 3. The primary development cohort was the external development (CCHS cycles 2000–2006). In the primary development cohort, the female model achieved better predictive performance than the male model. The female external development model achieved a Brier score: 0.007, discrimination slope: 0.096 and c-statistic: 0.895. The male external development model also performed well (Brier score: 0.010) and had good discriminative ability (discrimination slope: 0.105 and c-statistic: 0.853).

Model split sample and external validation

The female model in the external validation cohort produced the best overall predictive performance (Brier score: 0.006) and discrimination (discrimination slope: 0.109) (table 3). The female model also performed well in the split set validation (Brier score: 0.007 and discrimination slope: 0.094). The male model performed slightly better in the split set validation in terms of predictive performance (Brier score: 0.010) and discrimination (discrimination slope: 0.096) in comparison to the external validation (Brier score: 0.009 and discrimination slope: 0.108). The c-statistic for the female split set validation was 0.852 and the male was 0.846 (table 3). The female and male split set calibration curves (figures 1A,C) exhibit similar shapes, with strong adherence to the 45° line for low levels of predicted risk and a tendency to overpredict risk among the small proportion of high-risk individuals. The c-statistic for the female and males models in the external validation cohorts was 0.856 and 0.846, respectively (table 3). The calibration curves for the female and male models in external validation (figure 1B,D) diverge from the 45° line at a lower predicted risk than in the split set validation. However, given that <2% of the overall cohort has a predicted risk above 20%, there was minimal impact of the overprediction on overall model calibration. Calibration curves and predictive performance measures for the minimal and full model in the split set and external validation are found in online supplemental figures S3 and S4 and table S7, respectively.

Request permission

Request permission

Subgroup calibration

The model was further evaluated on more than 20 important policy or programme-relevant population subgroups. In figure 2, we report the observed incidence of premature mortality compared with average model prediction among females and males for three of the 20 subgroups we assessed, including education, income and immigration status. Overall, calibration across subgroups was strong. Among both sexes, we observed minor underprediction among those with less secondary school graduation and post-secondary education. Among those with less than secondary school graduation, there is slight overprediction for males and underprediction for females. With the exception of income quintile 2 for males, there is slight underprediction for both sexes across income quintiles. Among both sexes, there was minor overprediction in immigrants and visible minorities and slight underprediction for people born in Canada.

Principal findings

We developed and externally validated a prediction model for population health applications to determine individual risk of premature mortality within 5 years. Model predictors from routinely collected survey data included sociodemographic characteristics, self-perceived measures, health behaviours and chronic conditions. The most influential predictors, with the largest contributions to model performance, included age, self-rated health, smoking, COPD/emphysema, heart disease, diabetes, cancer and stroke. The final model showed reproducibility and transportability through different validation settings (ie, internal and external validation). Further, PreMPoRT demonstrated robust discrimination and calibration across different subpopulations. Thus, we anticipate that this tool will produce valid and accurate predictions of premature mortality when applied to support interventions and health policies aimed at reducing premature mortality in the population.

Strengths and limitations

One of the main strengths of this study is the ability to validate a robust prediction model, based on routinely collected national population-based survey data that is representative of the national population. This dramatically increases the ability for this model to be used widely and regularly, compared with many existing models that rely on the use of highly restricted data. Due to the use of a national mortality registry, we were able to comprehensively capture all premature deaths over the study period. In addition, to reduce the possibility of overfitting and to improve transparency in the development of PreMPoRT we prespecified our analytical plan in our study protocol.22 Beyond use within Canada’s population and public health system, countries internationally are routinely employing population health surveys with increasing standardisation.49 Given that PreMPoRT was developed using data from the Canadian population, findings are likely generalisable to other high-income nations with similar population profiles. However, it is important to consider other contextual factors, including variations in healthcare systems, social determinants of health and health status, which may influence the performance and applicability of the tool in different countries. We recommend users update the baseline hazard function before employing this tool in other countries. Lastly, this study is enhanced by the adherence to best practices and reported in line with TRIPOD guidelines.23

This prognostic study should be interpreted considering a few limitations. Although the CCHS is representative of 98% of the population, due to the nature of the sampling methodology the data does not include people living in correctional facilities, long-term care and retirement homes, and those living in Aboriginal settlements.24 Although these groups represent <3% of the CCHS target population, it is important given that these populations have different premature mortality risk than the general population.50 51 Additionally, baseline risk factors for the development and validation of PreMPoRT were collected through the CCHS, which is a cross-sectional, self-reported survey. As a result, there is potential for misclassification (systematic and non-directional). Finally, miscalibration was present in <2% of the population with the highest predicted risk. In these populations, public health decision-makers may interpret the precise probabilities with caution but should still treat these individuals as higher risk than the general population. Additionally, this minor overprediction may be corrected with recalibration techniques.

Comparison with previous literature

To the best of our knowledge, there are no systematic reviews that capture development and validation studies of prediction models for premature mortality. There is one study that describes the development and validation of a prediction model for premature mortality by Weng et al, that compared the use of traditional survival methods to machine learning approaches.21 Predictors were from the UK Biobank and included clinical data (eg, demographic, biometric and lifestyle factors).21 The authors used a multivariable Cox model that achieved an AUC of 0.751, with machine learning methods resulting in an improvement in comparison to survival methods (random forest AUC of 0.783; deep learning AUC of 0.790).21 The authors observed overprediction with survival models, which we also observed in the present study. The authors did not examine subgroup calibration, so it is unclear how their model performs among different population groups (eg, by income or visible minority status), which is important in understanding fairness in prediction models. Our study was reported in line with the TRIPOD guidelines and demonstrated excellent discrimination through external validation with a c-statistic of 0.856 for the female model and 0.846 for the male model. We also examined subgroup calibration, which shows robust performance across different population characteristics. Using only self-reported survey data, PreMPoRT achieved the strongest discrimination and calibration of existing prediction models for premature mortality to date.

Implications for policy and practice

Premature mortality is an important health outcome that is frequently reported on through monitoring and surveillance initiatives to determine overall population health and to assess health system functioning.1 The use of prediction models in applied public health settings is becoming more commonplace.15 Prediction models may be used to identify individuals at high risk and are an important evidence-based planning tool to support targeted public health interventions.15 This prediction model provides public health decision-makers with a tool to estimate individual risk of premature mortality using widely available population-based survey data.

Given that most premature deaths are preventable through targeted policy, intervention or treatment, this model may be used to deliver targeted or population-wide interventions to reduce premature mortality. Understanding the future distribution of risk and incidence of premature morality is important information for individuals working in both public health and healthcare. Collectively, those working in our health system play an important role in the prevention and management of chronic diseases and overall patient care, which impacts premature mortality rates. Interventions for those at high risk of premature mortality may span across the health system and could include navigation support for mental health and substance use services or health promotion strategies to encourage physical activity or smoking cessation.52 53 Population-level public health interventions may include educational strategies to promote vaccine confidence and uptake or adaptation to the built environment to prevent pedestrian and cyclist injuries.54 55 At present, several other prediction models developed by our team are in use across local public health departments.35–41 Decision-makers report using these prediction models to understand the future risk of a specific health outcome and the distribution of risk within their community. The output of risk prediction models like the one described here is frequently used to support evidence-based planning and building a case for policy change or funding of public health programmes.