MPM Calibration & Analysis

About



This application is part of the online supplement for the article - J. Uthoff, N. Koehn, J. Larson, S.K.N. Dilger, E. Hammond, A. Schwartz, B. Mullan, R. Sanchez, R.M. Hoffman, J.C. Sieren, ''Post-imaging Mathematical Prediction Models: Are They Clinically Relevant?'' published in European Radiology. For more information about post-imaging mathematical prediction models (MPMs) and the methods used here, please see the article PMID: XXXXX




Along the top of this application, there are four tabs:


(1) About TAB

Current tab, overview of application and additional information about development.


(2) Templates and User Guide TAB

Here you can access the user guide and templates.


(3) Local Dataset TAB

Here you can load your own local cohort for analysis in concordance with the methods described in the publication.


(4) Sample Dataset TAB

Here you can try out the app and explore the environment before you assemble your local dataset.

The sample subjects used are not based on real subject data and should not be used to assess MPM performance or recommend thresholds for real-world cases.

Citing this work



Article

This application is included as supplemental material for the article - J. Uthoff, N. Koehn, J. Larson, S.K.N. Dilger, E. Hammond, A. Schwartz, B. Mullan, R. Sanchez, R.M. Hoffman, J.C. Sieren, ''Post-imaging Mathematical Prediction Models: Are They Clinically Relevant?'' published in European Radiology. PMID: XXXXXX

The authors would like to acknowledge Mark Escher and Patrick Thalken for their assistance in application deployment.


Source code

Following publication, the source code for this application will be available on GitHub (http://XXXX) under Creative Commons Attribution-ShareAlike (CC BY-SA) License. This license lets you remix, tweak, and build upon this work provided you cite the origin code and license new creations under identical terms. This license is similar to copyleft free and open source software licenses.


About Application Development

This application was developed using the R programming language version 3.5.1 (2018-07-02) -- 'Feather Spray'((C) 2018 The R Project for Statistical Computing).

R is free software environment for statistical computing and graphics. It complies and runs on a wide variety of UNIX platforms, Windows, and MacOS. It is available under the GNU General Public License. The authors of the associated publication and developers of this application would like to thank the R Development Core Team and all CRAN contributors for their work to make statistical and graphical libraries and tools available to all.

R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.

R libraries: 'shiny', 'pROC','PRROC', and 'dplyr'.

Winston Chang, Joe Cheng, JJ Allaire, Yihui Xie and Jonathan McPherson (2018). shiny: Web Application Framework for R. R package version 1.1.0. https://CRAN.R-project.org/package=shiny

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frederique Lisacek, Jean-Charles Sanchez and Markus Muller (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, p. 77. DOI: 10.1186/1471-2105-12-77 <http://www.biomedcentral.com/1471-2105/12/77/>

Jan Grau, Ivo Grosse, and Jens Keilwagen (2015). PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics (31) 15, pp. 2595-2597.R package version 1.3.1.

Hadley Wickham, Romain Francois, Lionel Henry and Kirill Muller (2018). dplyr: A Grammar of Data Manipulation. R package version 0.7.6. https://CRAN.R-project.org/package=dplyr

MPM Calibration & Analysis

Templates and User Guide



Here you can access the user guide and templates.


This application is part of the online supplement for the article - J. Uthoff, N. Koehn, J. Larson, S.K.N. Dilger, E. Hammond, A. Schwartz, B. Mullan, R. Sanchez, R.M. Hoffman, J.C. Sieren, ''Post-imaging Mathematical Prediction Models: Are They Clinically Relevant?'' published in European Radiology. For more information about post-imaging mathematical prediction models (MPMs) and the methods used here, please see the article PMID: XXXXX

Why download the file template?

It is necessary that your local data is formatted in a certain manner, specifically:

(1) - how variables are 'coded' (ex: age is coded in years, not months)

(2) - how the columns in the file should be named (ex: age column is named as ''Age'', not ''HowOld'')


Failure to correctly set up the local dataset file will result in either inaccurate assessments or an application failure.

For your convenience, we have provided a user guide and a template for your data file set-up.


User Guide

The user guide indicates (1) how variables should be coded and (2) how columns should be named. It also provides an overview of which variables/columns are used in each of the four MPMs. If you wish to only receive results from a single MPM, please feel free to use the templates provided on the 'Local Single MPM File Templates' tab above.

Save user guide as ''MPM_UserGuide.pdf'' to your computer's Downloads folder

Template File

The template file is a comma separated file or '.csv' which can open in Excel or a text editor. It includes the appropriate column names and a single default subject. To use the template, simply add local subjects as a new row per the directions in the user guide.

Save template as ''MPM_localData_template.csv'' to your computer's Downloads folder

Use these templates if you wish to examine a single MPM.

These files contain the column names for the variables used in each individual MPM. When they are uploaded to this application, any unused columns will be automatically populated with the default values. For information on which variables are used in the models see the user guide or the publications.
Failure to correctly set up the local dataset file will result in either inaccurate assessments or an application failure.

(MC) Mayo Clinic Model (PMID: 9129544)

Save MC MPM template as ''MPM_MC_template.csv'' to your computer's Downloads folder

(VA) Veteran's Affairs Model (PMID: 17296637)

Save VA MPM template as ''MPM_VA_template.csv'' to your computer's Downloads folder

(BU) Brock University Model (PMID: 24004118)

Save BU MPM template as ''MPM_BU_template.csv'' to your computer's Downloads folder

(PU) Peking University Model (PMID: 22297626)

Save PU MPM template as ''MPM_PU_template.csv'' to your computer's Downloads folder

MPM Calibration & Analysis

Local Dataset



Here you can load your own local cohort for analysis in concordance with the methods described in the publication.


We recommend a dataset size of at least 100 subjects.

As with all mathematical models, the more independent and complete data points these MPMs are provided the more accurate the assessment can be expected to be. All MPMs included in this analysis were developed using cohorts greater than 200 subjects.


Care should be taken to include representative proportions of malignant and benign cases as seen in your local population.

Deviation from the proportion could negatively impact the reliability of positive and negative predictive values. As stated in the publication, if your dataset contains a class imbalance (i.e. more benign cases than malignant cases), the use of the precision-recall (PR) curve is superior to AUC-ROC. AUC-ROC is best suited for cases with little-to-no imbalance in classifications.



This application is part of the online supplement for the article - J. Uthoff, N. Koehn, J. Larson, S.K.N. Dilger, E. Hammond, A. Schwartz, B. Mullan, R. Sanchez, R.M. Hoffman, J.C. Sieren, ''Post-imaging Mathematical Prediction Models: Are They Clinically Relevant?'' published in European Radiology. For more information about post-imaging mathematical prediction models (MPMs) and the methods used here, please see the article PMID: XXXXX

Provide .CSV file of local dataset.



Table 1: List of variables used in the MPM risk prediction calculation and performance comparison. Indications of which variables are used for each MPM are made with 'X' (e.g. all MPMs use Age as a variable). Your local dataset file should have the correct Column Names and be coded based on the Acceptable Values. Any omitted columns will be automatically populated with the Default value.


Visual inspection of data used in analysis.



Assess MPM Performance.

The table shows MPM performance using AUC-ROC and Precision-Recall. It also includes the Youden's J Statistic Threshold and associated sensitivity and specificity using the Youden threshold. Table 2 of the User Guide provides description of the performance measures.



Table 2: List of performance statistics in the application with calculation procedures and example meanings of values.


Visualize the risk predictions and MPM-recommended thresholds.


MPM Prediction Visualization Histograms

Similar to Figure 1 of Uthoff et al. (PMID: XXXXX)- histograms of MPM predictions split based on true nodule classification. Solid lines indicate MPM-associated thresholds. Note vertical axis autoscales to optimal for visualizatoin of MPM.


Explore the effect of threshold on sensitivity and specificity.


Use the slider to adjust the threshold and see the effect on sensitivity and specificity. Note vertical axis autoscales to optimal for visualizatoin of MPM.

Use the slider to adjust the threshold based on the desired level of sensitivity (True positive rate).

Use the slider to adjust the threshold based on the desired level of specificity (True negative rate).


Save the results from your local dataset to your computer's Downloads folder.



Save the MPM performance Youden measures

File will save as ''MPM_youdenSummary_localDataset.csv'' to your computer's Downloads folder

Youden Summary Statistics


Save custom MPM performance measures

File will save as ''MPM_customSummary_localDataset.csv'' to your computer's Downloads folder

Custom Summary Statistics


Save the MPM subject-specific risk predictions

File will save as ''MPM_predictions_localDataset.csv'' to your computer's Downloads folder

All four MPM risk predictions

MPM Calibration & Analysis

Sample Dataset Playground



Here you can familiarize yourself with the capabilities and environment of this App before you use your local dataset.


The sample subjects shown here are not based on real subject data and should not be used to assess MPM performance or to recommend thresholds for real-world cases.



This application is part of the online supplement for the article - J. Uthoff, N. Koehn, J. Larson, S.K.N. Dilger, E. Hammond, A. Schwartz, B. Mullan, R. Sanchez, R.M. Hoffman, J.C. Sieren, ''Post-imaging Mathematical Prediction Models: Are They Clinically Relevant?'' published in European Radiology. For more information about post-imaging mathematical prediction models (MPMs) and the methods used here, please see the article PMID: XXXXX

Visual inspection of data used in analysis.



Assess MPM Performance.


The table below shows MPM performance using AUC-ROC and Precision-Recall. It also includes the Youden's J Statistic Threshold and associated sensitivity and specificity using the Youden threshold.



Table 2: List of performance statistics in the application with calculation procedures and example meanings of values.


Visualize the risk predictions and MPM-recommended thresholds.


MPM Prediction Visualization Histograms

Similar to Figure 1 of Uthoff et al. (PMID: XXXX) - histograms of MPM predictions split based on true nodule classification. Solid black lines indicate MPM-associated thresholds. The gray dashed line is the Youden threshold. Note vertical axis autoscales to optimal for visualizatoin of MPM.


Explore the effect of threshold on sensitivity and specificity.


Use the slider to adjust the threshold and see the effect on sensitivity and specificity. Note vertical axis autoscales to optimal for visualizatoin of MPM.

Use the slider to adjust the threshold based on the desired level of sensitivity (True positive rate).

Use the slider to adjust the threshold based on the desired level of specificity (True negative rate).