Along the top of this application, there are five tabs:


(1) About TAB

Current tab, overview of application and additional information about development.


(2) Templates and User Guide TAB

Here you can access the user guide and templates.

(3) Single Case Use TAB

Here you can enter a single subject's features to analyze the risk score from each MPM.

(4) Local Dataset TAB

Here you can load your own local cohort for analysis in concordance with the methods described in the publication.

(5) Sample Dataset TAB

Here you can try out the app and explore the environment before you assemble your local dataset.

The sample subjects used are not based on real subject data and should not be used to assess MPM performance or recommend thresholds for real-world cases.

Citing this work


Articles

For more information about post-imaging mathematical prediction models (MPMs) and the methods used here, please see the articles:

K.E. Schroeder, K. Knoernschild, S.L. Averill, R.M. Hoffman, J.C. Sieren,''Comparison of four published cancer prediction equations performance applied to a cohort from the National Lung Screening Trial'' submitted to Radiology.

J. Uthoff, N. Koehn, J. Larson, S.K.N. Dilger, E. Hammond, A. Schwartz, B. Mullan, R. Sanchez, R.M. Hoffman, J.C. Sieren, ''Post-imaging Mathematical Prediction Models: Are They Clinically Relevant?'' published in European Radiology. PMID: 30937590

The authors would like to acknowledge Mark Escher and Patrick Thalken for their assistance in application deployment.


Source code

Following publication, the source code for this application will be available on GitLab (https://research-git.uiowa.edu/appil/MPM.git) under Creative Commons Attribution-ShareAlike (CC BY-SA) License. This license lets you remix, tweak, and build upon this work provided you cite the origin code and license new creations under identical terms. This license is similar to copyleft free and open source software licenses.


About Application Development

This application was developed using the R programming language version 3.5.1 (2018-07-02) -- 'Feather Spray'((C) 2018 The R Project for Statistical Computing).

Updates to the application used R version 4.3.0 (2023-04-21) -- 'Already Tomorrow'

R is free software environment for statistical computing and graphics. It complies and runs on a wide variety of UNIX platforms, Windows, and MacOS. It is available under the GNU General Public License. The authors of the associated publication and developers of this application would like to thank the R Development Core Team and all CRAN contributors for their work to make statistical and graphical libraries and tools available to all.

R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.

R libraries: 'shiny', 'pROC','PRROC', and 'dplyr'.

Winston Chang, Joe Cheng, JJ Allaire, Yihui Xie and Jonathan McPherson (2018). shiny: Web Application Framework for R. R package version 1.1.0. https://CRAN.R-project.org/package=shiny

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frederique Lisacek, Jean-Charles Sanchez and Markus Muller (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, p. 77. DOI: 10.1186/1471-2105-12-77 <http://www.biomedcentral.com/1471-2105/12/77/>

Jan Grau, Ivo Grosse, and Jens Keilwagen (2015). PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics (31) 15, pp. 2595-2597.R package version 1.3.1.

Hadley Wickham, Romain Francois, Lionel Henry and Kirill Muller (2018). dplyr: A Grammar of Data Manipulation. R package version 0.7.6. https://CRAN.R-project.org/package=dplyr


In the Single Case Use tab, enter the values for the variables in the input boxes. Please note that non-accepted values may not result in an error message, so double check that values are correct or results will be an inaccurate assessment. See the table below for the accepted values.



Why download the file template?

It is necessary that your local data is formatted in a certain manner, specifically:

(1) - how variables are 'coded' (ex: age is coded in years, not months)

(2) - how the columns in the file should be named (ex: age column is named as ''Age'', not ''HowOld'')


Failure to correctly set up the local dataset file will result in either inaccurate assessments or an application failure.

For your convenience, we have provided a user guide and a template for your data file set-up.


User Guide

The user guide indicates (1) how variables should be coded and (2) how columns should be named. It also provides an overview of which variables/columns are used in each of the four MPMs. If you wish to only receive results from a single MPM, please feel free to use the templates provided on the 'Local Single MPM File Templates' tab above.

Save user guide as ''MPM_UserGuide.pdf'' to your computer's Downloads folder

Template File

The template file is a comma separated file or '.csv' which can open in Excel or a text editor. It includes the appropriate column names and a single default subject. To use the template, simply add local subjects as a new row per the directions in the user guide.

Save template as ''MPM_localData_template.csv'' to your computer's Downloads folder

Use these templates if you wish to examine a single MPM.

These files contain the column names for the variables used in each individual MPM. When they are uploaded to this application, any unused columns will be automatically populated with the default values. For information on which variables are used in the models see the user guide or the publications.
Failure to correctly set up the local dataset file will result in either inaccurate assessments or an application failure.

(MC) Mayo Clinic Model (PMID: 9129544)

Save MC MPM template as ''MPM_MC_template.csv'' to your computer's Downloads folder

(VA) Veteran's Affairs Model (PMID: 17296637)

Save VA MPM template as ''MPM_VA_template.csv'' to your computer's Downloads folder

(BU) Brock University Model (PMID: 24004118)

Save BU MPM template as ''MPM_BU_template.csv'' to your computer's Downloads folder

(PU) Peking University Model (PMID: 22297626)

Save PU MPM template as ''MPM_PU_template.csv'' to your computer's Downloads folder

Provide values for the subject's features.

Please see Table 1 below for description of each variable and the accepted values. Once the values have been entered, please click the Add button; this will calculate the results which can be viewed in the MPM Results tab.

If you wish to change the values, please click the Clear button, enter in the new values, and then click Add. New results can be viewed in the MPM Results tab.





Table 1: List of variables used in the MPM risk prediction calculation and performance comparison. Indications of which variables are used for each MPM are made with 'X' (e.g. all MPMs use Age as a variable).


Assess MPM Performance.



Visualization of single case's risk predictions to NLST MPM-recommended thresholds.

Histograms of MPM predictions split based on true nodule classification. Blue bars indicates benign nodules, and pink bars indicates malignant nodules. The black dashed lines are the NLST calibrated thresholds. The red dashed line is the prediction value calculated from the input values. Note vertical axis autoscales to optimal for visualization of MPM.


Provide .CSV file of local dataset.



Table 1: List of variables used in the MPM risk prediction calculation and performance comparison. Indications of which variables are used for each MPM are made with 'X' (e.g. all MPMs use Age as a variable). Your local dataset file should have the correct Column Names and be coded based on the Acceptable Values. Any omitted columns will be automatically populated with the Default value.


Visual inspection of data used in analysis.



Assess MPM Performance.

The table shows MPM performance using AUC-ROC and Precision-Recall. It also includes the Youden's J Statistic Threshold and associated sensitivity and specificity using the Youden threshold. Table 2 of the User Guide provides description of the performance measures.



Table 2: List of performance statistics in the application with calculation procedures and example meanings of values.


Visualize the risk predictions and MPM-recommended thresholds.


MPM Prediction Visualization Histograms

Similar to Figure 1 of Uthoff et al. (PMID: 30937590)- histograms of MPM predictions split based on true nodule classification. Solid lines indicate MPM-associated thresholds. Note vertical axis autoscales to optimal for visualization of MPM.


Explore the effect of threshold on sensitivity and specificity.


Use the slider to adjust the threshold and see the effect on sensitivity and specificity. Note vertical axis autoscales to optimal for visualization of MPM.

Use the slider to adjust the threshold based on the desired level of sensitivity (True positive rate).

Use the slider to adjust the threshold based on the desired level of specificity (True negative rate).


Save the results from your local dataset to your computer's Downloads folder.



Save the MPM performance Youden measures

File will save as ''MPM_youdenSummary_localDataset.csv'' to your computer's Downloads folder

Youden Summary Statistics


Save custom MPM performance measures

File will save as ''MPM_customSummary_localDataset.csv'' to your computer's Downloads folder

Custom Summary Statistics


Save the MPM subject-specific risk predictions

File will save as ''MPM_predictions_localDataset.csv'' to your computer's Downloads folder

All four MPM risk predictions

Visual inspection of data used in analysis.



Assess MPM Performance.


The table below shows MPM performance using AUC-ROC and Precision-Recall. It also includes the Youden's J Statistic Threshold and associated sensitivity and specificity using the Youden threshold.



Table 2: List of performance statistics in the application with calculation procedures and example meanings of values.


Visualize the risk predictions and MPM-recommended thresholds.


MPM Prediction Visualization Histograms

Similar to Figure 1 of Uthoff et al. (PMID: 30937590) - histograms of MPM predictions split based on true nodule classification. Solid black lines indicate MPM-associated thresholds. The gray dashed line is the Youden threshold. Note vertical axis autoscales to optimal for visualization of MPM.



Explore the effect of threshold on sensitivity and specificity.


Use the slider to adjust the threshold and see the effect on sensitivity and specificity. Note vertical axis autoscales to optimal for visualization of MPM.

Use the slider to adjust the threshold based on the desired level of sensitivity (True positive rate).

Use the slider to adjust the threshold based on the desired level of specificity (True negative rate).