Along the top of this application, there are four tabs:
(1) About TAB
Current tab, overview of application and additional information about development.
(2) Templates and User Guide TAB
Here you can access the user guide and templates.
(3) Local Dataset TAB
Here you can load your own local cohort for analysis in concordance with the methods described in the publication.
(4) Sample Dataset TAB
Here you can try out the app and explore the environment before you assemble your local dataset.
The sample subjects used are not based on real subject data and should not be used to assess MPM performance or recommend thresholds for real-world cases.
Citing this work
Article
This application is included as supplemental material for the article - J. Uthoff, N. Koehn, J. Larson, S.K.N. Dilger, E. Hammond, A. Schwartz, B. Mullan, R. Sanchez, R.M. Hoffman, J.C. Sieren, ''Post-imaging Mathematical Prediction Models: Are They Clinically Relevant?'' published in European Radiology. PMID: XXXXXX
The authors would like to acknowledge Mark Escher and Patrick Thalken for their assistance in application deployment.
Source code
Following publication, the source code for this application will be available on GitHub (http://XXXX) under Creative Commons Attribution-ShareAlike (CC BY-SA) License. This license lets you remix, tweak, and build upon this work provided you cite the origin code and license new creations under identical terms. This license is similar to copyleft free and open source software licenses.
About Application Development
This application was developed using the R programming language version 3.5.1 (2018-07-02) -- 'Feather Spray'((C) 2018 The R Project for Statistical Computing).
R is free software environment for statistical computing and graphics. It complies and runs on a wide variety of UNIX platforms, Windows, and MacOS. It is available under the GNU General Public License. The authors of the associated publication and developers of this application would like to thank the R Development Core Team and all CRAN contributors for their work to make statistical and graphical libraries and tools available to all.
R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.
R libraries: 'shiny', 'pROC','PRROC', and 'dplyr'.
Winston Chang, Joe Cheng, JJ Allaire, Yihui Xie and Jonathan McPherson (2018). shiny: Web Application Framework for R. R package version 1.1.0. https://CRAN.R-project.org/package=shiny
Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frederique Lisacek, Jean-Charles Sanchez and Markus Muller (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, p. 77. DOI: 10.1186/1471-2105-12-77 <http://www.biomedcentral.com/1471-2105/12/77/>
Jan Grau, Ivo Grosse, and Jens Keilwagen (2015). PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics (31) 15, pp. 2595-2597.R package version 1.3.1.
Hadley Wickham, Romain Francois, Lionel Henry and Kirill Muller (2018). dplyr: A Grammar of Data Manipulation. R package version 0.7.6. https://CRAN.R-project.org/package=dplyr
Why download the file template?
It is necessary that your local data is formatted in a certain manner, specifically:
(1) - how variables are 'coded' (ex: age is coded in years, not months)
(2) - how the columns in the file should be named (ex: age column is named as ''Age'', not ''HowOld'')
Failure to correctly set up the local dataset file will result in either inaccurate assessments or an application failure.
For your convenience, we have provided a user guide and a template for your data file set-up.
User Guide
The user guide indicates (1) how variables should be coded and (2) how columns should be named. It also provides an overview of which variables/columns are used in each of the four MPMs. If you wish to only receive results from a single MPM, please feel free to use the templates provided on the 'Local Single MPM File Templates' tab above.
Save user guide as ''MPM_UserGuide.pdf'' to your computer's Downloads folderTemplate File
The template file is a comma separated file or '.csv' which can open in Excel or a text editor. It includes the appropriate column names and a single default subject. To use the template, simply add local subjects as a new row per the directions in the user guide.
Save template as ''MPM_localData_template.csv'' to your computer's Downloads folderUse these templates if you wish to examine a single MPM.
These files contain the column names for the variables used in each individual MPM. When they are uploaded to this application, any unused columns will be automatically populated with the default values. For information on which variables are used in the models see the user guide or the publications.
Failure to correctly set up the local dataset file will result in either inaccurate assessments or an application failure.
(MC) Mayo Clinic Model (PMID: 9129544)
Save MC MPM template as ''MPM_MC_template.csv'' to your computer's Downloads folder(VA) Veteran's Affairs Model (PMID: 17296637)
Save VA MPM template as ''MPM_VA_template.csv'' to your computer's Downloads folder(BU) Brock University Model (PMID: 24004118)
Save BU MPM template as ''MPM_BU_template.csv'' to your computer's Downloads folder(PU) Peking University Model (PMID: 22297626)
Save PU MPM template as ''MPM_PU_template.csv'' to your computer's Downloads folderProvide .CSV file of local dataset.
Table 1: List of variables used in the MPM risk prediction calculation and performance comparison. Indications of which variables are used for each MPM are made with 'X' (e.g. all MPMs use Age as a variable). Your local dataset file should have the correct Column Names and be coded based on the Acceptable Values. Any omitted columns will be automatically populated with the Default value.
Visual inspection of data used in analysis.
Assess MPM Performance.
The table shows MPM performance using AUC-ROC and Precision-Recall. It also includes the Youden's J Statistic Threshold and associated sensitivity and specificity using the Youden threshold. Table 2 of the User Guide provides description of the performance measures.
Table 2: List of performance statistics in the application with calculation procedures and example meanings of values.
Visualize the risk predictions and MPM-recommended thresholds.
MPM Prediction Visualization Histograms
Similar to Figure 1 of Uthoff et al. (PMID: XXXXX)- histograms of MPM predictions split based on true nodule classification. Solid lines indicate MPM-associated thresholds. Note vertical axis autoscales to optimal for visualizatoin of MPM.
Explore the effect of threshold on sensitivity and specificity.
Use the slider to adjust the threshold and see the effect on sensitivity and specificity. Note vertical axis autoscales to optimal for visualizatoin of MPM.
Use the slider to adjust the threshold based on the desired level of sensitivity (True positive rate).
Use the slider to adjust the threshold based on the desired level of specificity (True negative rate).
Save the results from your local dataset to your computer's Downloads folder.
Save the MPM performance Youden measures
File will save as ''MPM_youdenSummary_localDataset.csv'' to your computer's Downloads folder
Youden Summary StatisticsSave custom MPM performance measures
File will save as ''MPM_customSummary_localDataset.csv'' to your computer's Downloads folder
Custom Summary StatisticsSave the MPM subject-specific risk predictions
File will save as ''MPM_predictions_localDataset.csv'' to your computer's Downloads folder
All four MPM risk predictionsVisual inspection of data used in analysis.
Assess MPM Performance.
The table below shows MPM performance using AUC-ROC and Precision-Recall. It also includes the Youden's J Statistic Threshold and associated sensitivity and specificity using the Youden threshold.
Table 2: List of performance statistics in the application with calculation procedures and example meanings of values.
Visualize the risk predictions and MPM-recommended thresholds.
MPM Prediction Visualization Histograms
Similar to Figure 1 of Uthoff et al. (PMID: XXXX) - histograms of MPM predictions split based on true nodule classification. Solid black lines indicate MPM-associated thresholds. The gray dashed line is the Youden threshold. Note vertical axis autoscales to optimal for visualizatoin of MPM.