Diagnostic Signature Challenge - Scoring

Diagnostic Signature Challenge - Scoring

Scoring Methodology

The challenge scoring was performed as follows:

Compute the sub-challenge score, by comparing the predictions to the Gold Standard and computing Quality Score with following metrics:

  • Belief Confusion Metric (BCM)
  • Area Under Precision Recall curve (AUPR)
  • Correct Class Enrichment Metric (CCEM)

Details on the metrics are presented here.

The sub-challenge score is obtained by ranking the teams according to the sum of ranks attained in each metric.

Compute the overall rank, by ranking of the sum of ranks over all* sub-challenges (COPD, Lung Cancer, Multiple Sclerosis Diagnostic, Psoriasis).

 

*Statistical Significance Requirement

The predictions on the Multiple Sclerosis Stage sub-challenge did not show statistically significant scores.  This sub-challenge was therefore removed from overall scoring. A minimum of 5 submissions were required, of which, at least one submission needed to be statistically significant in at least one metric, at a level of significance given by a false discovery rate of 0.05 .

 

Participation in all sub-challenges

Participants were asked to submit predictions in all sub-challenges. If a team did not participate in one of the sub-challenges, that team was ranked last on that sub challenge. The other sub-challenge scores were not affected.

 

Tie Resolution

No ties occurred for the best performing teams.

 

Procedure

Double-blind scoring

The submissions were anonymized before scoring, so that the team names were not known to the scoring team.

The detailed scoring methods were defined before challenge closure.

The detailed metrics were not disclosed to participants prior to closure. High-level guidelines were provided to participants during the open phase, these can be found here.

Independent oversight and guidance

An independent IMPROVER Scoring Review Panelguided, reviewed and agreed to the selected scoring methods and rankings. Their names, affiliations and relationships with the challenge organizers were made public (see “Scorers and Scoring Review Panel” below).

Transparent methods

Methods, procedures and rankings are made public.

This scoring procedure, combined with the fact that submissions were made on unseen test data, allows for an independent, methodical and transparent verification.

 

Scorers and Scoring Review Panel

A team of researchers from the IBM T.J. Watson Research Center in New York (USA) will establish a scoring methodology and perform the scoring on the blinded submissions under the review of an independent  Scoring Review Panel.

The IMPROVER Scoring Review Panel will review the scoring process of the Diagnostic Signature challenge and consists of following members:

  • Dr. Richard A. Bonneau, New York University
  • Dr. Alberto de la Fuente, CRS4 Bioinformatica
  • Dr. Igor Jurisica, University of Toronto
  • Dr. Daniel Marbach, MIT, Computational Biology Group
  • Dr. Tamir Tuller, Tel Aviv University 

The members of the Scoring Review Panel were selected for their expertise in the field of systems biology. None of the members of the Scoring Review Panel has declared any conflict of interest which may arise from their participation in the scoring process of the IMPROVER Diagnostic Signature Challenge.

For the sake of transparency, we publish their associations with the challenge organizers (PMI and IBM), as follows:

Member of the Scoring Review Panel Association(s) with the challenge organizers (PMI and IBM)
Dr. Richard A. Bonneau, New York University • Joint research with IBM on the Human Proteome Folding Project (Grid Computing)
Dr. Alberto de la Fuente, CRS4 Bioinformatica

• IMPROVER SBV Interest Group member
• Joint research papers with the challenge organizers

Dr. Igor Jurisica, University of Toronto • Visiting scientist at the IBM Toronto Laboratory, Centre for Advanced Studies, since 1998.
• Work on research projects and publications funded by IBM:
– Knowledge Management in Biological Domains
– Managing Protein Crystallization Experiences
– Knowledge Warehousing in Biology
– IBM World Community Grid
– Various IBM Shared University Research (SUR) grants
Dr. Daniel Marbach, MIT, Computational Biology Group • Joint research papers with challenge organize
Dr. Tamir Tuller, Tel Aviv University      • IMPROVER SBV Interest Group member
• Data provider for the IMPROVER Diagnostic Signature Challenge

 

Scoring Programs

The scoring programs are available for download. Please click the following links to download these programs:

Please contact us should you have further questions.

Share this page