Online Crowd Verification

Online Crowd Verification

To see all videos, click here.

The sbv IMPROVER Network Verification Challenge (NVC) asks participants to verify, modify or create edges in selected biological network models. The aim is to build a consensus around what parts of the networks are accurate, incorrect or incomplete.

Biological network models are a representation of known biology within defined contextual boundaries (e.g. species, tissue, and disease). Networks consist of nodes (e.g. DNA, RNA, proteins, etc.) and edges, where edges are causal or correlative relationships between the nodes. For instance, the protein MDM2 negatively regulates the activity of the protein p53. MDM2 and p53 are the nodes, and “negatively regulates” is the edge. NVC participants are requested to verify this kind of relationship.

The NVC website visualizes the available networks, enabling participants to scrutinize relationships, and make submissions that will either extend the network or verify existing parts of the network:

Extend the network:

  • Create a new edge between two existing nodes, an existing and a new node, or two new nodes.

Guidelines & tips: Each new edge creation must respect the network’s contextual boundary conditions and be submitted with a supporting peer-reviewed academic article. New nodes can only be created as part of creating an edge. Participants can capture new edges using Biological Expression Language (BEL). The website provides a BEL statement editor and tutorials to support participants.

Verify the network:

  • Add additional evidence to provide support for an existing edge.
  • Approve or reject evidence for edges, based on whether the provided reference supports the edge and whether an evidence form has been filled in accurately.

Guidelines & tips: When submitting additional evidence or voting on edges, participants should fill in the evidence and vote form as completely and accurately as possible. This will help other participants understand the rationale for submissions in the network, and help the consensus-building process.

The outcome of the online verification process is the result of the combination of submissions by different participants. Based on this, each edge can have four possible states at the end of the challenge:

  • Verified:
    There is at least one verified piece of evidence associated with the edge. A piece of evidence is verified if the overwhelming majority of participants approved rather than rejected the evidence.
  • Ambiguous:
    Participants are divided on whether a piece of evidence supports the edge (less than 80 % of participants approve or reject the edge).
  • Rejected:
    All evidence that has been suggested in favor of an edge has been rejected by the overwhelming majority of participants during the course of the challenge.
  • Not verified:
    The evidence for an edge did not receive sufficient submissions from participants to be considered verified.

A selection of edges that attracted a lot of attention and controversy from challenge participants will be reviewed and discussed at the Jamboree. This face-to-face meeting takes place once the online verification process is closed.


Overview of Actions in the Network Verification Challenge


Share this page