-
Notifications
You must be signed in to change notification settings - Fork 128
Example_iris
This example was generated with Chalab. All code is written in Python 2.7.
This is a very simple example of competition in 2 phases, with either result or code submission, using an ingestion program.
- View the competition -- To be moved to public server.
- View the competition bundle
- Download the competition bundle
- Download the sample code submission
- Download the sample result submission
It has a special workflow: submissions are made only during the first phase and they are automatically forwarded to the second phase:
The competition bundle contains the following files:
input_data_1.zip The input data (to code submitted by participants); same for both phases
reference_data_1.zip The solution to the problem, in the 1st phase
reference_data_2.zip The solution to the problem, in the 2nd phase
ingestion_program_1.zip The ingestion program
scoring_program_1.zip The program evaluating the solution (same in both phases)
starting_kit_1.zip The starting kit to generate sample submissions
logo.png The logo
competition.yaml The YAML configuration file
data.html HTML documentation pages
evaluation.html
overview.html
terms.html
get_starting_kit.html
The YAML file is the configuration file. It consist in a number of attribute: value
pairs. For a full list of attributes, see the Codalab competition YAML definition language.
- First section: general settings and HTML pages
title: Iris
description: The well known Iris dataset from Fisher's classic paper (Fisher, 1936).
image: logo.png
has_registration: false
html:
data: data.html
evaluation: evaluation.html
get_starting_kit: get_starting_kit.html
overview: overview.html
terms: terms.html
- Second section: phases (3 phases in this example)
phases:
0:
color: green
ingestion_program: ingestion_program_1.zip
input_data: input_data_1.zip
label: Development
max_submissions: 100
max_submissions_per_day: 5
phasenumber: 1
public_data: input_data_1.zip
reference_data: reference_data_1.zip
scoring_program: scoring_program_1.zip
start_date: 2017-10-22 18:53:00+00:00
starting_kit: starting_kit_1.zip
1:
color: purple
ingestion_program: ingestion_program_1.zip
input_data: input_data_1.zip
label: Final
max_submissions: 100
max_submissions_per_day: 0
phasenumber: 2
reference_data: reference_data_2.zip
scoring_program: scoring_program_1.zip
start_date: 2018-04-30 18:53:00+00:00
- Third section: leaderboard configuration
leaderboard:
columns:
Duration:
label: Duration
leaderboard: &id001
label: Results
rank: 1
rank: 7
sort: desc
set1_score: # This is the name of the score returned in score.txt written by the scoring program
label: Prediction score # This will be the name of the column showing the results
leaderboard: *id001
rank: 2
sort: desc # Change that to "asc" if you want to sort the score in ascending order
leaderboards:
Results: *id001
In this case, the scoring program writes a file scores.txt
containing:
set1_score: 0.5
Duration: 0.123
We have several zip files providing data:
-
Public data:
input_data_1.zip
is provided to the participants for download in the first phase. -
Input data:
input_data_1.zip
is also provided to the submitted code on the Codalab platform. -
Reference data:
reference_data_1.zip
is the solution to the problem in the 1st phase (occluded to the participants).reference_data_2.zip
is the solution to the problem in the 2nd phase.
This competition uses an Ingestion Program.
ingestion.py
- In this example, datasets are loaded and a class model.py
supplied by the participants is called to train and test a model.
metadata
- instruct Codalab to execute the ingestion program when a participant submits code.
These pieces and other library functions are packaged as ingestion_program_1.zip to process the submissions for this competition.
model.py
- is a class containing a predictive model with 2 methods: fit
and predict
. This is inspired by the interface of the scikit-learn Python library of machine learning.
metadata
- instruct Codalab that this is a code submission. There is no command though because the code is not executable.
description: this is a non executable code submission.
For examples of code submissions that are executable and do not require an ingestion program, see Yellow World and Compute Pi.
score.py
- is an example that computes the performance of prediction using the metric given in metric.txt
.
metadata
- this is the file that provides the command to run the scoring program.
These pieces and other library functions are packaged as scoring_program_1.zip to evaluate the predictions made by code submitted by the participants.
This competition also accepts result submissions. All the participants need to do is to provide properly formatted results (similar to the output of their code submission) and submit a zipfile containing them, without metadata
file. This can be done since the participants can download the input data (which is the same as the public data).
You can easily turn this competition into a result submission ONLY competition:
- In the YAML file, change to
is_scoring_only: False
, or - In the editor, check the box:
[-] Results Scoring Only