HOW to add new reference data to a phase?

[johann-petrak]

I have uploaded 4 competitions, each having three phases as a bundle each.
See:

• https://www.codabench.org/competitions/2744/
• https://www.codabench.org/competitions/2745/
• https://www.codabench.org/competitions/2746/
• https://www.codabench.org/competitions/2747/

For the first phase (the trial phase) I have included the reference data in the bundle within the directory trial_phase as
trial_phase/reference_data/targets.json

But I have not added this file for the other two phases because we did not have those files yet.

Now I have the file for the next phase, the development phase.

But how can I just upload a file into that directory so that the scorer will find it at the correct place? How can I test this as an admin as long as the phase did not start yet?
I find the way how the GUI works very confusing, TBH.

Could you please provide help on this because the competition really needs to have that file uploaded to the correct place when the phase starts?

[ihsaan-ullah]

A phase actually runs a Task, which is composed of scoring_program, ingestion_program, input_data, reference data. You can upload your files in the resource interface

You can upload the files in Datasets and programs tab and create a new task in the Tasks tab using these files.
After that you can go to your competition settings (Edit Competiiton), select phases tab, edit the phase you want to add this task to, delete previous task and select the new task you have created. To test that this works, you can use the same task (same files) locally AND/OR you can change the phase dates to test if the task is ok on Codabench.

Please do suggest possible improvements to the UI in another discussion thread. Something which you think could be easier for users to understand in another way rather than the current UI.

[johann-petrak]

There is no way to just update the reference data for an existing task which is already part of an existing phase? You have to go through the whole process of creating a new task with new data and replace the existing task just to add the missing reference data? Is that correct?

I would have expected to be able to just upload the missing reference data file and replace whatever the current task has set with this in the editor. Having to do it this way is incredibly tedious and error prone.

Ideally one would be able to just edit the bundle structure as one uploads it, with the same directory layout and file names.

[johann-petrak]

OR you can change the phase dates to test if the task is ok on Codabench

I just wanted to do this but the Editor does not allow me to change the start date of the phase to today. Before I did that I changed the end date of the previous phase to April 27th and saved (it was set to 29th originally because of the bug where the bundle cannot have the end date of the previous phase be the same as the start date of the next phase).

you can use the same task (same files) locally

What do you mean by this? I know that my file contains the correct data, what I need to test if the task setup on codabench is configured in a way which allows to submission get scored correctly.

[ihsaan-ullah]

You can edit an existing task. Upload the new reference data, click the edit button in front of the task you want to update, select the uploaded reference data in the reference data input, and save changes. NOW your phase will run the updated task

[ihsaan-ullah]

I understand that you want to upload the file to the directory but once you upload your bundle to Codabench, there are no more directories that is why you cannot upload something to a folder

[ihsaan-ullah]

To be able to distinguish several files, you have to name your zip files differently e.g. comp_1_ref_phase_1, comp_1_ref_phase_2, ...

It does not matter if the content of files are same or have same names but the zip names should be distinct. I think you cannot upload two files with the same name

[johann-petrak]

To be able to distinguish several files, you have to name your zip files differently e.g. comp_1_ref_phase_1, comp_1_ref_phase_2, ...

It does not matter if the content of files are same or have same names but the zip names should be distinct. I think you cannot upload two files with the same name

Can you please clarify this? As I said I will have to do this for the next phase in the same way again: there is another task, for the competition phase where I have to upload the targets file which is, using the same scorer, expected to be named targets.json in the reference_data directory (as the scorer sees it).

How does uploading a zip file for the reference data actually impact the name of the file that is used? If I would already have had the file available, I would certainly have been able to use the same file name for all phase-directories so what do I have to do to make the targets.json file show up for the scorer correctly in the next phase?

[ihsaan-ullah]

You have one scoring program that is used in all the 3 phases. Your reference data is different for the 3 phases. You can do either of the following to use the right reference data in the right phase.

• Create 3 tasks one for each phase with same scoring program but different reference data
• Create 1 task that uses ref_Data1 for now and for next phase update the phase to use ref_DAta2 and so on

[johann-petrak]

Thanks ... what I would like to understand is how the file name I use when I upload the reference data corresponds to the filename that my scoring program sees when I run it during the phase.

I found that my scorer will always see the directory "/app/input/ref" and for the first phase I placed the file "targets.json" directly into the reference directory for the task used by that phase.

Now I have uploaded a file "targets.json" (within a zip file called targets.zip) and assigned that file to the task for my current phase, as you suggested.

When the final phase arrives, with that method I will have to assign a NEW target.json file to the task for the last phase. So I will not have ref_Data1 and ref_Data2 (different names) since the scores is the same for both tasks and expects the same file name to be present.

So if I upload another file "targets.json", how can I make sure I select the correct one when I have to update the task for the final phase? Or if it is not even possible to upload another file with the same name, what is the alternative - does this mean I have to upload a modified scorer because I cannot use the same file name for that phase/task?

I also found that although I have defined other users as admins for our projects, they do not have access to the files that are part of the competition bundles (or the one I uploaded now) for which they are admins, since all these files are apparently just global resources private to the creator or public to everyone.

[Didayolo]

So if I upload another file "targets.json", how can I make sure I select the correct one when I have to update the task for the final phase? Or if it is not even possible to upload another file with the same name, what is the alternative - does this mean I have to upload a modified scorer because I cannot use the same file name for that phase/task?

When you upload the new scoring program (as a zip file) you chose a name and a description for it. This allows you to distinguish it from others.