Weights only save every 100 iterations until 900, then saves every 10,000

[allendorf]

Hello,

I am training YOLO on the COCO dataset and I have noticed that the weights only save in increments of 100 until iteration 900. After iteration 900, the next set of saved weights is at iteration 10,000, then 20,000 after that.

Is there a way to change how often weights are saved to $DARKNET_ROOT/backup? I would request there be a variable in cfg/yolo.cfg that allows the user to choose how often weights are saved to backup.

[pluto16]

please see function train_detector in examples/detector.c

[allendorf]

Thank you @pluto16 for this helpful comment. Indeed, line 136 in detector.c describes when the weights should be saved:

if(i%10000==0 || (i < 1000 && i%100 == 0))

[zaher88abd]

Hi,
After I chang the values
if(i%10000==0 || (i < 10000 && i%100 == 0)){
I need to rebuild / remake the yolo and How I could do that.

[elliotwutingfeng]

@zaher88abd At the terminal, cd to the darknet folder and run make to re-compile.

[zaher88abd]

Thanks.

[shm007g]

This should be post in the readme file, so everyone could reach it before training.
I want to train about 4000 iterations and take a look about it. Then i MUST stop the process running for hours and rebuild.

[lisa676]

@pluto16 @shm007g For AlexeyAB Darknet where we can change iterations because there is no Example folder containing detector.c file.

[lisa676]

@pluto16 @shm007g I found a file yolo.c in src folder, I think we can change iterations in this file in line 83. Kindly have a look.
if(i%1000==0 || (i < 1000 && i%100 == 0))

[VanessaMachi]

@ahsan3803 I change to if(i%1==0), suppose will save the file in 1st iteration, but it is no success.

[srhtyldz]

The code has been changed how could we implement again for every 100 cycle ? @AlexeyAB

[srhtyldz]

What are we gonna edit ? Edit yolo.c file or detector. c ?? Can anyone inform me pleasE?

[ivder]

@srhtyldz pretty sure you have to modify the detector.c

[srhtyldz]

@srhtyldz pretty sure you have to modify the detector.c

I couldn't find the similar line in the new code . Can you help me please about that issue ?

[ivder]

@srhtyldz It's still there. Look at darknet/examples/detector.c line 138
if(i%10000==0 || (i < 1000 && i%100 == 0)){

[srhtyldz]

@srhtyldz It's still there. Look at darknet/examples/detector.c line 138
if(i%10000==0 || (i < 1000 && i%100 == 0)){

I changed only this line and it worked.
https://github.com/AlexeyAB/darknet/blob/master/src/detector.c#L313

[ivder]

Oh I thought you were talking about pjreddie's repo

[srhtyldz]

Oh I thought you were talking about pjreddie's repo

Sorry that i did not mention about it.

[dbersan]

Do I need to just make or make clean; make ?
I'm doing a make clean anyway, I'm just curious.

[ultimatecrack]

@pluto16 @shm007g For AlexeyAB Darknet where we can change iterations because there is no Example folder containing detector.c file.

for AlexeyAB Darknet
I think below is the condition. The previous condition is commented

//if (i % 1000 == 0 || (i < 1000 && i % 100 == 0)) {
//if (i % 100 == 0) {

    if (iteration >= (iter_save + 1000) || iteration % 1000 == 0) {
        iter_save = iteration;

#ifdef GPU
if (ngpus != 1) sync_nets(nets, ngpus, 0);
#endif
char buff[256];
sprintf(buff, "%s/%s_%d.weights", backup_directory, base, iteration);
save_weights(net, buff);
}

    if (iteration >= (iter_save_last + 100) || (iteration % 100 == 0 && iteration > 1)) {
        iter_save_last = iteration;

#ifdef GPU
if (ngpus != 1) sync_nets(nets, ngpus, 0);
#endif
char buff[256];
sprintf(buff, "%s/%s_last.weights", backup_directory, base);
save_weights(net, buff);
}

[ketand334]

@srhtyldz It's still there. Look at darknet/examples/detector.c line 138
if(i%10000==0 || (i < 1000 && i%100 == 0)){

I changed only this line and it worked.
https://github.com/AlexeyAB/darknet/blob/master/src/detector.c#L313

What did you change on this line that worked for you?

[XZLeo]

Is it possible to make it a parameter in the .cfg file? @it is a bit inconvenient to rebuild every time before running a new network.

[sandipan]

It's now the lines 383-387 in https://github.com/AlexeyAB/darknet/blob/master/src/detector.c#L383 that need be changed to e.g., like the following, to create a checkpoint every 100 iterations

    //if (i % 1000 == 0 || (i < 1000 && i % 100 == 0)) {
    if (iteration % 100 == 0) {
    //if ((iteration >= (iter_save + 10000) || iteration % 10000 == 0) ||
    //    (iteration >= (iter_save + 1000) || iteration % 1000 == 0) && net.max_batches < 10000)
    // {

along with a subsequent invocation of make

[arielkantorovich]

@sandipan
your answer will allow me to save every 1000 iteration?

[sandipan]

@arielkantorovich every 100