We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Checklist
Describe the bug hello @tyomj @ternaus @hellock @yhcao6 ,when i use albu data augmentation like https://github.com/open-mmlab/mmdetection/blob/master/configs/albu_example/mask_rcnn_r50_fpn_1x.py. I modified the original setting of albu and used the random 30 degree rotation and extended anchor ratio, but the loss will suddenly increase after the 15th epon, and this error also exists in htc_x101_32x4d_fpn without semantic, could you please fix this error?
【Note that I have modified the Settings in these places】
# model settings model = dict( type='MaskRCNN', pretrained='open-mmlab://resnext101_32x4d', backbone=dict( type='ResNeXt', depth=101, groups=32, base_width=4, num_stages=4, out_indices=(0, 1, 2, 3), frozen_stages=1, style='pytorch', dcn=dict( type='DCN', # groups=32, deformable_groups=1, fallback_on_stride=False), stage_with_dcn=(False, True, True, True)), neck=dict( type='FPN', in_channels=[256, 512, 1024, 2048], out_channels=256, num_outs=5), rpn_head=dict( type='RPNHead', in_channels=256, feat_channels=256, anchor_scales=[8], anchor_ratios=[0.2, 0.5, 1.0, 2.0, 5.0], anchor_strides=[4, 8, 16, 32, 64], target_means=[.0, .0, .0, .0], target_stds=[1.0, 1.0, 1.0, 1.0], loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0), loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0)), bbox_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict(type='RoIAlign', out_size=7, sample_num=2), out_channels=256, featmap_strides=[4, 8, 16, 32]), bbox_head=dict( type='SharedFCBBoxHead', num_fcs=2, in_channels=256, fc_out_channels=1024, roi_feat_size=7, num_classes=2, target_means=[0., 0., 0., 0.], target_stds=[0.1, 0.1, 0.2, 0.2], reg_class_agnostic=False, loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0), loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)), mask_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict(type='RoIAlign', out_size=14, sample_num=2), out_channels=256, featmap_strides=[4, 8, 16, 32]), mask_head=dict( type='FCNMaskHead', num_convs=4, in_channels=256, conv_out_channels=256, num_classes=2, loss_mask=dict( type='CrossEntropyLoss', use_mask=True, loss_weight=1.0))) # model training and testing settings train_cfg = dict( rpn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.7, neg_iou_thr=0.3, min_pos_iou=0.3, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=256, pos_fraction=0.5, neg_pos_ub=-1, add_gt_as_proposals=False), allowed_border=0, pos_weight=-1, debug=False), rpn_proposal=dict( nms_across_levels=False, nms_pre=2000, nms_post=2000, max_num=2000, nms_thr=0.7, min_bbox_size=0), rcnn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.5, neg_iou_thr=0.5, min_pos_iou=0.5, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=512, pos_fraction=0.25, neg_pos_ub=-1, add_gt_as_proposals=True), mask_size=28, pos_weight=-1, debug=False)) test_cfg = dict( rpn=dict( nms_across_levels=False, nms_pre=2000, nms_post=1000, max_num=1000, nms_thr=0.7, min_bbox_size=0), rcnn=dict( score_thr=0.05, nms=dict(type='nms', iou_thr=0.5), max_per_img=100, mask_thr_binary=0.5)) # dataset settings dataset_type = 'ScenetextDataset' data_root = '/data/' img_norm_cfg = dict( mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True) albu_train_transforms = [ dict( type='ShiftScaleRotate', shift_limit=0.0625, scale_limit=0.0, rotate_limit=30, interpolation=1, p=0.5), dict( type='RandomBrightnessContrast', brightness_limit=[0.1, 0.3], contrast_limit=[0.1, 0.3], p=0.2), dict( type='OneOf', transforms=[ dict( type='RGBShift', r_shift_limit=10, g_shift_limit=10, b_shift_limit=10, p=1.0), dict( type='HueSaturationValue', hue_shift_limit=20, sat_shift_limit=30, val_shift_limit=20, p=1.0) ], p=0.1), dict(type='JpegCompression', quality_lower=85, quality_upper=95, p=0.2), dict(type='ChannelShuffle', p=0.1), dict( type='OneOf', transforms=[ dict(type='Blur', blur_limit=3, p=1.0), dict(type='MedianBlur', blur_limit=3, p=1.0) ], p=0.1), ] train_pipeline = [ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True, with_mask=True), dict(type='Resize', img_scale=(1024, 960), keep_ratio=True), dict(type='Pad', size_divisor=32), dict( type='Albu', transforms=albu_train_transforms, bbox_params=dict( type='BboxParams', format='pascal_voc', label_fields=['gt_labels'], min_visibility=0.0, filter_lost_elements=True), keymap={ 'img': 'image', 'gt_masks': 'masks', 'gt_bboxes': 'bboxes' }, update_pad_shape=False, skip_img_without_anno=True), dict(type='Normalize', **img_norm_cfg), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor')) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1024, 960), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict(type='Normalize', **img_norm_cfg), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']), ]) ] data = dict( imgs_per_gpu=2, workers_per_gpu=2, train=dict( type=dataset_type, ann_file=data_root + 'annotations/gd_xiaode_no_aug_train.json', img_prefix=data_root + 'images/', pipeline=train_pipeline), val=dict( type=dataset_type, ann_file=data_root + 'annotations/gd_xiaode_no_aug_val.json', img_prefix=data_root + 'images/', pipeline=test_pipeline), test=dict( type=dataset_type, ann_file=data_root + 'annotations/gd_xiaode_no_aug_val.json', img_prefix=data_root + 'images/', pipeline=test_pipeline)) evaluation = dict(interval=36, metric=['bbox', 'segm']) # optimizer optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2)) # learning policy lr_config = dict( policy='step', warmup='linear', warmup_iters=500, warmup_ratio=1.0 / 3, step=[24, 33]) checkpoint_config = dict(interval=1) # yapf:disable log_config = dict( interval=50, hooks=[ dict(type='TextLoggerHook'), # dict(type='TensorboardLoggerHook') ]) # yapf:enable # runtime settings total_epochs = 36 dist_params = dict(backend='nccl') log_level = 'INFO' work_dir = './work_dirs/mask_rcnn_x101_fpn_3x_albu' load_from = None resume_from = None workflow = [('train', 1)]
Reproduction
#!/bin/bash export PYTHONPATH=`pwd`:$PYTHONPATH CUDA_VISIBLE_DEVICES=2 python tools/train.py mask_rcnn_x101_fpn_3x_expend_anchor_albu.py
Error traceback If applicable, paste the error trackback here.
2020-02-28 22:42:37,631 - mmdet - INFO - Epoch [15][50/1597] lr: 0.02000, eta: 9:03:08, time: 1.144, data_time: 0.240, memory: 6915, loss_rpn_cls: 0.0123, loss_rpn_bbox: 0.0160, loss_cls: 0.1203, acc: 95.0410, loss_bbox: 0.0759, loss_mask: 0.1827, loss: 0.4072 2020-02-28 22:43:25,793 - mmdet - INFO - Epoch [15][100/1597] lr: 0.02000, eta: 9:02:25, time: 0.963, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.0137, loss_rpn_bbox: 0.0148, loss_cls: 0.1201, acc: 95.1555, loss_bbox: 0.0725, loss_mask: 0.1803, loss: 0.4013 2020-02-28 22:44:14,066 - mmdet - INFO - Epoch [15][150/1597] lr: 0.02000, eta: 9:01:41, time: 0.965, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0117, loss_rpn_bbox: 0.0153, loss_cls: 0.1128, acc: 95.3838, loss_bbox: 0.0713, loss_mask: 0.1833, loss: 0.3945 2020-02-28 22:45:02,124 - mmdet - INFO - Epoch [15][200/1597] lr: 0.02000, eta: 9:00:57, time: 0.961, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0118, loss_rpn_bbox: 0.0165, loss_cls: 0.1187, acc: 95.1553, loss_bbox: 0.0772, loss_mask: 0.1881, loss: 0.4122 2020-02-28 22:45:49,831 - mmdet - INFO - Epoch [15][250/1597] lr: 0.02000, eta: 9:00:12, time: 0.954, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.0128, loss_rpn_bbox: 0.0164, loss_cls: 0.1198, acc: 95.1387, loss_bbox: 0.0775, loss_mask: 0.1942, loss: 0.4207 2020-02-28 22:46:36,517 - mmdet - INFO - Epoch [15][300/1597] lr: 0.02000, eta: 8:59:26, time: 0.934, data_time: 0.033, memory: 6915, loss_rpn_cls: 0.0120, loss_rpn_bbox: 0.0160, loss_cls: 0.1191, acc: 95.1389, loss_bbox: 0.0768, loss_mask: 0.1787, loss: 0.4026 2020-02-28 22:47:24,363 - mmdet - INFO - Epoch [15][350/1597] lr: 0.02000, eta: 8:58:42, time: 0.956, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.0134, loss_rpn_bbox: 0.0164, loss_cls: 0.1173, acc: 95.1985, loss_bbox: 0.0747, loss_mask: 0.1795, loss: 0.4014 2020-02-28 22:48:11,572 - mmdet - INFO - Epoch [15][400/1597] lr: 0.02000, eta: 8:57:57, time: 0.945, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.0127, loss_rpn_bbox: 0.0162, loss_cls: 0.1184, acc: 95.1401, loss_bbox: 0.0774, loss_mask: 0.1847, loss: 0.4094 2020-02-28 22:48:58,651 - mmdet - INFO - Epoch [15][450/1597] lr: 0.02000, eta: 8:57:11, time: 0.941, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0131, loss_rpn_bbox: 0.0166, loss_cls: 0.1125, acc: 95.4646, loss_bbox: 0.0741, loss_mask: 0.1919, loss: 0.4081 2020-02-28 22:49:46,141 - mmdet - INFO - Epoch [15][500/1597] lr: 0.02000, eta: 8:56:26, time: 0.950, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0113, loss_rpn_bbox: 0.0163, loss_cls: 0.1198, acc: 95.1125, loss_bbox: 0.0790, loss_mask: 0.1846, loss: 0.4109 2020-02-28 22:50:33,445 - mmdet - INFO - Epoch [15][550/1597] lr: 0.02000, eta: 8:55:41, time: 0.946, data_time: 0.046, memory: 6915, loss_rpn_cls: 0.0133, loss_rpn_bbox: 0.0151, loss_cls: 0.1175, acc: 95.1824, loss_bbox: 0.0707, loss_mask: 0.1825, loss: 0.3992 2020-02-28 22:51:20,188 - mmdet - INFO - Epoch [15][600/1597] lr: 0.02000, eta: 8:54:55, time: 0.935, data_time: 0.034, memory: 6915, loss_rpn_cls: 0.0125, loss_rpn_bbox: 0.0148, loss_cls: 0.1220, acc: 94.9873, loss_bbox: 0.0750, loss_mask: 0.1792, loss: 0.4035 2020-02-28 22:52:08,212 - mmdet - INFO - Epoch [15][650/1597] lr: 0.02000, eta: 8:54:11, time: 0.961, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.0125, loss_rpn_bbox: 0.0155, loss_cls: 0.1183, acc: 95.1719, loss_bbox: 0.0734, loss_mask: 0.1854, loss: 0.4050 2020-02-28 22:52:55,178 - mmdet - INFO - Epoch [15][700/1597] lr: 0.02000, eta: 8:53:25, time: 0.939, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0134, loss_rpn_bbox: 0.0146, loss_cls: 0.1218, acc: 95.0764, loss_bbox: 0.0739, loss_mask: 0.1954, loss: 0.4191 2020-02-28 22:53:42,082 - mmdet - INFO - Epoch [15][750/1597] lr: 0.02000, eta: 8:52:39, time: 0.939, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0129, loss_rpn_bbox: 0.0166, loss_cls: 0.1217, acc: 95.0298, loss_bbox: 0.0768, loss_mask: 0.1796, loss: 0.4075 2020-02-28 22:54:29,907 - mmdet - INFO - Epoch [15][800/1597] lr: 0.02000, eta: 8:51:55, time: 0.957, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0133, loss_rpn_bbox: 0.0160, loss_cls: 0.1218, acc: 94.9822, loss_bbox: 0.0792, loss_mask: 0.1890, loss: 0.4193 2020-02-28 22:55:17,470 - mmdet - INFO - Epoch [15][850/1597] lr: 0.02000, eta: 8:51:10, time: 0.951, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0135, loss_rpn_bbox: 0.0161, loss_cls: 0.1229, acc: 94.9690, loss_bbox: 0.0795, loss_mask: 0.1904, loss: 0.4223 2020-02-28 22:56:05,244 - mmdet - INFO - Epoch [15][900/1597] lr: 0.02000, eta: 8:50:25, time: 0.955, data_time: 0.045, memory: 6915, loss_rpn_cls: 0.0135, loss_rpn_bbox: 0.0179, loss_cls: 0.1201, acc: 95.0413, loss_bbox: 0.0770, loss_mask: 0.1780, loss: 0.4066 2020-02-28 22:56:53,064 - mmdet - INFO - Epoch [15][950/1597] lr: 0.02000, eta: 8:49:41, time: 0.956, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0136, loss_rpn_bbox: 0.0171, loss_cls: 0.1201, acc: 95.0701, loss_bbox: 0.0767, loss_mask: 0.1869, loss: 0.4143 2020-02-28 22:57:40,798 - mmdet - INFO - Epoch [15][1000/1597] lr: 0.02000, eta: 8:48:56, time: 0.955, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.0119, loss_rpn_bbox: 0.0154, loss_cls: 0.1244, acc: 94.8640, loss_bbox: 0.0765, loss_mask: 0.1923, loss: 0.4205 2020-02-28 22:58:27,871 - mmdet - INFO - Epoch [15][1050/1597] lr: 0.02000, eta: 8:48:11, time: 0.941, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0151, loss_rpn_bbox: 0.0168, loss_cls: 0.1245, acc: 95.0212, loss_bbox: 0.0778, loss_mask: 0.1875, loss: 0.4216 2020-02-28 22:59:16,352 - mmdet - INFO - Epoch [15][1100/1597] lr: 0.02000, eta: 8:47:27, time: 0.969, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.0145, loss_rpn_bbox: 0.0166, loss_cls: 0.1229, acc: 94.9785, loss_bbox: 0.0765, loss_mask: 0.1904, loss: 0.4209 2020-02-28 23:00:04,640 - mmdet - INFO - Epoch [15][1150/1597] lr: 0.02000, eta: 8:46:43, time: 0.966, data_time: 0.047, memory: 6915, loss_rpn_cls: 0.0160, loss_rpn_bbox: 0.0187, loss_cls: 0.1324, acc: 94.5950, loss_bbox: 0.0819, loss_mask: 0.1932, loss: 0.4422 2020-02-28 23:00:52,217 - mmdet - INFO - Epoch [15][1200/1597] lr: 0.02000, eta: 8:45:58, time: 0.952, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0138, loss_rpn_bbox: 0.0172, loss_cls: 0.1214, acc: 95.0952, loss_bbox: 0.0787, loss_mask: 0.1928, loss: 0.4240 2020-02-28 23:01:38,856 - mmdet - INFO - Epoch [15][1250/1597] lr: 0.02000, eta: 8:45:12, time: 0.933, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0142, loss_rpn_bbox: 0.0164, loss_cls: 0.1237, acc: 94.9583, loss_bbox: 0.0805, loss_mask: 0.1981, loss: 0.4330 2020-02-28 23:02:27,509 - mmdet - INFO - Epoch [15][1300/1597] lr: 0.02000, eta: 8:44:28, time: 0.973, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0145, loss_rpn_bbox: 0.0168, loss_cls: 0.1221, acc: 95.0525, loss_bbox: 0.0794, loss_mask: 0.1952, loss: 0.4280 2020-02-28 23:03:13,700 - mmdet - INFO - Epoch [15][1350/1597] lr: 0.02000, eta: 8:43:41, time: 0.924, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0204, loss_rpn_bbox: 0.0167, loss_cls: 0.1295, acc: 94.7988, loss_bbox: 0.0797, loss_mask: 0.1961, loss: 0.4425 2020-02-28 23:04:00,766 - mmdet - INFO - Epoch [15][1400/1597] lr: 0.02000, eta: 8:42:56, time: 0.941, data_time: 0.032, memory: 6915, loss_rpn_cls: 0.0152, loss_rpn_bbox: 0.0186, loss_cls: 0.1317, acc: 94.5842, loss_bbox: 0.0831, loss_mask: 0.1908, loss: 0.4394 2020-02-28 23:04:47,592 - mmdet - INFO - Epoch [15][1450/1597] lr: 0.02000, eta: 8:42:10, time: 0.937, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0140, loss_rpn_bbox: 0.0158, loss_cls: 0.1272, acc: 94.8086, loss_bbox: 0.0807, loss_mask: 0.1872, loss: 0.4249 2020-02-28 23:05:34,585 - mmdet - INFO - Epoch [15][1500/1597] lr: 0.02000, eta: 8:41:24, time: 0.940, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0153, loss_rpn_bbox: 0.0150, loss_cls: 0.1278, acc: 94.8647, loss_bbox: 0.0795, loss_mask: 0.1838, loss: 0.4215 2020-02-28 23:06:22,516 - mmdet - INFO - Epoch [15][1550/1597] lr: 0.02000, eta: 8:40:39, time: 0.959, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0166, loss_rpn_bbox: 0.0177, loss_cls: 0.1294, acc: 94.7173, loss_bbox: 0.0838, loss_mask: 0.1951, loss: 0.4425 2020-02-28 23:08:09,128 - mmdet - INFO - Epoch [16][50/1597] lr: 0.02000, eta: 8:38:25, time: 1.169, data_time: 0.241, memory: 6915, loss_rpn_cls: 0.0169, loss_rpn_bbox: 0.0173, loss_cls: 0.1327, acc: 94.5635, loss_bbox: 0.0825, loss_mask: 0.2026, loss: 0.4521 2020-02-28 23:08:56,801 - mmdet - INFO - Epoch [16][100/1597] lr: 0.02000, eta: 8:37:40, time: 0.953, data_time: 0.050, memory: 6915, loss_rpn_cls: 0.0166, loss_rpn_bbox: 0.0161, loss_cls: 0.1234, acc: 94.9868, loss_bbox: 0.0759, loss_mask: 0.1901, loss: 0.4222 2020-02-28 23:09:44,695 - mmdet - INFO - Epoch [16][150/1597] lr: 0.02000, eta: 8:36:56, time: 0.958, data_time: 0.048, memory: 6915, loss_rpn_cls: 0.0150, loss_rpn_bbox: 0.0165, loss_cls: 0.1207, acc: 95.1270, loss_bbox: 0.0761, loss_mask: 0.1861, loss: 0.4144 2020-02-28 23:10:32,384 - mmdet - INFO - Epoch [16][200/1597] lr: 0.02000, eta: 8:36:11, time: 0.954, data_time: 0.045, memory: 6915, loss_rpn_cls: 0.0141, loss_rpn_bbox: 0.0152, loss_cls: 0.1205, acc: 95.1179, loss_bbox: 0.0735, loss_mask: 0.1909, loss: 0.4141 2020-02-28 23:11:20,172 - mmdet - INFO - Epoch [16][250/1597] lr: 0.02000, eta: 8:35:26, time: 0.956, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0157, loss_rpn_bbox: 0.0163, loss_cls: 0.1336, acc: 94.5417, loss_bbox: 0.0827, loss_mask: 0.1949, loss: 0.4431 2020-02-28 23:12:07,609 - mmdet - INFO - Epoch [16][300/1597] lr: 0.02000, eta: 8:34:41, time: 0.949, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.0135, loss_rpn_bbox: 0.0160, loss_cls: 0.1211, acc: 95.0649, loss_bbox: 0.0762, loss_mask: 0.1870, loss: 0.4138 2020-02-28 23:12:54,386 - mmdet - INFO - Epoch [16][350/1597] lr: 0.02000, eta: 8:33:55, time: 0.935, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0142, loss_rpn_bbox: 0.0165, loss_cls: 0.1223, acc: 94.9717, loss_bbox: 0.0785, loss_mask: 0.1885, loss: 0.4200 2020-02-28 23:13:40,803 - mmdet - INFO - Epoch [16][400/1597] lr: 0.02000, eta: 8:33:09, time: 0.928, data_time: 0.044, memory: 6915, loss_rpn_cls: 0.0141, loss_rpn_bbox: 0.0170, loss_cls: 0.1251, acc: 94.8159, loss_bbox: 0.0819, loss_mask: 0.1901, loss: 0.4281 2020-02-28 23:14:27,643 - mmdet - INFO - Epoch [16][450/1597] lr: 0.02000, eta: 8:32:23, time: 0.937, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0147, loss_rpn_bbox: 0.0189, loss_cls: 0.1269, acc: 94.7927, loss_bbox: 0.0837, loss_mask: 0.2025, loss: 0.4466 2020-02-28 23:15:14,972 - mmdet - INFO - Epoch [16][500/1597] lr: 0.02000, eta: 8:31:37, time: 0.947, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.0178, loss_rpn_bbox: 0.0221, loss_cls: 0.1535, acc: 93.6848, loss_bbox: 0.1009, loss_mask: 0.2110, loss: 0.5052 2020-02-28 23:16:01,848 - mmdet - INFO - Epoch [16][550/1597] lr: 0.02000, eta: 8:30:51, time: 0.938, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0177, loss_rpn_bbox: 0.0195, loss_cls: 0.1309, acc: 94.6230, loss_bbox: 0.0845, loss_mask: 0.1893, loss: 0.4419 2020-02-28 23:16:50,853 - mmdet - INFO - Epoch [16][600/1597] lr: 0.02000, eta: 8:30:08, time: 0.980, data_time: 0.034, memory: 6915, loss_rpn_cls: 0.0168, loss_rpn_bbox: 0.0182, loss_cls: 0.1389, acc: 94.2625, loss_bbox: 0.0876, loss_mask: 0.2042, loss: 0.4657 2020-02-28 23:17:37,525 - mmdet - INFO - Epoch [16][650/1597] lr: 0.02000, eta: 8:29:22, time: 0.933, data_time: 0.033, memory: 6915, loss_rpn_cls: 0.0169, loss_rpn_bbox: 0.0172, loss_cls: 0.1358, acc: 94.5029, loss_bbox: 0.0831, loss_mask: 0.1952, loss: 0.4481 2020-02-28 23:18:24,291 - mmdet - INFO - Epoch [16][700/1597] lr: 0.02000, eta: 8:28:36, time: 0.935, data_time: 0.034, memory: 6915, loss_rpn_cls: 0.0145, loss_rpn_bbox: 0.0172, loss_cls: 0.1280, acc: 94.7725, loss_bbox: 0.0805, loss_mask: 0.1892, loss: 0.4294 2020-02-28 23:19:12,031 - mmdet - INFO - Epoch [16][750/1597] lr: 0.02000, eta: 8:27:51, time: 0.955, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0172, loss_rpn_bbox: 0.0199, loss_cls: 0.1366, acc: 94.3987, loss_bbox: 0.0901, loss_mask: 0.2052, loss: 0.4690 2020-02-28 23:19:58,810 - mmdet - INFO - Epoch [16][800/1597] lr: 0.02000, eta: 8:27:05, time: 0.936, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0139, loss_rpn_bbox: 0.0163, loss_cls: 0.1243, acc: 94.9497, loss_bbox: 0.0811, loss_mask: 0.1800, loss: 0.4156 2020-02-28 23:20:46,010 - mmdet - INFO - Epoch [16][850/1597] lr: 0.02000, eta: 8:26:20, time: 0.944, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0176, loss_rpn_bbox: 0.0191, loss_cls: 0.1345, acc: 94.5435, loss_bbox: 0.0856, loss_mask: 0.2046, loss: 0.4614 2020-02-28 23:21:33,313 - mmdet - INFO - Epoch [16][900/1597] lr: 0.02000, eta: 8:25:34, time: 0.946, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0140, loss_rpn_bbox: 0.0168, loss_cls: 0.1254, acc: 94.8811, loss_bbox: 0.0819, loss_mask: 0.1886, loss: 0.4267 2020-02-28 23:22:20,652 - mmdet - INFO - Epoch [16][950/1597] lr: 0.02000, eta: 8:24:49, time: 0.947, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0165, loss_rpn_bbox: 0.0186, loss_cls: 0.1370, acc: 94.3828, loss_bbox: 0.0901, loss_mask: 0.1965, loss: 0.4588 2020-02-28 23:23:08,513 - mmdet - INFO - Epoch [16][1000/1597] lr: 0.02000, eta: 8:24:04, time: 0.957, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0159, loss_rpn_bbox: 0.0189, loss_cls: 0.1328, acc: 94.4641, loss_bbox: 0.0856, loss_mask: 0.1901, loss: 0.4433 2020-02-28 23:23:56,384 - mmdet - INFO - Epoch [16][1050/1597] lr: 0.02000, eta: 8:23:20, time: 0.958, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0150, loss_rpn_bbox: 0.0186, loss_cls: 0.1323, acc: 94.5413, loss_bbox: 0.0846, loss_mask: 0.1915, loss: 0.4422 2020-02-28 23:24:43,955 - mmdet - INFO - Epoch [16][1100/1597] lr: 0.02000, eta: 8:22:35, time: 0.951, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0147, loss_rpn_bbox: 0.0174, loss_cls: 0.1341, acc: 94.5359, loss_bbox: 0.0864, loss_mask: 0.1933, loss: 0.4460 2020-02-28 23:25:31,061 - mmdet - INFO - Epoch [16][1150/1597] lr: 0.02000, eta: 8:21:49, time: 0.942, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.0140, loss_rpn_bbox: 0.0185, loss_cls: 0.1304, acc: 94.6208, loss_bbox: 0.0872, loss_mask: 0.1970, loss: 0.4472 2020-02-28 23:26:18,999 - mmdet - INFO - Epoch [16][1200/1597] lr: 0.02000, eta: 8:21:04, time: 0.959, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0131, loss_rpn_bbox: 0.0168, loss_cls: 0.1224, acc: 95.0242, loss_bbox: 0.0774, loss_mask: 0.1834, loss: 0.4131 2020-02-28 23:27:07,040 - mmdet - INFO - Epoch [16][1250/1597] lr: 0.02000, eta: 8:20:20, time: 0.961, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0141, loss_rpn_bbox: 0.0164, loss_cls: 0.1237, acc: 94.9224, loss_bbox: 0.0810, loss_mask: 0.1891, loss: 0.4244 2020-02-28 23:27:54,622 - mmdet - INFO - Epoch [16][1300/1597] lr: 0.02000, eta: 8:19:35, time: 0.951, data_time: 0.031, memory: 6915, loss_rpn_cls: 0.0170, loss_rpn_bbox: 0.0200, loss_cls: 0.1400, acc: 94.2537, loss_bbox: 0.0916, loss_mask: 0.1988, loss: 0.4675 2020-02-28 23:28:42,303 - mmdet - INFO - Epoch [16][1350/1597] lr: 0.02000, eta: 8:18:50, time: 0.954, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.0248, loss_rpn_bbox: 0.0226, loss_cls: 0.1594, acc: 93.3970, loss_bbox: 0.1073, loss_mask: 0.2028, loss: 0.5169 2020-02-28 23:29:26,618 - mmdet - INFO - Epoch [16][1400/1597] lr: 0.02000, eta: 8:18:01, time: 0.886, data_time: 0.030, memory: 6915, loss_rpn_cls: 291.5323, loss_rpn_bbox: 67.0836, loss_cls: 135.5149, acc: 91.7811, loss_bbox: 17.3593, loss_mask: 2178.4751, loss: 2689.9652 2020-02-28 23:30:09,803 - mmdet - INFO - Epoch [16][1450/1597] lr: 0.02000, eta: 8:17:10, time: 0.864, data_time: 0.042, memory: 6915, loss_rpn_cls: 659.2569, loss_rpn_bbox: 428.5792, loss_cls: 4334.8523, acc: 87.3309, loss_bbox: 1169.4782, loss_mask: 1962.3003, loss: 8554.4673 2020-02-28 23:30:52,354 - mmdet - INFO - Epoch [16][1500/1597] lr: 0.02000, eta: 8:16:18, time: 0.850, data_time: 0.038, memory: 6915, loss_rpn_cls: 2298.3458, loss_rpn_bbox: 941.3212, loss_cls: 6867.5134, acc: 87.2608, loss_bbox: 1659.7873, loss_mask: 8699.6353, loss: 20466.6026 2020-02-28 23:31:34,424 - mmdet - INFO - Epoch [16][1550/1597] lr: 0.02000, eta: 8:15:26, time: 0.842, data_time: 0.036, memory: 6915, loss_rpn_cls: 34560.5711, loss_rpn_bbox: 20196.0533, loss_cls: 134253.9739, acc: 84.7126, loss_bbox: 173239.8111, loss_mask: 168253.4768, loss: 530503.8719 2020-02-28 23:33:10,507 - mmdet - INFO - Epoch [17][50/1597] lr: 0.02000, eta: 8:13:09, time: 1.043, data_time: 0.229, memory: 6915, loss_rpn_cls: 888.9794, loss_rpn_bbox: 249.2877, loss_cls: 10332.9805, acc: 86.5781, loss_bbox: 2265.0971, loss_mask: 10465.6031, loss: 24201.9477 2020-02-28 23:33:53,067 - mmdet - INFO - Epoch [17][100/1597] lr: 0.02000, eta: 8:12:18, time: 0.851, data_time: 0.039, memory: 6915, loss_rpn_cls: 62187.1519, loss_rpn_bbox: 25625.9393, loss_cls: 1685080.2555, acc: 95.3147, loss_bbox: 1130550.9912, loss_mask: 535324.3485, loss: 3438768.6382 2020-02-28 23:34:35,836 - mmdet - INFO - Epoch [17][150/1597] lr: 0.02000, eta: 8:11:27, time: 0.855, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.5497, loss_rpn_bbox: 0.0657, loss_cls: 656.5872, acc: 97.5288, loss_bbox: 100.8471, loss_mask: 0.6937, loss: 758.7433 2020-02-28 23:35:18,105 - mmdet - INFO - Epoch [17][200/1597] lr: 0.02000, eta: 8:10:36, time: 0.846, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.4552, loss_rpn_bbox: 0.0637, loss_cls: 0.1110, acc: 97.6616, loss_bbox: 0.0148, loss_mask: 0.7255, loss: 1.3702 2020-02-28 23:36:00,735 - mmdet - INFO - Epoch [17][250/1597] lr: 0.02000, eta: 8:09:45, time: 0.853, data_time: 0.040, memory: 6915, loss_rpn_cls: 85.2085, loss_rpn_bbox: 16.9726, loss_cls: 190.6585, acc: 97.0966, loss_bbox: 160.9999, loss_mask: 334.0644, loss: 787.9038 2020-02-28 23:36:42,543 - mmdet - INFO - Epoch [17][300/1597] lr: 0.02000, eta: 8:08:53, time: 0.836, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.3951, loss_rpn_bbox: 0.0642, loss_cls: 0.1067, acc: 97.7776, loss_bbox: 0.0129, loss_mask: 0.6899, loss: 1.2689 2020-02-28 23:37:26,234 - mmdet - INFO - Epoch [17][350/1597] lr: 0.02000, eta: 8:08:03, time: 0.874, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.3938, loss_rpn_bbox: 0.0681, loss_cls: 0.1155, acc: 97.5400, loss_bbox: 0.0148, loss_mask: 0.6885, loss: 1.2807 2020-02-28 23:38:09,411 - mmdet - INFO - Epoch [17][400/1597] lr: 0.02000, eta: 8:07:13, time: 0.863, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.3869, loss_rpn_bbox: 0.0660, loss_cls: 0.1145, acc: 97.5669, loss_bbox: 0.0150, loss_mask: 0.6909, loss: 1.2733 2020-02-28 23:38:51,535 - mmdet - INFO - Epoch [17][450/1597] lr: 0.02000, eta: 8:06:21, time: 0.843, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.3744, loss_rpn_bbox: 0.0644, loss_cls: 0.1095, acc: 97.7017, loss_bbox: 0.0138, loss_mask: 0.6911, loss: 1.2532 2020-02-28 23:39:33,332 - mmdet - INFO - Epoch [17][500/1597] lr: 0.02000, eta: 8:05:30, time: 0.836, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.3831, loss_rpn_bbox: 0.0686, loss_cls: 0.1144, acc: 97.5696, loss_bbox: 0.0145, loss_mask: 0.6925, loss: 1.2731 2020-02-28 23:40:14,947 - mmdet - INFO - Epoch [17][550/1597] lr: 0.02000, eta: 8:04:38, time: 0.832, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.3771, loss_rpn_bbox: 0.0675, loss_cls: 0.1096, acc: 97.6973, loss_bbox: 0.0129, loss_mask: 0.6900, loss: 1.2572 2020-02-28 23:40:58,161 - mmdet - INFO - Epoch [17][600/1597] lr: 0.02000, eta: 8:03:48, time: 0.864, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.3712, loss_rpn_bbox: 0.0653, loss_cls: 0.1109, acc: 97.6641, loss_bbox: 0.0138, loss_mask: 0.6908, loss: 1.2520 2020-02-28 23:41:40,403 - mmdet - INFO - Epoch [17][650/1597] lr: 0.02000, eta: 8:02:56, time: 0.845, data_time: 0.043, memory: 6915, loss_rpn_cls: 0.3728, loss_rpn_bbox: 0.0647, loss_cls: 0.1109, acc: 97.6626, loss_bbox: 0.0135, loss_mask: 0.6902, loss: 1.2521 2020-02-28 23:42:22,660 - mmdet - INFO - Epoch [17][700/1597] lr: 0.02000, eta: 8:02:05, time: 0.845, data_time: 0.043, memory: 6915, loss_rpn_cls: 0.3761, loss_rpn_bbox: 0.0680, loss_cls: 0.1137, acc: 97.5886, loss_bbox: 0.0148, loss_mask: 0.6896, loss: 1.2622 2020-02-28 23:43:04,407 - mmdet - INFO - Epoch [17][750/1597] lr: 0.02000, eta: 8:01:14, time: 0.834, data_time: 0.033, memory: 6915, loss_rpn_cls: 0.3746, loss_rpn_bbox: 0.0670, loss_cls: 0.1115, acc: 97.6489, loss_bbox: 0.0133, loss_mask: 0.6905, loss: 1.2568 2020-02-28 23:43:46,908 - mmdet - INFO - Epoch [17][800/1597] lr: 0.02000, eta: 8:00:23, time: 0.851, data_time: 0.045, memory: 6915, loss_rpn_cls: 0.3834, loss_rpn_bbox: 0.0692, loss_cls: 0.1144, acc: 97.5701, loss_bbox: 0.0138, loss_mask: 0.6918, loss: 1.2725 2020-02-28 23:44:29,771 - mmdet - INFO - Epoch [17][850/1597] lr: 0.02000, eta: 7:59:33, time: 0.857, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.3768, loss_rpn_bbox: 0.0666, loss_cls: 0.1259, acc: 97.6091, loss_bbox: 0.0379, loss_mask: 0.6913, loss: 1.2985 2020-02-28 23:45:12,013 - mmdet - INFO - Epoch [17][900/1597] lr: 0.02000, eta: 7:58:41, time: 0.845, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.3734, loss_rpn_bbox: 0.0655, loss_cls: 0.1141, acc: 97.5771, loss_bbox: 0.0147, loss_mask: 0.6886, loss: 1.2564 2020-02-28 23:45:53,451 - mmdet - INFO - Epoch [17][950/1597] lr: 0.02000, eta: 7:57:50, time: 0.829, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.3679, loss_rpn_bbox: 0.0650, loss_cls: 0.1133, acc: 97.5999, loss_bbox: 0.0158, loss_mask: 0.6912, loss: 1.2532 2020-02-28 23:46:34,998 - mmdet - INFO - Epoch [17][1000/1597] lr: 0.02000, eta: 7:56:58, time: 0.831, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.3787, loss_rpn_bbox: 0.0687, loss_cls: 0.1152, acc: 97.5464, loss_bbox: 0.0148, loss_mask: 0.6891, loss: 1.2665 2020-02-28 23:47:15,764 - mmdet - INFO - Epoch [17][1050/1597] lr: 0.02000, eta: 7:56:05, time: 0.815, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.3618, loss_rpn_bbox: 0.0626, loss_cls: 0.1080, acc: 97.7522, loss_bbox: 0.0141, loss_mask: 0.6911, loss: 1.2376 2020-02-28 23:47:57,786 - mmdet - INFO - Epoch [17][1100/1597] lr: 0.02000, eta: 7:55:14, time: 0.840, data_time: 0.033, memory: 6915, loss_rpn_cls: 0.3679, loss_rpn_bbox: 0.0649, loss_cls: 0.1119, acc: 97.6370, loss_bbox: 0.0142, loss_mask: 0.6915, loss: 1.2503 2020-02-28 23:48:39,630 - mmdet - INFO - Epoch [17][1150/1597] lr: 0.02000, eta: 7:54:23, time: 0.837, data_time: 0.033, memory: 6915, loss_rpn_cls: 0.3693, loss_rpn_bbox: 0.0666, loss_cls: 0.1136, acc: 97.5908, loss_bbox: 0.0146, loss_mask: 0.6891, loss: 1.2533 2020-02-28 23:49:21,353 - mmdet - INFO - Epoch [17][1200/1597] lr: 0.02000, eta: 7:53:31, time: 0.835, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.3704, loss_rpn_bbox: 0.0650, loss_cls: 0.1108, acc: 97.6685, loss_bbox: 0.0132, loss_mask: 0.6900, loss: 1.2494 2020-02-28 23:50:03,601 - mmdet - INFO - Epoch [17][1250/1597] lr: 0.02000, eta: 7:52:41, time: 0.845, data_time: 0.031, memory: 6915, loss_rpn_cls: 0.3668, loss_rpn_bbox: 0.0659, loss_cls: 0.1081, acc: 97.7375, loss_bbox: 0.0132, loss_mask: 0.6896, loss: 1.2436 2020-02-28 23:50:46,104 - mmdet - INFO - Epoch [17][1300/1597] lr: 0.02000, eta: 7:51:50, time: 0.850, data_time: 0.045, memory: 6915, loss_rpn_cls: 0.3716, loss_rpn_bbox: 0.0673, loss_cls: 0.1156, acc: 97.5381, loss_bbox: 0.0153, loss_mask: 0.6897, loss: 1.2594 2020-02-28 23:51:27,660 - mmdet - INFO - Epoch [17][1350/1597] lr: 0.02000, eta: 7:50:59, time: 0.831, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.3822, loss_rpn_bbox: 0.0689, loss_cls: 0.1141, acc: 97.5771, loss_bbox: 0.0133, loss_mask: 0.6891, loss: 1.2676 2020-02-28 23:52:08,417 - mmdet - INFO - Epoch [17][1400/1597] lr: 0.02000, eta: 7:50:06, time: 0.815, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.3673, loss_rpn_bbox: 0.0656, loss_cls: 0.1120, acc: 97.6343, loss_bbox: 0.0144, loss_mask: 0.6914, loss: 1.2506 2020-02-28 23:52:50,171 - mmdet - INFO - Epoch [17][1450/1597] lr: 0.02000, eta: 7:49:15, time: 0.835, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.3702, loss_rpn_bbox: 0.0650, loss_cls: 0.1166, acc: 97.5090, loss_bbox: 0.0155, loss_mask: 0.6923, loss: 1.2595 2020-02-28 23:53:32,175 - mmdet - INFO - Epoch [17][1500/1597] lr: 0.02000, eta: 7:48:24, time: 0.840, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.3677, loss_rpn_bbox: 0.0655, loss_cls: 0.1138, acc: 97.5862, loss_bbox: 0.0158, loss_mask: 0.6918, loss: 1.2547 2020-02-28 23:54:13,574 - mmdet - INFO - Epoch [17][1550/1597] lr: 0.02000, eta: 7:47:33, time: 0.828, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.3673, loss_rpn_bbox: 0.0646, loss_cls: 0.1089, acc: 97.7170, loss_bbox: 0.0132, loss_mask: 0.6889, loss: 1.2429
The text was updated successfully, but these errors were encountered:
i have sovle this problem, rotate_limit led to the problem
Sorry, something went wrong.
@whuhangzhang How did you set the rotate_limit and fix the problem? Set the rotate_limit=10, or other value?
Fix hyperparameter overloading in HPO for time series models (open-mm…
f44b138
…lab#2189)
No branches or pull requests
Checklist
Describe the bug
hello @tyomj @ternaus @hellock @yhcao6 ,when i use albu data augmentation like https://github.com/open-mmlab/mmdetection/blob/master/configs/albu_example/mask_rcnn_r50_fpn_1x.py.
I modified the original setting of albu and used the random 30 degree rotation and extended anchor ratio, but the loss will suddenly increase after the 15th epon, and this error also exists in htc_x101_32x4d_fpn without semantic, could you please fix this error?
【Note that I have modified the Settings in these places】
Reproduction
custom datasets (like coco )
Environment
Error traceback
If applicable, paste the error trackback here.
The text was updated successfully, but these errors were encountered: