Calling for volunteers for adding new metrics! #50

ice-tong · 2022-11-04T03:30:11Z

Dear community,

We are excited to introduce our new evaluation library: MMEval, a unified evaluation library for multiple machine learning libraries.

With the release of MMEval, we have some metrics from the OpenMMLab algorithm library that have not yet been added. We list the metrics below and call for community help.

If you are interested, you can claim a metric by replying to this issue in the following format:

Metric No. : <The Metric No. you claim>
Status :  Apply | Submit
Links : The repo links you fork (Apply status) / The PR links you open (Submit status)

No.	Metric	Algorithm library	Level of Difficulty	Assigned to	Status / PR
1	CityScapesMetric	mmdet / mmseg	⭐️⭐️⭐️	@Muyun99	Apply
2	CocoPanopticMetric	mmdet	⭐️⭐️⭐️	@Even-ok	Apply
3	LVISMetric	mmdet	⭐️⭐️	@dmucby	Apply
4	CrowdHumanMetric	mmdet	⭐️⭐️	@tianleiSHI	Apply
5	DOTAMetric	mmrotate	⭐️⭐️	@YanxingLiu	#65
6	RotatedCocoMetric	mmrotate	⭐️⭐️	@YanxingLiu
7	IndoorMetric	mmdet3d	⭐️⭐️⭐️
8	InstanceSegMetric	mmdet3d	⭐️⭐️⭐️	@Pzzzzz5142	Apply
9	KittiMetric	mmdet3d	⭐️⭐️⭐️
10	LyftMetric	mmdet3d	⭐️⭐️⭐️
11	NuScenesMetric	mmdet3d	⭐️⭐️⭐️
12	WaymoMetric	mmdet3d	⭐️⭐️⭐️
13	OneMinusNEDMetric	mmocr	⭐️⭐️
14	WordMetric	mmocr	⭐️⭐️
15	CharMetric	mmocr	⭐️⭐️
16	CocoWholeBodyMetric	mmpose	⭐️⭐️
17	NME	mmpose	⭐️⭐️
18	AUC	mmpose	⭐️⭐️
19	EPE	mmpose	⭐️⭐️
20	PoseTrack18Metric	mmpose	⭐️⭐️
21	ANetMetric	mmaction2	⭐️⭐️
22	AccMetric	mmaction2	⭐️⭐️
23	FlowOutliers	mmflow	⭐️⭐️	@fengsxy	Apply
24	MOTChallengeMetrics	mmtracking	⭐️⭐️
25	ReIDMetrics	mmtracking	⭐️⭐️
26	SOTMetric	mmtracking	⭐️⭐️
27	TAOMetric	mmtracking	⭐️⭐️
28	YouTubeVISMetric	mmtracking	⭐️⭐️	@qianlian-mozi	Apply
29	MultiScaleStructureSimilarity	mmediting	⭐️⭐️
30	FrechetInceptionDistance & TransFID	mmediting	⭐️⭐️
31	InceptionScore & TransIS	mmediting	⭐️⭐️
32	SAD	mmediting	⭐️⭐️	@xuan07472	#76
33	MattingMSE	mmediting	⭐️⭐️	@xuan07472	#71
34	ConnectivityError	mmediting	⭐️⭐️	@xuan07472	#79
35	GradientError	mmediting	⭐️⭐️	@xuan07472	#78
36	PerceptualPathLength	mmediting	⭐️⭐️
37	PrecisionAndRecall	mmediting	⭐️⭐️
38	SlicedWassersteinDistance	mmediting	⭐️⭐️
39	NIQE	mmediting	⭐️⭐️
40	Equivariance	mmediting	⭐️⭐️

NOTES:

Documents of MMEval can be found at: https://mmeval.readthedocs.io/en/latest/
The contributing guides can be found at: https://github.com/open-mmlab/mmeval/blob/main/CONTRIBUTING.md

Examples:

We provide some examples showing how to add Metric from the OpenMMLab algorithm library to MMEval

mmeval.Accuracy from mmcsl

PR in mmeval: [Feat] Add Accuracy #6
PR in mmcls: [Feature] Use mmeval.Accuracy mmpretrain#1046

mmeval.MeanIoU from mmseg

PR in mmeval: [Feat] Add MeanIoU #7
PR in mmseg: [WIP][Feat] Use mmeval.MeanIoU mmsegmentation#2003

mmeval.VOCMeanAP from mmdet

PR in mmeval: [Feature] Add VOCMeanAP and OIDMeanAP #8
PR in mmdet: Use MMEval VOCMeanAP and OIDMeanAP mmdetection#8897

mmeval.COCODetectionMetric from mmdet

PR in mmeval: [Feature] Add COCO Metric #23
PR in mmdet: [Refactor] Refactor COCOMetric, CocoOccludedSeparatedMetric, and ProposalRecallMetric by using MMEval mmdetection#9079

Submitted Content:

The PR in MMEval that adds the new metric (with complete documentation and testing)
The PR in original algorithm library that uses the new metric (with evaluation comparison between using MMEval and before)

No1. CityScapesMetric

Add the CityScapesMetric from mmdet / mmseg to MMEval, and use the MMEval's CityScapesMetric in mmdet / mmseg.
Technical Tags: Python; Object detection; Semantic segmentation;
The original CityScapesMetric in mmdet
The original CityScapesMetric in mmseg

No2. CocoPanopticMetric

Add the CocoPanopticMetric from mmdet to MMEval, and use the MMEval's CocoPanopticMetric in mmdet.
Technical Tags: Python; Object detection; Panoptic segmentation;
The original CocoPanopticMetric in mmdet

No3. LVISMetric

Add the LVISMetric from mmdet to MMEval, and use the MMEval's LVISMetric in mmdet.
Technical Tags: Python; Object detection;
The original LVISMetric in mmdet

No4. CrowdHumanMetric

Add the CrowdHumanMetric from mmdet to MMEval, and use the MMEval's CrowdHumanMetric in mmdet.
Technical Tags: Python; Object detection;
The original CrowdHumanMetric in mmdet

No5. DOTAMetric

Add the DOTAMetric from mmrotate to MMEval, and use the MMEval's DOTAMetric in mmrotate.
Technical Tags: Python; Rotated object detection;
The original DOTAMetric in mmrotate

No6. RotatedCocoMetric

Add the RotatedCocoMetric from mmrotate to MMEval, and use the MMEval's RotatedCocoMetric in mmrotate.
Technical Tags: Python; Rotated object detection;
The original RotatedCocoMetric in mmrotate

No7. IndoorMetric

Add the IndoorMetric from mmdet3d to MMEval, and use the MMEval's IndoorMetric in mmdet3d.
Technical Tags: Python; 3D object detection;
The original IndoorMetric in mmdet3d

No8. InstanceSegMetric

Add the InstanceSegMetric from mmdet3d to MMEval, and use the MMEval's InstanceSegMetric in mmdet3d.
Technical Tags: Python; 3D instance segmentation;
The original InstanceSegMetric in mmdet3d

No9. KittiMetric

Add the KittiMetric from mmdet3d to MMEval, and use the MMEval's KittiMetric in mmdet3d.
Technical Tags: Python; 3D object detection;
The original KittiMetric in mmdet3d

No10. LyftMetric

Add the LyftMetric from mmdet3d to MMEval, and use the MMEval's LyftMetric in mmdet3d.
Technical Tags: Python; 3D object detection;
The original LyftMetric in mmdet3d

No11. NuScenesMetric

Add the NuScenesMetric from mmdet3d to MMEval, and use the MMEval's NuScenesMetric in mmdet3d.
Technical Tags: Python; 3D object detection;
The original NuScenesMetric in mmdet3d

No12. WaymoMetric

Add the WaymoMetric from mmdet3d to MMEval, and use the MMEval's WaymoMetric in mmdet3d.
Technical Tags: Python; 3D object detection;
The original WaymoMetric in mmdet3d

No13. OneMinusNEDMetric

Add the OneMinusNEDMetric from mmocr to MMEval, and use the MMEval's OneMinusNEDMetric in mmocr.
Technical Tags: Python; OCR;
The original OneMinusNEDMetric in mmocr

No14. WordMetric

Add the WordMetric from mmocr to MMEval, and use the MMEval's WordMetric in mmocr.
Technical Tags: Python; OCR;
The original WordMetric in mmocr

No15. CharMetric

Add the CharMetric from mmocr to MMEval, and use the MMEval's CharMetric in mmocr.
Technical Tags: Python; OCR;
The original CharMetric in mmocr

No16. CocoWholeBodyMetric

Add the CocoWholeBodyMetric from mmpose to MMEval, and use the MMEval's CocoWholeBodyMetric in mmpose.
Technical Tags: Python; Pose estimation;
The original CocoWholeBodyMetric in mmpose

No17. NME

Add the NME from mmpose to MMEval, and use the MMEval's NME in mmpose.
Technical Tags: Python; Pose estimation;
The original NME in mmpose

No18. AUC

Add the AUC from mmpose to MMEval, and use the MMEval's AUC in mmpose.
Technical Tags: Python; Pose estimation;
The original AUC in mmpose

No19. EPE

Add the EPE from mmpose to MMEval, and use the MMEval's EPE in mmpose.
Technical Tags: Python; Pose estimation;
The original EPE in mmpose

No20. PoseTrack18Metric

Add the PoseTrack18Metric from mmpose to MMEval, and use the MMEval's PoseTrack18Metric in mmpose.
Technical Tags: Python; Pose estimation;
The original PoseTrack18Metric in mmpose

No21. ANetMetric

Add the ANetMetric from mmaction2 to MMEval, and use the MMEval's ANetMetric in mmaction2.
Technical Tags: Python; Video understanding;
The original ANetMetric in mmaction2

No22. AccMetric

Add the AccMetric from mmaction2 to MMEval, and use the MMEval's AccMetric in mmaction2.
Technical Tags: Python; Video understanding;
The original AccMetric in mmaction2

No23. FlowOutliers

Add the FlowOutliers from mmflow to MMEval, and use the MMEval's FlowOutliers in mmflow.
Technical Tags: Python; Optical flow;
The original FlowOutliers in mmflow

No24. MOTChallengeMetrics

Add the MOTChallengeMetrics from mmtracking to MMEval, and use the MMEval's MOTChallengeMetrics in mmtracking.
Technical Tags: Python; Multiple object tracking;
The original MOTChallengeMetrics in mmtracking

No25. ReIDMetrics

Add the ReIDMetrics from mmtracking to MMEval, and use the MMEval's ReIDMetrics in mmtracking.
Technical Tags: Python; Re-identification;
The original ReIDMetrics in mmtracking

No26. SOTMetric

Add the SOTMetric from mmtracking to MMEval, and use the MMEval's SOTMetric in mmtracking.
Technical Tags: Python; Single object tracking;
The original SOTMetric in mmtracking

No27. TAOMetric

Add the TAOMetric from mmtracking to MMEval, and use the MMEval's TAOMetric in mmtracking.
Technical Tags: Python; Object tracking;
The original TAOMetric in mmtracking

No28. YouTubeVISMetric

Add the YouTubeVISMetric from mmtracking to MMEval, and use the MMEval's YouTubeVISMetric in mmtracking.
Technical Tags: Python; Video instance segmentation;
The original YouTubeVISMetric in mmtracking

No29. MultiScaleStructureSimilarity

Add the MultiScaleStructureSimilarity from mmediting to MMEval, and use the MMEval's MultiScaleStructureSimilarity in mmediting.
Technical Tags: Python; GAN;
The original MultiScaleStructureSimilarity in mmediting

No30. FrechetInceptionDistance & TransFID

Add the FrechetInceptionDistance and TransFID from mmediting to MMEval, and use the MMEval's FrechetInceptionDistance and TransFID in mmediting.
Technical Tags: Python; GAN;
The original FrechetInceptionDistance in mmediting
The original TransFID in mmediting

No31. InceptionScore & TransIS

Add the InceptionScore and TransIS from mmediting to MMEval, and use the MMEval's InceptionScore and TransIS in mmediting.
Technical Tags: Python; GAN;
The original InceptionScore in mmediting
The original TransIS in mmediting

No32. SAD

Add the SAD from mmediting to MMEval, and use the MMEval's SAD in mmediting.
Technical Tags: Python; GAN;
The original SAD in mmediting

No33. MattingMSE

Add the MattingMSE from mmediting to MMEval, and use the MMEval's MattingMSE in mmediting.
Technical Tags: Python; Matting;
The original MattingMSE in mmediting

No34. ConnectivityError

Add the ConnectivityError from mmediting to MMEval, and use the MMEval's ConnectivityError in mmediting.
Technical Tags: Python; Matting;
The original ConnectivityError in mmediting

No35. GradientError

Add the GradientError from mmediting to MMEval, and use the MMEval's GradientError in mmediting.
Technical Tags: Python; Matting;
The original GradientError in mmediting

No36. PerceptualPathLength

Add the PerceptualPathLength from mmediting to MMEval, and use the MMEval's PerceptualPathLength in mmediting.
Technical Tags: Python; GAN;
The original PerceptualPathLength in mmediting

No37. PrecisionAndRecall

Add the PrecisionAndRecall from mmediting to MMEval, and use the MMEval's PrecisionAndRecall in mmediting.
Technical Tags: Python; GAN;
The original PrecisionAndRecall in mmediting

No38. SlicedWassersteinDistance

Add the SlicedWassersteinDistance from mmediting to MMEval, and use the MMEval's SlicedWassersteinDistance in mmediting.
Technical Tags: Python; GAN;
The original SlicedWassersteinDistance in mmediting

No39. NIQE

Add the NIQE from mmediting to MMEval, and use the MMEval's NIQE in mmediting.
Technical Tags: Python; GAN;
The original NIQE in mmediting

No40. Equivariance

Add the Equivariance from mmediting to MMEval, and use the MMEval's Equivariance in mmediting.
Technical Tags: Python; GAN;
The original Equivariance in mmediting

The text was updated successfully, but these errors were encountered:

Muyun99 · 2022-11-17T14:07:19Z

Metric No.1 CityScapesMetric
Status : Apply
Links : https://github.com/Muyun99/mmeval

YanxingLiu · 2022-11-29T02:47:55Z

Metric No.5 DOTAMetric
Status : Apply
Links : https://github.com/YanxingLiu/mmeval.git

fengsxy · 2022-12-08T02:52:33Z

Metric No.24 FlowOutliers
Status : Apply
Links : https://github.com/fengsxy/mmeval

dmucby · 2022-12-09T08:45:08Z

Metric No.3 LVISMetric
Status : Apply
Links : https://github.com/dmucby/mmeval

Even-ok · 2022-12-14T07:32:23Z

Metric No.2 CocoPanopticMetric
Status : Apply
Links : https://github.com/Even-ok/mmeval.git

xuan07472 · 2022-12-19T14:03:14Z

Metric No.33 MattingMSE
Status : Apply
Links : https://github.com/xuan07472/mmeval

tianleiSHI · 2022-12-20T07:31:13Z

Metric No.4 CrowdHumanMetric
Status: Apply
Links: https://github.com/tianleiSHI/mmeval

Pzzzzz5142 · 2022-12-27T14:19:09Z

Metric No.8 InstanceSegMetric
Status: Apply
Links: https://github.com/Pzzzzz5142/mmeval

xuan07472 · 2023-01-09T07:14:23Z

Metric No.32 SAD
Status : Apply
Links : https://github.com/xuan07472/mmeval

xuan07472 · 2023-01-11T05:47:50Z

Metric No.33 MattingMSE
Status : Submit
Links : #71

xuan07472 · 2023-01-11T05:51:13Z

Metric No.32 SAD
Status : Submit
Links : #76

YanxingLiu · 2023-01-11T06:07:46Z

Metric No.5 DOTAMetric
Status : Submit
Links : #65

qianlian-mozi · 2023-01-12T10:57:33Z

Metric No.28 : YouTubeVISMetric
Status : Apply
Links : https://github.com/qianlian-mozi/mmeval

xuan07472 · 2023-01-12T11:05:17Z

Metric No.35 GradientError
Status : Apply
Links : https://github.com/xuan07472/mmeval

xuan07472 · 2023-01-13T10:17:21Z

Metric No.35 GradientError
Status : Submit
Links : #78

xuan07472 · 2023-01-13T10:18:47Z

Metric No.34 ConnectivityError
Status : Apply
Links : https://github.com/xuan07472/mmeval

xuan07472 · 2023-01-15T13:03:06Z

Metric No.34 ConnectivityError
Status : Submit
Links : #79

YanxingLiu · 2023-01-31T08:56:29Z

Metric No.6 RotatedCocoMetric
Satus: Apply
Links: https://github.com/YanxingLiu/mmeval.git

zhouzaida pinned this issue Nov 7, 2022

This was referenced Nov 17, 2022

[Attention] 超级视客营 MMEval 🥇🥈🥉 #54

Closed

[Attention] OpenMMLab Codecamp 🥇🥈🥉 #55

Closed

zytx121 mentioned this issue Dec 2, 2022

[Attention] OpenMMLab Codecamp open-mmlab/mmrotate#626

Open

zhouzaida closed this as completed Feb 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Calling for volunteers for adding new metrics! #50

Calling for volunteers for adding new metrics! #50

ice-tong commented Nov 4, 2022 •

edited

Loading

Muyun99 commented Nov 17, 2022

YanxingLiu commented Nov 29, 2022 •

edited

Loading

fengsxy commented Dec 8, 2022

dmucby commented Dec 9, 2022

Even-ok commented Dec 14, 2022 •

edited

Loading

xuan07472 commented Dec 19, 2022 •

edited

Loading

tianleiSHI commented Dec 20, 2022

Pzzzzz5142 commented Dec 27, 2022

xuan07472 commented Jan 9, 2023

xuan07472 commented Jan 11, 2023 •

edited by ice-tong

Loading

xuan07472 commented Jan 11, 2023

YanxingLiu commented Jan 11, 2023 •

edited by ice-tong

Loading

qianlian-mozi commented Jan 12, 2023 •

edited by ice-tong

Loading

xuan07472 commented Jan 12, 2023 •

edited

Loading

xuan07472 commented Jan 13, 2023

xuan07472 commented Jan 13, 2023 •

edited

Loading

xuan07472 commented Jan 15, 2023 •

edited

Loading

YanxingLiu commented Jan 31, 2023

Calling for volunteers for adding new metrics! #50

Calling for volunteers for adding new metrics! #50

Comments

ice-tong commented Nov 4, 2022 • edited Loading

NOTES:

Examples:

Submitted Content:

No1. CityScapesMetric

No2. CocoPanopticMetric

No3. LVISMetric

No4. CrowdHumanMetric

No5. DOTAMetric

No6. RotatedCocoMetric

No7. IndoorMetric

No8. InstanceSegMetric

No9. KittiMetric

No10. LyftMetric

No11. NuScenesMetric

No12. WaymoMetric

No13. OneMinusNEDMetric

No14. WordMetric

No15. CharMetric

No16. CocoWholeBodyMetric

No17. NME

No18. AUC

No19. EPE

No20. PoseTrack18Metric

No21. ANetMetric

No22. AccMetric

No23. FlowOutliers

No24. MOTChallengeMetrics

No25. ReIDMetrics

No26. SOTMetric

No27. TAOMetric

No28. YouTubeVISMetric

No29. MultiScaleStructureSimilarity

No30. FrechetInceptionDistance & TransFID

No31. InceptionScore & TransIS

No32. SAD

No33. MattingMSE

No34. ConnectivityError

No35. GradientError

No36. PerceptualPathLength

No37. PrecisionAndRecall

No38. SlicedWassersteinDistance

No39. NIQE

No40. Equivariance

Muyun99 commented Nov 17, 2022

YanxingLiu commented Nov 29, 2022 • edited Loading

fengsxy commented Dec 8, 2022

dmucby commented Dec 9, 2022

Even-ok commented Dec 14, 2022 • edited Loading

xuan07472 commented Dec 19, 2022 • edited Loading

tianleiSHI commented Dec 20, 2022

Pzzzzz5142 commented Dec 27, 2022

xuan07472 commented Jan 9, 2023

xuan07472 commented Jan 11, 2023 • edited by ice-tong Loading

xuan07472 commented Jan 11, 2023

YanxingLiu commented Jan 11, 2023 • edited by ice-tong Loading

qianlian-mozi commented Jan 12, 2023 • edited by ice-tong Loading

xuan07472 commented Jan 12, 2023 • edited Loading

xuan07472 commented Jan 13, 2023

xuan07472 commented Jan 13, 2023 • edited Loading

xuan07472 commented Jan 15, 2023 • edited Loading

YanxingLiu commented Jan 31, 2023

ice-tong commented Nov 4, 2022 •

edited

Loading

YanxingLiu commented Nov 29, 2022 •

edited

Loading

Even-ok commented Dec 14, 2022 •

edited

Loading

xuan07472 commented Dec 19, 2022 •

edited

Loading

xuan07472 commented Jan 11, 2023 •

edited by ice-tong

Loading

YanxingLiu commented Jan 11, 2023 •

edited by ice-tong

Loading

qianlian-mozi commented Jan 12, 2023 •

edited by ice-tong

Loading

xuan07472 commented Jan 12, 2023 •

edited

Loading

xuan07472 commented Jan 13, 2023 •

edited

Loading

xuan07472 commented Jan 15, 2023 •

edited

Loading