Add metric MultiManager and MetricPlotter #811

rcgorman-dstl · 2023-06-01T16:06:31Z

Have replaced SimpleManager metric manager with new MultiManager class so this will be a breaking change.
MultiManager can take multiple metric generators of the same type and multiple sets of the same type to generate metrics.
All metrics that take in track and truth can now also take in 2 tracks.
Additional functionality added to MultiManager through new methods to help easier extraction of metrics and insights.
New MetricPlotter class created to make plotting metrics easier.

Update:

Have added SimpleManager metric manager back in to avert breaking change. It should operate in the same way as before.

nperree-dstl

Seems to work really well, comments are mostly documentation issues. I think you also need to remove the old SimpleManager from test_pcrbmetric and test_uncertaintymetric to stop tests failing.

docs/examples/Metrics.py

stonesoup/metricgenerator/ospametric.py

stonesoup/plotter.py

…racks input to SimpleManager

…nager class with no changes yet.

…ack-truth metrics. change inputs to MultiManager to dicts instead of sets

… merge them together

…e_metric() method to switch if statements

…ultiManager

… from SimpleManager

add new plot_metrics method to Plotter class. add new plot_timeseries parameter to Plotter.__init__ which defaults to False and sets fig axes to 'equal'. must be set to True to display axes correctly when plotting timeseries data including metrics.

rcgorman-dstl

Adding overall comments for each file to summarise changes

rcgorman-dstl · 2023-07-05T08:26:01Z

docs/tutorials/sensormanagement/01_SingleSensorManagement.py

update single sensor management tutorial to work with updated metrics, MultiManager, and MetricPlotter

rcgorman-dstl · 2023-07-05T08:26:14Z

docs/tutorials/sensormanagement/02_MultiSensorManagement.py

update multi sensor management tutorial to work with updated metrics, MultiManager, and MetricPlotter

rcgorman-dstl · 2023-07-05T08:26:27Z

docs/tutorials/sensormanagement/03_OptimisedSensorManagement.py

update optimised sensor management tutorial to work with updated metrics, MultiManager, and MetricPlotter

rcgorman-dstl · 2023-07-05T08:26:45Z

stonesoup/dataassociator/base.py

correct typos in DataAssociator base class

rcgorman-dstl · 2023-07-05T08:27:05Z

stonesoup/metricgenerator/base.py

correct typos in MetricGenerator base class

rcgorman-dstl · 2023-07-05T08:53:44Z

docs/examples/Metrics.py

new Metrics example to show updated metrics, MultiManager, and MetricPlotter

rcgorman-dstl · 2023-07-05T08:54:00Z

docs/examples/Multi_Tracker_Example.py

update Multitracker example to work with updated metrics, MultiManager, and MetricPlotter

rcgorman-dstl · 2023-07-05T08:54:11Z

docs/examples/ParticleFlow.py

update Gromov Particle flow example to work with updated metrics, MultiManager, and MetricPlotter

rcgorman-dstl · 2023-07-05T08:54:22Z

docs/examples/Track2Track_Fusion_Example.py

update multi-sensor fusion example to work with updated metrics, MultiManager, and MetricPlotter

rcgorman-dstl · 2023-07-05T08:54:35Z

docs/examples/Track_Stitcher_Example.py

update track stitching example to work with updated metrics, MultiManager, and MetricPlotter

…metric test

jswright-dstl · 2023-07-13T17:23:55Z

docs/examples/MTT_3D_Platform.py

-           ylabel="Sum of covariance matrix norms")
+fig3 = MetricPlotter()
+fig3.plot_metrics(metrics, generator_names=['Uncertainty metric'])
+fig3.fig.axes[0].set_title(label='Track uncertainty over time')


Could you add set_title as a MetricPlotter method? This will allow a user not to have to worry about "fig.axes[0]" and allows the MetricPlotter to be extended to plotly in the future.

docs/examples/Metrics.py

Co-authored-by: James Wright <69153443+jswright-dstl@users.noreply.github.com>

minor additions/corrections in metrics example Co-authored-by: James Wright <69153443+jswright-dstl@users.noreply.github.com>

…ration

…ne-Soup into add-metric-multimanager

sdhiscocks · 2023-08-09T11:10:05Z

The breaking change makes this difficult to merge at this point. However, it may be possible to make it backwards compatible fairly easy:

Add back in the SimpleManager, inheriting from MultiManager, with "tracks", "groundtruth_paths" and "detections" as default keys in states_set.
Have property on SimpleManager for tracks, groundtruth_paths and detections that returns values from those dictionary keys mentioned above.
Change metrics so default for tracks_key and truths_key are "tracks" and "groundtruth_paths".
Also provide sensible default generator_name for each metric.

I've not tried this, but something like this should work or at least be a good start.

sdhiscocks · 2023-08-10T13:49:09Z

stonesoup/metricgenerator/manager.py

+        generators = self.generators if isinstance(self.generators, list) else [self.generators]
+
+        for generator in generators:
+            if isinstance(generator, SIAPMetrics):


Referencing to specific metric here doesn't seem appropriate. Why does this need to be done?

I believe I added it in as an attempt to save unnecessary computation but not entirely sure. I think it can be removed easily. Will do so and test.

edit: nope, was to avoid an error. now updated with improved code, no hard-coding.

sdhiscocks · 2023-08-10T13:50:06Z

stonesoup/metricgenerator/manager.py

+    def display_basic_metrics(self):
+        """
+        Print basic metrics generated for each :class:`BasicMetrics` generator.
+        """
+        for generator_name, generator in self.metrics.items():
+            if "Number of targets" in generator.keys():
+                print(f'\nGenerator: {generator_name}')
+                for metric_key, metric in generator.items():
+                    if isinstance(metric.generator, BasicMetrics):
+                        print(f"{metric.title}: {metric.value}")
+
+    def get_siap_averages(self, generator_name):
+        """
+        Get SIAP averages metrics from SIAP metric generator specified by generator_name
+
+        Returns
+        ----------
+        : dict of :class`SIAPMetrics` averages
+        """
+        siap_metrics = self.metrics[generator_name]
+        siap_averages = {siap_metrics.get(metric) for metric in siap_metrics
+                         if metric.startswith("SIAP") and not metric.endswith(" at times")}
+
+        return siap_averages


These methods also seem specific to the name or specific metric. A different approach is needed here.

Had to remove the methods as they're too specific to each metric type.

update documentation of MetricManager and Metrics to include information about input parameters for different methods. Also remove hard-coded reference to SIAP metrics in associate_tracks method of MetricManager as requested in review.

Add SimpleManager MetricManager class back in and make backwards compatible with new updates to all Metrics Generators related to MultiManager. Update MultiManager parent class to work with SimpleManager and return data from SimpleManager in original format.

allow simplemanager add_data method to add data with any number of kwargs instead of requiring all kwargs.

add simplemanager tests back into test_manager and update all metrics tests files so that they use default generator_name, tracks_key, and truths_key arguments where relevant now that default values are available.

update indentation in manager.py and fix compute_metric method of abstract dummy metric generator class in test_manager.

remove methods from MultiManager and instances of use in tutorials/examples to avoid hard-coding any references to specific MetricGenerators in MetricManager code.

- MultiManager can take multiple metric generators of the same type and multiple sets of the same type to generate metrics. - Have made SimpleManager metric manager subclass of MultiManager class so this will be a breaking change. - All metrics that take in track and truth can now also take in 2 tracks. - Additional functionality added to MultiManager through new methods to help easier extraction of metrics and insights. - New MetricPlotter class created to make plotting metrics easier.

rcgorman-dstl added enhancement documentation breaking A breaking change that required other to modify their code labels Jun 1, 2023

rcgorman-dstl requested review from jswright-dstl and nperree-dstl June 1, 2023 16:06

rcgorman-dstl requested a review from a team as a code owner June 1, 2023 16:06

nperree-dstl reviewed Jun 23, 2023

View reviewed changes

rcgorman-dstl added 23 commits July 3, 2023 16:42

Correct typos. Restore manager.py to original. Remove second set of t…

a32b8d1

…racks input to SimpleManager

add notes for updates to manager.py and ospametric.py. create MultiMa…

2eff70f

…nager class with no changes yet.

update compute_metric() return statement to compute track-track or tr…

82d01c8

…ack-truth metrics. change inputs to MultiManager to dicts instead of sets

update add_data() method for MultiManager to take dicts as inputs and…

b8ae8e8

… merge them together

update variable names and doc strings

7be15b7

update imports so that manager.py imports successfully. update comput…

9474712

…e_metric() method to switch if statements

remove extra blank line

2dc361b

update generate_metrics() to generate and store metrics in dict for M…

f30230f

…ultiManager

add generator_name parameter to OSPA metric class

0c2302e

remove duplicate methods from MultiManager class - they are inherited…

579fe3a

… from SimpleManager

fix typos and errors in plotter documentation

7e22a1d

correct documentation for TwoDPlotter class

6b47a68

update and clean documentation for MultiManager

b4b7076

remove notes to self from code

1813a5f

add raise error if generator_name attribute missing from generators

e8f5fc4

update docstrings for ospametric

b972497

correct documentation for MultiManager

5db1068

create new MetricPlotter class with plot_metrics method

a1f951d

update ospametric to take groundtruth and tracks keys

0367086

update new ospametric property names to tracks_keys and groundtruth_key

3728cd1

fix bugs in ospametric related to keys

2c795d0

update multimanager add_data to take 1 dict as arg

7bd0ce2

update multi tracker example to work with multimanager

3b9ee4c

rcgorman-dstl commented Jul 5, 2023

View reviewed changes

correct typo in Track2Track fusion example and delete comment in pcrb…

f14f63a

…metric test

nperree-dstl approved these changes Jul 5, 2023

View reviewed changes

jswright-dstl approved these changes Jul 14, 2023

View reviewed changes

rcgorman-dstl and others added 7 commits August 8, 2023 13:42

add set_title methods to metric plotter and update MTT example

b0ba4fc

update MTT 3D and Metrics examples with new fig and ax title methods

9c0e3ba

Update docs/examples/Metrics.py

254d82e

Co-authored-by: James Wright <69153443+jswright-dstl@users.noreply.github.com>

Update docs/examples/Metrics.py

d400277

Co-authored-by: James Wright <69153443+jswright-dstl@users.noreply.github.com>

Apply suggestions from code review

1d894a3

minor additions/corrections in metrics example Co-authored-by: James Wright <69153443+jswright-dstl@users.noreply.github.com>

update Metrics example with clarification on track-track metrics gene…

c592971

…ration

Merge branch 'add-metric-multimanager' of https://github.com/dstl/Sto…

b0063b1

…ne-Soup into add-metric-multimanager

jswright-dstl approved these changes Aug 8, 2023

View reviewed changes

sdhiscocks removed enhancement documentation labels Aug 9, 2023

sdhiscocks reviewed Aug 10, 2023

View reviewed changes

rcgorman-dstl added 4 commits August 10, 2023 15:58

remove hard code in MetricManager associate_tracks

e01a8e2

update documentation of MetricManager and Metrics to include information about input parameters for different methods. Also remove hard-coded reference to SIAP metrics in associate_tracks method of MetricManager as requested in review.

fix bug in SimpleManager add_data method

10c2c3c

allow simplemanager add_data method to add data with any number of kwargs instead of requiring all kwargs.

add simplemanager tests to test_manager.py

e97710f

add simplemanager tests back into test_manager and update all metrics tests files so that they use default generator_name, tracks_key, and truths_key arguments where relevant now that default values are available.

rcgorman-dstl removed the breaking A breaking change that required other to modify their code label Aug 11, 2023

rcgorman-dstl added 4 commits August 11, 2023 16:51

fix flake8 error in manager.py and bug in test_manager.py

9655bc2

update indentation in manager.py and fix compute_metric method of abstract dummy metric generator class in test_manager.

remove MultiManager display_basic_metrics and get_siap_averages methods

dc182c2

remove methods from MultiManager and instances of use in tutorials/examples to avoid hard-coding any references to specific MetricGenerators in MetricManager code.

fix flake8 error in manager.py

6e6eb46

fix reference error in Metrics example

0ee7ea8

sdhiscocks added the enhancement label Sep 1, 2023

sdhiscocks merged commit f24090c into main Sep 1, 2023
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add metric MultiManager and MetricPlotter #811

Add metric MultiManager and MetricPlotter #811

rcgorman-dstl commented Jun 1, 2023 •

edited

Loading

nperree-dstl left a comment

rcgorman-dstl left a comment

rcgorman-dstl Jul 5, 2023

rcgorman-dstl Jul 5, 2023

rcgorman-dstl Jul 5, 2023

rcgorman-dstl Jul 5, 2023

rcgorman-dstl Jul 5, 2023

rcgorman-dstl Jul 5, 2023

rcgorman-dstl Jul 5, 2023

rcgorman-dstl Jul 5, 2023

rcgorman-dstl Jul 5, 2023

rcgorman-dstl Jul 5, 2023

jswright-dstl Jul 13, 2023

sdhiscocks commented Aug 9, 2023

sdhiscocks Aug 10, 2023

rcgorman-dstl Aug 10, 2023 •

edited

Loading

sdhiscocks Aug 10, 2023

rcgorman-dstl Aug 21, 2023

Add metric MultiManager and MetricPlotter #811

Add metric MultiManager and MetricPlotter #811

Conversation

rcgorman-dstl commented Jun 1, 2023 • edited Loading

nperree-dstl left a comment

Choose a reason for hiding this comment

rcgorman-dstl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdhiscocks commented Aug 9, 2023

Choose a reason for hiding this comment

rcgorman-dstl Aug 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rcgorman-dstl commented Jun 1, 2023 •

edited

Loading

rcgorman-dstl Aug 10, 2023 •

edited

Loading