Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metric MultiManager and MetricPlotter #811

Merged
merged 117 commits into from
Sep 1, 2023
Merged
Show file tree
Hide file tree
Changes from 102 commits
Commits
Show all changes
117 commits
Select commit Hold shift + click to select a range
a32b8d1
Correct typos. Restore manager.py to original. Remove second set of t…
rcgorman-dstl Apr 12, 2023
2eff70f
add notes for updates to manager.py and ospametric.py. create MultiMa…
rcgorman-dstl Apr 20, 2023
82d01c8
update compute_metric() return statement to compute track-track or tr…
rcgorman-dstl Apr 20, 2023
b8ae8e8
update add_data() method for MultiManager to take dicts as inputs and…
rcgorman-dstl Apr 20, 2023
7be15b7
update variable names and doc strings
rcgorman-dstl Apr 20, 2023
9474712
update imports so that manager.py imports successfully. update comput…
rcgorman-dstl Apr 20, 2023
2dc361b
remove extra blank line
rcgorman-dstl Apr 20, 2023
f30230f
update generate_metrics() to generate and store metrics in dict for M…
rcgorman-dstl Apr 21, 2023
0c2302e
add generator_name parameter to OSPA metric class
rcgorman-dstl Apr 21, 2023
579fe3a
remove duplicate methods from MultiManager class - they are inherited…
rcgorman-dstl Apr 21, 2023
7e22a1d
fix typos and errors in plotter documentation
rcgorman-dstl May 1, 2023
a9aba24
add plot_metrics method to stonesoup/plotter.py
rcgorman-dstl May 2, 2023
6b47a68
correct documentation for TwoDPlotter class
rcgorman-dstl May 2, 2023
b4b7076
update and clean documentation for MultiManager
rcgorman-dstl May 2, 2023
1813a5f
remove notes to self from code
rcgorman-dstl May 2, 2023
e8f5fc4
add raise error if generator_name attribute missing from generators
rcgorman-dstl May 3, 2023
b972497
update docstrings for ospametric
rcgorman-dstl May 3, 2023
5db1068
correct documentation for MultiManager
rcgorman-dstl May 4, 2023
a1f951d
create new MetricPlotter class with plot_metrics method
rcgorman-dstl May 4, 2023
0367086
update ospametric to take groundtruth and tracks keys
rcgorman-dstl May 5, 2023
3728cd1
update new ospametric property names to tracks_keys and groundtruth_key
rcgorman-dstl May 5, 2023
2c795d0
fix bugs in ospametric related to keys
rcgorman-dstl May 5, 2023
7bd0ce2
update multimanager add_data to take 1 dict as arg
rcgorman-dstl May 5, 2023
fbc1026
change basicmetrics to work with multimanager
rcgorman-dstl May 5, 2023
264a636
update multimanager to work with basic metrics
rcgorman-dstl May 5, 2023
7b80610
correct ospametric groundtruths_key spelling
rcgorman-dstl May 5, 2023
41773d9
update GOSPA metric to work with multimanager
rcgorman-dstl May 5, 2023
c4dfb3e
update SIAP metrics to work with multimanager
rcgorman-dstl May 8, 2023
0cb7dae
update list_timestamps and associate_tracks to work with multimanager
rcgorman-dstl May 8, 2023
36cf85f
update groundtruth_keys to truths_keys in basic and ospa metrics
rcgorman-dstl May 8, 2023
7b1e144
update IDSIAPMetrics to work with MultiManager
rcgorman-dstl May 8, 2023
4ae3cad
update PCRB metric to work with multimanager
rcgorman-dstl May 8, 2023
121a6ed
delete unnecessary code - update PCRB metric to work with multimanager
rcgorman-dstl May 8, 2023
3c8f927
delete unnecessary code again - update PCRB metric to work with multi…
rcgorman-dstl May 8, 2023
7ffb267
update _CovarianceNormsMetric to work with multimanager
rcgorman-dstl May 8, 2023
5117684
add _get_data staticmethod to MetricGenerator base
rcgorman-dstl May 8, 2023
8a7fe87
update multimanager generate_metrics to create dict output
rcgorman-dstl May 8, 2023
153aeb4
update metric generator plotter to work with multimanager
rcgorman-dstl May 8, 2023
e2fc1df
correct raised error type for no generator_name
rcgorman-dstl May 16, 2023
1949f38
update to work only with multimanager and not simplemanager
rcgorman-dstl May 18, 2023
9676a42
update basic metrics to produce track-to-track scores
rcgorman-dstl May 18, 2023
2cde7bd
update generator_name documentation to show unique name required
rcgorman-dstl May 18, 2023
4e74c30
update siaps to produce track-to-track scores
rcgorman-dstl May 18, 2023
b8664fe
set tracks_key argument to only take one string for all metrics
rcgorman-dstl May 19, 2023
d8aa4dc
add methods to plot metrics on either separate or same plots
rcgorman-dstl May 19, 2023
dbab180
update metric plotter to plot metrics separate or individually
rcgorman-dstl May 22, 2023
bc83ace
fix bugs in plotting individually and separately
rcgorman-dstl May 22, 2023
2007eba
update method names and add graph labels to combine_plots
rcgorman-dstl May 22, 2023
410deca
update and clean MetricPlotter documentation
rcgorman-dstl May 23, 2023
8ef3be0
fix graph colour bug
rcgorman-dstl May 23, 2023
419d811
clean create generator_names list code
rcgorman-dstl May 23, 2023
66a45ba
remove SimpleManager
rcgorman-dstl May 23, 2023
eb708e4
add display_basic_metrics method
rcgorman-dstl May 25, 2023
071c481
correct formatting
rcgorman-dstl May 25, 2023
5110afa
update plotter metricgenerator to show title and tracks label
rcgorman-dstl May 26, 2023
820e4ac
update display_basic_metrics method to add generator name label
rcgorman-dstl May 26, 2023
70e32f5
return self.metrics from generate_metrics
rcgorman-dstl May 26, 2023
48d4d1d
add new get_siap_averages method and fix SIAP generating bug
rcgorman-dstl May 31, 2023
a91e3a0
add default colour to plotting, update plottable metrics to incl cov …
rcgorman-dstl May 31, 2023
272339d
update longest track segment documentation and target value
rcgorman-dstl Jun 1, 2023
e568023
update longest track segment length
rcgorman-dstl Jun 1, 2023
0db2488
fix colouring bug in metric plotter
rcgorman-dstl Jun 1, 2023
78cdd41
update metrics example documentation
rcgorman-dstl Jun 1, 2023
85e08d9
correct typo
rcgorman-dstl Jun 1, 2023
ccdc096
delete blank space
rcgorman-dstl Jun 1, 2023
b836b84
fix error in circleCI test 310
rcgorman-dstl Jun 2, 2023
c806860
add metric_names argument to plot_metrics
rcgorman-dstl Jun 2, 2023
ac91b9d
demonstrate metric_names arg in MetricPlotter example
rcgorman-dstl Jun 2, 2023
9a94d0c
fix ParticleFlow.py circleCI doc fail
rcgorman-dstl Jun 2, 2023
275f1fd
fix 03_optimisedSensorManagement circleCI docs fail
rcgorman-dstl Jun 2, 2023
3eda813
fix 01_singleSensorManagement docs fail
rcgorman-dstl Jun 2, 2023
8d12538
fix 02_MultiSensorManagement docs fail
rcgorman-dstl Jun 5, 2023
bb64eb1
fix introduced error in 03_OptimisedSensorManagement
rcgorman-dstl Jun 5, 2023
db3b147
fix MTT_3D_Platform example docs fail
rcgorman-dstl Jun 5, 2023
87f946b
fix Track2Track_Fusion_Example docs fail
rcgorman-dstl Jun 5, 2023
1f71a46
additional corrections on Track2Track_Fusion_Example text
rcgorman-dstl Jun 5, 2023
c03b760
fix basicmetrics tests to work with multimanager
rcgorman-dstl Jun 5, 2023
1f29d93
update test_manager to test multimanager
rcgorman-dstl Jun 5, 2023
3b8dc53
update multimanager to convert all states set to sets
rcgorman-dstl Jun 5, 2023
228fc8b
update ospametric tests to work with multimanager
rcgorman-dstl Jun 5, 2023
1be11bd
update pcrbmetric tests to work with multimanager
rcgorman-dstl Jun 5, 2023
de909dc
delete unnecessary code from test_manager.py
rcgorman-dstl Jun 26, 2023
ddfa7c5
update SIAP tables tests to work with multimanager
rcgorman-dstl Jun 26, 2023
1ba1998
update tracktotruthmetrics tests to work with multimanager
rcgorman-dstl Jun 26, 2023
b2e21ce
fix Flake8 errors
rcgorman-dstl Jun 27, 2023
d85f137
fix plotter tests to work with multimanager
rcgorman-dstl Jun 27, 2023
0382ab5
update pcrb metric to work with single ground truth path
rcgorman-dstl Jun 27, 2023
876de1e
fix pcrb metric tests to work with multimanager
rcgorman-dstl Jun 27, 2023
532aff4
change plotter to make tracks/truths/detections optional
rcgorman-dstl Jun 27, 2023
e1cdb97
minor corrections in metrics example
rcgorman-dstl Jun 28, 2023
6962d98
correct formatting of example
rcgorman-dstl Jun 27, 2023
610cf9b
fix PEP8 formatting in metrics example
rcgorman-dstl Jun 28, 2023
1228c1e
fix uncertaintymetric tests to work with multimanager
rcgorman-dstl Jun 28, 2023
a4569e3
attempt unexpected indentation error in metrics example
rcgorman-dstl Jun 29, 2023
5b6c798
fix circleCI indentation docstrings errors for manager and SIAPs
rcgorman-dstl Jun 29, 2023
6667de6
add uncertainty ellipses to plots in example
rcgorman-dstl Jun 30, 2023
822d2ad
add documentation to MetricPlotter methods
rcgorman-dstl Jun 30, 2023
fb50bd3
fix Flake8 errors in test_pcrb and stonesoup/plotter
rcgorman-dstl Jul 3, 2023
c7ad47a
update track stitcher example to work with MultiManager
rcgorman-dstl Jul 4, 2023
4c8bbbc
update multi-target tracking example to work with multimanager
rcgorman-dstl Jul 4, 2023
3b9ee4c
update multi tracker example to work with multimanager
rcgorman-dstl Jul 4, 2023
f14f63a
correct typo in Track2Track fusion example and delete comment in pcrb…
rcgorman-dstl Jul 5, 2023
b0ba4fc
add set_title methods to metric plotter and update MTT example
rcgorman-dstl Aug 8, 2023
9c0e3ba
update MTT 3D and Metrics examples with new fig and ax title methods
rcgorman-dstl Aug 8, 2023
254d82e
Update docs/examples/Metrics.py
rcgorman-dstl Aug 8, 2023
d400277
Update docs/examples/Metrics.py
rcgorman-dstl Aug 8, 2023
1d894a3
Apply suggestions from code review
rcgorman-dstl Aug 8, 2023
c592971
update Metrics example with clarification on track-track metrics gene…
rcgorman-dstl Aug 8, 2023
b0063b1
Merge branch 'add-metric-multimanager' of https://github.com/dstl/Sto…
rcgorman-dstl Aug 8, 2023
e01a8e2
remove hard code in MetricManager associate_tracks
rcgorman-dstl Aug 10, 2023
d3b63e4
add SimpleManager and make backwards compatible
rcgorman-dstl Aug 10, 2023
10c2c3c
fix bug in SimpleManager add_data method
rcgorman-dstl Aug 11, 2023
e97710f
add simplemanager tests to test_manager.py
rcgorman-dstl Aug 11, 2023
9655bc2
fix flake8 error in manager.py and bug in test_manager.py
rcgorman-dstl Aug 11, 2023
dc182c2
remove MultiManager display_basic_metrics and get_siap_averages methods
rcgorman-dstl Aug 18, 2023
6e6eb46
fix flake8 error in manager.py
rcgorman-dstl Aug 18, 2023
0ee7ea8
fix reference error in Metrics example
rcgorman-dstl Aug 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 25 additions & 38 deletions docs/examples/MTT_3D_Platform.py
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update Multi target tracking example to work with updated metrics, MultiManager, and MetricPlotter

Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,11 @@

# OSPA metric
from stonesoup.metricgenerator.ospametric import OSPAMetric
ospa_generator = OSPAMetric(c=40, p=1)
ospa_generator = OSPAMetric(c=40, p=1,
generator_name='OSPA metrics',
tracks_key='tracks',
truths_key='truths'
)

# SIAP metrics
from stonesoup.metricgenerator.tracktotruthmetrics import SIAPMetrics
Expand All @@ -329,77 +333,60 @@
SIAPvel_measure = Euclidean(mapping=np.array([1, 3]))
siap_generator = SIAPMetrics(
position_measure=SIAPpos_measure,
velocity_measure=SIAPvel_measure
velocity_measure=SIAPvel_measure,
generator_name='SIAP metrics',
tracks_key='tracks',
truths_key='truths'
)

# Uncertainty metric
from stonesoup.metricgenerator.uncertaintymetric import \
SumofCovarianceNormsMetric
uncertainty_generator = SumofCovarianceNormsMetric()

uncertainty_generator = SumofCovarianceNormsMetric(generator_name='Uncertainty metric',
tracks_key='tracks')

# %%
# The metric manager requires us to define an associator. Here we want to
# compare the track estimates with the ground truth.

from stonesoup.dataassociator.tracktotrack import TrackToTruth
associator = TrackToTruth(association_threshold=30)

from stonesoup.metricgenerator.manager import SimpleManager
metric_manager = SimpleManager(
from stonesoup.metricgenerator.manager import MultiManager
metric_manager = MultiManager(
[ospa_generator, siap_generator, uncertainty_generator],
associator=associator
)


# %%
# Since we saved the groundtruth and tracks before, we can easily add them
# to the metric manager now, and then tell it to generate the metrics.
metric_manager.add_data(groundtruth_plot, tracks_plot)
metric_manager.add_data({'truths': groundtruth_plot, 'tracks': tracks_plot})
metrics = metric_manager.generate_metrics()


# %%
# The first metric we will look at is the OSPA metric.
ospa_metric = metrics["OSPA distances"]
from stonesoup.plotter import MetricPlotter

fig, ax = plt.subplots()
ax.plot([i.timestamp for i in ospa_metric.value],
[i.value for i in ospa_metric.value])
ax.set_ylabel("OSPA distance")
_ = ax.set_xlabel("Time")
fig1 = MetricPlotter()
fig1.plot_metrics(metrics, generator_names=['OSPA metrics'])


# %%
# Next are the SIAP metrics. Specifically, we will look at the position and
# velocity accuracy.
position_accuracy = metrics['SIAP Position Accuracy at times']
velocity_accuracy = metrics['SIAP Velocity Accuracy at times']
times = metric_manager.list_timestamps()

# Make a figure with 2 subplots.
fig, axes = plt.subplots(2)

# The first subplot will show the position accuracy
axes[0].set(title='Positional Accuracy Over Time', xlabel='Time',
ylabel='Accuracy')
axes[0].plot(times, [metric.value for metric in position_accuracy.value])

# The second subplot will show the velocity accuracy
axes[1].set(title='Velocity Accuracy Over Time', xlabel='Time',
ylabel='Accuracy')
axes[1].plot(times, [metric.value for metric in velocity_accuracy.value])
plt.tight_layout()
fig2 = MetricPlotter()
fig2.plot_metrics(metrics, metric_names=['SIAP Position Accuracy at times',
'SIAP Velocity Accuracy at times'])


# %%
# Finally, we will examine a general uncertainty metric. This is calculated as
# the sum of the norms of the covariance matrices of each estimated state.
# Since the sum is not normalized for the number of estimated states, it is
# most important to look at the trends of this graph rather than the values.
uncertainty_metric = metrics["Sum of Covariance Norms Metric"]

fig, ax = plt.subplots()
ax.plot([i.timestamp for i in uncertainty_metric.value],
[i.value for i in uncertainty_metric.value])
_ = ax.set(title="Track Uncertainty Over Time", xlabel="Time",
ylabel="Sum of covariance matrix norms")
fig3 = MetricPlotter()
fig3.plot_metrics(metrics, generator_names=['Uncertainty metric'])
fig3.fig.axes[0].set_title(label='Track uncertainty over time')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add set_title as a MetricPlotter method? This will allow a user not to have to worry about "fig.axes[0]" and allows the MetricPlotter to be extended to plotly in the future.

plt.show()
Loading