MCorr example notebook

[AlibiZhenis]

Description

This PR adds an example notebook that demonstrates the usage of MCorr algorithm, as well as corresponding documentation update.

Issues Resolved

Closes #157

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #186 (462ab1b) into main (57f0cf8) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #186   +/-   ##
=======================================
  Coverage   91.15%   91.15%           
=======================================
  Files          37       37           
  Lines        4091     4091           
=======================================
  Hits         3729     3729           
  Misses        362      362           

[dhrubo-os]

"Demo Notebooks for In-house python based models"

[dhrubo-os]

We added a new workflow changelog verifier. Please add a line here in 1.1.0 section

[dhrubo-os]

Can you share any notebook url from your repo which has the same output? It's not quite possible to review from here.

[AlibiZhenis]

Can you see it from here?

[AlibiZhenis]

Can you see it?

[dhrubo-os]

Yeah, I can see it. Sorry I forgot to reply. @greaa-aws will also review your PR and comment here as well.

[dhrubo-os]

Looks like we didn't use SMD_small_labels.csv.zip file that much? If this is not required, may be we can remove it?

[AlibiZhenis]

I removed the labels. Anything else needs that needs to be changed?

[dhrubo-os]

@greaa-aws is going to send you a dataset which works better shortly.

[greaa-aws]

I'm going to share my comments section-by-section. I think this is an awesome first effort and I really appreciate the work involved in creating an example notebook for a new algorithm!

I'd like to offer some feedback and ideas for how to expand it from here so that we can end up with something that is easy-to-follow and informative for users. To do this, we need to communicate what the algorithm is doing and why, not just how to use the execute API.

I have written some material elsewhere for this, so I’ll share it here along with some suggestions for where to use it. Please feel free to ask questions!

[greaa-aws]

Introduction

Let's add a little text here to orient our users. I'd suggest:

This notebook contains an introduction to the metrics correlation algorithm released in OpenSearch 2.8. We will work through an example using data from the Server Machine Dataset and include commentary on the objective, configuration, and output of the algorithm itself.

[greaa-aws]

Step 0: Imports

Looks good to me, just need to correct a typo in the second line: “uncommenting the line and execute” -> “uncommenting and executing the line in the second code block”.

Step 1: Set up client

Looks good to me.

[greaa-aws]

Step 2: Preparing the data

I like the use of the Server Machine Dataset data, but let’s use a different example subset. This one should provide a nice illustration of the algorithm:
smd_data.csv
It has $M = 38$ metrics and $T = 1000$ timestamps. The algorithm should find two separate events in the data. The data are already in (row, column) format so it's not necessary to transpose. Rather than printing the dataframe, lets visualize in a plot.

For the writing, this is where we need a proper introduction to the method. Please see below for an intro I have previously written.

In this case study, we consider an example from the recently-released Server Machine Dataset. As the name suggests, the data are performance metrics collected from industrial servers. The servers occasionally suffer performance degradations that appear in the metrics data. Here, we take the role of an ops engineer investigating recent metrics data to understand whether and how these issues have occurred.

The individual metrics can display very different behavior, yet correlating issues across metrics is essential for detecting and diagnosing issues with the server. To start, let’s plot the data that we’ll be working with in this case study. It consists of 38 metrics observed across 1000 timestamps. We have normalized each metrics time series such that its values lie between 0 and 1.

This can be followed by a code block that plots the data:

M, T = smd_data.shape
cm = plt.get_cmap('tab20')
vals = np.linspace(0, 1, M)

for i in range(M):
    plt.plot(smd_data[i,:], color = cm(vals[i]))

plt.title('Example metrics from the Server Machine Dataset')
plt.ylabel('Normalized metrics value')
plt.xlabel('Timestamp')

[greaa-aws]

Step 3: Metrics correlation

Again, let’s add some more detailed writing to share the idea of the algorithm. Here’s what I have:

The metrics correlation algorithm finds events in a set of metrics data, where an event is defined as a window in time where multiple metrics simultaneously display anomalous behavior. Given a time span and a total set of metrics, the metrics correlation algorithm automatically determines how many events occurred, when they occurred, and which metrics were involved in each event. This algorithm requires no prior training and minimal configuration.

Let’s see how it works for the Server Machine Dataset example.

Then we can convert to JSON, as you already have in the notebook. Note: please do NOT print the JSON. It leads to hundreds (for the new data, it would be thousands) of lines of output that are not informative and make the notebook hard to navigate.

First, we’ll reformat the data to JSON:

input_json = {"metrics": smd_data.values.tolist()}

Then we can show how the algorithm is run. Again, we don’t need to print the raw results. It’s not an informative way to share or discuss the output.

Now we can run the metrics correlation algorithm on this data via the execute API:

results = ml_client.execute(
    algorithm_name = "METRICS_CORRELATION",
    input_json = input_json
)

Instead, I’d suggest the following text and code blocks to describe the output.

The results are given as a list, where each element of the list is an event that the algorithm has discovered in the data. If the length of the list is zero, then the algorithm has discovered no events in the given input data.

In this case, the algorithm discovers two events:

len(results[‘output’][‘inference_results’])

Every event is represented as a dictionary with three fields:

• The event window, a list consisting of the start and end index values for the event. This describes when the event occurred.
• The suspected metrics, a list with the indices of all metrics that are part of the event. This describes which metrics were correlated during the time of the event.
• The event pattern, an array of length T where each value indicates the relative intensity of the event at the corresponding timestamp. This is the algorithm's underlying "idea" of the event; roughly, it indicates where in time the algorithm has focused to define the event.

results[‘output’][‘inference_results’][0].keys()

[greaa-aws]

Step 4: Result visualization

This is a great idea. Below, I suggest some writing for this section and some expanded code for event plotting.

Below, we implement a plotting function that uses each of these fields to visually display an event. This function plots all of the input data as above, but highlights the metrics that were involved in the event and shades the interval when the event occurred. It also plots the event pattern.

from matplotlib.lines import Line2D

def plot_event(event, metrics):
    M, _ = metrics.shape
    _, axs = plt.subplots(2, 1, figsize = (12,6))

    # plot non-event metrics in grayscale
    for i in range(M):
        if i not in event['suspected_metrics']:
            axs[0].plot(metrics[i,:], color = 'lightgray')

    # highlight event metrics
    for i in range(M):
        if i in event['suspected_metrics']:
            axs[0].plot(metrics[i,:], color = 'maroon')

    # shade event window    
    axs[0].fill_between(event['event_window'], 0, 1, color='skyblue', alpha=0.6, label = 'Event window')
    axs[0].set_ylabel('Normalized metrics value')

    # add legend
    sus_met = Line2D([0],[0], label='Suspected metrics', color='maroon')
    unsus_met = Line2D([0],[0], label='Other metrics', color='lightgray')
    handles, _ = axs[0].get_legend_handles_labels()
    handles.extend([sus_met, unsus_met])
    axs[0].legend(loc='upper left', handles=handles, facecolor='white', framealpha=1)

    # plot (scaled) event pattern
    axs[1].plot(event['event_pattern'] / event['event_pattern'].max(), color = 'darkorange', alpha=0.6, linewidth=3.0, label='Intensity')
    axs[1].set_ylabel('Event pattern')
    axs[1].set_xlabel('Timestamp')

    plt.show()

Now we can easily visualize both events discovered in the data:

plot_event(smd_res[0], smd_data)

plot_event(smd_res[1], smd_data)

Lastly, we should conclude with some discussion of the results.

From the above, we see that the metrics correlation algorithm has found two events.

The first event occurs between timestamps $345$ and $450$ and it features the large rise that was clearly visible from the first plot of the data. However, the algorithm has found and associated with this rise several other metrics that display more subtle changes during this window.

The second event occurs between timestamps $690$ and $777$, capturing the sharp drop that was again clearly visible in the initial plot. Here, the drop has been automatically correlated with a second, smaller drop, along with several metrics whose behavior changes from flat to a slow rise during this interval.

[AlibiZhenis]

Thanks for the suggestions, @greaa-aws. I added all of them to the notebook. However, the MCorr algorithm is timing out on my side. I suspect it's because the size of the dataset exceeds the limit (ref). The total number of data points M * T <= 10000.

Is it successfully running on your side?

[greaa-aws]

Hi @AlibiZhenis, good catch! It looks like I had tested this dataset out on a local build without this limit on data size. I've attached a reduced version of the dataset that satisfies $M*T &lt; 10000$. It has the same number of timestamps and the same key metrics / events as above, so the results and suggested commentary text should be the same. Let me know if you have any difficulties.

smd_data.csv

[AlibiZhenis]

@dhrubo-os @greaa-aws Can you review again

[dhrubo-os]

@dhrubo-os @greaa-aws Can you review again

I tested in my end and looks good to me. Just address the comment. Thanks.

[greaa-aws]

It's 9 metrics now, after we reduced the dataset size.

[greaa-aws]

Can we list these as 3 bullet points or add another newline between field descriptions?

[greaa-aws]

Are these intervals consistent with the new results from the reduced data? They should match the start and end times in the event window field for each event.

[dhrubo-os]

Opensearch 2.7.

[dhrubo-os]

Thanks @AlibiZhenis for your contribution. Merging your PR.

[opensearch-trigger-bot]

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.x 1.x
# Navigate to the new working tree
cd .worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-186-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0a0fa63bb896b8ae506caea78b490af24be8566c
# Push it to GitHub
git push --set-upstream origin backport/backport-186-to-1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-186-to-1.x.