Skip to content

Commit

Permalink
add coremark pro support
Browse files Browse the repository at this point in the history
  • Loading branch information
@ewchong
ewchong committed Oct 15, 2021
1 parent 487ff98 commit 4f87bda
Show file tree
Hide file tree
Showing 6 changed files with 498 additions and 0 deletions.
1 change: 1 addition & 0 deletions ci/common.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ function update_operator_image() {
sed -i "s#${default_ripsaw_image_prefix}/pgbench:latest#${SNAFU_WRAPPER_IMAGE_PREFIX}/pgbench:${SNAFU_IMAGE_TAG}#g" roles/pgbench/defaults/main.yml
sed -i "s#${default_ripsaw_image_prefix}/smallfile:master#${SNAFU_WRAPPER_IMAGE_PREFIX}/smallfile:${SNAFU_IMAGE_TAG}#g" roles/smallfile/templates/* roles/smallfile/tasks/*
sed -i "s#${default_ripsaw_image_prefix}/sysbench:latest#${SNAFU_WRAPPER_IMAGE_PREFIX}/sysbench:${SNAFU_IMAGE_TAG}#g" roles/sysbench/templates/*
sed -i "s#${default_ripsaw_image_prefix}/coremark-pro:latest#${SNAFU_WRAPPER_IMAGE_PREFIX}/sysbench:${SNAFU_IMAGE_TAG}#g" roles/coremarkpro/templates/*
sed -i "s#${default_ripsaw_image_prefix}/uperf:latest#${SNAFU_WRAPPER_IMAGE_PREFIX}/uperf:${SNAFU_IMAGE_TAG}#g" roles/uperf/templates/*
sed -i "s#${default_ripsaw_image_prefix}/ycsb-server:latest#${SNAFU_WRAPPER_IMAGE_PREFIX}/ycsb-server:${SNAFU_IMAGE_TAG}#g" roles/ycsb/templates/*
sed -i "s#${default_ripsaw_image_prefix}/vegeta:latest#${SNAFU_WRAPPER_IMAGE_PREFIX}/vegeta:${SNAFU_IMAGE_TAG}#g" roles/vegeta/templates/*
Expand Down
13 changes: 13 additions & 0 deletions snafu/benchmarks/coremarkpro/Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
FROM registry.access.redhat.com/ubi8:latest
MAINTAINER Sai Sindhur Malleni <smalleni@redhat.org>

RUN dnf install -y --nodocs make git gcc && dnf clean all
RUN git clone https://github.com/eembc/coremark-pro.git && cd coremark-pro && ls util/make && make TARGET=linux64 build
WORKDIR /output/

RUN dnf install -y --nodocs python3.8 python38-devel procps-ng iproute net-tools ethtool nmap iputils && dnf clean all
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN pip3 install --upgrade pip
#RUN echo -e '[global]\nextra-index-url=https://www.piwheels.org/simple' > /etc/pip.conf
COPY . /opt/snafu
RUN pip3 install -r /opt/snafu/requirements/py38-reqs/install.txt -e /opt/snafu
207 changes: 207 additions & 0 deletions snafu/benchmarks/coremarkpro/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
# CoreMark-Pro

Wrapper for [CoreMark-Pro](https://github.com/eembc/coremark-pro) which is a CPU benchmarking tool that provides a single number score for easy comparison across runs.

## Overview of Operations

- A path to where CoreMark-Pro has been cloned is provided to benchmark-wrapper since there is no install
mechanism.
- Executing the benchmark is done with `make` and also compiles the benchmark if not already done.
- A log folder is created from the exection, where only the `.log` and `.mark` file are processed:

```
coremark-pro/builds/linux64/gcc64/logs
├── linux64.gcc64.log # Raw logs of the CoreMark Pro run
├── linux64.gcc64.mark # Results: both individual workloads and overall score
├── progress.log # Does not process
├── zip-test.run.log # Does not process
└── zip-test.size.log # Does not process
```
- The results are ingested into two different Elasticsarch indexes:
- `*-coremark-pro-summary`: Results from the `.mark` file. Provides the calculated results from CoreMark-Pro.
- `*-coremark-pro-raw`: Raw logs from the `.log` file. Intended for analyzing the logs manually.

## Arguments

### Required

- `-p` / `--path` Directory where CoreMark Pro is located.

### Optional

- `-c` / `--context`: CoreMark Pro's context argument. Defaults to `0`.
- `-w` / `--workload`: CoreMark Pro's workload argument. Defaults to `1`.
- `-s` / `--sample`: Number of samples to run. Defaults to `1`.
- `-r` / `--result-name`: The name of CoreMark Pro's result files. This includes the path relative to `--path` and does not include the extension. Defaults to `builds/linux64/gcc64/logs/linux64.gcc64`
- `-u` / `--upload`: Parses existing results in a CoreMark-Pro log directory. No support for multiple samples and `sample_starttime` is based on when benchmark-wrapper is run. Mainly used for debugging.


## Running inside a container

The Dockerfile has CoreMark-Pro pre-built and is located at `/coremark-pro/`. This will need
to be passed to benchmark-wrapper with the `--path` command.

Rest of this section will cover common use-cases that need additional parameters.

### Archive file

To create an archive file and make it accessible to the host system, the `WORKDIR` is set to
`/output/` and can be mounted to the host system. Example:

```
podman run -it \
-v ./FOLDER_TO_SAVE_ARCHIVE/:/output/ \
coremark-pro run_snafu -t coremark-pro --path /coremark-pro/ --create-archive \
--archive coremarkpro.archive
```

### Raw logs

To retrieve the raw logs, mount the log folder from CoreMark-Pro to a **dedicated** folder on the host system otherwise all contents of the folder will be **deleted** when CoreMark-Pro is executed. Default folder is
`/coremark-pro/builds/linux64/gcc/logs/`.

Example of logs folder being saved to a `output` folder in the current directory:

```
podman run -it \
-w /coremark-pro/builds/linux64/gcc64/logs/ \
-v ./output/:/coremark-pro/builds/linux64/gcc64/logs/ \
coremark-pro run_snafu -t coremark-pro -p /coremark-pro/
```

## Parsing

This section gives a general idea of how CoreMark-Pro output matches with the Elasticsearch fields.

### Results

These results are calculated by CoreMark-Pro and read from the `*.mark` file. Each row of the table is ingested as its own record.

#### Example `.mark` file

```
WORKLOAD RESULTS TABLE
MultiCore SingleCore
Workload Name (iter/s) (iter/s) Scaling
----------------------------------------------- ---------- ---------- ----------
cjpeg-rose7-preset 178.57 192.31 0.93
.... truncated rest of the workloads ...
MARK RESULTS TABLE
Mark Name MultiCore SingleCore Scaling
----------------------------------------------- ---------- ---------- ----------
CoreMark-PRO 5708.35 5714.89 1.00
```

#### Benchmark-wrapper's archive file output

```
{
"_source": {
"test_config": {
"workload": 0, # `--workload`
"context": 1 # `--context`
},
"sample_starttime": "2021-10.1..", # Time when benchmark-wrapper was executed.
"sample": 1, # `--sample`
"name": "cjpeg-rose7-preset, # Name of the CoreMark-Pro workload
"multicore": 178.57, # Multi Core result
"singlecore": 192.31, # Single Core result
"scaling": 0.93, # Scaling result
"type": "workload", # Type of result, determined by the table header
# - `workload`: Data from 'Workload Results Table'
# - `mark`: Data from 'Mark Results Table'
"cluster_name": "laptop",
"user": "ed",
"uuid": "3cc2e4a9-bd7f-4394-8d8c-66415ceeb02f",
"workload": "coremark-pro",
"run_id": "NA"
},
}
... The above is repeated for the rest of the workloads and the mark result ...
```

### Raw logs

These are the raw logs parsed from the `.log` file. The median results are dropped since they can be derived using Elasticsearch. Each row of results is ingested as its own record.


#### Excerpt of a log file

```
#UID Suite Name Ctx Wrk Fails t(s) Iter Iter/s Codesize Datasize
#Results for verification run started at 21285:10:58:22 XCMD=-c1 -w0
236760500 MLT cjpeg-rose7-preset 1 1 0 0.010 1 100.00 105616 267544
#Results for performance runs started at 21285:10:58:23 XCMD=-c1 -w0
236760500 MLT cjpeg-rose7-preset 1 1 0 0.081 10 123.46 105616 267544
... truncated rest of the log ...
```

CoreMark-Pro performs two sets of runs for each workload that are marked by the same `uid`. Each set of runs has a single verification run and three performance runs. The number of runs is non-configurable. Structure of the runs:
```
Set 1: Context = 1 Workload = 1
├─── Workload Verification Run
├─── Performance run #1
├─── Performance run #2
└─── Performance run #3
... Repeat for all workloads ...
Set 2: Context and workload specified by user through -w / -c
├─── Workload Verification Run
├─── Performance run #1
├─── Performance run #2
└─── Performance run #3
... Repeat for all workloads ...
```
#### Benchmark-wrapper's archive file output

A `run_index` field was added to ensure performance runs with the same results are not marked as duplicates.

```
{
"_source": {
## Same as the `.mark` file
"test_config": {
"workload": 0,
"context": 1
},
"sample_starttime": "2021-10.1..", # Time when benchmark-wrapper was executed.
"sample": 1,
## Results from the logs
"uid": "236760500", # A UID generated per workload by CoreMark-Pro
"suite": "MLT",
"name": "cjpeg-rose7-preset",
"ctx": 1,
"wrk": 1,
"fails": 0,
"t(s)": 0.01,
"iter": 1,
"iter/s": 100.0,
"codesize": 105616,
"datasize": 267544,
"type": "verification", # Possible types: verification / performance
"starttime": "2021-10....", # The start time for the runs as recorded by CoreMark Pro
"run_index": 0, # An index of how many runs of the same type. Always
# 0 for verification, between 0-2 for performance runs.
## Same as the `.mark` file
"cluster_name": "laptop",
"user": "ed",
"uuid": "816f7fe9-ab04-45a4-8a1f-ce61c2fe11e6",
"workload": "coremark-pro",
"run_id": "NA"
},
}
```
## Limitations

- Limited ability to visualize the data from `*-coremark-pro-raw`, requires additional fields to aggregate the runs.
5 changes: 5 additions & 0 deletions snafu/benchmarks/coremarkpro/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# flake8: noqa
"""Uperf benchmark"""
from snafu.benchmarks.coremarkpro.coremarkpro import Coremarkpro
11 changes: 11 additions & 0 deletions snafu/benchmarks/coremarkpro/ci_test.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/bin/bash

set -x

source ci/common.sh

# Build image for ci
image_spec=$SNAFU_WRAPPER_IMAGE_PREFIX/coremark-pro:$SNAFU_IMAGE_TAG
build_and_push snafu/coremarkpro/Dockerfile $image_spec
pushd ripsaw
source tests/test_coremarkpro.sh
Loading

0 comments on commit 4f87bda

Please sign in to comment.