Skip to content

Commit

Permalink
feat(doc): update README and bug finding (#123)
Browse files Browse the repository at this point in the history
  • Loading branch information
ganler committed Oct 5, 2023
1 parent 8010210 commit 018e759
Show file tree
Hide file tree
Showing 5 changed files with 396 additions and 268 deletions.
3 changes: 3 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ jobs:
with:
mode: all

- name: Test Doc
run: |
cd doc && python bug_summary.py
- name: Test core
run: |
pytest tests/core
Expand Down
39 changes: 31 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

[![](https://github.com/ise-uiuc/nnsmith/actions/workflows/ci.yaml/badge.svg)](https://github.com/ise-uiuc/nnsmith/actions/workflows/ci.yaml)
[![](https://img.shields.io/pypi/v/nnsmith?color=g)](https://pypi.org/project/nnsmith/)
[![](https://static.pepy.tech/badge/nnsmith)](https://pepy.tech/project/nnsmith)
[![](https://img.shields.io/pypi/l/nnsmith)](https://github.com/ise-uiuc/nnsmith/blob/main/LICENSE)

🌟NNSmith🌟 is a random DNN generator and a fuzzing infrastructure, primarily designed for automatically validating deep-learning frameworks and compilers.
Expand All @@ -17,10 +18,10 @@
| Models | [`tvm`](https://github.com/apache/tvm) | [`pt2`](https://pytorch.org/get-started/pytorch-2.0/) | [`torchjit`](https://pytorch.org/docs/stable/jit.html) | [`tensorrt`](https://github.com/NVIDIA/TensorRT) | [`onnxruntime`](https://github.com/microsoft/onnxruntime) | [`xla`](https://www.tensorflow.org/xla) | [`tflite`](https://www.tensorflow.org/lite) |
| ------------ | ------------------------------------ | ----------------------------------------------- | ---------------------------------------------- | ----------------------------------------- | ------------------------------------- | ----------------------------------------------------- | ------------ |
| ONNX || | ||| | |
| PyTorch | 🔨 | ✅📈 | ✅📈 | | | | |
| TensorFlow | 🔨 | | | | |||
| PyTorch | | ✅📈 | ✅📈 | | | | |
| TensorFlow | | | | | |||

✅: Supported; 📈: Supports gradient check; 🔨: Coming soon;
✅: Supported; 📈: Supports gradient check;

</div>

Expand Down Expand Up @@ -79,14 +80,36 @@ nnsmith.model_gen model.type=onnx debug.viz=true

## Learning More

- Bugs: [links to reports](doc/bugs.md).
- Documentation: [CLI](doc/cli.md), [concept](doc/concept.md), [logging](doc/log-and-err.md), and [known issues](doc/known-issues.md).
- Contributions: [`doc/CONTRIBUTING.md`](doc/CONTRIBUTING.md)
- We use [hydra](https://hydra.cc/) to manage configurations. See `nnsmith/config/main.yaml`.
- 🐛 [**Uncovered bugs**](doc/bugs.md).
- 📚 [**Documentation**](doc/): [CLI](doc/cli.md), [concept](doc/concept.md), [logging](doc/log-and-err.md), and [known issues](doc/known-issues.md).
- 🤗 [**Contributing to NNSmith**](doc/CONTRIBUTING.md)
- 📝 We use [hydra](https://hydra.cc/) to manage configurations. See `nnsmith/config/main.yaml`.

## Papers

<details><summary><b> 📜 NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers.</b> <i>[click :: citation]</i></summary>
<details><summary><b> 📜 NeuRI: Diversifying DNN Generation via Inductive Rule Inference </b> <i>[click :: citation]</i></summary>
<div>

```bibtex
@article{liu2023neuri,
title = {NeuRI: Diversifying DNN Generation via Inductive Rule Inference},
author = {Liu, Jiawei and Peng, Jinjun and Wang, Yuyao and Zhang, Lingming},
journal = {arXiv preprint arXiv:2302.02261},
year = {2023},
}
```

</div>
</details>

<p align="center">
<a href="https://arxiv.org/abs/2302.02261"><img src="https://img.shields.io/badge/Paper-FSE'23-a55fed.svg"></a>
<a href="https://arxiv.org/abs/2302.02261"><img src="https://img.shields.io/badge/arXiv-2302.02261-b31b1b.svg"></a>
<a href="https://github.com/ise-uiuc/neuri-artifact"><img src="https://img.shields.io/badge/artifact-git-black.svg"></a>
<a href="https://doi.org/10.5281/zenodo.8319975"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.8319975.svg"></a>
</p>

<details><summary><b> 📜 NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers </b> <i>[click :: citation]</i></summary>
<div>

```bibtex
Expand Down
107 changes: 107 additions & 0 deletions doc/bug_summary.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# pip install marko
import marko
import marko.inline
from rich.console import Console
from rich.table import Table
from tomark import Tomark

if __name__ == "__main__":
text = open("bugs.md").read()
doc = marko.parse(text)

systems = [
"PyTorch",
"PyTorch-ONNX Converter",
"ONNX",
"ONNXRuntime",
"TVM",
"TensorRT",
"TensorFlow",
"Hidet",
]

table = Table(title="Bug Summary")
table.add_column("System", justify="right", style="cyan", no_wrap=True)
table.add_column("#Fixed", style="magenta")
table.add_column("#Confirmed", style="magenta")
table.add_column("#Pending", style="magenta")
table.add_column("#Total", justify="right", style="green")

# Level-2 headings must be:
# - One of the system names
# - Url'ed
bugs = {}
system = None
for child in doc.children:
if isinstance(child, marko.block.Heading):
if child.level == 2:
assert len(child.children) == 1
url = child.children[0]
assert isinstance(
url, marko.inline.Link
), "Level-2 headings must be url'ed to the system homepage"
text: marko.inline.RawText = url.children[0]
system = text.children
assert system in systems, f"Parsed name: {system} not in {systems}"
elif isinstance(child, marko.block.List):
if system is None:
continue

bugs[system] = {"#Fixed": 0, "#Confirmed": 0, "#Pending": 0, "#Total": 0}
for item in child.children:
url = item.children[0].children[-1]
assert isinstance(
url, marko.inline.Link
), f"No link found in the list {item.children[0].children}"

text = item.children[0].children[0]
if isinstance(text, marko.inline.RawText):
text = text.children
if "✅" in text:
bugs[system]["#Fixed"] += 1
elif "🔵" in text:
bugs[system]["#Confirmed"] += 1
else:
bugs[system]["#Pending"] += 1
else:
bugs[system]["#Pending"] += 1

bugs[system]["#Total"] += 1
system = None

for system, item in bugs.items():
table.add_row(
system,
str(item["#Fixed"]),
str(item["#Confirmed"]),
str(item["#Pending"]),
str(item["#Total"]),
)

# Add a total row
table.add_row(
"Sum",
str(sum(item["#Fixed"] for item in bugs.values())),
str(sum(item["#Confirmed"] for item in bugs.values())),
str(sum(item["#Pending"] for item in bugs.values())),
str(sum(item["#Total"] for item in bugs.values())),
style="on green",
)

# Create a markdown table
md_table = [{"System": k, **v} for k, v in bugs.items()]
md_table.append(
{
"System": "Sum",
"#Fixed": sum(item["#Fixed"] for item in bugs.values()),
"#Confirmed": sum(item["#Confirmed"] for item in bugs.values()),
"#Pending": sum(item["#Pending"] for item in bugs.values()),
"#Total": sum(item["#Total"] for item in bugs.values()),
}
)

# TODO(@ganler): consistency check
print(Tomark.table(md_table))

console = Console()
console.print(table)
Loading

0 comments on commit 018e759

Please sign in to comment.