OpenVINO resnet_v1_50 python example with VNNI (#1582)

* This Python resnet_v1_50 example support both fp32 and int8 OpenVINO model. Replacing fp32 model path with int8 model path will give you better performance.
* VNNI: If the running platform has VNNI instruction, then using int8 quantized models can generally give you additional performance boost.

pyzoo/zoo/examples/vnni/openvino/README.md

@@ -0,0 +1,47 @@
+# OpenVINO ResNet_v1_50 example
+This example illustrates how to use a pre-trained OpenVINO optimized model to make inferences with OpenVINO toolkit as backend using Analytics Zoo. We hereby illustrate the support of [VNNI](https://en.wikichip.org/wiki/x86/avx512vnni) using [OpenVINO](https://software.intel.com/en-us/openvino-toolkit) as backend in Analytics Zoo, which aims at accelerating inference by utilizing low numerical precision (Int8) computing. Int8 quantized models can generally give you better performance on Intel Xeon scalable processors.
+
+## Environment
+* Apache Spark (This version needs to be same with the version you use to build Analytics Zoo)
+* [Analytics Zoo](https://analytics-zoo.github.io/master/#PythonUserGuide/install/)
+
+- Set `ZOO_NUM_MKLTHREADS` to determine cores used by OpenVINO, e.g, `export ZOO_NUM_MKLTHREADS=10`. If it is set to `all`, e.g., `export ZOO_NUM_MKLTHREADS=all`, then OpenVINO will utilize all physical cores for Prediction.
+- Set `KMP_BLOCKTIME=200`, i.e., `export KMP_BLOCKTIME=200`
+
+## PrepareOpenVINOResNet
+TensorFlow models cannot be directly loaded by OpenVINO. It should be converted to OpenVINO optimized model and int8 optimized model first. You can use [PrepareOpenVINOResNet](https://github.com/intel-analytics/analytics-zoo/tree/master/zoo/src/main/scala/com/intel/analytics/zoo/examples/vnni/openvino) or [OpenVINO toolkit](https://docs.openvinotoolkit.org/2018_R5/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html) to finish this job.
+
+__Sample Result files in MODEL_PATH__:
+```
+resnet_v1_50.ckpt
+resnet_v1_50_inference_graph.bin
+resnet_v1_50_inference_graph-calibrated.bin
+resnet_v1_50_inference_graph-calibrated.xml
+resnet_v1_50_inference_graph.mapping
+resnet_v1_50_inference_graph.xml
+```
+
+Among them, `resnet_v1_50_inference_graph.xml` and `resnet_v1_50_inference_graph.bin` are OpenVINO optimized ResNet_v1_50 model and weight, `resnet_v1_50_inference_graph-calibrated.xml` and `resnet_v1_50_inference_graph-calibrated.bin` are OpenVINO int8 optimized ResNet_v1_50 model and weight. Both of them can be loaded by OpenVINO or Zoo.
+
+## Image Classification with ResNet_v1_50
+
+```
+python predict.py --image ${image} --model ${model}
+```
+
+### Options
+* `--image` The path where the images are stored. It can be either a folder or an image path. Local file system, HDFS and Amazon S3 are supported.
+* `--model` The path to the TensorFlow object detection model.
+
+### Results
+We print the inference result of each batch.
+```
+[ INFO ] Start inference (1 iterations)
+
+batch_0
+* Predict result {'Top-1': '67'}
+* Predict result {'Top-1': '65'}
+* Predict result {'Top-1': '334'}
+* Predict result {'Top-1': '795'}
+```
+

pyzoo/zoo/examples/vnni/openvino/predict.py

@@ -0,0 +1,74 @@
+#
+# Copyright 2018 Analytics Zoo Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from optparse import OptionParser
+
+from zoo.pipeline.inference import InferenceModel
+from zoo.common.nncontext import init_nncontext
+from zoo.feature.image import *
+from zoo.pipeline.nnframes import *
+
+BATCH_SIZE = 4
+
+
+def predict(model_path, img_path):
+    model = InferenceModel()
+    model.load_openvino(model_path,
+                        weight_path=model_path[:model_path.rindex(".")] + ".bin",
+                        batch_size=BATCH_SIZE)
+    sc = init_nncontext("OpenVINO Python resnet_v1_50 Inference Example")
+    # pre-processing
+    infer_transformer = ChainedPreprocessing([ImageBytesToMat(),
+                                             ImageResize(256, 256),
+                                             ImageCenterCrop(224, 224),
+                                             ImageMatToTensor(format="NHWC", to_RGB=True)])
+    image_set = ImageSet.read(img_path, sc).\
+        transform(infer_transformer).get_image().collect()
+    image_set = np.expand_dims(image_set, axis=1)
+
+    for i in range(len(image_set) // BATCH_SIZE + 1):
+        index = i * BATCH_SIZE
+        # check whether out of index
+        if index >= len(image_set):
+            break
+        batch = image_set[index]
+        # put 4 images in one batch
+        for j in range(index + 1, min(index + BATCH_SIZE, len(image_set))):
+            batch = np.vstack((batch, image_set[j]))
+        batch = np.expand_dims(batch, axis=0)
+        # predict batch
+        predictions = model.predict(batch)
+        result = predictions[0]
+
+        # post-processing for Top-1
+        print("batch_" + str(i))
+        for r in result:
+            output = {}
+            max_index = np.argmax(r)
+            output["Top-1"] = str(max_index)
+            print("* Predict result " + str(output))
+
+
+if __name__ == "__main__":
+    parser = OptionParser()
+    parser.add_option("--image", type=str, dest="img_path",
+                      help="The path where the images are stored, "
+                           "can be either a folder or an image path")
+    parser.add_option("--model", type=str, dest="model_path",
+                      help="Zoo Model Path")
+
+    (options, args) = parser.parse_args(sys.argv)
+    predict(options.model_path, options.img_path)