This notebook provides an example of serving a model built in PyTorch with ONNX Runtime, a cross-platform, high performance scoring engine for machine learning models.
The Open Neural Network Exchange (ONNX) format is supported by a number of deep learning frameworks, including PyTorch, CNTK and MXNet.
import os
from urllib.request import urlretrieve
import sys
import numpy as np
from PIL import Image
import onnx
from onnx import optimizer
from konduit.utils import default_python_path
This page documents two ways to create Konduit Serving configurations with the Python SDK:
Using Python to create a configuration, and
Writing the configuration as a YAML file, then serving it using the Python SDK.
These approaches are documented in separate tabs throughout this page. For example, the following code block shows the imports for each approach in separate tabs:
from konduit import PythonConfig, ServingConfig, InferenceConfiguration, \
PythonStep
from konduit.server import Server
from konduit.client import Client
from konduit.load import server_from_file, client_from_file
Download file
dl_path = os.path.abspath("../data/facedetector/facedetector.onnx")
DOWNLOAD_URL = "https://raw.githubusercontent.com/Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB/master/models/onnx/version-RFB-320.onnx"
if not os.path.isfile(dl_path):
urlretrieve(DOWNLOAD_URL, filename=dl_path)
We start by loading the model and running onnx.checker.check_model to check whether the model has a valid schema.
# Load the ONNX model
model = onnx.load(dl_path)
# model is a onnx.ModelProto object
onnx.checker.check_model(model)
Optimize
When loading some models, ONNX may return warnings that the model can be further optimized by removing some unused nodes.
steps:
python_step:
type: PYTHON
python_path: C:\\Users\\Skymind AI Berhad\\Documents\\konduit-serving-examples\\notebooks;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\python37.zip;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\DLLs;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch;;C:\\Users\\Skymind AI Berhad\\AppData\\Roaming\\Python\\Python37\\site-packages;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\konduit-0.1.4-py3.7.egg;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\pyyaml-5.1.2-py3.7-win-amd64.egg;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\win32;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\win32\\lib;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\Pythonwin;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\IPython\\extensions;C:\\Users\\Skymind AI Berhad\\.ipython;C:\\Users\\Skymind AI Berhad\\Documents\\konduit-serving-examples\\notebooks
python_code: |
from PIL import Image
import torchvision.transforms as transforms
import onnxruntime
import os
dl_path = os.path.abspath("../data/facedetector/facedetector.onnx")
image = Image.fromarray(image.astype('uint8'), 'RGB')
resize = transforms.Resize([240, 320])
img_y = resize(image)
to_tensor = transforms.ToTensor()
img_y = to_tensor(img_y)
img_y.unsqueeze_(0)
def to_numpy(tensor):
return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()
ort_session = onnxruntime.InferenceSession(dl_path)
ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(img_y)}
ort_outs = ort_session.run(None, ort_inputs)
_, boxes = ort_outs
python_inputs:
image: NDARRAY
python_outputs:
boxes: NDARRAY
We define a single python_step of type PYTHON.
python_path specifies the location of Python modules.
python_code specifies the Python code to be run. Here, we use a YAML literal block scalar.
python_inputs and python_outputsspecifies the data type of the objects in the Python script to be used as input(s) and output(s) respectively.
Models loaded from a YAML configuration do not currently support input and output names for Python steps. To construct configurations with custom input and output names, use the Python SDK.
To locate your Python path, run the following:
from konduit.utils import default_python_path
work_dir = os.path.abspath('.')
print(default_python_path(work_dir))
Configure the server
port = np.random.randint(1000, 65535)
server = Server(
steps=onnx_step,
serving_config=ServingConfig(http_port=port)
)
For the purposes of this example, we use ONNX model files from by Linzaer, a lightweight facedetection model designed for edge computing devices.
The following content is based on the PyTorch tutorial , with modifications.
Use ONNX's optimizer to optimize your ONNX file. The code below is adapted from this .
Note that the API for optimizing models in ONNX Runtime is experimental, and .
transforms a image into a 240 x 320 image,
adds an extra dimension with ,
Define the inputs and outputs as dictionaries, where the keys represent objects in the server's Python environment, and the values represent data types (Python data structures), defined as strings. See for supported data types.
The default Python path includes NumPy and a basic set of modules. However, for this example, we also require the Pillow, PyTorch and ONNX Runtime modules. See the for additional documentation on Python paths, and refer to the for recommended installation steps.