> For the complete documentation index, see [llms.txt](https://serving.konduit.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://serving.konduit.ai/examples/ipython-notebook/onnx/onnx-pytorch-iris.md). # Pytorch (IRIS) ### Overview This documentation shows Konduit-Serving can serve a custom model and include post-processing in the pipeline to give a direct output label understood by a human. Iris model is used in this example to deploy on the server as a classifier through custom endpoints. ### Adding package to the classpath First of all we need to add the main package to the classpath so that the notebook can load all the necessary libraries from Konduit-Serving into the Jupyter Notebook kernel. Classpaths can be considered similar to `site-packages` in the python ecosystem where each library that's to be imported to your code is loaded from. We package almost everything you need to get started with the `konduit.jar` package so you can just start working on the actual code, without having to care about any boilerplate configuration. ```bash %classpath add jar ../../konduit.jar ``` Let's ensure the working directory is correct and list all the file available in the directory. ```bash %%bash echo "Current directory $(pwd)" && tree ``` You'll be able to view as the following. ``` Current directory /root/konduit/demos/1-pytorch-onnx-iris . ├── dataset │ └── iris.csv ├── iris.onnx ├── onnx-iris.ipynb ├── onnx.yaml └── train.py 1 directory, 5 files ``` ### Main model script code We're creating a Pytorch model from scratch here and then converting that into ONNX format. ```bash %%bash less train.py ``` You'll be able to browse the source code how the training takes places. ``` import pandas as pd from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score, precision_score, recall_score import torch import torch.nn as nn import torch.nn.functional as F from torch.autograd import Variable class Net(nn.Module): # define nn def __init__(self): super(Net, self).__init__() self.fc1 = nn.Linear(4, 100) self.fc2 = nn.Linear(100, 100) self.fc3 = nn.Linear(100, 3) self.softmax = nn.Softmax(dim=1) def forward(self, X): X = F.relu(self.fc1(X)) X = self.fc2(X) X = self.fc3(X) X = self.softmax(X) return X # load IRIS dataset dataset = pd.read_csv('dataset/iris.csv') # transform species to numerics dataset.loc[dataset.species == 'Iris-setosa', 'species'] = 0 dataset.loc[dataset.species == 'Iris-versicolor', 'species'] = 1 dataset.loc[dataset.species == 'Iris-virginica', 'species'] = 2 train_X, test_X, train_y, test_y = train_test_split(dataset[dataset.columns[0:4]].values, dataset.species.values, test_size=0.8) # wrap up with Variable in pytorch train_X = Variable(torch.Tensor(train_X).float()) test_X = Variable(torch.Tensor(test_X).float()) train_y = Variable(torch.Tensor(train_y).long()) test_y = Variable(torch.Tensor(test_y).long()) net = Net() criterion = nn.CrossEntropyLoss() # cross entropy loss optimizer = torch.optim.SGD(net.parameters(), lr=0.01) for epoch in range(1000): optimizer.zero_grad() out = net(train_X) loss = criterion(out, train_y) loss.backward() optimizer.step() if epoch % 100 == 0: print('number of epoch', epoch, 'loss', loss.item()) predict_out = net(test_X) _, predict_y = torch.max(predict_out, 1) print('prediction accuracy', accuracy_score(test_y.data, predict_y.data)) print('macro precision', precision_score(test_y.data, predict_y.data, average='macro')) print('micro precision', precision_score(test_y.data, predict_y.data, average='micro')) print('macro recall', recall_score(test_y.data, predict_y.data, average='macro')) print('micro recall', recall_score(test_y.data, predict_y.data, average='micro')) # Input to the model x = torch.randn(1, 4, requires_grad=True) # Export the model torch.onnx.export(net, # model being run x, # model input (or a tuple for multiple inputs) "iris.onnx", # where to save the model (can be a file or file-like object) export_params=True, # store the trained parameter weights inside the model file opset_version=10, # the ONNX version to export the model to do_constant_folding=True, # whether to execute constant folding for optimization input_names=['input'], # the model's input names output_names=['output'], # the model's output names dynamic_axes={'input': {0: 'batch_size'}, # variable length axes 'output': {0: 'batch_size'}}) ``` ### Viewing the configuration file The configuration for the custom endpoint is as follow: ```bash %%bash less onnx.yaml ``` The output shows configurations in YAML, in which you can see two steps in the pipeline, serving a model and post-processing to make it directly understood by a human. ``` --- host: "localhost" port: 0 protocol: "HTTP" pipeline: steps: - '@type': "ONNX" modelUri: "iris.onnx" inputNames: - "input" outputNames: - "output" - '@type': "CLASSIFIER_OUTPUT" input_name: "output" labels: - Setosa - Versicolor - Virginica ``` ### Starting the server `konduit serve` can be used together with ID's name (use any name) and configuration file to start the server. ```bash %%bash konduit serve -id onnx-iris -c onnx.yaml -rwm -b ``` You'll get the following message. ``` Starting konduit server... Using classpath: /root/konduit/bin/../konduit.jar INFO: Running command /root/miniconda/jre/bin/java -Dkonduit.logs.file.path=/root/.konduit-serving/command_logs/onnx-iris.log -Dlogback.configurationFile=/tmp/logback-run_command_80a3902b721c4c3f.xml -jar /root/konduit/bin/../konduit.jar run --instances 1 -s inference -c onnx.yaml -Dserving.id=onnx-iris For server status, execute: 'konduit list' For logs, execute: 'konduit logs onnx-iris' ``` To view the logs of the running server, use `konduit logs` commands. ```bash %%bash konduit logs onnx-iris -l 100 ``` You'll be able to see the following from logging. ``` 15:01:50.334 [main] INFO a.k.s.c.l.command.KonduitRunCommand - Processing configuration: /root/konduit/demos/1-pytorch-onnx-iris/onnx.yaml . . . #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### . . . 15:01:50.919 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:01:50.919 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 35761 with 2 pipeline steps ``` ### Sending inputs Now we can send our inputs through `cURL` for inference ``` %%bash konduit predict onnx-iris "{\"input\":[[5.1,3.5,1.4,0.2]]}" ``` So, the server will print out the output with a label. ``` { "output" : [ [ 0.99312085, 0.0068791825, 6.1220806E-9 ] ], "prob" : 0.9931208491325378, "index" : 0, "label" : "Setosa" } ``` Try once again with `--input-type` flag. ``` %%bash konduit predict onnx-iris --input-type multipart "input=[[5.1,3.5,1.4,0.2]]" ``` You'll see the output of the prediction. ``` { "output" : [ [ 0.99312085, 0.0068791825, 6.1220806E-9 ] ], "prob" : 0.9931208491325378, "index" : 0, "label" : "Setosa" } ``` ### Stopping the server Now after we're done with the server, we can stop it through the `konduit stop` command ``` %%bash konduit stop onnx-iris ``` You'll receive this once the server is terminated. ``` Stopping konduit server 'onnx-iris' Application 'onnx-iris' terminated with status 0 ``` Let's take a look at the following example, where we are going to give an image input and doing pre-processing step before fetching it into the model. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://serving.konduit.ai/examples/ipython-notebook/onnx/onnx-pytorch-iris.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.