Using Docker
Guide to start using Konduit-Serving with Docker
In this section, we provide guidance on how to demonstrate Konduit-Serving with Docker. Konduit-Serving is a platform to serve the ML/DL models directly through a single line of code. We'll start building and installing the Konduit-Serving from docker image and follow by deploying the trained model in Konduit-Serving.
Prerequisites
You will need following prerequisites to follow along
To ensure Konduit-Serving works properly, install these prerequisites. We’ll be using Konduit-Serving to deploy machine/deep learning pipelines easier using a few lines of code. We've prepared a Github repository to simplify showcasing Konduit-Serving.
Introduction to Repository
The repository contains simple examples of deploying a pipeline using different model types. To clone the repository, run the following command:
git clone https://github.com/ShamsUlAzeem/konduit-serving-demo
Build the CPU version of docker image by running the following command in the root directory:
bash build.sh CPU
After a successful build, run docker image with docker-compose at current working directory:
docker-compose up
Now, open JupyterLab in the browser.
Explore Konduit-Serving in Repository
Let's take a look inside the demos directory from konduit-serving-demo. Each folder inside the demos folder demonstrate serving a different kind of model, using a configuration file in either JSON or YAML, through the Konduit-Serving CLI.
The examples use different frameworks, including Keras, Tensorflow, Pytorch, and DL4J. These examples can be run in IPython Notebook (.ipynb) with Java-based kernel. Konduit-Serving provides a platform for users to take advantage of models using konduit
CLI such as serve
, list
, logs
, predict
and stop
. You can also build your model and start using Konduit-Serving.
Let's look at training and serving a model with Konduit-Serving.
Build your model
These are steps to train a model for Keras and DL4J from scratch:
First, import library that will be use:
import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adam
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import seaborn as sns
import pandas as pd
import numpy as np
Load Iris data set:
dataset = sns.load_dataset("iris")
print(dataset)
Display distribution of data set:
sns.set(style="ticks")
sns.set_palette("husl")
sns.pairplot(dataset.iloc[:,0:6], hue="species")
Apply data pre-processing and split data into training and testing:
X = dataset.iloc[:,0:4].values
y = dataset.iloc[:,4].values
encoder = LabelEncoder()
y1 = encoder.fit_transform(y)
Y = pd.get_dummies(y1).values
X_train,X_test, y_train,y_test = train_test_split(X,Y,test_size=0.3,random_state=0)
Configure the model and print summary:
model = Sequential()
model.add(Dense(4,input_shape=(4,),activation='relu', name="input"))
model.add(Dense(3,activation='softmax', name='output'))
model.compile(Adam(lr=0.01),'categorical_crossentropy',metrics=['accuracy'])
model.summary()
Train the model by training data with 800 epochs:
model.fit(X_train,y_train,epochs=800)
Test the model by predicting testing data:
y_pred = model.predict(X_test)
y_test_class = np.argmax(y_test,axis=1)
y_pred_class = np.argmax(y_pred,axis=1)
Display confusion matrix to see more details of model prediction between actual and predicted result (if satisfied can save the model):
cm = confusion_matrix(y_test_class, y_pred_class)
print(cm)
Also, you can try to predict by using your value in the trained model:
X_test2 = np.array([[5, 3.9, 2, 0.5],[5,2.5,3,1],[8,3.5,6,2]])#setosa, versicolor, virginica
y_pred2 = model.predict(X_test2)
print(y_pred2)
Print the result of classification:
print(np.argmax(y_pred2,axis=1)) #0 = setosa 1 = versicolor 2 = virginica
Then, save the trained model in HDF5 (.h5) format which will be used in Konduit-Serving later:
model.save_weights("model.h5")
print("Saved model to disk")
The model is now ready to be deployed.
Deploy your model in Konduit-Serving
Now, you are ready to deploy a model in Konduit-Serving by using konduit
CLI. This step needs a saved model file (h5 or zip file) and JSON/YAML configuration file. Let's begin with:
Create a new folder in the demos directory (for example, 10-iris-model):

Drag the model file into the demos directory folder and create an IPython Notebook file with Java kernel (for example, iris-model.ipynb).
In this notebook, we will use konduit
CLI.
Check the version of Konduit-Serving, and either is installed or not:
%%bash
konduit -version
Create the JSON/YAML configuration file by using
konduit config
command:
YAML configuration:
%%bash
konduit config -p keras -o iris-keras.yaml -y
Or, you can try this command to get a JSON configuration file:
%%bash
konduit config -p keras -o iris-keras.json
In the configuration file, you need to edit the YAML/JSON file in the pipeline section (for this example, we will use YAML with DL4J):
---
host: "localhost"
port: 0
use_ssl: false
protocol: "HTTP"
static_content_root: "static-content"
static_content_url: "/static-content"
static_content_index_page: "/index.html"
kafka_configuration:
start_http_server_for_kafka: true
http_kafka_host: "localhost"
http_kafka_port: 0
consumer_topic_name: "inference-in"
consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
consumer_group_id: "konduit-serving-consumer-group"
consumer_auto_offset_reset: "earliest"
consumer_auto_commit: "true"
producer_topic_name: "inference-out"
producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
producer_acks: "1"
mqtt_configuration: {}
custom_endpoints: []
pipeline:
steps:
- '@type': "DEEPLEARNING4J"
modelUri: "<path_to_model>"
inputNames:
- "1"
- "2"
outputNames:
- "11"
- "22"
To determine the name of input and output, you can use Netron to read ML/DL model, for example:
In node properties, use the name's value of first weights as "inputNames"
.

In node properties, use the name's value of last weights as "outputname"
.

Start the server by using
konduit serve
command and give the id's name based on your own:
%%bash
konduit serve -id dl4j-iris -c iris-dl4j.yaml -rwm -b
Listing the active server in Konduit-Serving,
konduit list
:
%%bash
konduit list
Show the log of the selected server’s id for 100 lines by
konduit logs
:
%%bash
konduit logs dl4j-iris -l 100
Test the prediction of ML/DL model by
konduit predict
in the Konduit-Serving at the selected id (in this example: dl4j-iris):
%%bash
konduit predict dl4j-iris "{\"input\":[[1,2,3,4]]}"
You can test again with another input value to get another result:
%%bash
konduit predict dl4j-iris "{\"input\":[[5.1,3.5,1.4,0.2]]}"
And, the result will be like this:

For more interactive result, you can edit the JSON/YAML file in the pipeline section as below:
---
host: "localhost"
port: 0
use_ssl: false
protocol: "HTTP"
static_content_root: "static-content"
static_content_url: "/static-content"
static_content_index_page: "/index.html"
kafka_configuration:
start_http_server_for_kafka: true
http_kafka_host: "localhost"
http_kafka_port: 0
consumer_topic_name: "inference-in"
consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
consumer_group_id: "konduit-serving-consumer-group"
consumer_auto_offset_reset: "earliest"
consumer_auto_commit: "true"
producer_topic_name: "inference-out"
producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
producer_acks: "1"
mqtt_configuration: {}
custom_endpoints: []
pipeline:
steps:
- '@type': "DEEPLEARNING4J"
modelUri: "dl4j_iris_model.zip"
inputNames:
- "input"
outputNames:
- "output"
- '@type': "CLASSIFIER_OUTPUT"
input_names: "layer2"
labels:
- Sentosa
- Versicolor
- Virginica
So, you will get the result of classification straightforward with prediction's label:

Lastly but not least, use
konduit stop
to terminate selected id in Konduit-Serving:
%%bash
konduit stop dl4j-iris
Congratulation! You have deployed Konduit-Serving on your own. What's next?
Last updated
Was this helpful?