Using Docker

Guide to start using Konduit-Serving with Docker

In this section, we provide guidance on how to demonstrate Konduit-Serving with Docker. Konduit-Serving is a platform to serve the ML/DL models directly through a single line of code. We'll start building and installing the Konduit-Serving from docker image and follow by deploying the trained model in Konduit-Serving.

Prerequisites

You will need following prerequisites to follow along

• Docker 19.03.14

• Docker-Compose 1.27.4

To ensure Konduit-Serving works properly, install these prerequisites. We’ll be using Konduit-Serving to deploy machine/deep learning pipelines easier using a few lines of code. We've prepared a Github repository to simplify showcasing Konduit-Serving.

Introduction to Repository

The repository contains simple examples of deploying a pipeline using different model types. To clone the repository, run the following command:

git clone https://github.com/ShamsUlAzeem/konduit-serving-demo

Build the CPU version of docker image by running the following command in the root directory:

bash build.sh CPU

After a successful build, run docker image with docker-compose at current working directory:

docker-compose up

Now, open JupyterLab in the browser.

Explore Konduit-Serving in Repository

Let's take a look inside the demos directory from konduit-serving-demo. Each folder inside the demos folder demonstrate serving a different kind of model, using a configuration file in either JSON or YAML, through the Konduit-Serving CLI.

The examples use different frameworks, including Keras, Tensorflow, Pytorch, and DL4J. These examples can be run in IPython Notebook (.ipynb) with Java-based kernel. Konduit-Serving provides a platform for users to take advantage of models using konduit CLI such as serve, list, logs, predict and stop. You can also build your model and start using Konduit-Serving.

Let's look at training and serving a model with Konduit-Serving.

Build your model

These are steps to train a model for Keras and DL4J from scratch:

Keras

• First, import library that will be use:

import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adam
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import seaborn as sns
import pandas as pd
import numpy as np

• Load Iris data set:

dataset = sns.load_dataset("iris")
print(dataset)

• Display distribution of data set:

sns.set(style="ticks")
sns.set_palette("husl")
sns.pairplot(dataset.iloc[:,0:6], hue="species")

• Apply data pre-processing and split data into training and testing:

X = dataset.iloc[:,0:4].values
y = dataset.iloc[:,4].values

encoder =  LabelEncoder()
y1 = encoder.fit_transform(y)
Y = pd.get_dummies(y1).values

X_train,X_test, y_train,y_test = train_test_split(X,Y,test_size=0.3,random_state=0)

• Configure the model and print summary:

model = Sequential()
model.add(Dense(4,input_shape=(4,),activation='relu', name="input"))
model.add(Dense(3,activation='softmax', name='output'))
model.compile(Adam(lr=0.01),'categorical_crossentropy',metrics=['accuracy'])
model.summary()

• Train the model by training data with 800 epochs:

model.fit(X_train,y_train,epochs=800)

• Test the model by predicting testing data:

y_pred = model.predict(X_test)
y_test_class = np.argmax(y_test,axis=1)
y_pred_class = np.argmax(y_pred,axis=1)

• Display confusion matrix to see more details of model prediction between actual and predicted result (if satisfied can save the model):

cm = confusion_matrix(y_test_class, y_pred_class)
print(cm)

• Also, you can try to predict by using your value in the trained model:

X_test2 = np.array([[5, 3.9, 2, 0.5],[5,2.5,3,1],[8,3.5,6,2]])#setosa, versicolor, virginica
y_pred2 = model.predict(X_test2)
print(y_pred2)

• Print the result of classification:

print(np.argmax(y_pred2,axis=1)) #0 = setosa 1 = versicolor 2 = virginica

• Then, save the trained model in HDF5 (.h5) format which will be used in Konduit-Serving later:

model.save_weights("model.h5")
print("Saved model to disk")

The model is now ready to be deployed.

DL4J

Deploy your model in Konduit-Serving

Now, you are ready to deploy a model in Konduit-Serving by using konduit CLI. This step needs a saved model file (h5 or zip file) and JSON/YAML configuration file. Let's begin with:

• Create a new folder in the demos directory (for example, 10-iris-model):

• Drag the model file into the demos directory folder and create an IPython Notebook file with Java kernel (for example, iris-model.ipynb).

In this notebook, we will use konduit CLI.

• Check the version of Konduit-Serving, and either is installed or not:

%%bash
konduit -version

• Create the JSON/YAML configuration file by using konduit config command:

Keras

YAML configuration:

%%bash
konduit config -p keras -o iris-keras.yaml -y

Or, you can try this command to get a JSON configuration file:

%%bash
konduit config -p keras -o iris-keras.json

DL4J

• In the configuration file, you need to edit the YAML/JSON file in the pipeline section (for this example, we will use YAML with DL4J):

Before

---
host: "localhost"
port: 0
use_ssl: false
protocol: "HTTP"
static_content_root: "static-content"
static_content_url: "/static-content"
static_content_index_page: "/index.html"
kafka_configuration:
  start_http_server_for_kafka: true
  http_kafka_host: "localhost"
  http_kafka_port: 0
  consumer_topic_name: "inference-in"
  consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
  consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
  consumer_group_id: "konduit-serving-consumer-group"
  consumer_auto_offset_reset: "earliest"
  consumer_auto_commit: "true"
  producer_topic_name: "inference-out"
  producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
  producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
  producer_acks: "1"
mqtt_configuration: {}
custom_endpoints: []
pipeline:
  steps:
  - '@type': "DEEPLEARNING4J"
    modelUri: "<path_to_model>"
    inputNames:
    - "1"
    - "2"
    outputNames:
    - "11"
    - "22"

After

• To determine the name of input and output, you can use Netron to read ML/DL model, for example:

Keras

In node properties, use the name's value of first weights as "inputNames".

In node properties, use the name's value of last weights as "outputname".

DL4J

• Start the server by using konduit serve command and give the id's name based on your own:

%%bash
konduit serve -id dl4j-iris -c iris-dl4j.yaml -rwm -b

• Listing the active server in Konduit-Serving, konduit list:

%%bash
konduit list

• Show the log of the selected server’s id for 100 lines by konduit logs:

%%bash
konduit logs dl4j-iris -l 100

• Test the prediction of ML/DL model by konduit predict in the Konduit-Serving at the selected id (in this example: dl4j-iris):

%%bash
konduit predict dl4j-iris "{\"input\":[[1,2,3,4]]}"

• You can test again with another input value to get another result:

%%bash
konduit predict dl4j-iris "{\"input\":[[5.1,3.5,1.4,0.2]]}"

• And, the result will be like this:

• For more interactive result, you can edit the JSON/YAML file in the pipeline section as below:

---
host: "localhost"
port: 0
use_ssl: false
protocol: "HTTP"
static_content_root: "static-content"
static_content_url: "/static-content"
static_content_index_page: "/index.html"
kafka_configuration:
  start_http_server_for_kafka: true
  http_kafka_host: "localhost"
  http_kafka_port: 0
  consumer_topic_name: "inference-in"
  consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
  consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
  consumer_group_id: "konduit-serving-consumer-group"
  consumer_auto_offset_reset: "earliest"
  consumer_auto_commit: "true"
  producer_topic_name: "inference-out"
  producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
  producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
  producer_acks: "1"
mqtt_configuration: {}
custom_endpoints: []
pipeline:
  steps:
  - '@type': "DEEPLEARNING4J"
    modelUri: "dl4j_iris_model.zip"
    inputNames:
    - "input"
    outputNames:
    - "output"
  - '@type': "CLASSIFIER_OUTPUT"
    input_names: "layer2"
    labels:
      - Sentosa
      - Versicolor
      - Virginica

• So, you will get the result of classification straightforward with prediction's label:

• Lastly but not least, use konduit stop to terminate selected id in Konduit-Serving:

%%bash
konduit stop dl4j-iris

Congratulation! You have deployed Konduit-Serving on your own. What's next?