Using Docker

Search…
Using Docker
Guide to start using Konduit-Serving with Docker
In this section, we provide guidance on how to demonstrate Konduit-Serving with Docker. Konduit-Serving is a platform to serve the ML/DL models directly through a single line of code. We'll start building and installing the Konduit-Serving from docker image and follow by deploying the trained model in Konduit-Serving.
Prerequisites
You will need following prerequisites to follow along
​Docker 19.03.14 
​Docker-Compose 1.27.4​
To ensure Konduit-Serving works properly, install these prerequisites. We’ll be using Konduit-Serving to deploy machine/deep learning pipelines easier using a few lines of code. We've prepared a Github repository to simplify showcasing Konduit-Serving.
Introduction to Repository
The repository contains simple examples of deploying a pipeline using different model types. To clone the repository, run the following command:
1
git clone https://github.com/ShamsUlAzeem/konduit-serving-demo
Copied!
Build the CPU version of docker image by running the following command in the root directory:
1
bash build.sh CPU
Copied!
After a successful build, run docker image with docker-compose at current working directory:
1
docker-compose up
Copied!
Now, open JupyterLab in the browser.
Explore Konduit-Serving in Repository
Let's take a look inside the demos directory from konduit-serving-demo. Each folder inside the demos folder demonstrate serving a different kind of model, using a configuration file in either JSON or YAML, through the Konduit-Serving CLI.
The examples use different frameworks, including Keras, Tensorflow, Pytorch, and DL4J. These examples can be run in IPython Notebook (.ipynb) with Java-based kernel. Konduit-Serving provides a platform for users to take advantage of models using konduit CLI such as serve, list, logs, predict and stop. You can also build your model and start using Konduit-Serving.
Let's look at training and serving a model with Konduit-Serving.
Build your model
These are steps to train a model for Keras and DL4J from scratch:
Keras
DL4J
First, import library that will be use:
1
import keras
2
from keras.models import Sequential
3
from keras.layers import Dense
4
from keras.optimizers import Adam
5
from sklearn.preprocessing import LabelEncoder
6
from sklearn.model_selection import train_test_split
7
from sklearn.metrics import confusion_matrix
8
import seaborn as sns
9
import pandas as pd
10
import numpy as np
Copied!
Load Iris data set:
1
dataset = sns.load_dataset("iris")
2
print(dataset)
Copied!
Display distribution of data set:
1
sns.set(style="ticks")
2
sns.set_palette("husl")
3
sns.pairplot(dataset.iloc[:,0:6], hue="species")
Copied!
Apply data pre-processing and split data into training and testing:
1
X = dataset.iloc[:,0:4].values
2
y = dataset.iloc[:,4].values
3
​
4
encoder =  LabelEncoder()
5
y1 = encoder.fit_transform(y)
6
Y = pd.get_dummies(y1).values
7
​
8
X_train,X_test, y_train,y_test = train_test_split(X,Y,test_size=0.3,random_state=0)
Copied!
Configure the model and print summary:
1
model = Sequential()
2
model.add(Dense(4,input_shape=(4,),activation='relu', name="input"))
3
model.add(Dense(3,activation='softmax', name='output'))
4
model.compile(Adam(lr=0.01),'categorical_crossentropy',metrics=['accuracy'])
5
model.summary()
Copied!
Train the model by training data with 800 epochs:
1
model.fit(X_train,y_train,epochs=800)
Copied!
Test the model by predicting testing data:
1
y_pred = model.predict(X_test)
2
y_test_class = np.argmax(y_test,axis=1)
3
y_pred_class = np.argmax(y_pred,axis=1)
Copied!
Display confusion matrix to see more details of model prediction between actual and predicted result (if satisfied can save the model):
1
cm = confusion_matrix(y_test_class, y_pred_class)
2
print(cm)
Copied!
Also, you can try to predict by using your value in the trained model:
1
X_test2 = np.array([[5, 3.9, 2, 0.5],[5,2.5,3,1],[8,3.5,6,2]])#setosa, versicolor, virginica
2
y_pred2 = model.predict(X_test2)
3
print(y_pred2)
Copied!
Print the result of classification:
1
print(np.argmax(y_pred2,axis=1)) #0 = setosa 1 = versicolor 2 = virginica
Copied!
Then, save the trained model in HDF5 (.h5) format which will be used in Konduit-Serving later:
1
model.save_weights("model.h5")
2
print("Saved model to disk")
Copied!
The model is now ready to be deployed.
First, import library that will be use (auto generated if you are using IntelliJ - proceed to next step) :
1
import org.datavec.api.records.reader.RecordReader;
2
import org.datavec.api.records.reader.impl.csv.CSVRecordReader;
3
import org.datavec.api.split.FileSplit;
4
import org.deeplearning4j.api.storage.StatsStorage;
5
import org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator;
6
import org.deeplearning4j.nn.conf.BackpropType;
7
import org.deeplearning4j.nn.conf.MultiLayerConfiguration;
8
import org.deeplearning4j.nn.conf.NeuralNetConfiguration;
9
import org.deeplearning4j.nn.conf.layers.DenseLayer;
10
import org.deeplearning4j.nn.conf.layers.OutputLayer;
11
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
12
import org.deeplearning4j.nn.weights.WeightInit;
13
import org.deeplearning4j.ui.api.UIServer;
14
import org.deeplearning4j.ui.stats.StatsListener;
15
import org.deeplearning4j.ui.storage.InMemoryStatsStorage;
16
import org.deeplearning4j.util.ModelSerializer;
17
import org.nd4j.evaluation.classification.Evaluation;
18
import org.nd4j.linalg.activations.Activation;
19
import org.nd4j.linalg.dataset.DataSet;
20
import org.nd4j.linalg.dataset.SplitTestAndTrain;
21
import org.nd4j.linalg.dataset.api.iterator.DataSetIterator;
22
import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization;
23
import org.nd4j.linalg.dataset.api.preprocessor.NormalizerStandardize;
24
import org.nd4j.linalg.io.ClassPathResource;
25
import org.nd4j.linalg.learning.config.Nesterovs;
26
import org.nd4j.linalg.lossfunctions.LossFunctions;
27
​
28
import java.io.File;
29
import java.util.Arrays;
Copied!
Declare the variables that will be used in the training process (this code starts under main body):
1
int numLinesToSkip = 0;
2
char delimiter = ',';
3
​
4
int batchSize = 150; // Iris data set: 150 examples total. We are loading all of them into one DataSet (not recommended for large data sets)
5
int labelIndex = 4; // index of label/class column
6
int numClasses = 3;
7
​
8
int seed = 1234;
9
int epochs = 800;
10
double learningRate = 0.1;
11
int nHidden = 4;
Copied!
Load Iris data set from resources file:
1
File inputFile = new ClassPathResource("datavec/iris.txt").getFile();
2
FileSplit fileSplit = new FileSplit(inputFile);
Copied!
Get data set using record reader (to handle loading or parsing):
1
RecordReader recordReader = new CSVRecordReader(numLinesToSkip, delimiter);
2
recordReader.initialize(fileSplit);
Copied!
Create iterator from record reader:
1
DataSetIterator iterator = new RecordReaderDataSetIterator(recordReader, batchSize, labelIndex, numClasses);
2
DataSet allData = iterator.next();
Copied!
Shuffling the arrangement of data and splitting into training and testing:
1
allData.shuffle(seed);
2
SplitTestAndTrain testAndTrain = allData.splitTestAndTrain(0.7);
3
DataSet trainingData = testAndTrain.getTrain();
4
DataSet testData = testAndTrain.getTest();
Copied!
Apply data pre-processing by normalization:
1
DataNormalization normalizer = new NormalizerStandardize();
2
normalizer.fit(trainingData);
3
normalizer.transform(trainingData);
4
normalizer.transform(testData);
Copied!
Configure and initiate the model that will be used:
1
MultiLayerConfiguration config = new NeuralNetConfiguration.Builder()
2
       .seed(seed)
3
       .weightInit(WeightInit.XAVIER)
4
       .activation(Activation.TANH)
5
       .updater(new Nesterovs(learningRate, Nesterovs.DEFAULT_NESTEROV_MOMENTUM))
6
       .l2(1e-4)
7
       .list()
8
       .layer(0, new DenseLayer.Builder()
9
               .nIn(labelIndex)
10
               .nOut(nHidden)
11
               .build())
12
       .layer(1, new DenseLayer.Builder()
13
               .nOut(nHidden)
14
               .build())
15
       .layer(2, new OutputLayer.Builder(LossFunctions.LossFunction.MCXENT)
16
               .activation(Activation.SOFTMAX)
17
               .nOut(numClasses)
18
               .name("output")
19
               .build())
20
       .backpropType(BackpropType.Standard)
21
       .build();
22
​
23
MultiLayerNetwork model = new MultiLayerNetwork(config);
24
model.init();
Copied!
Display in UI of training process which can be seen when open in browser (optional):
1
StatsStorage storage = new InMemoryStatsStorage();
2
UIServer server = UIServer.getInstance();
3
server.attach(storage);
4
model.setListeners(new StatsListener(storage, 10));
Copied!
Train the model using training data:
1
for (int i=0; i < epochs; i++) {
2
   model.fit(trainingData);
3
}
Copied!
Evaluate the model by using testing data (save the model if satisfied):
1
Evaluation eval = new Evaluation(3);
2
eval.eval(model.output(testData.getFeatures()),testData.getLabels());
3
System.out.println(eval.stats());
Copied!
Then, save the model at the location you want to keep with the name in Zip format. The model is ready to be used (for example, the model saved in the current working directory):
1
File locationToSave = new File("./dl4j_iris_model.zip");
2
boolean saveUpdater = true;
3
ModelSerializer.writeModel(model,locationToSave,saveUpdater);
4
System.out.println("******PROGRAM IS FINISHED******");
Copied!
The model is now ready to be deployed.
Deploy your model in Konduit-Serving
Now, you are ready to deploy a model in Konduit-Serving by using konduit CLI. This step needs a saved model file (h5 or zip file) and JSON/YAML configuration file. Let's begin with:
Create a new folder in the demos directory (for example, 10-iris-model):
Drag the model file into the demos directory folder and create an IPython Notebook file with Java kernel (for example, iris-model.ipynb). 
In this notebook, we will use konduit CLI.
Check the version of Konduit-Serving, and either is installed or not:
1
%%bash
2
konduit -version
Copied!
Create the JSON/YAML configuration file by using konduit config command:
Keras
DL4J
YAML configuration:
1
%%bash
2
konduit config -p keras -o iris-keras.yaml -y
Copied!
Or, you can try this command to get a JSON configuration file:
1
%%bash
2
konduit config -p keras -o iris-keras.json
Copied!
YAML configuration:
1
%%bash
2
konduit config -p dl4j -o iris-dl4j.yaml -y
Copied!
Or, you can try this command to get a JSON configuration file:
1
%%bash
2
konduit config -p dl4j -o iris-dl4j.json
Copied!
In the configuration file, you need to edit the YAML/JSON file in the pipeline section (for this example, we will use YAML with DL4J):
Before
After
1
---
2
host: "localhost"
3
port: 0
4
use_ssl: false
5
protocol: "HTTP"
6
static_content_root: "static-content"
7
static_content_url: "/static-content"
8
static_content_index_page: "/index.html"
9
kafka_configuration:
10
  start_http_server_for_kafka: true
11
  http_kafka_host: "localhost"
12
  http_kafka_port: 0
13
  consumer_topic_name: "inference-in"
14
  consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
15
  consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
16
  consumer_group_id: "konduit-serving-consumer-group"
17
  consumer_auto_offset_reset: "earliest"
18
  consumer_auto_commit: "true"
19
  producer_topic_name: "inference-out"
20
  producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
21
  producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
22
  producer_acks: "1"
23
mqtt_configuration: {}
24
custom_endpoints: []
25
pipeline:
26
  steps:
27
  - '@type': "DEEPLEARNING4J"
28
    modelUri: "<path_to_model>"
29
    inputNames:
30
    - "1"
31
    - "2"
32
    outputNames:
33
    - "11"
34
    - "22"
Copied!
1
---
2
host: "localhost"
3
port: 0
4
use_ssl: false
5
protocol: "HTTP"
6
static_content_root: "static-content"
7
static_content_url: "/static-content"
8
static_content_index_page: "/index.html"
9
kafka_configuration:
10
  start_http_server_for_kafka: true
11
  http_kafka_host: "localhost"
12
  http_kafka_port: 0
13
  consumer_topic_name: "inference-in"
14
  consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
15
  consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
16
  consumer_group_id: "konduit-serving-consumer-group"
17
  consumer_auto_offset_reset: "earliest"
18
  consumer_auto_commit: "true"
19
  producer_topic_name: "inference-out"
20
  producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
21
  producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
22
  producer_acks: "1"
23
mqtt_configuration: {}
24
custom_endpoints: []
25
pipeline:
26
  steps:
27
  - '@type': "DEEPLEARNING4J"
28
    modelUri: "dl4j_iris_model.zip"
29
    inputNames:
30
    - "input"
31
    outputNames:
32
    - "output"
Copied!
To determine the name of input and output, you can use Netron to read ML/DL model, for example:
Keras
DL4J
In node properties, use the name's value of first weights as "inputNames".
In node properties, use the name's value of last weights as "outputname".
In model properties, use the name's value of input for "inputNames" and output for "outputNames".
Start the server by using konduit serve command and give the id's name based on your own:
1
%%bash
2
konduit serve -id dl4j-iris -c iris-dl4j.yaml -rwm -b
Copied!
Listing the active server in Konduit-Serving, konduit list:
1
%%bash
2
konduit list
Copied!
Show the log of the selected server’s id for 100 lines by konduit logs:
1
%%bash
2
konduit logs dl4j-iris -l 100
Copied!
Test the prediction of ML/DL model by konduit predict in the Konduit-Serving at the selected id (in this example: dl4j-iris):
1
%%bash
2
konduit predict dl4j-iris "{\"input\":[[1,2,3,4]]}"
Copied!
You can test again with another input value to get another result:
1
%%bash
2
konduit predict dl4j-iris "{\"input\":[[5.1,3.5,1.4,0.2]]}"
Copied!
And, the result will be like this:
For more interactive result, you can edit the JSON/YAML file in the pipeline section as below:
1
---
2
host: "localhost"
3
port: 0
4
use_ssl: false
5
protocol: "HTTP"
6
static_content_root: "static-content"
7
static_content_url: "/static-content"
8
static_content_index_page: "/index.html"
9
kafka_configuration:
10
  start_http_server_for_kafka: true
11
  http_kafka_host: "localhost"
12
  http_kafka_port: 0
13
  consumer_topic_name: "inference-in"
14
  consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
15
  consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
16
  consumer_group_id: "konduit-serving-consumer-group"
17
  consumer_auto_offset_reset: "earliest"
18
  consumer_auto_commit: "true"
19
  producer_topic_name: "inference-out"
20
  producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
21
  producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
22
  producer_acks: "1"
23
mqtt_configuration: {}
24
custom_endpoints: []
25
pipeline:
26
  steps:
27
  - '@type': "DEEPLEARNING4J"
28
    modelUri: "dl4j_iris_model.zip"
29
    inputNames:
30
    - "input"
31
    outputNames:
32
    - "output"
33
  - '@type': "CLASSIFIER_OUTPUT"
34
    input_names: "layer2"
35
    labels:
36
      - Sentosa
37
      - Versicolor
38
      - Virginica
Copied!
So, you will get the result of classification straightforward with prediction's label:
Lastly but not least, use konduit stop to terminate selected id in Konduit-Serving:
1
%%bash
2
konduit stop dl4j-iris
Copied!
Congratulation! You have deployed Konduit-Serving on your own. What's next?
Quickstart
Using Java SDK
Last modified 7mo ago
Copy link
Contents
Prerequisites
Introduction to Repository
Explore Konduit-Serving in Repository
Build your model
Deploy your model in Konduit-Serving