This page illustrates a simple client-server interaction to perform inference on a DL4J image classification model using the Python SDK for Konduit Serving.
import numpy as np
import os
This page documents two ways to create Konduit Serving configurations with the Python SDK:
Using Python to create a configuration, and
Writing the configuration as a YAML file, then serving it using the Python SDK.
These approaches are documented in separate tabs throughout this page. For example, the following code block shows the imports for each approach in separate tabs:
from konduit import ModelConfig, TensorDataTypesConfig, ModelConfigType, \
ModelStep, ParallelInferenceConfig, ServingConfig, InferenceConfiguration
from konduit.server import Server
from konduit.client import Client
from konduit.load import server_from_file, client_from_file
Saving models in Deeplearning4j
The following is a short Java program that loads a simple CNN model from DL4J's model zoo, initializes weights, then saves the model to a new file, SimpleCNN.zip.
import org.deeplearning4j.nn.conf.MultiLayerConfiguration;
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
import org.deeplearning4j.zoo.ZooModel;
import org.deeplearning4j.zoo.model.SimpleCNN;
import java.io.File;
public class SaveSimpleCNN {
private static int nClasses = 5;
private static boolean saveUpdater = false;
public static void main(String[] args) throws Exception {
ZooModel zooModel = SimpleCNN.builder()
.numClasses(nClasses)
.inputShape(new int[]{3, 224, 224})
.build();
MultiLayerConfiguration conf = ((SimpleCNN) zooModel).conf();
MultiLayerNetwork net = new MultiLayerNetwork(conf);
net.init();
System.out.println(net.summary());
File locationToSave = new File("SimpleCNN.zip");
net.save(locationToSave, saveUpdater);
}
}
A reference Java project using DL4J 1.0.0-beta6 is provided in this repository with a Maven pom.xml dependencies file. If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of the SaveSimpleCNN class.
Overview
Konduit Serving works by defining a series of steps. These include operations such as
Pre- or post-processing steps
One or more machine learning models
Transforming the output in a way that can be understood by humans
If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single ModelStep.
Configure the step
Define the DL4J configuration as a ModelConfig object.
model_config_type: This argument requires a ModelConfigType object. In the Java program above, we recognized that SimpleCNN is configured as a MultiLayerNetwork, in contrast with the ComputationGraph class, which is used for more complex networks. Specify model_type as MULTI_LAYER_NETWORK, and model_loading_path to point to the location of DL4J weights saved in the ZIP file format.
In the Java program above, we recognised that SimpleCNN is configured as a MultiLayerNetwork, in contrast with the ComputationGraph class, which is used for more complex networks. Hence, we create a dl4j_mln_step of type MULTI_LAYER_NETWORK.
model_loading_path denotes the location of the model file.
input_names and output_names denote the names of the input and output nodes, as lists.
parallel_inference_config: specify the number of workers to run in parallel. Here, we specify workers=1.
Accepted input and output data formats are as follows:
Input: JSON, ARROW, IMAGE, ND4J (not yet implemented) and NUMPY.
Output: NUMPY, JSON, ND4J (not yet implemented) and ARROW.
Start the server
server.start()
Starting server...
Server has started successfully.
<subprocess.Popen at 0x2723b619ac8>
konduit_yaml_path = "../yaml/deeplearning4j.yaml"
server = server_from_file(konduit_yaml_path)
server.start()
Configure the client
To configure the client, create a Client object with the following arguments:
input_data_format: data format passed to the server for inference.
output_data_format: data format returned by the server endpoint.
return_output_data_format: data format to be returned to the client. Note that this argument can be used to convert the output returned from the server to the client into a different format, e.g. NUMPY to JSON.
We generate a (3, 224, 224) array of random numbers between 0 and 255 as input to the model for prediction.
NDARRAY inputs to ModelSteps must be specified with a preceding batchSize dimension. For batches with a single observation, this can be done by using numpy.expand_dims() to add an additional dimension to your array.
Before requesting for a prediction, we normalize the image to be between 0 and 1:
Before running this notebook, run the build_jar.py script or the konduit init command. Refer to the page for details.
tensor_data_types_config: The ModelConfig object requires a dictionary input_data_types. Its keys should represent column names, and the values should represent data types as strings, e.g. "INT32". See for a list of supported data types.
input_data_types maps the data types of the input nodes to the data type. See for a list of supported data types.
