EN master
Examples
Change Logs
Components
Describes the components that make up Konduit-Serving
Components
Konduit-Serving has many internal components that work together to achieve the desired result. Following are the main concepts and components that make up Konduit-Serving:
- 1.CLI
- 2.Jar Package
- 3.Pipelines
- 4.Pipeline Step
- 5.Inference Configuration
This page will go through each one of them and explain what they are used for.
CLI Interface
Konduit-Serving comes with a CLI interface (with a
konduit alias) that's responsible for taking care of most aspects of the application. The help command will describe most of what we are able to do with Konduit-Serving. Executing konduit --help command will show us the following output:1
$ konduit --help
2
---------------------------------------------------------------------------
3
Usage: konduit [COMMAND] [OPTIONS] [arg...]
4
​
5
Commands:
6
build Command line interface for performing Konduit Serving builds.
7
config A helper command for creating boiler plate json/yaml for
8
inference configuration
9
inspect Inspect the details of a particular konduit server.
10
list Lists the running konduit servers.
11
logs View the logs of a particular konduit server
12
predict Run inference on konduit servers using given inputs
13
profile Command to List, view, edit, create and delete konduit
14
serving run profiles.
15
pythonpaths A utility command to manage system installed and manually
16
registered python binaries.
17
serve Start a konduit server application
18
stop Stop a running konduit server
19
version Displays konduit-serving version.
20
​
21
Run 'konduit COMMAND --help' for more information on a command.
22
---------------------------------------------------------------------------
Copied!
Each command describes its shorthand description right in front of it. If you want to look at an individual command in detail, you can use the corresponding --help command with them. For example, the help menu for the logs command can be seen by executing, konduit logs --help:
1
$ konduit logs --help
2
---------------------------------------------------------------------------
3
Usage: konduit logs [-f] [-l <value>] server-id
4
​
5
View the logs of a particular konduit server
6
​
7
View the logs of a particular konduit server given an id.
8
​
9
Example usages:
10
--------------
11
- Outputs the log file contents of server with an id of 'inf_server':
12
$ konduit logs inf_server
13
​
14
- Outputs and tail the log file contents of server with an id of 'inf_server':
15
$ konduit logs inf_server -f
16
​
17
- Outputs and tail the log file contents of server with an id of 'inf_server'
18
from the last 10 lines:
19
$ konduit logs inf_server -l 10 -f
20
--------------
21
​
22
Options and Arguments:
23
-f,--follow Follow the logs output.
24
-l,--lines <value> Sets the number of lines to be printed. Default is '10'.
25
Use -1 for outputting everything.
26
​
27
<server-id> Konduit server id
28
---------------------------------------------------------------------------
Copied!
As can be seen, the --help command for an individual help command describes its functionality in detail along with some explicit examples and use cases. It also describes each individual optional/non-optional argument that can be used with it. This can come in very handy while learning about konduit-serving for the first time and is a useful starting place to play around with a specific command. You can do the same for the rest of the commands. Which are:
- build
- config
- inspect
- list
- logs
- predict
- metrics
- profile
- pythonpaths
- serve
- stop
- version
Jar File Package
Each Konduit-Serving distribution whether it is for Windows, Linux or MacOS comes contained in a JAR file. So, you'll need a Java Virtual Machine present in the system where you're using Konduit-Serving as a Model Pipeline Server. The CLI itself is linked with the jar file and utilizes a java runtime internally to interact with the Konduit-Serving package. If you look at the konduit serving distribution, you'll see the following folder architecture in the root folder:
Could not load image
The main CLI logic is places under bin/konduit file, which contains the following content:
1
#!/usr/bin/env bash
2
​
3
SCRIPT_DIR="$(dirname "$0")"
4
​
5
. ${SCRIPT_DIR}/../conf/konduit-serving-env.sh
6
​
7
java -jar -Dvertx.cli.usage.prefix=konduit ${SCRIPT_DIR}/../konduit.jar "[email protected]"
Copied!
As you can see, it uses the java command which is available through a Java runtime environment. The java command itself uses the konduit.jar file which is the main application package inside a Konduit-Serving distribution.
This JAR file will be used as a Java application dependency while creating custom endpoints logic for a Konduit-Serving pipeline. We'll get to how we can do that later in this notebook.
Introduction to Pipelines
Throughout this document the term "Model Serving Pipeline" has been used. This refers to how Machine Learning or Deep Learning models get served on an application server. Machine/Deep Learning models work on n-dimensional arrays (also known as ND-Arrays). They don't know how to convert a JPEG or PNG image into numbers directly. Instead, they expect pre-processed data in the form of a multidimensional array. Also, any other form of data, be it text, audio or video, gets converted into numbers ND-Arrays before getting fed into a machine learning model.
The process during which the data is converted from one form to another is called pre-processing and is done just before it's fed as a model input. So, in a sense you can see this as being Lego blocks fitting into each other. One part takes input in a specific form and outputs it into another form, which in turn gets fed into the next part. This chaining of processes creates a series of steps which have specific jobs to perform before the next step and the end result is a machine learning Pipeline. The typical flow of the pipeline looks like the following:
Could not load image
A pipeline can also be in the form of a directed acyclic graph or DAG where data can flow into the graph and can give multiple outputs. In Konduit-Serving a Pipeline graph can also contain optional graph branches and can also concatenate outputs from multiple graph nodes. For the sake of the current goal (BMI Model Serving) we'll stick to a Sequential Pipeline which only has one input and one output.
Pipeline Steps
Inside Konduit-Serving, a pipeline can be broken down into steps, where each step is responsible for performing a specific function. A pipeline step is the smallest component of a whole pipeline and can be used for a whole list of operations. To see the list of available pipeline steps you can use the config command in Konduit-Serving CLI.
1
$ konduit config --help
2
---------------------------------------------------------------------------
3
Usage: konduit config [-m] [-o <output-file>] -p <config> [-pr <value>] [-y]
4
​
5
A helper command for creating boilerplate json/yaml for inference configuration
6
​
7
This command is a utility to create boilerplate json/yaml configurations that can be conveniently modified to start konduit servers.
8
​
9
Example usages:
10
--------------
11
-- FOR SEQUENCE PIPELINES--
12
- Prints 'logging -> tensorflow -> logging' config in pretty format:
13
$ konduit config -p logging,tensorflow,logging
14
​
15
- Prints 'logging -> tensorflow -> logging' config with gRPC protocol
16
in pretty format:
17
$ konduit config -p logging,tensorflow,logging -pr grpc
18
​
19
- Prints 'dl4j -> logging' config in minified format:
20
$ konduit config -p dl4j,logging -m
21
​
22
- Saves 'dl4j -> logging' config in a 'config.json' file:
23
$ konduit config -p dl4j,logging -o config.json
24
​
25
- Saves 'dl4j -> logging' config in a 'config.yaml' file:
26
$ konduit config -p dl4j,logging -y -o config.json
27
​
28
​
29
-- FOR GRAPH PIPELINES --
30
- Generates a config that logs the input(1) then flow them through two
31
tensorflow models(2,3) and merges the output(4):
32
$ konduit config -p
33
1=logging(input),2=tensorflow(1),3=tensorflow(1),4=merge(2,3)
34
​
35
- Generates a config that logs the input(1) then channels(2) them through one
36
of the two tensorflow models(3,4) and then selects the output(5) based
37
on the value of the selection integer field 'select'
38
$ konduit config -p
39
1=logging(input),[2_1,2_2]=switch(int,select,1),3=tensorflow(2_1),4=tensorflow(2_2),5=any(3,4)
40
​
41
- Generates a config that logs the input(1) then channels(2) them through one
42
of the two tensorflow models(3,4) and then selects the output(5) based
43
on the value of the selection string field 'select' in the selection map
44
(x:0,y:1).
45
$ konduit config -p 1=logging(input),[2_1,2_2]=switch(string,select,x:0,y:1,1),3=tensorflow(2_1),4=tensorflow(2_2),5=any(3,4)
46
--------------
47
​
48
Options and Arguments:
49
-m,--minified If set, the output json will be printed in a
50
single line, without indentations. (Ignored
51
for yaml)
52
​
53
-o,--output <output-file> Optional: If set, the generated json/yaml will
54
be saved here. Otherwise, it's printed on the
55
console.
56
​
57
-p,--pipeline <config> A comma-separated list of sequence/graph
58
pipeline
59
steps to create boilerplate configuration
60
from. For
61
sequences, allowed values are: [crop_grid,
62
crop_fixed_grid, dl4j, keras,
63
Draw_bounding_box, draw_fixed_grid, draw_grid,
64
draw_segmentation,extract_bounding_box,
65
Camera_frame_capture, video_frame_capture,
66
Image_to_ndarray, logging,
67
ssd_to_bounding_box, samediff, show_image,
68
tensorflow, nd4jtensorflow, python, onnx].
69
For graphs, the list item should be in the
70
format
71
'<output>=<type>(<inputs>)' or
72
'[outputs]=switch(<inputs>)' for switches. The
73
pre-defined root input is named, 'input'.
74
Examples
75
are ==> Pipeline step:
76
'a=tensorflow(input),b=dl4j(input)' Merge
77
Step:
78
'c=merge(a,b)' Switch Step (int):
79
'[d1,d2,d3]=switch(int,select,input)' Switch
80
Step
81
(string):
82
'[d1,d2,d3]=switch(string,select,x:1,y:2,z:3,input)'
83
'Any Step: 'e=any(d1,d2,d3)' See the examples
84
above for more usage information.
85
-pr,--protocol <value> Protocol to use with the server. Allowed
86
values are
87
[http, grpc, mqtt]
88
-y,--yaml Set if you want the output to be a yaml
89
configuration.
90
---------------------------------------------------------------------------
Copied!
config command can take a pipeline pattern string and creates a pipeline configuration as well as server configuration. The server configuration combined with pipeline step configuration is collectively called Inference Configuration in Konduit-Serving. This is the final output from the config command.
The following pipeline step types can be used to configure a pipeline configuration.
#
Pipeline Step Type Name
Description
01
CROP_GRID
A pipeline step that crops sub images out of a larger image, based on a grid.
02
CROP_FIXED_GRID
This step is similar to CROP_GRID with the difference that the x/y location values are hardcoded into the configuration, instead of coming dynamically from the input Data instance.
03
DL4J
A model pipelinestep that serves a DL4J model.
04
KERAS
A model pipeline step that serves a Keras model.
05
DRAW_BOUNDING_BOX
A pipeline step that configures how to draw a bounding box onto an image. The bounding box data, that's to be drawn, is taken from the previous step's data instance.
06
DRAW_GRID
Draw a grid on the specified image, based on the x/y coordinates of the corners, and the number of segments within the grid in both directions.
07
DRAW_FIXED_GRID
A pipeline step that draws a grid on an image. This is similar to DRAW_GRID but the corner x/y location values are hardcoded into the configuration (via points), instead of coming dynamically from the input Data instance.
08
DRAW_SEGMENTATION
A pipeline step that configures how to draw a segmentation mask, optionally on an image.
09
EXTRACT_BOUNDING_BOX
A pipeline step that extracts sub-images from an input image, based on the locations of input bounding boxes.
10
CAMERA_FRAME_CAPTURE
A pipeline step that specifies an input that's taken from a camera feed.
11
VIDEO_FRAME_CAPTURE
A pipeline step that configures how to extract a single frame from a video each time inference is called. The video path is hardcoded, mainly used for testing/demo purposes.
12
IMAGE_TO_NDARRAY
A PipelineStep for converting images to n-dimensional arrays.
13
LOGGING
A step that logs the key and value pairs coming from the previous steps.
14
SSD_TO_BOUNDING_BOX
A pipeline step that configures extraction of bounding boxes from an SSD model output.
15
SAMEDIFF
A model pipeline step that serves a SameDiff model.
16
SHOW_IMAGE
A pipeline step that configures how to show/render an image from a previous step in an application frame. Usually only used for testing and debugging locally, not when serving from HTTP/GRPC etc endpoints.
17
TENSORFLOW
A model pipeline step that serves a TensorFlow model using Tensorflow Java API. This is packaged into Konduit-Serving through JavaCPP.
18
ND4JTENSORFLOW
A pipeline step that configures a TensorFlow model that is to be executed based on an ND4J graph runner. This has performance benefits over native TensorFlow Java distribution.
19
PYTHON
This pipeline step can take an arbitrary python script and serve that through Konduit-Serving.
Inference Configuration
Inference configuration contains all the details about how the pipeline server should be setup, which protocol it’s supposed to use for sending in/out data, whether we have a sequential or graph pipeline and finally how the pipeline is configured. The config command outputs inference configuration for us that can be directly used with a serve command after necessary changes are made. An Example of inference configuration through config command is as follows:
1
$ konduit config --pipeline logging --yaml
2
---------------------------------------------------------------------------
3
---
4
host: "localhost"
5
port: 0
6
use_ssl: false
7
protocol: "HTTP"
8
kafka_configuration:
9
start_http_server_for_kafka: true
10
http_kafka_host: "localhost"
11
http_kafka_port: 0
12
consumer_topic_name: "inference-in"
13
consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
14
consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer"
15
consumer_group_id: "konduit-serving-consumer-group"
16
consumer_auto_offset_reset: "earliest"
17
consumer_auto_commit: "true"
18
producer_topic_name: "inference-out"
19
producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
20
producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer"
21
producer_acks: "1"
22
mqtt_configuration: {}
23
custom_endpoints: []
24
pipeline:
25
steps:
26
- '@type': "LOGGING"
27
logLevel: "INFO"
28
log: "KEYS_AND_VALUES"
29
---------------------------------------------------------------------------
Copied!
The above command creates a yaml configuration for a logging pipeline step. Logging pipeline step just takes in an input from the previous pipeline step and outputs (and logs on console) the same information. This is useful for debugging issues in the running servers.
This configuration can be saved inside a file by executing the following:
1
$ konduit config --pipeline logging --yaml --output server-conf.yaml
2
---------------------------------------------------------------------------
3
Config file created successfully at /root/server-conf.yaml
4
---------------------------------------------------------------------------
Copied!
Afterwards, this configuration can be used with the serve command to start a Konduit-Serving server. Example is following:
1
$ konduit serve --config server-conf.yaml -id server --background --runWithoutManifest
2
---------------------------------------------------------------------------
3
Starting konduit server...
4
Using classpath: /root/konduit/bin/../konduit.jar
5
INFO: Running command /root/miniconda/jre/bin/java -Dkonduit.logs.file.path=/root/.konduit-serving/command_logs/server.log -Dlogback.configurationFile=/tmp/logback-run_command_4b1acb956d0d436c.xml -jar /root/konduit/bin/../konduit.jar run --instances 1 -s inference -c server-conf.yaml -Dserving.id=server
6
For server status, execute: 'konduit list'
7
For logs, execute: 'konduit logs server'
8
---------------------------------------------------------------------------
Copied!
Last modified 1yr ago