# Introduction Konduit Serving is a serving system and framework focused on deploying machine learning pipelines to production. ## Overview Konduit Serving provides building blocks for developers to write their own production machine learning pipelines from pre-processing to model serving, exposable as a simple REST API. The core abstraction is an idea called a **pipeline step**. A pipeline step performs a task such as: 1. pre-processing steps; 2. running one or more machine learning models; and 3. post-processing steps: transforming the output in a way that can be understood by humans, such as labels in a classification example, as part of using a machine learning model in a deployment scenario. For instance, a `ModelStep` performs inference on a (mix of) TensorFlow, Keras, Deeplearning4j (DL4J) or Predictive Model Markup Language (PMML) models. {% content-ref url="/pages/-Lupw-aLrZSPXWeB9hMA" %} [TensorFlow (1.x)](/0.1.0-snapshot/examples/python/tensorflow-model-serving) {% endcontent-ref %} {% content-ref url="/pages/-LuptbQpkK76qsdqbAd6" %} [Broken mention](broken://pages/-LuptbQpkK76qsdqbAd6) {% endcontent-ref %} {% content-ref url="/pages/-Lv4jqWJf9I0LtMWQkJ3" %} [Keras (TensorFlow 2.0)](/0.1.0-snapshot/examples/python/keras) {% endcontent-ref %} A custom pipeline step can be built using a `PythonStep`. This allows you to embed pre- or post-processing steps into your machine learning pipeline, or to serve models built in frameworks that do not have built-in`ModelStep`s such as scikit-learn and PyTorch. {% content-ref url="/pages/-LuptfSTVbseUO9gX9uP" %} [Open Neural Network Exchange (ONNX)](/0.1.0-snapshot/examples/python/onnx) {% endcontent-ref %} Konduit Serving also contains functionality for other pre-processing tasks, such as DataVec transform processes and image transforms. {% content-ref url="/pages/-LuptvmFAcwPZJOvHxpT" %} [DataVec](/0.1.0-snapshot/examples/python/datavec) {% endcontent-ref %} ## Usage One way to configure a Konduit Serving instance is by using a [YAML file](/0.1.0-snapshot/yaml-configurations). The following YAML file configures a Konduit Serving instance to run a short Python script as specified in the `python_code` argument: ```yaml serving: http_port: 1337 input_data_format: NUMPY output_data_format: NUMPY steps: python_step: type: PYTHON python_code: | first += 2 second = first python_inputs: first: NDARRAY python_outputs: second: NDARRAY client: port: 1337 ``` [Installing the Konduit Serving Python SDK](/0.1.0-snapshot/installation) exposes the `konduit` command line interface (CLI). Assuming the YAML file above is saved in the current directory as `hello-world.yaml`, start a Konduit Serving instance by running the following code in the command line: ```bash konduit serve --config hello-world.yaml ``` This exposes a REST API for sending data to the server for inference. Inputs can be sent using the CLI, the Python SDK or any other application that supports sending HTTP POST requests such as [requests ](https://requests.readthedocs.io/en/master/)or [UiPath](https://docs.uipath.com/activities/docs/http-client) (for RPA-based workflows). Finally, stop the Konduit Serving instance: ```bash konduit stop-server --config hello-world.yaml ``` To get started with Konduit Serving, check out the Quickstart page. {% content-ref url="/pages/-LwHPx\_6wFFVL3XCNywk" %} [Quickstart](/0.1.0-snapshot/quickstart) {% endcontent-ref %} ## Why Konduit Serving? ### Python-first We strive to provide a Python-first SDK that makes it easy to integrate Konduit Serving into a Python-first workflow. {% content-ref url="/pages/-Lve8rd7Huj5i35VZwzg" %} [Monitoring with Grafana](/0.1.0-snapshot/model-monitoring/monitoring-grafana) {% endcontent-ref %} ### Modern visualization standards We want to expose [modern standards](http://prometheus.io/) for monitoring everything from your GPU to your inference time. Konduit Serving supports visualization applications such as [Grafana](http://grafana.com) that support the [Prometheus](https://prometheus.io/) standard for visualizing data. ### Performance and security Konduit Serving was built with the goal of providing proper low-level interoperability with native math libraries such as TensorFlow and DL4J's core math library libnd4j. At the core of Konduit Serving are the [JavaCPP Presets](https://github.com/bytedeco/javacpp-presets), [Vert.x](http://vertx.io) and DL4J for running Keras models in Java. Combining JavaCPP's low-level access to C-like APIs from Java with Java's robust server-side application development (Vert.x on top of [netty](http://netty.io/)) allows for better access to faster math code in production while minimizing the surface area where native code = more security flaws (mainly in server side networked applications). This allows us to do things like zero-copy memory access of NumPy arrays or Arrow records for consumption straight from the server without copy or serialization overhead. Extending that to Python SDK, we know when to return a raw Arrow record and return it as a pandas DataFrame. When dealing with deep learning, we can handle proper inference on the GPU (batching large workloads). ### Java microservices A Vert.x-based model server and pipeline development framework allows a thin abstraction that can be embedded in a Java microservice. ### Enterprise integration We aim to provide integrations with more enterprise platforms typically seen outside the big data space. # Quickstart Konduit Serving is a framework-agnostic model serving solution focused on deploying machine learning pipelines to production. The Python SDK allows data scientists to quickly test machine learning deployment scenarios, bridging the gap between data science teams and DevOps. Before running these commands, set up Konduit Serving according to the installation instructions on the Installation page: {% content-ref url="/pages/-LtTMd5ReUOXuuHoM\_rH" %} [Installation](/0.1.0-snapshot/installation) {% endcontent-ref %} Konduit Serving configuration files consist of `serving`, `steps` and `client` components. Save the configuration below as a text file named `hello-world.yaml` in your current directory: ```yaml serving: http_port: 1337 input_data_format: NUMPY output_data_format: NUMPY steps: python_step: type: PYTHON python_code: | first += 2 second = first python_inputs: first: NDARRAY python_outputs: second: NDARRAY client: port: 1337 ``` The pages in this section show you how to start and interact with a Konduit Serving instance. For these examples, the Konduit Serving instance and client are on the same machine. For quick experimentation, check out the quickstart for the command line interface (CLI): {% content-ref url="/pages/-LwHXeis5Wv0a3Y9FToQ" %} [Command line interface (CLI)](/0.1.0-snapshot/quickstart/quickstart-cli) {% endcontent-ref %} To access additional options, you will want to configure Konduit Serving instances with the Python SDK. Start with the Python quickstart: {% content-ref url="/pages/-LwHXqwszcR9gLBWn2qw" %} [Python SDK](/0.1.0-snapshot/quickstart/quickstart-python) {% endcontent-ref %} # Pip package To install the `konduit` pip package, you can run: ```bash pip install konduit ``` After installing the pip package, you can initialize the `konduit` CLI in the following way ## Initializing the CLI ### For git users If you use git you can build the binaries by running #### CPU ```bash konduit-init --chip cpu ``` #### GPU ```bash konduit-init --chip gpu ``` ### For non-git users If you want to download the pre-build binaries, you can #### CPU ```bash konduit-init --chip cpu -d ``` #### GPU ``` konduit-init --chip gpu -d ``` ## Verification After doing the above process, you can verify the installation by doing: ```bash konduit --version ``` If everything went alright then you should see something like ``` Konduit serving version: 0.1.0-SNAPSHOT Commit hash: 3f9ac52f ``` ## What's next? Try out how to work with the `konduit` CLI with an example workflow [here](/0.1.0-snapshot/quickstart/quickstart-cli). # Command line interface (CLI) A brief overview of konduit-serving command line interface. [konduit-serving](https://github.com/KonduitAI/konduit-serving) comes with a handy CLI that you can use to manage your serving instances. Konduit CLI comes with the [konduit](https://pypi.org/project/konduit/) pip package. After installing the pip package and [initializing the CLI](/0.1.0-snapshot/quickstart/pip-package#initializing-the-cli), you can use the `konduit` command line tool by typing the following on the terminal: ``` konduit --help ``` which should prompt all currently available commands: ``` Usage: konduit [COMMAND] [OPTIONS] [arg...] Commands: config A helper command for creating JSON for inference configuration inspect Inspect the details of a particular konduit server. list Lists the running konduit servers. logs View the logs of a particular konduit server predict Run inference on konduit servers using given inputs serve Start a konduit server application stop Stop a running konduit server version Displays konduit-serving version. Run 'konduit COMMAND --help' for more information on a command. ``` For more help on individual commands you can run `konduit [COMMAND] --help`. For example: ``` konduit serve --help ``` to view the usage of `serve` command: ``` Usage: konduit serve [-b] [-cp ] -c [-i ] [-jo ] [-s ] [-id ] Start a konduit server application Start a konduit server application. The application is identified with an id that can be set using the `--serving-id` or `-id` option. The application can be stopped with the `stop` command. This command takes the `run` command parameters. To see the run command parameters, execute `run --help` Example usages: -------------- - Starts a server in the foreground with an id of 'inf_server' using 'config.json' as configuration file: $ konduit serve -id inf_server -c config.json - Starts a server in the background with an id of 'inf_server' using 'config.json' as configuration file: $ konduit serve -id inf_server -c config.json -b -------------- Options and Arguments: -b,--background Runs the process in the background, if set. -cp,--classpath Provides an extra classpath to be used for the verticle deployment. -c,--config Specifies configuration that should be provided to the verticle. should reference either a text file containing a valid JSON object which represents the configuration OR be a JSON string. -i,--instances Specifies how many instances of the server will be deployed. Defaults to 1. -jo,--java-opts Java Virtual Machine options to pass to the spawned process such as "-Xmx1G -Xms256m -XX:MaxPermSize=256m". If not set the `JAVA_OPTS` environment variable is used. -s,--service Service type that needs to be deployed. Defaults to "inference" -id,--serving-id Id of the serving process. This will be visible in the 'list' command. This id can be used to call 'predict' and 'stop' commands on the running servers. If not given then an 8 character UUID is created automatically. ``` The `--help` argument for the individual commands gives you a quick summary and a detailed description of what the command is about along with a few examples of common usage patterns. ## **Example Workflow** Following is an example workflow of how to use the `konduit` CLI for serving an [ImageLoadingStep](/0.1.0-snapshot/steps/image-loading-pipeline-steps). #### 1. Create a configuration Before running any server, you'll have to configure a json configuration for the serving pipeline. The `config` command is a very handy tool to create a baseline configuration that you can edit later based on your requirements. In this workflow, you'll see how to create a basic configuration for reading image file and return the loaded image in `JSON` format with the `predict` command. To create an image configuration you can run the `config` command as follows: ``` konduit config -t image ``` You'll see the following output from it (might differ based on your local environment): ``` { "servingConfig" : { "createLoggingEndpoints" : false, "httpPort" : 0, "listenHost" : "localhost", "logTimings" : false, "metricTypes" : [ "CLASS_LOADER", "JVM_MEMORY", "JVM_GC", "PROCESSOR", "JVM_THREAD", "LOGGING_METRICS", "NATIVE" ], "metricsConfigurations" : [ ], "outputDataFormat" : "JSON", "uploadsDirectory" : "C:\\Users\\konduit\\AppData\\Local\\Temp" }, "steps" : [ { "@type" : "ImageLoadingStep", "dimensionsConfigs" : { }, "imageProcessingInitialLayout" : "NCHW", "imageProcessingRequiredLayout" : "NCHW", "imageTransformProcesses" : { }, "inputColumnNames" : { }, "inputNames" : [ "default" ], "inputSchemas" : { }, "originalImageHeight" : 0, "originalImageWidth" : 0, "outputColumnNames" : { }, "outputNames" : [ "default" ], "outputSchemas" : { }, "updateOrderingBeforeTransform" : false } ] } ``` {% hint style="warning" %} Note A port equal to `0` means that a random port will be selected for the server when it's run. {% endhint %} To save the configuration in a file, you can run: ```bash konduit config -t image -o image-config.json ``` You'll see the following output from it: ``` Config file created successfully at C:\Users\konduit\image-config.json ``` #### 2. Start a server For starting the server, you can use the `serve` command: ```bash konduit serve -b -id image-server -c image-config.json ``` This will start a konduit server with the given configuration in the background. #### 3. List the running servers To view the running servers, you can use the `list` command: ```bash konduit list ``` You can an output like the following: ``` Listing konduit servers... # | ID | TYPE | URL | PID | STATUS 1 | image-server | inference | localhost:58663 | 23756 | started ``` {% hint style="warning" %} Note You might see a different port based on your running environment. {% endhint %} #### 4. View the logs You can view the logs of the running server with the `logs` command: ```bash konduit logs image-server ``` which will show you the following logs for the running server (truncated for brevity): ``` 16:36:09.491 [main] INFO ai.konduit.serving.util.LogUtils - Logging file at: C:\Users\shams\.konduit-serving\command_logs\image-server.log 16:36:09.631 [main] INFO a.k.s.l.KonduitServingLauncher - Setup micro meter options. 16:36:10.154 [main] INFO a.k.s.l.command.KonduitRunCommand - Starting konduit server with an id of 'image-server' 16:36:10.397 [vert.x-eventloop-thread-0] INFO a.k.s.routers.PipelineRouteDefiner - Using metrics registry io.micrometer.prometheus.PrometheusMeterRegistry for inference 16:36:10.760 [vert.x-eventloop-thread-0] DEBUG o.h.common.AbstractCentralProcessor - Oracle MXBean detected. 16:36:10.811 [vert.x-eventloop-thread-0] DEBUG o.d.windows.PerfCounterWildcardQuery - Localized Processor to Processor 16:36:10.852 [vert.x-eventloop-thread-0] DEBUG o.h.p.w.WindowsCentralProcessor - Initialized Processor 16:36:11.896 [vert.x-eventloop-thread-0] DEBUG o.s.o.windows.WindowsOSVersionInfoEx - Initialized OSVersionInfoEx . . . . 16:36:15.164 [vert.x-eventloop-thread-0] INFO a.k.s.v.inference.InferenceVerticle - Inference server is listening on host: 'localhost' 16:36:15.164 [vert.x-eventloop-thread-0] INFO a.k.s.v.inference.InferenceVerticle - Inference server started on port 58663 with 1 pipeline steps 16:36:15.164 [vert.x-eventloop-thread-1] INFO i.v.c.i.l.c.VertxIsolatedDeployer - Succeeded in deploying verticle ``` #### 5. Running predictions After a server is successfully started you can use the `predict` command to run inferences on the server: ```bash konduit predict -it IMAGE image-server C:\Users\konduit\mnist-5_10x10.png ``` The output json will look similar to (truncated for brevity): ``` { "default" : { "batchId" : "7d0db4c3-2cb4-4da4-b750-6e6435cadcab", "ndArray" : { "dataType" : "FLOAT", "shape" : [ 1, 3, 10, 10 ], "data" : [ 0.0, 28.0, 61.0, 25.0, , ..., 55.0, 0.0, 0.0, 0.0, 0.0, 11.0 ] } } } ``` #### 6. Stop a server Finally for stopping a server you can use the `stop` command: ```bash konduit stop image-server ``` which will output: ``` Stopping konduit server 'image-server' Application 'image-server' terminated with status 0 ``` ## What's next? You can look at the description for each of the `konduit` CLI commands and try out different combination of configuration. # Python SDK A Konduit Serving instance can be created by: 1. creating a Python object of the `Server` class using 1. the `Server()` function; or 2. the `server_from_file()` function from the `konduit.load` module; and 2. starting the server using the `.start()` method of the `Server` object created in step 1. We will use the `server_from_file()` function to configure Konduit Serving in this example. In Python, specify the path to your configuration in `konduit_yaml_path`: ```python konduit_yaml_path = "hello-world.yaml" ``` Initialize a Konduit Serving instance with the following code: ```python from konduit.load import server_from_file server = server_from_file(konduit_yaml_path) server.start() ``` Note that the file also contains Client configuration. To create a `Client` object, use the `client_from_file()` function from the `konduit.load` module: ```python from konduit.load import client_from_file client = client_from_file(konduit_yaml_path) ``` The `Client` class provides a `.predict()` method that sends data to the Serving instance. First, create some sample data as a NumPy array: ```python import numpy as np data_input = np.ones(5) ``` Assuming your data is declared in the `data_input` object, data can be passed to `client` for prediction using: ```python client.predict(data_input) ``` The `.predict()` method takes a single argument `data_input` which is typically a dictionary. A NumPy array can be directly passed to the `.predict()` method if the input name is `default`. ## Next steps To build configurations using the YAML format, check out the YAML configurations page: {% content-ref url="/pages/-Lv4m-FMeX0mEGNtY8JE" %} [YAML configurations](/0.1.0-snapshot/yaml-configurations) {% endcontent-ref %} YAML configurations are sufficient for most use cases. In particular, if your use case: * does not involve DataVec transformations, * for Python steps: has one transformation script at each pipeline step, then you should use a YAML configuration. For more complex configurations, you should use the Python SDK. To build configurations with Python steps, start with the Python pipeline steps page: {% content-ref url="/pages/-LtNeobIL9\_JE8lXIz6S" %} [Python pipeline steps](/0.1.0-snapshot/steps/python) {% endcontent-ref %} To build configurations in Python with TensorFlow, DL4J and Keras models using DL4J and JavaCPP Presets, refer to the example for the respective framework: {% content-ref url="/pages/-Lupw-aLrZSPXWeB9hMA" %} [TensorFlow (1.x)](/0.1.0-snapshot/examples/python/tensorflow-model-serving) {% endcontent-ref %} {% content-ref url="/pages/-LuptbQpkK76qsdqbAd6" %} [Broken mention](broken://pages/-LuptbQpkK76qsdqbAd6) {% endcontent-ref %} {% content-ref url="/pages/-Lv4jqWJf9I0LtMWQkJ3" %} [Keras (TensorFlow 2.0)](/0.1.0-snapshot/examples/python/keras) {% endcontent-ref %} To build ETL processes into your serving pipeline, refer to the DataVec example: {% content-ref url="/pages/-LuptvmFAcwPZJOvHxpT" %} [DataVec](/0.1.0-snapshot/examples/python/datavec) {% endcontent-ref %} # Installation [![PyPI](https://img.shields.io/pypi/v/konduit?style=for-the-badge)](https://pypi.org/project/konduit/0.1.2/)[![Conda (channel only)](https://img.shields.io/conda/vn/konduitai/konduit?color=%233EB049\&style=for-the-badge)](https://anaconda.org/konduitai/konduit) ## System requirements **Operating systems** Konduit Serving is supported on Linux, macOS and Windows. **Dependencies** Ensure that you have JDK 8.0 installed. To use the Python SDK, install Python 3.7 and above. **Hardware requirements** Binaries are provided for Intel/x86 architectures. For ARM support, see the [*Building from source*](/0.1.0-snapshot/building-from-source#manual-build) page. **GPU**: Hardware acceleration with CUDA version 10.1 (included in GPU build) is supported. ## Installation Install `konduit` from PyPI with ```bash pip install konduit ``` {% hint style="warning" %} A version of Konduit Serving with the command line interface (CLI) is not currently available on PyPI. To obtain the CLI, clone the [konduit-serving](https://github.com/KonduitAI/konduit-serving) repository, and in the `python` folder run `pip install`. {% endhint %} If using the Anaconda distribution, you may install `konduit` from the `konduitai` Anaconda channel. First add the `konduitai` channel: ``` conda config --add channels konduitai ``` then install `konduit` with: ``` conda install -c konduitai konduit ``` You may need to install Cython before installing `konduit` using ``` pip install cython ``` We recommend using Python 3.7+. {% hint style="warning" %} `konduit` PyPI wheels and conda packages do not currently ship with Konduit Serving JARs. Refer to the [*Building from source*](/0.1.0-snapshot/building-from-source#manual-build) page for instructions on compiling a Konduit Serving JAR. {% endhint %} ## Set environment variables manually In the absence of the `KONDUIT_JAR_PATH` environment variable, the Python SDK looks for the Konduit Serving JAR file in `~/.konduit/konduit-serving`. To overwrite this default, you can set a default location for the Konduit Serving JAR using environment variables. {% tabs %} {% tab title="Windows" %} Use [setx.exe](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/setx): ```bash setx KONDUIT_JAR_PATH "C:\Users\User\konduit-serving\konduit.jar" ``` {% endtab %} {% tab title="Linux, macOS" %} ```bash export KONDUIT_JAR_PATH="~/konduit-serving/konduit.jar" ``` {% endtab %} {% endtabs %} ## Common installation issues 1. Installing `pyjnius` returns ``` WARNING: Not able to assign machine() = AMD64 to a cpu value! Using cpu = 'i386' instead! ``` Fix: Ensure your JAVA environment variables point to a 64-bit version of Java if you're using a 64-bit version of Python, or a 32-bit version of Java if you're using a 32-bit version of Python (see [kivy/pyjnius#390](https://github.com/kivy/pyjnius/issues/390)). 2. When running `konduit` commands on Windows, the following error message is returned: ``` ImportError: DLL load failed: The specified module could not be found. ``` Fix: On Windows, `pyjnius` requires an additional PATH variable to locate `jvm.dll`. Refer to the [pyjnius documentation](https://pyjnius.readthedocs.io/en/stable/installation.html#installation-for-windows) for details. # Building from source Konduit Serving sources are hosted on GitHub. If you have `git` installed, clone the [konduit-serving repository](https://github.com/KonduitAI/konduit-serving) using the `git clone` command: ``` git clone https://github.com/KonduitAI/konduit-serving.git ``` ## Python module To install the `konduit` Python module from source, in the `python` directory, after installing Cython, run ``` pip install . ``` To install all extensions needed for development run ``` pip install -e '.[tests,codegen,dev]' ``` The `dev` dependencies use `black` as a pre-commit hook to lint your code automatically. To activate this functionality, run `pre-commit install` on the command line first. ### Running tests Install test dependencies using `pip install 'konduit[tests]'` if you want to run tests. On Windows, compiling the test dependencies requires Visual Studio Build Tools 14.0, which can be installed from [here](https://visualstudio.microsoft.com/downloads/). You may also need to install the Windows 8.1 / 10 SDK. See Python's [*WindowsCompilers*](https://wiki.python.org/moin/WindowsCompilers) page for details. The tests also require `bert_mrpc_frozen.pb` to be placed in the `python/tests` folder. Run the following code in `python/tests`: ``` curl https://deeplearning4jblob.blob.core.windows.net/testresources/bert_mrpc_frozen_v1.zip --output bert.zip unzip bert.zip ``` The resulting JAR will be generated at the base of the `konduit` project. To copy that JAR into the `tests` folder and prepare the documentation (in the `docs` folder) to be tested within the testing framework, run: ``` cd tests ./prepare_doc_tests.sh ``` The tests are then run with `pytest`: ``` cd python/tests python -m pytest . ``` To quickly run unit tests (recommended before each commit), or run the full set of integration tests, you can do: ``` pytest -m unit pytest -m integration ``` To also run documentation tests with `doctest` for an individual file, simply run: ``` python -m doctest ../konduit/server.py -v ``` ## JAR A Java Archive (JAR) file is used to bundle a Java program. {% hint style="info" %} Building the Konduit Serving JAR requires Maven and JDK 8. {% endhint %} ### Manual build Run the following commands in the root directory of konduit-serving: ``` python build_jar.py --os ``` where `` is picked from `windows-x86_64`,`linux-x86_64`,`linux-x86_64-gpu`, `macosx-x86_64`, `linux-armhf` and `windows-x86_64-gpu`, depending on your operating system and architecture. Use the `--help` flag to view the full list of arguments. An additional `--spin` argument provides the option to package Python (`python`), PMML (`pmml`), both (`all`) or neither (`minimal`). By default, both Python and PMML are packaged. Python bundling is not encouraged on ARM platforms, and PMML bundling is not encouraged if [AGPL licensing](https://www.gnu.org/licenses/agpl-3.0.en.html) is an issue. ### Building with the command line interface Once the `konduit` Python package is installed, you have access to a command line interface (CLI) tool called `konduit`. The `init` command: 1. gets the latest Konduit Serving code, then 2. builds the Java dependencies needed for`konduit`. It assumes that you have `git` installed on your system and that `python` is available. Run: ```bash konduit init --os ``` where `` is picked from `windows-x86_64`, `linux-x86_64`, `linux-x86_64-gpu`, `macosx-x86_64`, `linux-armhf` and `windows-x86_64-gpu`, depending on your operating system and architecture. An additional `--spin` argument provides the option to package Python (`python`), PMML (`pmml`), both (`all`) or neither (`minimal`). By default, both Python and PMML are packaged. Python bundling is not encouraged on ARM platforms, and PMML bundling is not encouraged if [AGPL licensing](https://www.gnu.org/licenses/agpl-3.0.en.html) is an issue. To rebuild the Konduit Serving JAR without re-downloading sources, run `build` instead of `init` with the appropriate flags. {% hint style="info" %} ### Known issues * `konduit init` fails for `linux-86_64-gpu` ([#115](https://github.com/KonduitAI/konduit-serving/issues/115)) {% endhint %} ## Linux builds Generally, the Linux builds of Konduit Serving perform the following tasks on installation: 1. Copy Konduit Serving JAR file to `/opt/konduit/serving/`; 2. Create the necessary environment variables; and 3. Install a Konduit Serving-specific Conda distribution with `install-python.sh`. ### RPM (CentOS, Redhat, etc.) Konduit Serving RPM packages are generated using the [RPM Maven Plugin](https://www.mojohaus.org/rpm-maven-plugin/). First, install required packages with `yum`: ``` sudo yum install -y java-1.8.0-openjdk-devel which rpm-build redhat-rpm-config ``` This command installs the developer tools for developing Java programs using JDK 8, the `which` package to locate a program file's path, tools to build RPM files and Red Hat-specific RPM configuration files. In the root folder of the `konduit-serving` project, run the following command to build RPM files using Maven Wrapper: ``` ./mvnw clean package -Ppython,pmml,uberjar,tar,rpm -Dmaven.test.skip=true -Djavacpp.platform=linux-x86_64 -Dchip=cpu ``` The Maven Wrapper `mvnw` script allows Maven to be used even if `mvn` is not available on the system PATH. This command runs the Maven goals `clean` and `install` with the following arguments: * `maven.test.skip=true` * Profiles: `uberjar,tar,rpm` (ensure this is specified without spaces in between). The profiles `python` and `pmml` are optional. * `chip`: `cpu` (use `gpu` to enable CUDA support) * `javacpp.platform`: `linux-x86_64` The `clean install` command first deletes previously compiled Java sources and resources; then compiles, tests and packages the Java project and copies it into the relevant folder. The path where the RPM file is saved depends on the `spin.version` (default `custom`) and the chip (`cpu` or `gpu`) . Use the YUM command `yum localinstall`to install the RPM file. ``` # replace with the spin version specified cd konduit-serving-rpm/target/rpm/konduit-serving--cpu/RPMS/x86_64/ sudo yum localinstall -y *.rpm ``` ### DEB (for Ubuntu and other Debian-based systems) Konduit Serving Debian packages are generated with the [jdeb](https://github.com/tcurdt/jdeb) library. Install JDK 8 using `apt-get`: ``` sudo apt-get install openjdk-8-jdk curl ``` In the root directory of the Konduit Serving project, run the `mvnw` script with parameters: ``` ./mvnw clean package -Ppython,pmml,uberjar,tar,deb -Dmaven.test.skip=true -Djavacpp.platform=linux-x86_64 -Dchip=cpu ``` * `maven.test.skip=true` * Enable the profiles `uberjar,tar,deb` (ensure this is specified without spaces in between). The `python` and `pmml` profiles are optional. * `chip`: `cpu` (use `gpu` to enable CUDA support) * `javacpp.platform`: `linux-x86_64` {% hint style="info" %} The error `java.io.IOException: This archives contains unclosed entries.` usually indicates insufficient disk space (see [tcurdt/jdeb#234](https://github.com/tcurdt/jdeb/issues/234)). {% endhint %} Finally, use `dpkg` to install the built package: ``` sudo dpkg -i konduit-serving-deb/target/konduit-serving-custom-cpu_0.1.0-SNAPSHOT.deb ``` Note that `dpkg` does not support dependencies. If you run into missing dependencies, run ``` sudo apt-get install -f ``` to install dependencies. Alternately, use the `gdebi` package to install the local DEB package (see this [StackExchange thread ](https://unix.stackexchange.com/questions/159094/how-to-install-a-deb-file-by-dpkg-i-or-by-apt)for details), or simply [use `apt-get install` to install the local package](https://askubuntu.com/a/795048) (apt 1.1 and above): ``` cd konduit-serving-deb/target sudo apt-get install ./*.deb ``` ### Tarball Konduit Serving can also be built as a tarball, where the JAR file and associated scripts are packaged in a gzip-compressed tar file. To build a Konduit Serving tar file, run the following Maven Wrapper command in the root folder of the Konduit Serving project: ``` ./mvnw clean package -Ppython,pmml,uberjar,tar -Dmaven.test.skip=true -Djavacpp.platform=linux-x86_64 -Dchip=cpu ``` This generates two compressed files in the `target` directory of the `konduit-serving-tar`folder: a tar (`.tar.gz`) and a zip (`.zip)` file. In addition to the JAR file, the tar file contains a script to install a Conda distribution (`ìnstall-python.sh`) and a script to set environment variables (`bin/konduit-serving`). After extracting the tar file, first run the `konduit-serving`shell script: ``` cd bin chmod u+x konduit-serving # allow user to execute script ./konduit-serving ``` then the `ìnstall-python.sh` script: ``` cd .. chmod u+x install-python.sh ./install-python.sh ``` ## Konduit Serving Conda distribution The following packages are included in this Conda distribution: | Package | Version | | ------------ | -------- | | NumPy | 1.16.4 | | Jupyter | 1.0.0 | | SciPy | 1.3.3 | | requests | 2.22.0 | | pandas | 0.24.2 | | TensorFlow | 1.15.0 | | Keras | 2.2.4 | | konduit | 0.1.3rc1 | | scikit-learn | 0.22 | | matplotlib | 3.1.2 | | PyTorch | 1.3.1 | | torchvision | 0.4.2 | | CUDA Toolkit | 10.1 | | OpenJDK | 8 | Note that packages are sourced from the following Anaconda channels, in descending order of priority: [pytorch](https://anaconda.org/pytorch), [conda-forge](https://anaconda.org/conda-forge), [anaconda](https://anaconda.org/anaconda), [konduitai](https://anaconda.org/konduitai). # YAML configurations Konduit Serving supports specifying server configurations as YAML files. This allows you to serve simple server configurations using the Konduit Python CLI and the konduit.load module. ## YAML components A Konduit Serving YAML configuration file has three top-level entities: 1. `serving` 2. `steps` 3. `client` The following is a sample YAML file for serving a Python script located at `simple.py` which takes a NumPy array `first` as input and returns a NumPy array `second` as output: ```yaml serving: http_port: 1337 output_data_format: NUMPY log_timings: True extra_start_args: -Xmx8g steps: python_step: type: PYTHON python_path: . python_code_path: ./simple.py python_inputs: first: NDARRAY python_outputs: second: NDARRAY client: port: 1337 ``` ### Serving The server configuration, `serving` takes the following arguments: * `http_port`: specify the port number * `listen_host`: the host of the Konduit Serving instance. Defaults to `http://localhost`. Additional arguments include: * `uploads_directory`: Directory to store file uploads. Defaults to `'file-uploads/'`. * `jar_path`: Path to the Konduit Serving uberjar. Defaults to the `KONDUIT_JAR_PATH` environment variable, or if unavailable, `~/.konduit/konduit-serving/konduit.jar`. * `log_timings`: Whether to log timings for this config. Defaults to `False`. * `extra_start_args`: Java Virtual Machine (JVM) arguments. In this case, `-Xmx8g` specifies that the maximum memory allocation for the JVM is 8GB. Refer to the [Server](/0.1.0-snapshot/server/inference) documentation for other arguments. ### Client The client configuration takes the following arguments: * `port`: specify the same HTTP port as `serving`. * `host`: defaults to `http://localhost`. Ignore this argument for local instances. Typically it is sufficient to specify the `port` and `host`as the remaining attributes are obtained from the Server. Refer to the [Client](/0.1.0-snapshot/client/python-client) documentation for details. * `input_names`, `output_names`: names of the first and final nodes of the Konduit Serving pipeline configuration defined in the Server. * `output_data_format`: the format of the server's input and output. Specify one of the following: `JSON`, `NUMPY`, `ARROW`, `IMAGE`, `ND4J`. ### Steps #### Python steps Python steps run code specified in the `python_code` (a string) or `python_code_path` (a `.py` script) argument. Python steps defined in the YAML configuration default to input name and output name `"default"`. Importantly, the `python_inputs` argument maps each input and output variable to the respective data type. Accepted data types are `INT`, `STR`, `FLOAT`, `BOOL`, `NDARRAY`. ```yaml steps: python_step: type: PYTHON python_code: | first += 2 second = first python_inputs: first: NDARRAY python_outputs: second: NDARRAY ``` If no Python path is specified, NumPy will still be available in the environment where the Python step is run. To further customize Python steps, refer to the YAML configuration section of the Python pipeline steps page. {% content-ref url="/pages/-LtNeobIL9\_JE8lXIz6S" %} [Python pipeline steps](/0.1.0-snapshot/steps/python) {% endcontent-ref %} A more comprehensive example is available on the following page: {% content-ref url="/pages/-LuptfSTVbseUO9gX9uP" %} [Open Neural Network Exchange (ONNX)](/0.1.0-snapshot/examples/python/onnx) {% endcontent-ref %} #### Model steps Use model steps when you want to use pre-packaged modules such as TensorFlow, DL4J and PMML for inference. ```yaml steps: tensorflow_step: type: TENSORFLOW model_loading_path: ../data/mnist/mnist_2.0.0.pb input_names: - input_layer output_names: - output_layer/Softmax input_data_types: input_layer: FLOAT ``` The following parameters should be specified: * `type`: one of `TENSORFLOW`, `KERAS`, `COMPUTATION_GRAPH`, `MULTI_LAYER_NETWORK`, `PMML`, `SAMEDIFF`; * `model_loading_path`: location of your model file; * `input_names`: list of the names of input nodes of your model file; * `output_names`: list of the names of output nodes of your model file; * `input_data_types`: map each of the input nodes to one of the following data types using the input names as keys: `INT`, `STR`, `FLOAT`, `BOOL`, `NDARRAY`. Refer to the model-specific example for details on configuring model steps. {% content-ref url="/pages/-Lupw-aLrZSPXWeB9hMA" %} [TensorFlow (1.x)](/0.1.0-snapshot/examples/python/tensorflow-model-serving) {% endcontent-ref %} {% content-ref url="/pages/-Lv4jqWJf9I0LtMWQkJ3" %} [Keras (TensorFlow 2.0)](/0.1.0-snapshot/examples/python/keras) {% endcontent-ref %} {% content-ref url="/pages/-LuptbQpkK76qsdqbAd6" %} [Broken mention](broken://pages/-LuptbQpkK76qsdqbAd6) {% endcontent-ref %} ## Usage On the **server**, start a Konduit Serving instance by: 1. creating a Server object using `server_from_file`, 2. starting the server using the `.start()` method. ```python from konduit.load import server_from_file konduit_yaml_path = "../yaml/simple.yaml" server = server_from_file(konduit_yaml_path) server.start() ``` ``` Starting server.. Server has started successfully. ``` After the server has started, on the **client**: 1. create a Client object using `client_from_file`; and 2. use the `.predict()` method to perform inference on a NumPy array (note that the input name of this Server configuration is `default`, therefore we can pass a NumPy array directly to the `.predict()` method.). ```python import numpy as np import os from konduit.load import client_from_file konduit_yaml_path = "../yaml/simple.yaml" input_arr = np.array(33) client = client_from_file(konduit_yaml_path) print(client.predict(input_arr)) ``` ``` [35] ``` Finally, stop the server with the `.stop()` method: ```python server.stop() ``` This can also be run in the **command line**. In the root folder of [konduit-serving-examples](https://github.com/KonduitAI/konduit-serving-examples), initialize the Konduit Serving instance with ```bash konduit serve --config yaml/simple.yaml ``` Send the NPY file to the server for inference with ```bash konduit predict-numpy --config yaml/simple.yaml --numpy_data data/simple/input_arr.npy ``` and finally, stop the server with ```bash konduit stop-server --config yaml/simple.yaml ``` ## Resources Some resources on the YAML format: * * * # Python # TensorFlow (1.x) # MNIST This notebook illustrates a simple client-server interaction to perform inference on a TensorFlow model using the Python SDK for Konduit Serving. This tutorial is split into three parts: 1. Freezing models 2. Configuration 3. Running the server {% hint style="info" %} This tutorial is tested on TensorFlow 1.14, 1.15 and 2.00. {% endhint %} ```python from konduit import ParallelInferenceConfig, ServingConfig, ModelConfigType, TensorFlowConfig from konduit import TensorDataTypesConfig, ModelStep, InferenceConfiguration from konduit.server import Server from konduit.client import Client import tensorflow as tf if tf.__version__[0] == "1": from tensorflow import keras elif tf.__version__[0] == "2": import tensorflow.compat.v1 as tf from tensorflow.compat.v1 import keras else: print("No valid TensorFlow version detected") from keras.layers import Flatten, Dense, Dropout, Lambda from keras.models import Sequential from keras.datasets import mnist from PIL import Image import numpy as np import imageio import os import matplotlib.pyplot as plt import pandas as pd ``` ``` Using TensorFlow backend. ``` ```python tensorflow_version = tf.__version__ print(tensorflow_version) ``` ``` 2.0.0 ``` ## Creating frozen models (Tensorflow 1.x) In TensorFlow 1.x, "frozen" models can be exported in the TensorFlow Graph format. For deployment, we only need information about the graph and checkpoint variables. Freezing a model allows you to discard information that is not required for deploying your model. {% hint style="warning" %} TensorFlow 2.0 introduces the [SavedModel format](https://www.tensorflow.org/guide/saved_model) as the universal format for saving models. Even though the deployable protobuff (PB) files have the same file extension as frozen TensorFlow Graph files, SavedModel protobuff files are not currently supported in Konduit Serving. A workaround for TensorFlow 2.0 is to adapt the code from this tutorial for your use case to create TensorFlow Graph protobuffs, or save your models as Keras HDF5 files and serve as Keras models (refer to the Keras tutorial for details). {% endhint %} The following code is adapted from `tf-import-examples` in the [`deeplearning4j-examples`](https://github.com/eclipse/deeplearning4j-examples/) repository. In the following code, we build a model using TensorFlow's Keras API and save it as a TensorFlow Graph. The architecture is adapted from the following Kaggle kernel: . ```python # Load data train_data, test_data = mnist.load_data() x_train, y_train = train_data x_test, y_test = test_data # Normalize x_train = x_train / 255.0 x_test = x_test / 255.0 weights = None def get_model(training=False): inputs = keras.layers.Input(shape=(28, 28), name="input_layer") x = keras.layers.Flatten()(inputs) x = keras.layers.Dense(200, activation="relu")(x) x = keras.layers.Dense(100, activation="relu")(x) x = keras.layers.Dense(60, activation="relu")(x) x = keras.layers.Dense(30, activation="relu")(x) outputs = keras.layers.Dense(10, activation="softmax", name="output_layer")(x) model = tf.keras.Model(inputs=inputs, outputs=outputs) model.compile( optimizer='sgd', loss='sparse_categorical_crossentropy', metrics=['accuracy'] ) if training: print(model.inputs[0].op.name) print(model.outputs[0].op.name) return model def train(): with tf.Session() as sess: keras.backend.set_session(sess) model = get_model(True) model.fit(x_train, y_train, epochs=8) weights = model.get_weights() return weights def save(weights): # save model to a protobuff keras.backend.clear_session() with tf.Session() as sess: keras.backend.set_session(sess) model = get_model(False) model.set_weights(weights) model.evaluate(x_test, y_test) output_node_name = model.output.name.split(':')[0] output_graph_def = tf.graph_util.convert_variables_to_constants( sess, sess.graph.as_graph_def(), [output_node_name] ) with tf.gfile.GFile( name=f"../data/mnist/mnist_{tensorflow_version}.pb", mode="wb" ) as f: f.write(output_graph_def.SerializeToString()) weights = train() save(weights) ``` ``` WARNING:tensorflow:From C:\Users\Skymind AI Berhad\AppData\Local\Continuum\miniconda3\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. input_layer output_layer/Softmax Train on 60000 samples Epoch 1/8 60000/60000 [==============================] - 3s 53us/sample - loss: 0.6992 - accuracy: 0.7869 Epoch 2/8 60000/60000 [==============================] - 3s 52us/sample - loss: 0.2445 - accuracy: 0.9294 Epoch 3/8 60000/60000 [==============================] - 3s 50us/sample - loss: 0.1782 - accuracy: 0.9481 Epoch 4/8 60000/60000 [==============================] - 3s 52us/sample - loss: 0.1425 - accuracy: 0.9588s Epoch 5/8 60000/60000 [==============================] - 3s 51us/sample - loss: 0.1180 - accuracy: 0.9651 Epoch 6/8 60000/60000 [==============================] - 3s 49us/sample - loss: 0.1020 - accuracy: 0.9699s - l Epoch 7/8 60000/60000 [==============================] - 3s 48us/sample - loss: 0.0882 - accuracy: 0.9741 Epoch 8/8 60000/60000 [==============================] - 3s 47us/sample - loss: 0.0778 - accuracy: 0.9771 10000/10000 [==============================] - 0s 31us/sample - loss: 0.1160 - accuracy: 0.9640 WARNING:tensorflow:From :54: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.compat.v1.graph_util.convert_variables_to_constants` WARNING:tensorflow:From C:\Users\Skymind AI Berhad\AppData\Local\Continuum\miniconda3\lib\site-packages\tensorflow_core\python\framework\graph_util_impl.py:275: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.compat.v1.graph_util.extract_sub_graph` INFO:tensorflow:Froze 10 variables. INFO:tensorflow:Converted 10 variables to const ops. ``` ## Overview Konduit Serving works by defining a series of **steps**. These include operations such as 1. Pre- or post-processing steps 2. One or more machine learning models 3. Transforming the output in a way that can be understood by humans If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single `ModelStep`. Before running this notebook, run the `build_jar.py` script or the `konduit init` command. Refer to the [Building from source](/0.1.0-snapshot/building-from-source#manual-build) page for details. ## Configure the step {% tabs %} {% tab title="Python" %} Define the TensorFlow configuration as a `TensorFlowConfig` object. * `tensor_data_types_config`: The `TensorFlowConfig` object requires a dictionary `input_data_types`. Its keys should represent column names, and the values should represent data types as strings, e.g. `"INT32"`. See [here](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-api/src/main/java/ai/konduit/serving/model/TensorDataType.java) for a list of supported data types. * `model_config_type`: This argument requires a `ModelConfigType` object. Specify `model_type` as `TENSORFLOW`, and `model_loading_path` to point to the location of TensorFlow weights saved in the PB file format. ```python tensorflow_config = TensorFlowConfig( tensor_data_types_config = TensorDataTypesConfig( input_data_types=input_data_types ), model_config_type = ModelConfigType( model_type='TENSORFLOW', model_loading_path=os.path.abspath( f'../data/mnist/mnist_{tensorflow_version}.pb' ) ) ) ``` ```python tensorflow_config.as_dict() ``` ``` {'@type': 'TensorFlowConfig', 'tensorDataTypesConfig': {'@type': 'TensorDataTypesConfig', 'inputDataTypes': {'input_layer': 'FLOAT'}}, 'modelConfigType': {'@type': 'ModelConfigType', 'modelType': 'TENSORFLOW', 'modelLoadingPath': 'C:\\Users\\Skymind AI Berhad\\Documents\\konduit-serving-examples\\data\\mnist\\mnist_2.0.0.pb'}} ``` Now that we have a `TensorFlowConfig` defined, we can define a `ModelStep`. The following parameters are specified: * `model_config`: pass the TensorFlowConfig object here * `parallel_inference_config`: specify the number of workers to run in parallel. Here, we specify `workers=1`. * `input_names`: names for the input data * `output_names`: names for the output data ```python tf_step = ModelStep( model_config=tensorflow_config, parallel_inference_config=ParallelInferenceConfig(workers=1), input_names=input_names, output_names=output_names ) ``` {% endtab %} {% tab title="YAML" %} In the YAML configuration file, we define a single `tensorflow_step` with * `type`: TENSORFLOW * `model_loading_path` pointing to the location of the weights * `input_names` and `output_names`: names of the input and output nodes. Define this as a list. * `input_data_types`: maps each of the inputs to a corresponding data type. Values should represent data types as strings, e.g. `INT32`. See [here](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-api/src/main/java/ai/konduit/serving/model/TensorDataType.java) for a list of supported data types. ```yaml steps: tensorflow_step: type: TENSORFLOW model_loading_path: ../data/mnist/mnist_2.0.0.pb input_names: - input_layer output_names: - output_layer/Softmax input_data_types: input_layer: FLOAT parallel_inference_config: workers: 1 ``` {% endtab %} {% endtabs %} {% hint style="info" %} Konduit Serving requires input and output names to be specified. In TensorFlow, you can find the names of your input and output nodes by printing `model.inputs[0].op.name` and `model.outputs[0].op.name` respectively. For more details, please refer to this [StackOverflow answer](https://stackoverflow.com/a/49154874/12260518). ```python # make note of hwo to obtain input_name and output_name input_data_types = {'input_layer': 'FLOAT'} input_names = list(input_data_types.keys()) output_names = ["output_layer/Softmax"] ``` {% endhint %} ## Configure the server {% tabs %} {% tab title="Python" %} Specify the following: * `http_port`: select a random port. * `input_data_format`, `output_data_format`: Specify input and output data formats as strings. ```python port = np.random.randint(1000, 65535) serving_config = ServingConfig( http_port=port, input_data_format='NUMPY', output_data_format='NUMPY' ) ``` The `ServingConfig` has to be passed to `Server` in addition to the steps as a Python list. In this case, there is a single step: `tf_step`. ```python server = Server( serving_config=serving_config, steps=[tf_step] ) ``` By default, `Server()` looks for the Konduit Serving JAR `konduit.jar` in the directory the script is run in. To change this default, use the `jar_path` argument. {% endtab %} {% tab title="YAML" %} The following parameters should be specified to `serving`: * `http_port`: specify an integer as port number * `input_data_format`, `output_data_format`: Input and output data formats ```yaml serving: http_port: 1337 input_data_format: NUMPY output_data_format: NUMPY ``` {% endtab %} {% endtabs %} {% hint style="info" %} Accepted input and output data formats are as follows: * Input: JSON, ARROW, IMAGE, ND4J (not yet implemented) and NUMPY. * Output: NUMPY, JSON, ND4J (not yet implemented) and ARROW. {% endhint %} ## Start the server {% tabs %} {% tab title="Python" %} Start the server: ```python server.start() ``` ``` Starting server..... Server has started successfully. ``` {% endtab %} {% tab title="Python from YAML" %} ```python konduit_yaml_path = "../yaml/tensorflow-mnist.yaml" server = server_from_file(konduit_yaml_path) server.start() ``` {% endtab %} {% endtabs %} ## Configure the client {% tabs %} {% tab title="Python" %} To configure the client, create a Client object by specifying the port number: ```python client = Client(port=port) ``` The `Client`'s attributes will be obtained from the Server. {% endtab %} {% tab title="YAML" %} Add the following to your YAML configuration file: ```yaml client: port: 1337 ``` In Python, use the `client_from_file` function to load the client configuration: ```python konduit_yaml_path = "../yaml/tensorflow-mnist.yaml" client = client_from_file(konduit_yaml_path) ``` {% endtab %} {% endtabs %} ## Inference {% hint style="warning" %} NDARRAY inputs to ModelSteps must be specified with a preceding `batchSize` dimension. For batches with a single observation, this can be done by using `numpy.expand_dims()` to add an additional dimension to your array. {% endhint %} We obtain test images from the test set defined by `keras.datasets`. ```python for img in x_test[0:3]: plt.imshow(img) predicted = client.predict( data_input={'input_layer': np.expand_dims(img.reshape(28, 28), axis=0)} ) plt.show() print(dict(zip(np.arange(10), predicted[0].round(3)))) ``` ![png](https://3414063056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fserving%2F-LuptFPbCw3XGeE8q9ww%2F-LuptM7j8KTcWbMx5Lx6%2Foutput_28_0.png?generation=1575009744548446\&alt=media) ``` {0: 0.0, 1: 0.0, 2: 0.001, 3: 0.001, 4: 0.0, 5: 0.0, 6: 0.0, 7: 0.998, 8: 0.0, 9: 0.0} ``` ![png](https://3414063056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fserving%2F-LuptFPbCw3XGeE8q9ww%2F-LuptM7kai5uB4VdbZMt%2Foutput_28_2.png?generation=1575009744501830\&alt=media) ``` {0: 0.0, 1: 0.0, 2: 0.998, 3: 0.002, 4: 0.0, 5: 0.0, 6: 0.0, 7: 0.0, 8: 0.0, 9: 0.0} ``` ![png](https://3414063056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fserving%2F-LuptFPbCw3XGeE8q9ww%2F-LuptM7lqoKhOgnsXVRs%2Foutput_28_4.png?generation=1575009744541542\&alt=media) ``` {0: 0.0, 1: 0.986, 2: 0.005, 3: 0.0, 4: 0.0, 5: 0.0, 6: 0.0, 7: 0.005, 8: 0.003, 9: 0.0} ``` ### Batch prediction To predict in batches, the `data_input` dictionary has to be specified differently for client images in NDARRAY format. To input a batch of observations, ensure that your inputs are in the **NCHW** format: number of observations, channels (optional if single channel), height and width. An example is as follows: ```python predicted = client.predict( data_input={'input_layer': x_test[0:3].reshape(3, 28, 28)} ) server.stop() ``` We compare the predicted probabilities and the corresponding labels: ```python pd.DataFrame(predicted).round(3) ``` | | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | | - | --- | ----- | ----- | ----- | --- | --- | --- | ----- | ----- | --- | | 0 | 0.0 | 0.000 | 0.001 | 0.001 | 0.0 | 0.0 | 0.0 | 0.998 | 0.000 | 0.0 | | 1 | 0.0 | 0.000 | 0.998 | 0.002 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.0 | | 2 | 0.0 | 0.986 | 0.005 | 0.000 | 0.0 | 0.0 | 0.0 | 0.005 | 0.003 | 0.0 | ```python y_test[0:3] ``` ``` array([7, 2, 1], dtype=uint8) ``` The configuration is stored as a dictionary. Note that the configuration can be converted to a dictionary using the `as_dict()` method: ```python server.config.as_dict() ``` ``` {'@type': 'InferenceConfiguration', 'steps': [{'@type': 'ModelStep', 'inputNames': ['input_layer'], 'outputNames': ['output_layer/Softmax'], 'modelConfig': {'@type': 'TensorFlowConfig', 'tensorDataTypesConfig': {'@type': 'TensorDataTypesConfig', 'inputDataTypes': {'input_layer': 'FLOAT'}}, 'modelConfigType': {'@type': 'ModelConfigType', 'modelType': 'TENSORFLOW', 'modelLoadingPath': 'C:\\Users\\Skymind AI Berhad\\Documents\\konduit-serving-examples\\data\\mnist\\mnist_2.0.0.pb'}}, 'parallelInferenceConfig': {'@type': 'ParallelInferenceConfig', 'workers': 1}}], 'servingConfig': {'@type': 'ServingConfig', 'httpPort': 4776, 'inputDataFormat': 'NUMPY', 'outputDataFormat': 'NUMPY', 'logTimings': True}} ``` # BERT This notebook illustrates a simple client-server interaction to perform inference on a TensorFlow model using the Python SDK for Konduit Serving. ```python import numpy as np import os ``` This page documents two ways to create Konduit Serving configurations with the Python SDK: 1. Using Python to create a configuration, and 2. Writing the configuration as a YAML file, then serving it using the Python SDK. These approaches are documented in separate tabs throughout this page. For example, the following code block shows the imports for each approach in separate tabs: {% tabs %} {% tab title="Python" %} ```python from konduit import ParallelInferenceConfig, ServingConfig, TensorFlowConfig, \ ModelConfigType, TensorDataTypesConfig, ModelStep, InferenceConfiguration from konduit.server import Server from konduit.client import Client ``` {% endtab %} {% tab title="Python from YAML" %} ```python from konduit.load import server_from_file, client_from_file ``` {% endtab %} {% endtabs %} ## Overview Konduit Serving works by defining a series of **steps**. These include operations such as 1. Pre- or post-processing steps 2. One or more machine learning models 3. Transforming the output in a way that can be understood by humans If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single `ModelStep`. Before running this notebook, run the `build_jar.py` script or the `konduit init` command. Refer to the [Building from source](/0.1.0-snapshot/building-from-source#manual-build) page for details. Start by downloading the model weights to the `data` folder. ```python from urllib.request import urlretrieve from zipfile import ZipFile dl_path = "../data/bert/bert.zip" if not os.path.isfile(dl_path): urlretrieve("https://deeplearning4jblob.blob.core.windows.net/testresources/bert_mrpc_frozen_v1.zip", dl_path) with ZipFile(dl_path, 'r') as zipObj: zipObj.extractall() ``` ## Configure the step {% tabs %} {% tab title="Python" %} Define the TensorFlow configuration as a `TensorFlowConfig` object. * `tensor_data_types_config`: The TensorFlowConfig object requires a dictionary `input_data_types`. Its keys should represent column names, and the values should represent data types as strings, e.g. `"INT32"`. See [here](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-api/src/main/java/ai/konduit/serving/model/TensorDataType.java) for a list of supported data types. * `model_config_type`: This argument requires a `ModelConfigType` object. Specify `model_type` as `TENSORFLOW`, and `model_loading_path` to point to the location of TensorFlow weights saved in the PB file format. ```python input_data_types = {'IteratorGetNext:0': 'INT32', 'IteratorGetNext:1': 'INT32', 'IteratorGetNext:4': 'INT32'} tensorflow_config = TensorFlowConfig( tensor_data_types_config = TensorDataTypesConfig( input_data_types=input_data_types ), model_config_type = ModelConfigType( model_type='TENSORFLOW', model_loading_path=os.path.abspath('bert_mrpc_frozen.pb') ) ) ``` Now that we have a `TensorFlowConfig` defined, we can define a `ModelStep`. The following parameters are specified: * `model_config`: pass the TensorFlowConfig object here * `parallel_inference_config`: specify the number of workers to run in parallel. Here, we specify `workers=1`. * `input_names`: names for the input data * `output_names`: names for the output data ```python input_names = list(input_data_types.keys()) output_names = ["loss/Softmax"] tf_step = ModelStep( model_config=tensorflow_config, parallel_inference_config=ParallelInferenceConfig(workers=1), input_names=input_names, output_names=output_names ) ``` {% endtab %} {% tab title="YAML" %} In the YAML file, we define `steps` with a single `tensorflow_step`. ```yaml steps: tensorflow_step: type: TENSORFLOW model_loading_path: bert_mrpc_frozen.pb input_names: - IteratorGetNext:0 - IteratorGetNext:1 - IteratorGetNext:4 output_names: - loss/Softmax input_data_types: IteratorGetNext:0: INT32 IteratorGetNext:1: INT32 IteratorGetNext:4: INT32 parallel_inference_config: workers: 1 ``` * `model_loading_path`: location of the model file * `input_names`, `output_names`: names of the input and output nodes respectively. **Important**: specify `input_names` and `output_names`as lists. * `input_data_types`: maps each of the `input_names` to the corresponding data type. The values should represent data types as strings, e.g. `INT32`. See [here](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-api/src/main/java/ai/konduit/serving/model/TensorDataType.java) for a list of supported data types. * `parallel_inference_config`: specify the number of workers to run in parallel. Here, we specify `workers=1`. {% endtab %} {% endtabs %} {% hint style="info" %} ### Names of input and output nodes In TensorFlow, you can find the names of your input and output nodes by iterating through`model.inputs`and `model.outputs`respectively and printing the `.os.name`attribute of each. For more details, please refer to this [StackOverflow answer](https://stackoverflow.com/a/49154874/12260518). {% endhint %} ## Configure the server Specify the following: * `http_port`: select a random port. * `input_data_format`, `output_data_format`: Specify input and output data formats as strings. {% hint style="info" %} Accepted input and output data formats are as follows: * Input: JSON, ARROW, IMAGE, ND4J (not yet implemented) and NUMPY. * Output: NUMPY, JSON, ND4J (not yet implemented) and ARROW. {% endhint %} {% tabs %} {% tab title="Python" %} ```python port = np.random.randint(1000, 65535) serving_config = ServingConfig( http_port=port, input_data_format='NUMPY', output_data_format='NUMPY' ) ``` The `ServingConfig` has to be passed to `Server` in addition to the steps as a Python list. In this case, there is a single step: `tf_step`. ```python server = Server( serving_config=serving_config, steps=[tf_step] ) ``` {% endtab %} {% tab title="YAML" %} ```yaml serving: http_port: 1337 input_data_format: NUMPY output_data_format: NUMPY ``` {% endtab %} {% endtabs %} By default, `Server()` looks for the Konduit Serving JAR `konduit.jar` in the directory the script is run in. To change this default, use the `jar_path` argument. ## Start the server {% tabs %} {% tab title="Python" %} Start the server: ```python server.start() ``` ``` Starting server................. Server has started successfully. ``` {% endtab %} {% tab title="Python from YAML" %} ```python konduit_yaml_path = "../yaml/tensorflow-bert.yaml" server = server_from_file(konduit_yaml_path) server.start() ``` {% endtab %} {% endtabs %} ## Configure the client To configure the client, create a Client object specifying the port number: {% tabs %} {% tab title="Python" %} ```python client = Client(port=port) ``` {% endtab %} {% tab title="YAML" %} Add the following to your YAML configuration file: ```yaml client: port: 1337 ``` Create a Client object using the `client_from_file` function: ```python konduit_yaml_path = "../yaml/tensorflow-bert.yaml" client = client_from_file(konduit_yaml_path) ``` {% endtab %} {% endtabs %} ## Inference {% hint style="warning" %} NDARRAY inputs to ModelSteps must be specified with a preceding `batchSize` dimension. For batches with a single observation, this can be done by using `numpy.expand_dims()` to add an additional dimension to your array. {% endhint %} Load some sample data from NumPy files. Note that these are NumPy arrays, each with shape (4, 128): ```python data_input = { 'IteratorGetNext:0': np.expand_dims(np.load('../data/bert/input-0.npy'), axis=0), 'IteratorGetNext:1': np.expand_dims(np.load('../data/bert/input-1.npy'), axis=0), 'IteratorGetNext:4': np.expand_dims(np.load('../data/bert/input-4.npy'), axis=0) } ``` ```python predicted = client.predict(data_input) print(predicted) server.stop() ``` ``` [[9.9860090e-01 1.3990625e-03] [7.0319971e-04 9.9929678e-01] [9.9866593e-01 1.3340610e-03] [9.7927457e-01 2.0725440e-02]] ``` The configuration is stored as a dictionary. Note that the configuration can be converted to a dictionary using the `as_dict()` method: ```python server.config.as_dict() ``` ``` {'@type': 'InferenceConfiguration', 'steps': [{'@type': 'ModelStep', 'inputNames': ['IteratorGetNext:0', 'IteratorGetNext:1', 'IteratorGetNext:4'], 'outputNames': ['loss/Softmax'], 'modelConfig': {'@type': 'TensorFlowConfig', 'tensorDataTypesConfig': {'@type': 'TensorDataTypesConfig', 'inputDataTypes': {'IteratorGetNext:0': 'INT32', 'IteratorGetNext:1': 'INT32', 'IteratorGetNext:4': 'INT32'}}, 'modelConfigType': {'@type': 'ModelConfigType', 'modelType': 'TENSORFLOW', 'modelLoadingPath': 'C:\\Users\\Skymind AI Berhad\\Documents\\konduit-serving-examples\\notebooks\\bert_mrpc_frozen.pb'}}, 'parallelInferenceConfig': {'@type': 'ParallelInferenceConfig', 'workers': 1}}], 'servingConfig': {'@type': 'ServingConfig', 'httpPort': 36846, 'inputDataFormat': 'NUMPY', 'outputDataFormat': 'NUMPY', 'logTimings': True}} ``` # Deeplearning4j (DL4J) This page illustrates a simple client-server interaction to perform inference on a DL4J image classification model using the Python SDK for Konduit Serving. ```python import numpy as np import os ``` This page documents two ways to create Konduit Serving configurations with the Python SDK: 1. Using Python to create a configuration, and 2. Writing the configuration as a YAML file, then serving it using the Python SDK. These approaches are documented in separate tabs throughout this page. For example, the following code block shows the imports for each approach in separate tabs: {% tabs %} {% tab title="Python" %} ```python from konduit import ModelConfig, TensorDataTypesConfig, ModelConfigType, \ ModelStep, ParallelInferenceConfig, ServingConfig, InferenceConfiguration from konduit.server import Server from konduit.client import Client ``` {% endtab %} {% tab title="Python from YAML" %} ```python from konduit.load import server_from_file, client_from_file ``` {% endtab %} {% endtabs %} ## Saving models in Deeplearning4j The following is a short Java program that loads a simple CNN model from DL4J's model zoo, initializes weights, then saves the model to a new file, `SimpleCNN.zip`. ```java import org.deeplearning4j.nn.conf.MultiLayerConfiguration; import org.deeplearning4j.nn.multilayer.MultiLayerNetwork; import org.deeplearning4j.zoo.ZooModel; import org.deeplearning4j.zoo.model.SimpleCNN; import java.io.File; public class SaveSimpleCNN { private static int nClasses = 5; private static boolean saveUpdater = false; public static void main(String[] args) throws Exception { ZooModel zooModel = SimpleCNN.builder() .numClasses(nClasses) .inputShape(new int[]{3, 224, 224}) .build(); MultiLayerConfiguration conf = ((SimpleCNN) zooModel).conf(); MultiLayerNetwork net = new MultiLayerNetwork(conf); net.init(); System.out.println(net.summary()); File locationToSave = new File("SimpleCNN.zip"); net.save(locationToSave, saveUpdater); } } ``` A reference Java project using DL4J 1.0.0-beta6 is provided in this repository with a Maven `pom.xml` dependencies file. If using the IntelliJ IDEA IDE, open the `java` folder as a Maven project and run the `main` function of the `SaveSimpleCNN` class. ## Overview Konduit Serving works by defining a series of **steps**. These include operations such as 1. Pre- or post-processing steps 2. One or more machine learning models 3. Transforming the output in a way that can be understood by humans If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single `ModelStep`. Before running this notebook, run the `build_jar.py` script or the `konduit init` command. Refer to the [Building from source](/0.1.0-snapshot/building-from-source#manual-build) page for details. ## Configure the step {% tabs %} {% tab title="Python" %} Define the DL4J configuration as a `ModelConfig` object. * `tensor_data_types_config`: The `ModelConfig` object requires a dictionary `input_data_types`. Its keys should represent column names, and the values should represent data types as strings, e.g. `"INT32"`. See [here](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-api/src/main/java/ai/konduit/serving/model/TensorDataType.java) for a list of supported data types. * `model_config_type`: This argument requires a `ModelConfigType` object. In the Java program above, we recognized that SimpleCNN is configured as a `MultiLayerNetwork`, in contrast with the `ComputationGraph` class, which is used for more complex networks. Specify `model_type` as `MULTI_LAYER_NETWORK`, and `model_loading_path` to point to the location of DL4J weights saved in the ZIP file format. ```python input_data_types = {"image_array": "FLOAT"} input_names = list(input_data_types.keys()) output_names = ["output"] port = np.random.randint(1000, 65535) dl4j_config = ModelConfig( tensor_data_types_config=TensorDataTypesConfig( input_data_types=input_data_types ), model_config_type=ModelConfigType( model_type="MULTI_LAYER_NETWORK", model_loading_path=os.path.abspath("../data/multilayernetwork/SimpleCNN.zip") ) ) ``` For the `ModelStep` object, the following parameters are specified: * `model_config`: pass the `ModelConfig` object here. * `parallel_inference_config`: specify the number of workers to run in parallel. Here, we specify `workers=1`. * `input_names`: names for the input data. * `output_names`: names for the output data. ```python dl4j_step = ModelStep( model_config=dl4j_config, parallel_inference_config=ParallelInferenceConfig(workers=1), input_names=input_names, output_names=output_names ) ``` {% endtab %} {% tab title="YAML" %} In the Java program above, we recognised that SimpleCNN is configured as a `MultiLayerNetwork`, in contrast with the `ComputationGraph` class, which is used for more complex networks. Hence, we create a `dl4j_mln_step` of type `MULTI_LAYER_NETWORK`. * `model_loading_path` denotes the location of the model file. * `input_names` and `output_names` denote the names of the input and output nodes, as lists. * `input_data_types` maps the data types of the input nodes to the data type. See [here](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-api/src/main/java/ai/konduit/serving/model/TensorDataType.java) for a list of supported data types. * `parallel_inference_config`: specify the number of workers to run in parallel. Here, we specify `workers=1`. ```yaml steps: dl4j_mln_step: type: MULTI_LAYER_NETWORK model_loading_path: ../data/multilayernetwork/SimpleCNN.zip input_names: - image_array output_names: - output input_data_types: image_array: FLOAT parallel_inference_config: workers: 1 ``` {% endtab %} {% endtabs %} {% hint style="info" %} To find the names of input and output nodes in DL4J, * for `input_names`: print the first element of `net.getLayerNames()`. * for `output_names`: check the last layer when printing `net.summary()`. {% endhint %} ## Configure the server Specify the following: * `http_port`: select a random port. * `input_data_format`, `output_data_format`: specify input and output data formats as strings. {% tabs %} {% tab title="Python" %} The `ServingConfig` has to be passed to `Server` in addition to the steps as a Python list. In this case, there is a single step: `dl4j_step`. ```python serving_config = ServingConfig( http_port=port, input_data_format='NUMPY', output_data_format='NUMPY' ) server = Server( serving_config=serving_config, steps=[dl4j_step] ) ``` {% endtab %} {% tab title="YAML" %} ```yaml serving: http_port: 1337 input_data_format: NUMPY output_data_format: NUMPY log_timings: True extra_start_args: -Xmx8 ``` {% endtab %} {% endtabs %} {% hint style="info" %} Accepted input and output data formats are as follows: * Input: `JSON`, `ARROW`, `IMAGE`, `ND4J` (not yet implemented) and `NUMPY`. * Output: `NUMPY`, `JSON`, `ND4J` (not yet implemented) and `ARROW`. {% endhint %} ## Start the server {% tabs %} {% tab title="Python" %} ```python server.start() ``` ``` Starting server... Server has started successfully. ``` {% endtab %} {% tab title="Python from YAML" %} ```python konduit_yaml_path = "../yaml/deeplearning4j.yaml" server = server_from_file(konduit_yaml_path) server.start() ``` {% endtab %} {% endtabs %} ## Configure the client To configure the client, create a Client object with the following arguments: * `input_data_format`: data format passed to the server for inference. * `output_data_format`: data format returned by the server endpoint. * `return_output_data_format`: data format to be returned to the client. Note that this argument can be used to convert the output returned from the server to the client into a different format, e.g. `NUMPY` to `JSON`. {% tabs %} {% tab title="Python" %} ```python client = Client( input_data_format='NUMPY', output_data_format='NUMPY', return_output_data_format="NUMPY", host='http://localhost', port=port ) ``` {% endtab %} {% tab title="YAML" %} Add the following to your YAML configuration file: ```yaml client: input_data_format: NUMPY output_data_format: NUMPY return_output_data_format: NUMPY host: http://localhost port: 1337 ``` Use `client_from_file` to create a `Client` object: ```python konduit_yaml_path = "../yaml/deeplearning4j.yaml" client = client_from_file(konduit_yaml_path) ``` {% endtab %} {% endtabs %} ## Inference We generate a (3, 224, 224) array of random numbers between 0 and 255 as input to the model for prediction. {% hint style="warning" %} `NDARRAY` inputs to `ModelStep`s must be specified with a preceding `batchSize` dimension. For batches with a single observation, this can be done by using `numpy.expand_dims()` to add an additional dimension to your array. {% endhint %} Before requesting for a prediction, we normalize the image to be between 0 and 1: ```python rand_image = np.random.randint(255, size=(1, 3, 224, 224)) / 255 ``` ```python prediction = client.predict({"image_array": rand_image}) print(prediction) server.stop() ``` ``` [[4.1741084e-02 3.2335979e-01 2.5368158e-02 3.9881383e-05 6.0949111e-01]] ``` Again, we can use the `as_dict()` method of the `config` attribute of `server` to view the overall configuration: ```python server.config.as_dict() ``` ``` {'@type': 'InferenceConfiguration', 'steps': [{'@type': 'ModelStep', 'inputNames': ['image_array'], 'outputNames': ['output'], 'modelConfig': {'@type': 'ModelConfig', 'tensorDataTypesConfig': {'@type': 'TensorDataTypesConfig', 'inputDataTypes': {'image_array': 'FLOAT'}}, 'modelConfigType': {'@type': 'ModelConfigType', 'modelType': 'MULTI_LAYER_NETWORK', 'modelLoadingPath': 'C:\\Users\\Skymind AI Berhad\\Documents\\konduit-serving-examples\\data\\multilayernetwork\\SimpleCNN.zip'}}, 'parallelInferenceConfig': {'@type': 'ParallelInferenceConfig', 'workers': 1}}], 'servingConfig': {'@type': 'ServingConfig', 'httpPort': 57441, 'inputDataFormat': 'NUMPY', 'outputDataFormat': 'NUMPY', 'logTimings': True}} ``` # DataVec Konduit Serving supports data transformations defined by the DataVec vectorization and ETL library. ```python from konduit import TransformProcessStep, ServingConfig from konduit.server import Server from konduit.client import Client from konduit.utils import is_port_in_use from pydatavec import Schema, TransformProcess from utils import load_java_tp import numpy as np import random import time import json import os ``` DataVec transformations can be defined in Python using the [PyDataVec](https://github.com/eclipse/deeplearning4j/tree/master/pydatavec) package, which can be installed from PyPi: ``` pip install pydatavec ``` Using PyDataVec requires [Docker](https://docs.docker.com/v17.09/engine/installation/#supported-platforms). For Windows 10 Home edition users, note that Docker Toolbox is not supported. Run the following cell to check that your Docker installation is successful: ```python !docker run hello-world ``` ``` Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/ ``` ## Data transformations with PyDataVec ### **Schema (**[**source**](https://github.com/eclipse/deeplearning4j/blob/master/pydatavec/pydatavec/schema.py)**)** A `Schema` specifies the structure of your data. In DataVec, a `TransformProcess` requires the `Schema` of the data to be specified. `Schema` objects have a number of methods that define different data types for columns: `add_string_column()`, `add_integer_column()`, `add_long_column()`, `add_float_column()`, `add_double_column()` and `add_categorical_column()`. ### **TransformProcess (**[**source**](https://github.com/eclipse/deeplearning4j/blob/master/pydatavec/pydatavec/transform_process.py)**)** `TransformProcess` provides a number of methods to manipulate your data. The following methods are available in the Python API: * Reduce the number of rows: `filter()` * General data transformations: `replace()`, * Type casting: `string_to_time()`, `derive_column_from_time()`, `categorical_to_integer()`, * Combining/reducing the values in each column: `reduce()` * String operations: `append_string()`, `lower()`, `upper()`, `concat()`, `remove_white_spaces()`, `replace_empty_string()`, `replace_string()`, `map_string()` * Column selection/renaming: `remove()`, `remove_columns_except()`, `rename_column()` * One-hot encoding: `one_hot()` In this short example, we append the string `two` to the end of values in the string column `first`. ```python schema = Schema() schema.add_string_column("first") tp = TransformProcess(schema) tp.append_string("first", "two") ``` The `TransformProcess` configuration has to be converted into JSON format to be passed to Konduit Serving. ```python java_tp = tp.to_java() tp_json = java_tp.toJson() load_java_tp(tp_json) as_python_json = json.loads(tp_json) ``` ## Configure the step The `TransformProcess` can now be defined in the Konduit Serving configuration with a `TransformProcessStep`. Here, we * **configure the inputs and outputs**: the schema, column names and data types should be defined here. * **declare the `TransformProcess`** using the `.transform_process()` method. Note that `Schema` data types are not defined in the same way as `PythonStep` data types. See the [source](https://github.com/KonduitAI/konduit-serving/blob/78851701004ebb3dbf079889d46b79a9db8fac60/konduit-serving-api/src/main/java/ai/konduit/serving/util/SchemaTypeUtils.java#L154-L195) for a complete list of supported Schema data types: * `NDArray` * `String` * `Boolean` * `Categorical` * `Float` * `Double` * `Integer` * `Long` * `Bytes` You should define the Schema data types in `TransformProcessStep()` as strings. ```python transform_step = (TransformProcessStep() .set_input(schema=None, column_names=["first"], types=["String"]) .set_output(schema=None, column_names=["first"], types=["String"]) .transform_process(as_python_json)) ``` ## Configure the server Configure the Server using `ServingConfig` to define the port using the `http_port` argument and data formats using the `input_data_type` and `output_data_type` arguments. ```python port = np.random.randint(1000, 65535) serving_config = ServingConfig( http_port=port, input_data_format='JSON', output_data_format='JSON', ) server = Server( serving_config=serving_config, steps=[transform_step] ) ``` The complete configuration is as follows: ```python server.config.as_dict() ``` ``` {'@type': 'InferenceConfiguration', 'steps': [{'@type': 'TransformProcessStep', 'inputSchemas': {'default': ['String']}, 'outputSchemas': {'default': ['String']}, 'inputNames': ['default'], 'outputNames': ['default'], 'inputColumnNames': {'default': ['first']}, 'outputColumnNames': {'default': ['first']}, 'transformProcesses': {'default': {'actionList': [{'transform': {'@class': 'org.datavec.api.transform.transform.string.AppendStringColumnTransform', 'columnName': 'first', 'toAppend': 'two'}}], 'initialSchema': {'@class': 'org.datavec.api.transform.schema.Schema', 'columns': [{'@class': 'org.datavec.api.transform.metadata.StringMetaData', 'name': 'first'}]}}}}], 'servingConfig': {'@type': 'ServingConfig', 'httpPort': 47964, 'inputDataFormat': 'JSON', 'outputDataFormat': 'JSON', 'logTimings': True}} ``` ## Start the server ```python server.start() ``` ## Configure the client Create a `Client` object and specify the port number as an argument: ```python client = Client(port=port) ``` ``` Starting server.. Server has started successfully. ``` ## Inference Finally, we run the Konduit Serving instance. Recall that the `TransformProcessStep()` appends a string `two` to strings in the column `first`: ```python data_input = {'first': 'value'} predicted = client.predict(data_input) print(predicted) server.stop() ``` ``` {'first': 'valuetwo'} ``` # Open Neural Network Exchange (ONNX) This notebook provides an example of serving a model built in PyTorch with ONNX Runtime, a cross-platform, high performance scoring engine for machine learning models. The Open Neural Network Exchange (ONNX) format is supported by a number of deep learning frameworks, including PyTorch, CNTK and MXNet. ```python import os from urllib.request import urlretrieve import sys import numpy as np from PIL import Image import onnx from onnx import optimizer from konduit.utils import default_python_path ``` This page documents two ways to create Konduit Serving configurations with the Python SDK: 1. Using Python to create a configuration, and 2. Writing the configuration as a YAML file, then serving it using the Python SDK. These approaches are documented in separate tabs throughout this page. For example, the following code block shows the imports for each approach in separate tabs: {% tabs %} {% tab title="Python" %} ```python from konduit import PythonConfig, ServingConfig, InferenceConfiguration, \ PythonStep from konduit.server import Server from konduit.client import Client ``` {% endtab %} {% tab title="YAML" %} ```python from konduit.load import server_from_file, client_from_file ``` {% endtab %} {% endtabs %} ## Download file For the purposes of this example, we use ONNX model files from [Ultra-Light-Fast-Generic-Face-Detector-1MB](https://github.com/Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB) by Linzaer, a lightweight facedetection model designed for edge computing devices. ```python dl_path = os.path.abspath("../data/facedetector/facedetector.onnx") DOWNLOAD_URL = "https://raw.githubusercontent.com/Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB/master/models/onnx/version-RFB-320.onnx" if not os.path.isfile(dl_path): urlretrieve(DOWNLOAD_URL, filename=dl_path) ``` The following content is based on the PyTorch tutorial [Exporting a Model from PyTorch to ONNX and Running it using ONNX Runtime](https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html), with modifications. We start by loading the model and running `onnx.checker.check_model` to check whether the model has a valid schema. ```python # Load the ONNX model model = onnx.load(dl_path) # model is a onnx.ModelProto object onnx.checker.check_model(model) ``` ## Optimize When loading some models, ONNX may return warnings that the model can be further optimized by removing some unused nodes. Use ONNX's optimizer to optimize your ONNX file. The code below is adapted from this [GitHub comment](https://github.com/microsoft/onnxruntime/issues/1899#issuecomment-534806537). Note that the API for optimizing models in ONNX Runtime is experimental, and [may change](https://github.com/onnx/onnx/blob/c08a7b76cf7c1555ae37186f12be4d62b2c39b3b/onnx/optimizer/optimize.h#L1-L2). ```python onnx_model = onnx.load(dl_path) passes = ["extract_constant_to_initializer", "eliminate_unused_initializer"] optimized_model = optimizer.optimize(onnx_model, passes) onnx.save(optimized_model, dl_path) ``` ## Python script with PyTorch and ONNX Runtime Now that we have an optimized ONNX file, we can serve our model. The following code: * transforms a [PIL](https://python-pillow.org/) image into a 240 x 320 image, * casts it into a PyTorch Tensor, * adds an extra dimension with [`unsqueeze`](https://pytorch.org/docs/stable/torch.html#torch.unsqueeze), * casts the Tensor into a NumPy array, then * returns the model's output with ONNX Runtime. ```python python_code = """ from PIL import Image import torchvision.transforms as transforms import onnxruntime import os dl_path = os.path.abspath("../data/facedetector/facedetector.onnx") image = Image.fromarray(image.astype('uint8'), 'RGB') resize = transforms.Resize([240, 320]) img_y = resize(image) to_tensor = transforms.ToTensor() img_y = to_tensor(img_y) img_y.unsqueeze_(0) def to_numpy(tensor): return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy() ort_session = onnxruntime.InferenceSession(dl_path) ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(img_y)} ort_outs = ort_session.run(None, ort_inputs) _, boxes = ort_outs """ ``` ## Configure the step {% tabs %} {% tab title="Python" %} ### Defining a `PythonConfig` * Here we use the `python_code` argument instead of `python_code_path`, since the code is defined as a string. * Define the inputs and outputs as dictionaries, where the keys represent objects in the server's Python environment, and the values represent data types (Python data structures), defined as strings. See for supported data types. ```python work_dir = os.path.abspath('.') python_config = PythonConfig( python_code=python_code, python_inputs={"image": "NDARRAY"}, python_outputs={"boxes": "NDARRAY"}, python_path=default_python_path(work_dir) ) ``` ### Define a pipeline step with the `PythonStep` class. In the `.step()` method, define a name for this step (`input1`) and the respective configuration (`python_config`). ```python onnx_step = (PythonStep() .step(input_name="input1", python_config=python_config)) ``` {% endtab %} {% tab title="YAML" %} ```yaml steps: python_step: type: PYTHON python_path: C:\\Users\\Skymind AI Berhad\\Documents\\konduit-serving-examples\\notebooks;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\python37.zip;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\DLLs;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch;;C:\\Users\\Skymind AI Berhad\\AppData\\Roaming\\Python\\Python37\\site-packages;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\konduit-0.1.4-py3.7.egg;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\pyyaml-5.1.2-py3.7-win-amd64.egg;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\win32;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\win32\\lib;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\Pythonwin;C:\\Users\\Skymind AI Berhad\\AppData\\Local\\Continuum\\miniconda3\\envs\\pytorch\\lib\\site-packages\\IPython\\extensions;C:\\Users\\Skymind AI Berhad\\.ipython;C:\\Users\\Skymind AI Berhad\\Documents\\konduit-serving-examples\\notebooks python_code: | from PIL import Image import torchvision.transforms as transforms import onnxruntime import os dl_path = os.path.abspath("../data/facedetector/facedetector.onnx") image = Image.fromarray(image.astype('uint8'), 'RGB') resize = transforms.Resize([240, 320]) img_y = resize(image) to_tensor = transforms.ToTensor() img_y = to_tensor(img_y) img_y.unsqueeze_(0) def to_numpy(tensor): return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy() ort_session = onnxruntime.InferenceSession(dl_path) ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(img_y)} ort_outs = ort_session.run(None, ort_inputs) _, boxes = ort_outs python_inputs: image: NDARRAY python_outputs: boxes: NDARRAY ``` We define a single `python_step` of type PYTHON. * `python_path` specifies the location of Python modules. * `python_code` specifies the Python code to be run. Here, we use a YAML literal block scalar. * `python_inputs` and `python_outputs`specifies the data type of the objects in the Python script to be used as input(s) and output(s) respectively. {% hint style="info" %} Models loaded from a YAML configuration do not currently support input and output names for Python steps. To construct configurations with custom input and output names, use the Python SDK. {% endhint %} {% hint style="info" %} The default Python path includes NumPy and a basic set of modules. However, for this example, we also require the Pillow, PyTorch and ONNX Runtime modules. See the [Python pipeline steps page](/0.1.0-snapshot/steps/python#python-modules-and-the-pythonpath-argument) for additional documentation on Python paths, and refer to the [PyTorch quickstart](https://pytorch.org/) for recommended installation steps. To locate your Python path, run the following: ```python from konduit.utils import default_python_path work_dir = os.path.abspath('.') print(default_python_path(work_dir)) ``` {% endhint %} {% endtab %} {% endtabs %} ## Configure the server {% tabs %} {% tab title="Python" %} ```python port = np.random.randint(1000, 65535) server = Server( steps=onnx_step, serving_config=ServingConfig(http_port=port) ) ``` {% endtab %} {% tab title="YAML" %} ```yaml serving: http_port: 1337 input_data_format: NUMPY output_data_format: NUMPY log_timings: True extra_start_args: -Xmx8g ``` {% endtab %} {% endtabs %} ## Start the server {% tabs %} {% tab title="Python" %} ```python server.start() ``` ``` Starting server......... Server has started successfully. ``` {% endtab %} {% tab title="YAML" %} ```python konduit_yaml_path = "../yaml/pytorch.yaml" server = server_from_file(konduit_yaml_path) server.start() ``` {% endtab %} {% endtabs %} ## Configure the client Make sure to configure the client after starting the server, so that the Client object can inherit the Server's attributes. Since the image is passed to the Server as a NumPy array, specify the input and output data format as `NUMPY`. {% tabs %} {% tab title="Python" %} ```python client = Client( input_data_format='NUMPY', return_output_data_format='NUMPY', output_data_format="RAW", port=port ) ``` {% endtab %} {% tab title="YAML" %} Add the following to your YAML configuration file: ```yaml client: input_data_format: NUMPY output_data_format: RAW return_output_data_format: NUMPY port: 1337 ``` Use `client_from_file` to create a `Client` object: ```python konduit_yaml_path = "../yaml/pytorch.yaml" client = client_from_file(konduit_yaml_path) ``` {% endtab %} {% endtabs %} ## Inference Load a sample image using PIL/Pillow and send the image to the server for prediction using the `predict()` method of the `Client` class. ```python im = Image.open("../data/facedetector/1.jpg") im = np.array(im).astype("int") ``` ```python output = client.predict( {"input1": im} ) print(output) ``` ``` [[[ 0.00601701 0.00688479 0.02177745 0.03408115] [-0.0018133 -0.00657785 0.03698186 0.05206966] [-0.01035942 -0.01786287 0.04902049 0.06799769] ... [ 0.7294515 0.6165271 1.0584102 1.1059598 ] [ 0.65046376 0.48442802 1.141786 1.2248938 ] [ 0.5633501 0.37209463 1.2047783 1.2747201 ]]] ``` Finally, we stop the server: ```python server.stop() ``` # Keras (TensorFlow 2.0) This page illustrates a simple client-server interaction to perform inference on a Keras LSTM model using the Python SDK for Konduit Serving. ```python from konduit import ModelConfig, ParallelInferenceConfig, ModelConfigType, \ ModelStep, ServingConfig from konduit.server import Server from konduit.client import Client import os import numpy as np ``` ## Saving models in Keras HDF5 (.h5) format HDF5 model files can be saved with the `.save()` method. Refer to the [TensorFlow documentation for Keras](https://www.tensorflow.org/guide/keras/save_and_serialize) for details. {% hint style="info" %} Keras model loading functionality in Konduit Serving converts Keras models to Deeplearning4J models. As a result, Keras models containing operations not supported in Deeplearning4J cannot be served in Konduit Serving. See [issue 8348](https://github.com/eclipse/deeplearning4j/issues/8348). {% endhint %} ## Overview Konduit Serving works by defining a series of **steps**. These include operations such as 1. Pre- or post-processing steps 2. One or more machine learning models 3. Transforming the output in a way that can be understood by humans If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single `ModelStep`. Before running this notebook, run the `build_jar.py` script or the `konduit init` command. Refer to the [Building from source](/0.1.0-snapshot/building-from-source#manual-build) page for details. ## Configure the step {% tabs %} {% tab title="Python" %} Define the Keras configuration as a `ModelConfig` object. * `model_config_type`: This argument requires a `ModelConfigType` object. Specify `model_type` as `KERAS`, and `model_loading_path` to point to the location of Keras weights saved in the HDF5 file format. For the `ModelStep` object, the following parameters are specified: * `model_config`: pass the `ModelConfig` object here. * `parallel_inference_config`: specify the number of workers to run in parallel. Here, we specify `workers=1`. * `input_names`: names for the input nodes. * `output_names`: names for the output nodes. ```python keras_config = ModelConfig( model_config_type = ModelConfigType( model_type='KERAS', model_loading_path=os.path.abspath( f'../data/keras/embedding_lstm_tensorflow_2.h5' ) ) ) keras_step = ModelStep( model_config=keras_config, parallel_inference_config=ParallelInferenceConfig(workers=1), input_names=["input"], output_names=["lstm_1"] ) ``` {% endtab %} {% tab title="YAML" %} ```yaml steps: keras_step: type: KERAS model_loading_path: ../data/keras/embedding_lstm_tensorflow_2.h5 input_names: - input output_names: - lstm_1 ``` * `type`: specify this as `KERAS`. * `model_loading_path`: location of the model weights. * `input_names`, `output_names`: names for the input and output nodes, as lists. {% endtab %} {% endtabs %} {% hint style="info" %} Input and output names can be obtained by visualizing the graph in [Netron](https://github.com/lutzroeder/netron). {% endhint %} ## Configure the server {% tabs %} {% tab title="Python" %} In the `ServingConfig`, specify a port number. The `ServingConfig` has to be passed to `Server` in addition to the steps as a Python list. In this case, there is a single step: `keras_step`. ```python serving_config = ServingConfig(http_port=1337) server = Server( serving_config=serving_config, steps=[keras_step] ) ``` {% endtab %} {% tab title="YAML" %} ```yaml serving: http_port: 1337 ``` {% endtab %} {% endtabs %} ## Start the server {% tabs %} {% tab title="Python" %} Use the `.start()` method: ```python server.start() ``` ``` Starting server.. Server has started successfully. ``` {% endtab %} {% tab title="YAML" %} ```python input_array = np.random.uniform(size = [10]) konduit_yaml_path = "../yaml/keras.yaml" server = server_from_file(konduit_yaml_path) server.start() ``` {% endtab %} {% endtabs %} ## Configure the client To configure the client, create a Client object with the `port` argument. Note that you should create the Client object after the Server has started, so that Client can inherit the Server's attributes. {% tabs %} {% tab title="Python" %} ```python client = Client(port=1337) ``` {% endtab %} {% tab title="YAML" %} Add the following to your YAML file: ```yaml client: port: 1337 ``` Use `client_from_file` to create a `Client` object in Python: ```python konduit_yaml_path = "../yaml/keras.yaml" client = client_from_file(konduit_yaml_path) ``` {% endtab %} {% endtabs %} ## Inference {% hint style="warning" %} NDARRAY inputs to ModelSteps must be specified with a preceding `batchSize` dimension. For batches with a single observation, this can be done by using `numpy.expand_dims()` to add an additional dimension to your array. {% endhint %} ```python input_array = np.random.uniform(size = [10]) prediction = client.predict({"input": np.expand_dims(input_array, axis=0)}) server.stop() ``` ```python print(prediction) prediction.shape ``` ``` [[[0.49911702 0.4983615 0.49773094 0.49721536 0.496801 0.49647287 0.49621654 0.49601877 0.495868 0.4957543 ] [0.49977526 0.49945772 0.49912393 0.4988142 0.49854672 0.49832645 0.49815112 0.49801567 0.49791327 0.49783763] [0.5001145 0.5001818 0.500209 0.50020653 0.50018466 0.50015163 0.50011355 0.5000747 0.5000376 0.5000038 ] [0.499907 0.49980363 0.4997118 0.499638 0.49958205 0.49954122 0.4995122 0.49949202 0.49947822 0.4994689 ] [0.49957263 0.4993314 0.49921444 0.49917603 0.4991838 0.499216 0.4992586 0.49930325 0.4993452 0.4993822 ] [0.50122684 0.5020251 0.5025376 0.50286067 0.50305897 0.50317615 0.5032414 0.50327414 0.50328714 0.50328875]]] (1, 6, 10) ``` # Java # TensorFlow (2.x) # MNIST This page illustrates a simple client-server interaction to perform inference on a TensorFlow model using the Java SDK for Konduit Serving. This tutorial is split into two parts: 1. Configuration 2. Running the server {% hint style="info" %} This tutorial is tested on TensorFlow 2.0.0. {% endhint %} ```java import ai.konduit.serving.InferenceConfiguration; import ai.konduit.serving.config.ParallelInferenceConfig; import ai.konduit.serving.config.ServingConfig; import ai.konduit.serving.configprovider.KonduitServingMain; import ai.konduit.serving.configprovider.KonduitServingMainArgs; import ai.konduit.serving.model.ModelConfig; import ai.konduit.serving.model.ModelConfigType; import ai.konduit.serving.model.TensorDataTypesConfig; import ai.konduit.serving.model.TensorFlowConfig; import ai.konduit.serving.pipeline.step.ImageLoadingStep; import ai.konduit.serving.pipeline.step.ModelStep; import ai.konduit.serving.verticles.inference.InferenceVerticle; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import org.apache.commons.io.FileUtils; import org.datavec.api.writable.NDArrayWritable; import org.datavec.api.writable.Writable; import org.datavec.image.transform.ImageTransformProcess; import org.nd4j.linalg.api.ndarray.INDArray; import org.nd4j.linalg.io.ClassPathResource; import org.nd4j.serde.binary.BinarySerde; import org.nd4j.tensorflow.conversion.TensorDataType; ``` ## Overview Konduit Serving works by defining a series of **steps**. These include operations such as 1. Pre- or post-processing steps 2. One or more machine learning models 3. Transforming the output in a way that can be understood by humans If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single `ModelStep`. Set the model file path to `mnistmodelfilePath`. ```java String tensorflow_version = "2.0.0"; String mnistmodelfilePath = new ClassPathResource("data/mnist/mnist_" + tensorflow_version + ".pb").getFile().getAbsolutePath(); ``` {% hint style="info" %} A reference Java project is provided in the Example repository ( ) with a Maven pom.xml dependencies file. If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of the InferenceModelStepMNIST class. {% endhint %} ## Configure the step ### Define the TensorFlow configuration as a `TensorFlowConfig` object * `tensorDataTypesConfig`: The `TensorFlowConfig` object requires a HashMap `input_data_types`. Its keys should represent column names, and the values should represent data types as strings, e.g. `"INT32"`,`"FLOAT"`,etc,. See [here](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-api/src/main/java/ai/konduit/serving/model/TensorDataType.java) for a list of supported data types. * `modelConfigType`: This argument requires a `ModelConfigType` object. Specify `modelType` as `TENSORFLOW`, and `modelLoadingPath` to point to the location of TensorFlow weights saved in the PB file format. ```java HashMap input_data_types = new HashMap(); input_data_types.put("input_layer", TensorDataType.FLOAT); ModelConfig mnistModelConfig = TensorFlowConfig.builder() .tensorDataTypesConfig(TensorDataTypesConfig.builder(). inputDataTypes(input_data_types).build()) .modelConfigType(ModelConfigType.builder(). modelLoadingPath(mnistmodelfilePath.toString()). modelType(ModelConfig.ModelType.TENSORFLOW).build()) .build(); ``` Now that we have a `TensorFlowConfig` defined, we can define a `ModelStep`. The following parameters are specified: * `modelConfig`: pass the mnistModelConfig object here * `parallelInferenceConfig`: specify the number of workers to run in parallel. Here, we specify `workers=1`. * `inputNames`: names for the input data * `outputNames`: names for the output data ```java List input_names = new ArrayList(input_data_types.keySet()); ArrayList output_names = new ArrayList<>(); output_names.add("output_layer/Softmax"); ModelStep mnistModelStep = ModelStep.builder() .modelConfig(mnistModelConfig) .inputNames(input_names) .outputNames(output_names) .parallelInferenceConfig(ParallelInferenceConfig.builder().workers(1).build()) .build(); ``` ## Configure the server Specify the following: * `httpPort`: Specify any port number that is not reserved. ```java int port = Util.randInt(1000, 65535); ServingConfig servingConfig = ServingConfig.builder() .httpPort(port) .build(); ``` The `ServingConfig` has to be passed to `Server` in addition to the steps as a list. In this case, there is a single step: `mnistModelStep`. ```java InferenceConfiguration inferenceConfiguration = InferenceConfiguration.builder() .servingConfig(servingConfig) .step(mnistModelStep) .build(); ``` The `inferenceConfiguration` is stored as a JSON File. Set the KonduitServingMainArgs with the saved **config.json** file path as `configPath` and other necessary server configuration arguments. ```java File configFile = new File("config.json"); FileUtils.write(configFile, inferenceConfiguration.toJson(), Charset.defaultCharset()); //Set and Start inference server as per the above configurations KonduitServingMainArgs args1 = KonduitServingMainArgs.builder() .configStoreType("file").ha(false) .multiThreaded(false).configPort(port) .verticleClassName(InferenceVerticle.class.getName()) .configPath(configFile.getAbsolutePath()) .build(); ``` Start server by calling KonduitServingMain with the configurations mentioned in the KonduitServingMainArgs using Callback Function(as per the code mentioned in the **Inference** Section below) ## Inference The image file(s) has to be converted into NDARRAY using `ImageLoadingStep` and passed as an input for inference. ```java ImageTransformProcess imageTransformProcess = new ImageTransformProcess.Builder() .scaleImageTransform(20.0f) .resizeImageTransform(28, 28) .build(); ImageLoadingStep imageLoadingStep = ImageLoadingStep.builder() .imageProcessingInitialLayout("NCHW") .imageProcessingRequiredLayout("NHWC") .inputName("default") .dimensionsConfig("default", new Long[]{240L, 320L, 3L}) // Height, width, channels .imageTransformProcess("default", imageTransformProcess) .build(); ArrayList imageArr = new ArrayList<>(); ArrayList inputString = new ArrayList<>(); inputString.add("data/facedetector/1.jpg"); for (String imagePathStr : inputString) { String tmpInput = new ClassPathResource(imagePathStr).getFile().getAbsolutePath(); Writable[][] tmpOutput = imageLoadingStep.createRunner().transform(tmpInput); INDArray tmpImage = ((NDArrayWritable) tmpOutput[0][0]).get(); imageArr.add(tmpImage); } ``` To configure the client, set the required URL to connect server and specify any port number that is not reserved (as used in server configuration). A Callback Function onSuccess is implemented in order to post the Client request and get the HttpResponse, only after the successful run of the KonduitServingMain Server. {% hint style="info" %} {% endhint %} Accepted input and output data formats are as follows: * Input: JSON, ARROW, IMAGE, ND4J and NUMPY. * Output: NUMPY, JSON, ND4J and ARROW. {% endhint %} Note that we consider only one test image in this example. ```java KonduitServingMain.builder() .onSuccess(() -> { try { for (INDArray indArray : imageArr) { File file = new File("src/main/resources/data/test-input.zip"); BinarySerde.writeArrayToDisk(indArray, file); String result = Unirest.post(String.format("http://localhost:%s/raw/nd4j", port)) .field("input_layer", file) .asString().getBody(); System.out.println(result); System.exit(0); } } catch (UnirestException | IOException e) { e.printStackTrace(); System.exit(0); } }) .build() .runMain(args1.toArgs()); ``` ## Confirm the output After executing the above, in order to confirm the successful start of the Server, check for the below output text: ``` Jan 07, 2020 2:31:37 PM ai.konduit.serving.configprovider.KonduitServingMain INFO: Deployed verticle ai.konduit.serving.verticles.inference.InferenceVerticle ``` The Output of the program is as follows: ```java System.out.println(result); ``` ![png](https://3414063056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-LthKq91pBS7x1fdSLub%2F-LthbWL_rpJQGG-_5B-Z%2Foutput_25_5.png?alt=media\&token=50db0717-5b12-4695-9eca-992337c7b0e5) ``` { "output_layer/Softmax" : { "batchId" : "8d7acc0d-5497-4882-89db-2b3a0772e480", "ndArray" : { "dataType" : "FLOAT", "shape" : [ 3, 10 ], "data" : [ 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ] } } } ``` The complete inference configuration in JSON format is as follows: ```java System.out.println(inferenceConfiguration.toJson()); ``` ``` { "memMapConfig" : null, "servingConfig" : { "httpPort" : 19947, "listenHost" : "localhost", "logTimings" : false, "metricTypes" : [ "CLASS_LOADER", "JVM_MEMORY", "JVM_GC", "PROCESSOR", "JVM_THREAD", "LOGGING_METRICS", "NATIVE" ], "outputDataFormat" : "JSON", "uploadsDirectory" : "file-uploads/" }, "steps" : [ { "@type" : "ModelStep", "inputColumnNames" : { }, "inputNames" : [ "input_layer" ], "inputSchemas" : { }, "modelConfig" : { "@type" : "TensorFlowConfig", "configProtoPath" : null, "modelConfigType" : { "modelLoadingPath" : "C:\\konduit-serving-examples\\java\\target\\classes\\data\\mnist\\mnist_2.0.0.pb", "modelType" : "TENSORFLOW" }, "savedModelConfig" : null, "tensorDataTypesConfig" : { "inputDataTypes" : { "input_layer" : "FLOAT" }, "outputDataTypes" : { } } }, "normalizationConfig" : null, "outputColumnNames" : { }, "outputNames" : [ "output_layer/Softmax" ], "outputSchemas" : { }, "parallelInferenceConfig" : { "batchLimit" : 32, "inferenceMode" : "BATCHED", "maxTrainEpochs" : 1, "queueLimit" : 64, "vertxConfigJson" : null, "workers" : 1 } } ] } ``` # BERT This page illustrates a simple client-server interaction to perform inference on a TensorFlow model using the Java SDK for Konduit Serving. ```java import ai.konduit.serving.InferenceConfiguration; import ai.konduit.serving.config.ParallelInferenceConfig; import ai.konduit.serving.config.ServingConfig; import ai.konduit.serving.configprovider.KonduitServingMain; import ai.konduit.serving.configprovider.KonduitServingMainArgs; import ai.konduit.serving.model.ModelConfig; import ai.konduit.serving.model.ModelConfigType; import ai.konduit.serving.model.TensorDataTypesConfig; import ai.konduit.serving.model.TensorFlowConfig; import ai.konduit.serving.pipeline.step.ModelStep; import ai.konduit.serving.verticles.inference.InferenceVerticle; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import org.apache.commons.io.FileUtils; import org.nd4j.linalg.io.ClassPathResource; import org.nd4j.tensorflow.conversion.TensorDataType; ``` ## Overview Konduit Serving works by defining a series of **steps**. These include operations such as 1. Pre- or post-processing steps 2. One or more machine learning models 3. Transforming the output in a way that can be understood by humans If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single `ModelStep`. Start by downloading the model weights to the `data` folder.The downloaded zip file can be unzipped using Util class(`Util.unzipBertFile`). ```java String bertmodelfilePath = new ClassPathResource("data/bert").getFile().getAbsolutePath(); String bertFileName = "bert_mrpc_frozen.pb"; File bertModelFile = new File(bertDataFolder, bertFileName); File bertFile = new File(bertmodelfilePath); if (!bertModelFile.exists()) { File bertDownloadedZipFile = Util.downloadBertModel(); Util.unzipBertFile(bertDownloadedZipFile.toString(), bertFileName); } ``` {% hint style="info" %} A reference Java project is provided in the Example repository ( ) with a Maven pom.xml dependencies file. If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of InferenceModelStepBERT the class. {% endhint %} ## Configure the step Define the TensorFlow configuration as a `TensorFlowConfig` object. * `tensorDataTypesConfig`: The TensorFlowConfig object requires a HashMap `input_data_types`. Its keys should represent column names, and the values should represent data types as strings, e.g. `"INT32"`. See [here](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-api/src/main/java/ai/konduit/serving/model/TensorDataType.java) for a list of supported data types. * `modelConfigType`: This argument requires a `ModelConfigType` object. Specify `modelType` as `TENSORFLOW`, and `modelLoadingPath` to point to the location of TensorFlow weights saved in the PB file format. ```java HashMap input_data_types = new LinkedHashMap<>(); input_data_types.put("IteratorGetNext:0", TensorDataType.INT32); input_data_types.put("IteratorGetNext:1", TensorDataType.INT32); input_data_types.put("IteratorGetNext:4", TensorDataType.INT32); ModelConfig bertModelConfig = TensorFlowConfig.builder() .tensorDataTypesConfig(TensorDataTypesConfig.builder(). inputDataTypes(input_data_types).build()) .modelConfigType(ModelConfigType.builder(). modelLoadingPath(bertModelFile.getAbsolutePath()). modelType(ModelConfig.ModelType.TENSORFLOW).build()) .build(); ``` Now that we have a `TensorFlowConfig` defined, we can define a `ModelStep`. The following parameters are specified: * `modelConfig`: pass the TensorFlowConfig object here * `parallelInferenceConfig`: specify the number of workers to run in parallel. Here, we specify `workers=1`. * `inputNames`: names for the input data * `outputNames`: names for the output data ```java List input_names = new ArrayList(input_data_types.keySet()); ArrayList output_names = new ArrayList<>(); output_names.add("loss/Softmax"); ModelStep bertModelStep = ModelStep.builder() .modelConfig(bertModelConfig) .inputNames(input_names) .outputNames(output_names) .parallelInferenceConfig(ParallelInferenceConfig.builder().workers(1).build()) .build(); ``` ## Configure the server Specify the following: * `httpPort`: specify any port number that is not reserved. ```java int port = Util.randInt(1000, 65535); ServingConfig servingConfig = ServingConfig.builder().httpPort(port) .build(); ``` The `ServingConfig` has to be passed to `Server` in addition to the steps as a Python list. In this case, there is a single step: `bertModelStep`. ```java InferenceConfiguration inferenceConfiguration = InferenceConfiguration.builder() .servingConfig(servingConfig) .step(bertModelStep) .build(); ``` The `inferenceConfiguration` is stored as a JSON File. Set the KonduitServingMainArgs with the saved **config.json** file path as `configPath` and other necessary server configuration arguments. ```java File configFile = new File("config.json"); FileUtils.write(configFile, inferenceConfiguration.toJson(), Charset.defaultCharset()); KonduitServingMainArgs args1 = KonduitServingMainArgs.builder() .configStoreType("file").ha(false) .multiThreaded(false).configPort(port) .verticleClassName(InferenceVerticle.class.getName()) .configPath(configFile.getAbsolutePath()) .build(); ``` Start server by calling KonduitServingMain with the configurations mentioned in the KonduitServingMainArgs using Callback Function(as per the code mentioned in the **Inference** Section below) ## Inference Load some sample data from NumPy files. Note that these are NumPy arrays, each with shape (4, 128): ```java File input0 = new ClassPathResource("data/bert/input-0.npy").getFile(); File input1 = new ClassPathResource("data/bert/input-1.npy").getFile(); File input4 = new ClassPathResource("data/bert/input-4.npy").getFile(); ``` To configure the client, set the required URL to connect server and specify any port number that is not reserved (as used in server configuration). A Callback Function onSuccess is implemented in order to post the Client request and get the HttpResponse, only after the successful run of the KonduitServingMain Server. {% hint style="info" %} Accepted input and output data formats are as follows: * Input: JSON, ARROW, IMAGE, ND4J (not yet implemented) and NUMPY. * Output: NUMPY, JSON, ND4J (not yet implemented) and ARROW. {% endhint %} ```java KonduitServingMain.builder() .onSuccess(()->{ try { String response = Unirest.post(String.format("http://localhost:%s/raw/numpy", port)) .field("IteratorGetNext:0", input0) .field("IteratorGetNext:1", input1) .field("IteratorGetNext:4", input4) .asString().getBody(); System.out.print(response); System.exit(0); } catch (UnirestException e) { e.printStackTrace(); System.exit(0); } }) .build() .runMain(args1.toArgs()); ``` ## Confirm the output After executing the above, in order to confirm the successful start of the Server, check for the below output text: ``` Jan 07, 2020 6:02:49 PM ai.konduit.serving.configprovider.KonduitServingMain INFO: Deployed verticle ai.konduit.serving.verticles.inference.InferenceVerticle ``` The Output of the program is as follows: ```java System.out.print(response); ``` ``` "loss/Softmax" : { "batchId" : "41600218-5fb7-401f-af7d-e7fe13313f5d", "ndArray" : { "dataType" : "FLOAT", "shape" : [ 4, 2 ], "data" : [ 0.9894917, 0.010508226, 0.8021635, 0.19783656, 0.9874369, 0.012563077, 0.99294597, 0.0070540793 ] } ``` The complete inference configuration in JSON format is as follows: ```java System.out.println(inferenceConfiguration.toJson()); ``` ``` { "memMapConfig" : null, "servingConfig" : { "httpPort" : 11805, "inputDataFormat" : "NUMPY", "listenHost" : "localhost", "logTimings" : false, "metricTypes" : [ "CLASS_LOADER", "JVM_MEMORY", "JVM_GC", "PROCESSOR", "JVM_THREAD", "LOGGING_METRICS", "NATIVE" ], "outputDataFormat" : "JSON", "predictionType" : "RAW", "uploadsDirectory" : "file-uploads/" }, "steps" : [ { "@type" : "ModelStep", "inputColumnNames" : { }, "inputNames" : [ "IteratorGetNext:0", "IteratorGetNext:1", "IteratorGetNext:4" ], "inputSchemas" : { }, "modelConfig" : { "@type" : "TensorFlowConfig", "configProtoPath" : null, "modelConfigType" : { "modelLoadingPath" : "C:\\konduit-serving-examples\\java\\target\\classes\\data\\bert\\bert_mrpc_frozen.pb", "modelType" : "TENSORFLOW" }, "savedModelConfig" : null, "tensorDataTypesConfig" : { "inputDataTypes" : { "IteratorGetNext:0" : "INT32", "IteratorGetNext:1" : "INT32", "IteratorGetNext:4" : "INT32" }, "outputDataTypes" : { } } }, "normalizationConfig" : null, "outputColumnNames" : { }, "outputNames" : [ "loss/Softmax" ], "outputSchemas" : { }, "parallelInferenceConfig" : { "batchLimit" : 32, "inferenceMode" : "BATCHED", "maxTrainEpochs" : 1, "queueLimit" : 64, "vertxConfigJson" : null, "workers" : 1 } } ] } ``` # Deeplearning4j (DL4J) This document illustrates how to create Konduit Serving configurations with the Java SDK: ## Deeplearning4j (DL4J) * Using Java to create a configuration ```java import ai.konduit.serving.InferenceConfiguration; import ai.konduit.serving.config.ServingConfig; import ai.konduit.serving.configprovider.KonduitServingMain; import ai.konduit.serving.configprovider.KonduitServingMainArgs; import ai.konduit.serving.model.ModelConfig; import ai.konduit.serving.model.ModelConfigType; import ai.konduit.serving.model.TensorDataTypesConfig; import ai.konduit.serving.pipeline.step.ModelStep; import ai.konduit.serving.verticles.inference.InferenceVerticle; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import org.apache.commons.io.FileUtils; import org.nd4j.linalg.api.ndarray.INDArray; import org.nd4j.linalg.io.ClassPathResource; import org.nd4j.serde.binary.BinarySerde; import org.nd4j.tensorflow.conversion.TensorDataType; ``` ### Overview Konduit Serving works by defining a series of **steps**. These include operations such as 1. Pre- or post-processing steps 2. One or more machine learning models 3. Transforming the output in a way that can be understood by humans If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single `ModelStep`. Set the dl4j model path to dl4jmodelfilePath. ```java String dl4jmodelfilePath = new ClassPathResource("data/multilayernetwork/SimpleCNN.zip"). getFile().getAbsolutePath(); ``` {% hint style="info" %} A reference Java project is provided in the Example repository ( ) with a Maven pom.xml dependencies file. If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of the InferenceModelStepDL4J class. {% endhint %} ### Configure the step Define the DL4J configuration as a `ModelConfig` object. * `tensorDataTypesConfig`: The ModelConfig object requires a HashMap `input_data_types`. Its keys should represent column names, and the values should represent data types as strings, e.g. `"INT32"`, `"FLOAT"`,etc,. See [here](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-api/src/main/java/ai/konduit/serving/model/TensorDataType.java) for a list of supported data types. * `modelConfigType`: This argument requires a `ModelConfigType` object. In the Java program above, we recognised that SimpleCNN is configured as a MultiLayerNetwork, in contrast with the ComputationGraph class, which is used for more complex networks. Specify `modelType` as `MULTI_LAYER_NETWORK`, and `modelLoadingPath` to point to the location of DL4J weights saved in the ZIP file format. ```java Map input_data_types = new HashMap<>(); input_data_types.put("image_array", TensorDataType.FLOAT); List input_names = new ArrayList(input_data_types.keySet()); List output_names = new ArrayList<>(); output_names.add("output"); ModelConfig dl4jModelConfig = ModelConfig.builder() .tensorDataTypesConfig(TensorDataTypesConfig.builder(). inputDataTypes(input_data_types).build()) .modelConfigType(ModelConfigType.builder(). modelLoadingPath(dl4jmodelfilePath.toString()). modelType(ModelConfig.ModelType.MULTI_LAYER_NETWORK).build()) .build(); ``` For the `ModelStep` object, the following parameters are specified: * `modelConfig`: pass the ModelConfig object here * `input_names`: names for the input data * `output_names`: names for the output data ```java ModelStep dl4jModelStep = ModelStep.builder() .modelConfig(dl4jModelConfig) .inputNames(input_names) .outputNames(output_names) .build(); ``` ### Configure the server Specify the following: * `httpPort`: specify any port number that is not reserved. ```java int port = Util.randInt(1000, 65535); ServingConfig servingConfig = ServingConfig.builder() .httpPort(port) .build(); ``` The `ServingConfig` has to be passed to `Server` in addition to the steps as a Java list. In this case, there is a single step: `dl4jModelStep`. ```java InferenceConfiguration inferenceConfiguration = InferenceConfiguration.builder() .servingConfig(servingConfig) .step(dl4jModelStep) .build(); ``` {% hint style="info" %} Accepted input and output data formats are as follows: * Input: JSON, ARROW, IMAGE, ND4J (not yet implemented) and NUMPY. * Output: NUMPY, JSON, ND4J (not yet implemented) and ARROW. {% endhint %} The `inferenceConfiguration` is stored as a JSON File. Set the KonduitServingMainArgs with the saved **config.json** file path as `configPath` and other necessary server configuration arguments. ```java File configFile = new File("config.json"); FileUtils.write(configFile, inferenceConfiguration.toJson(), Charset.defaultCharset()); KonduitServingMainArgs args1 = KonduitServingMainArgs.builder() .configStoreType("file").ha(false) .multiThreaded(false).configPort(port) .verticleClassName(InferenceVerticle.class.getName()) .configPath(configFile.getAbsolutePath()) .build(); ``` Start server by calling KonduitServingMain with the configurations mentioned in the KonduitServingMainArgs using Callback Function(as per the code mentioned in the **Inference** Section below) ### Inference We generate a (3, 224, 224) array of random numbers between 0 and 255 as input to the model for prediction. Before requesting for a prediction, we normalize the image to be between 0 and 1: ```java INDArray rand_image = Util.randInt(new int[]{1, 3, 244, 244}, 255); ``` ```java File file = new File("src/main/resources/data/test-dl4j.zip"); if(!file.exists()) file.createNewFile(); BinarySerde.writeArrayToDisk(rand_image, file); ``` To configure the client, set the required URL to connect server and specify any port number that is not reserved (as used in server configuration). A Callback Function onSuccess is implemented in order to post the Client request and get the HttpResponse, only after the successful run of the KonduitServingMain Server. {% hint style="info" %} Accepted input and output data formats are as follows: * Input: JSON, ARROW, IMAGE, ND4J (not yet implemented) and NUMPY. * Output: NUMPY, JSON, ND4J (not yet implemented) and ARROW. {% endhint %} ```java KonduitServingMain.builder() .onSuccess(() -> { try { String response = Unirest.post(String.format("http://localhost:%s/raw/nd4j", port)) .field("image_array", file).asString().getBody(); System.out.print(response); System.exit(0); } catch (UnirestException e) { e.printStackTrace(); System.exit(0); } }) .build() .runMain(args1.toArgs()); ``` ## Confirm the Output After executing the above, in order to confirm the successful start of the Server, check for the below output text: ``` Jan 08, 2020 3:03:50 PM ai.konduit.serving.configprovider.KonduitServingMain INFO: Deployed verticle ai.konduit.serving.verticles.inference.InferenceVerticle ``` The Output of the program is as follows: ```java System.out.print(response); ``` ``` { "output" : { "batchId" : "d5090c30-526d-4e1f-93e2-a918435ac1da", "ndArray" : { "dataType" : "FLOAT", "shape" : [ 1, 5 ], "data" : [ 0.028113496, 0.3778126, 0.023068674, 3.759411E-5, 0.5709677 ] } } } ``` The complete inference configuration in JSON format is as follows: ```java System.out.println(inferenceConfiguration.toJson()); ``` ``` { "memMapConfig" : null, "servingConfig" : { "httpPort" : 24229, "listenHost" : "localhost", "logTimings" : false, "metricTypes" : [ "CLASS_LOADER", "JVM_MEMORY", "JVM_GC", "PROCESSOR", "JVM_THREAD", "LOGGING_METRICS", "NATIVE" ], "outputDataFormat" : "JSON", "uploadsDirectory" : "file-uploads/" }, "steps" : [ { "@type" : "ModelStep", "inputColumnNames" : { }, "inputNames" : [ "image_array" ], "inputSchemas" : { }, "modelConfig" : { "@type" : "ModelConfig", "modelConfigType" : { "modelLoadingPath" : "C:\\konduit-serving-examples\\java\\target\\classes\\data\\multilayernetwork\\SimpleCNN.zip", "modelType" : "MULTI_LAYER_NETWORK" }, "tensorDataTypesConfig" : { "inputDataTypes" : { "image_array" : "FLOAT" }, "outputDataTypes" : { } } }, "normalizationConfig" : null, "outputColumnNames" : { }, "outputNames" : [ "output" ], "outputSchemas" : { }, "parallelInferenceConfig" : { "batchLimit" : 32, "inferenceMode" : "BATCHED", "maxTrainEpochs" : 1, "queueLimit" : 64, "vertxConfigJson" : null, "workers" : 1 } } ] } ``` # DataVec Konduit Serving supports data transformations defined by the DataVec vectorization and ETL library. ```java import ai.konduit.serving.InferenceConfiguration; import ai.konduit.serving.config.ServingConfig; import ai.konduit.serving.configprovider.KonduitServingMain; import ai.konduit.serving.pipeline.step.TransformProcessStep; import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.JsonNode; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import org.apache.commons.io.FileUtils; import org.datavec.api.transform.TransformProcess; import org.datavec.api.transform.schema.Schema; ``` {% hint style="info" %} A reference Java project is provided in the Example repository ( ) with a Maven pom.xml dependencies file. If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of InferenceModelStepDataVec the class. {% endhint %} ## Data transformations with DataVec ### **Schema (**[**source**](https://github.com/deeplearning4j/DataVec/blob/master/datavec-api/src/main/java/org/datavec/api/transform/schema/Schema.java)**)** A `Schema` specifies the structure of your data. In DataVec, a `TransformProcess` requires the `Schema` of the data to be specified. Both schema and transform process classes come with a helper Builder class which are useful for organizing code and avoiding complex constructors. `Schema` objects have a number of methods that define different data types for columns: `addColumnsString()`, `addColumnInteger()`, `addColumnLong()`, `addColumnFloat()`, `addColumnDouble()` and `addColumnCategorical()`. ### **TransformProcess (**[**source**](https://github.com/deeplearning4j/DataVec/blob/master/datavec-api/src/main/java/org/datavec/api/transform/TransformProcess.java)**)** `TransformProcess` provides a number of methods to manipulate your data. The following methods are available in the Datavec API: * Reduce the number of rows: `filter()` * General data transformations: `replaceStringTransform()`, `replaceMapTransform()`, * Type casting: `stringToTimeTransform()`, `transform()`, `categoricalToInteger()`, * Combining/reducing the values in each column: `reduce()` * String operations: `appendStringColumnTransform()`, `toLowerCase()`, `toUpperCase()`, `stringRemoveWhitespaceTransform()`, `replaceStringTransform()`, `stringMapTransform()` * Column selection/renaming: `removeColumns()`, `removeAllColumnsExceptFor()`, `renameColumn()` * One-hot encoding: `categoricalToOneHot()`, `integerToOneHot()` In this short example, we append the string `two` to the end of values in the string column `first`. As an initial step, define the input and output Schema with string column: ```java Schema inputSchema = new Schema.Builder() .addColumnString("first") .build(); Schema outputSchema = new Schema.Builder() .addColumnString("first") .build(); TransformProcess transformProcess = new TransformProcess.Builder(inputSchema). appendStringColumnTransform("first", "two").build(); ``` ## Configure the step The `TransformProcess` can now be defined in the Konduit Serving configuration with a `TransformProcessStep`. Here, we * **configure the inputs and outputs**: the schema, column names and data types should be defined here. * **declare the `TransformProcess`** using the `.transformProcess()` method. Note that `Schema` data types are not defined in the same way as `PythonStep` data types. See the [source](https://github.com/KonduitAI/konduit-serving/blob/78851701004ebb3dbf079889d46b79a9db8fac60/konduit-serving-api/src/main/java/ai/konduit/serving/util/SchemaTypeUtils.java#L154-L195) for a complete list of supported Schema data types: * `NDArray` * `String` * `Boolean` * `Categorical` * `Float` * `Double` * `Integer` * `Long` * `Bytes` You should define the Schema data types in `TransformProcessStep()` as strings. ```java TransformProcessStep transformProcessStep = new TransformProcessStep(transformProcess, outputSchema); ``` ## Configure the server Configure the Server using `ServingConfig` to define the port using the `httpPort` argument. ```java int port = Util.randInt(1000, 65535); ServingConfig servingConfig = ServingConfig.builder() .httpPort(port) .build(); InferenceConfiguration inferenceConfiguration = InferenceConfiguration.builder() .step(transformProcessStep).servingConfig(servingConfig).build(); ``` The `inferenceConfiguration` is stored as a JSON File. ```java File configFile = new File("config.json"); FileUtils.write(configFile, inferenceConfiguration.toJson(), Charset.defaultCharset()); ``` ## Inference The `Client` should be configured to match the Konduit Serving instance. As this example is run on a local computer, the server is located at host `'http://localhost'` and port `port`. And Finally, we run the Konduit Serving instance with the saved **config.json** file path as `configPath` and other necessary server configuration arguments.. Recall that the `TransformProcessStep()` appends a string `two` to strings in the column `first`. A Callback Function onSuccess is implemented in order to post the Client request and get the HttpResponse, only after the successful run of the KonduitServingMain Server. ```java KonduitServingMain.builder() .onSuccess(() -> { try { HttpResponse response = Unirest.post(String.format("http://localhost:%s/raw/json", port)) .header("Content-Type", "application/json") .body("{\"first\" :\"value\"}").asJson(); System.out.println(response.getBody().toString()); System.exit(0); } catch (UnirestException e) { e.printStackTrace(); System.exit(0); } }) .build() .runMain("--configPath", configFile.getAbsolutePath()); ``` ## Confirm the output After executing the above, in order to confirm the successful start of the Server, check for the below output text: ``` Jan 08, 2020 1:36:01 PM ai.konduit.serving.configprovider.KonduitServingMain INFO: Deployed verticle ai.konduit.serving.verticles.inference.InferenceVerticle ``` The Output of the program is as follows: ```java System.out.println(response.getBody().toString()); ``` ``` {"first":"valuetwo"} ``` The complete inference configuration in JSON format is as follows: ```java System.out.println(inferenceConfiguration.toJson()); ``` ``` SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] { "memMapConfig" : null, "servingConfig" : { "httpPort" : 15614, "listenHost" : "localhost", "logTimings" : false, "metricTypes" : [ "CLASS_LOADER", "JVM_MEMORY", "JVM_GC", "PROCESSOR", "JVM_THREAD", "LOGGING_METRICS", "NATIVE" ], "outputDataFormat" : "JSON", "uploadsDirectory" : "file-uploads/" }, "steps" : [ { "@type" : "TransformProcessStep", "inputColumnNames" : { "default" : [ "first" ] }, "inputNames" : [ "default" ], "inputSchemas" : { "default" : [ "String" ] }, "outputColumnNames" : { "default" : [ "first" ] }, "outputNames" : [ "default" ], "outputSchemas" : { "default" : [ "String" ] }, "transformProcesses" : { "default" : { "actionList" : [ { "transform" : { "@class" : "org.datavec.api.transform.transform.string.AppendStringColumnTransform", "columnName" : "first", "toAppend" : "two" } } ], "initialSchema" : { "@class" : "org.datavec.api.transform.schema.Schema", "columns" : [ { "@class" : "org.datavec.api.transform.metadata.StringMetaData", "name" : "first" } ] } } } } ] } ``` # Open Neural Network Exchange (ONNX) This page provides a Java example of inferencing a model, built in Python with ONNX Runtime, a cross-platform, high performance scoring engine for machine learning models. The Open Neural Network Exchange (ONNX) format is supported by a number of deep learning frameworks, including PyTorch, CNTK, MXNet, etc,. ```java import ai.konduit.serving.InferenceConfiguration; import ai.konduit.serving.config.ServingConfig; import ai.konduit.serving.configprovider.KonduitServingMain; import ai.konduit.serving.model.PythonConfig; import ai.konduit.serving.pipeline.step.ImageLoadingStep; import ai.konduit.serving.pipeline.step.PythonStep; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import org.apache.commons.io.FileUtils; import org.datavec.python.PythonVariables; import org.nd4j.linalg.io.ClassPathResource; ``` {% hint style="info" %} A reference Java project is provided in the Example repository ( ) with a Maven pom.xml dependencies file. If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of InferenceModelStepONNX class. {% endhint %} For the purposes of this example, we use ONNX model files from [Ultra-Light-Fast-Generic-Face-Detector-1MB](https://github.com/Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB) by Linzaer, a lightweight facedetection model designed for edge computing devices. ## Python script with PyTorch and ONNX Runtime Now that we have an optimized ONNX file, we can serve our model. The following is the python script `onnxFacedetect.py` : * transforms a [PIL](https://python-pillow.org/) image into a 240 x 320 image, * casts it into a PyTorch Tensor, * adds an extra dimension with [`unsqueeze`](https://pytorch.org/docs/stable/torch.html#torch.unsqueeze), * casts the Tensor into a NumPy array, then * returns the model's output with ONNX Runtime. ```python import os import numpy as np from PIL import Image import torchvision.transforms as transforms import onnxruntime from matplotlib.image import imread dl_path = os.path.abspath("./src/main/resources/data/facedetector/facedetector.onnx") sys.path.append(dl_path) a,b,c,d=inputimage.shape inputimage=inputimage.reshape(b,c,d) im=np.array(inputimage) image = Image.fromarray(im.astype('uint8'), 'RGB') resize = transforms.Resize([240, 320]) img_y = resize(image) to_tensor = transforms.ToTensor() img_y = to_tensor(img_y) img_y.unsqueeze_(0) def to_numpy(tensor): return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy() ort_session = onnxruntime.InferenceSession(dl_path) ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(img_y)} ort_outs = ort_session.run(None, ort_inputs) _, boxes = ort_outs ``` ## Configure the step ### Defining a `PythonConfig` * Here we use the `pythonCodePath` argument instead of `pythonCode`, in order to specify the location of the Python script. * Define the inputs and outputs name and type of the value as defined by the name() method of a PythonVariables.Type, here we use NDARRAY. See for supported data types. * To run this example please install (PIL 6.21,numpy,matplotlib 3.1.2,onnxruntime 1.1.0, torchvision 0.4.2)and set the python path as `pythonPath(pythonPath)` in the `python_config` to refer the required Python libraries. ```java String pythonCodePath = new ClassPathResource("scripts/onnxFacedetect.py").getFile().getAbsolutePath(); String pythonPath = Arrays.stream(cachePackages()) .filter(Objects::nonNull) .map(File::getAbsolutePath) .collect(Collectors.joining(File.pathSeparator)); PythonConfig python_config = PythonConfig.builder() .pythonCodePath(pythonCodePath) .pythonInput("inputimage", PythonVariables.Type.NDARRAY.name()) .pythonOutput("boxes", PythonVariables.Type.NDARRAY.name()) .pythonPath(pythonPath) .build(); ``` ### Define a pipeline step with the `PythonStep` class In the `.step()` method, define the input configuration (`python_config`). ```java PythonStep onnx_step = new PythonStep().step(python_config); ``` ### Define a pipeline step with `ImageLoadingStep` class A Pipeline Step for loading and transforming an image ```java ImageLoadingStep imageLoadingStep = ImageLoadingStep.builder() .inputName("inputimage") .dimensionsConfig("default", new Long[]{478L, 720L, 3L}) // Height, width, channels .build(); ``` ## Configure the server In the `ServingConfig`, specify any port number that is not reserved. ```java int port = Util.randInt(1000, 65535); ServingConfig servingConfig = ServingConfig.builder().httpPort(port). build(); ``` The `ServingConfig` has to be passed to `Server` in addition to the `imageLoadingStep` as a list. In this case, there is a single step: `onnx_step`. ```java InferenceConfiguration inferenceConfiguration = InferenceConfiguration.builder() .steps(Arrays.asList(imageLoadingStep, onnx_step)).servingConfig(servingConfig).build(); ``` The `inferenceConfiguration` is stored as a JSON File. ```java File configFile = new File("config.json"); FileUtils.write(configFile, inferenceConfiguration.toJson(), Charset.defaultCharset()); ``` ## Inference Load a sample image and send the image as NDARRAY to the server for prediction. {% hint style="info" %} {% endhint %} Accepted input and output data formats are as follows: * Input: JSON, ARROW, IMAGE, ND4J and NUMPY. * Output: NUMPY, JSON, ND4J and ARROW. {% endhint %} The `Client` should be configured to match the Konduit Serving instance. As this example is run on a local computer, the server is located at host `'http://localhost'` and port `port`. And Finally, we run the Konduit Serving instance with the saved **config.json** file path as `configPath` and other necessary server configuration arguments. A Callback Function onSuccess is implemented in order to post the Client request and get the HttpResponse, only after the successful run of the KonduitServingMain Server. ```java File imageOnnx = new ClassPathResource("data/facedetector/OnnxImageTest.jpg").getFile(); KonduitServingMain.builder() .onSuccess(() -> { try { HttpResponse response = Unirest.post(String.format("http://localhost:%s/raw/json", port)) .header("Content-Type", "application/json") .body("{\"first\" :\"value\"}").asJson(); System.out.println(response.getBody().toString()); System.exit(0); } catch (UnirestException e) { e.printStackTrace(); System.exit(0); } }) .build() .runMain("--configPath", configFile.getAbsolutePath()); ``` ## Confirm the output After executing the above, in order to confirm the successful start of the Server, check for the below output text: ``` Jan 08, 2020 6:33:47 PM ai.konduit.serving.configprovider.KonduitServingMain INFO: Deployed verticle ai.konduit.serving.verticles.inference.InferenceVerticle ``` The Output of the program is as follows: ```java System.out.println(response.getBody().toString()); ``` ``` { "default" : { "batchId" : "78590ecd-3dd0-4fe4-9b9f-c6046e691207", "ndArray" : { "dataType" : "FLOAT", "shape" : [ 1, 4420, 4 ], "data" : [ 0.0053688805, 0.0025114473, 0.02034211, 0.03720106, -0.0017913571, -0.008830132, 0.030541036, 0.062001586, -0.009721115, -0.02121478, 0.038849648, ...... ...... 0.95745945, 1.198462, 0.379943, 0.39496967, 1.0312535, 1.2312568, 0.7340108, 0.62931126, 1.0600785, 1.100086, 0.65668, 0.49809134, 1.1420089, 1.2198907, 0.57574236, 0.38997138, 1.2035433, 1.2634758 ] } } } ``` The complete inference configuration in JSON format is as follows: ```java System.out.println(inferenceConfiguration.toJson()); ``` ``` { "memMapConfig" : null, "servingConfig" : { "httpPort" : 26652, "listenHost" : "localhost", "logTimings" : false, "metricTypes" : [ "CLASS_LOADER", "JVM_MEMORY", "JVM_GC", "PROCESSOR", "JVM_THREAD", "LOGGING_METRICS", "NATIVE" ], "outputDataFormat" : "JSON", "uploadsDirectory" : "file-uploads/" }, "steps" : [ { "@type" : "ImageLoadingStep", "dimensionsConfigs" : { "default" : [ 478, 720, 3 ] }, "imageProcessingInitialLayout" : null, "imageProcessingRequiredLayout" : null, "imageTransformProcesses" : { }, "inputColumnNames" : { }, "inputNames" : [ "inputimage" ], "inputSchemas" : { }, "objectDetectionConfig" : null, "originalImageHeight" : 0, "originalImageWidth" : 0, "outputColumnNames" : { }, "outputNames" : [ ], "outputSchemas" : { }, "updateOrderingBeforeTransform" : false }, { "@type" : "PythonStep", "inputColumnNames" : { "default" : [ "inputimage" ] }, "inputNames" : [ "default" ], "inputSchemas" : { "default" : [ "NDArray" ] }, "outputColumnNames" : { "default" : [ "boxes" ] }, "outputNames" : [ "default" ], "outputSchemas" : { "default" : [ "NDArray" ] }, "pythonConfigs" : { "default" : { "extraInputs" : { }, "pythonCode" : null, "pythonCodePath" : "C:\\Projects\\konduit-serving-examples\\java\\target\\classes\\scripts\\onnxFacedetect.py", "pythonInputs" : { "inputimage" : "NDARRAY" }, "pythonOutputs" : { "boxes" : "NDARRAY" }, "pythonPath" : "C:\\Users\\AppData\\Local\\Programs\\Python\\Python37\\python37.zip;C:\\Users\\AppData\\Local\\Programs\\Python\\Python37\\DLLs;C:\\Users\\AppData\\Local\\Programs\\Python\\Python37\\lib;C:\\Users\\AppData\\Local\\Programs\\Python\\Python37;C:\\Users\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages;C:\\Users\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pyyaml-5.2-py3.7-win-amd64.egg;c:\\projects\\konduit-serving\\python", "returnAllInputs" : false, "setupAndRun" : false } } } ] } ``` # Keras (TensorFlow 2.0) This page illustrates a simple client-server interaction to perform inference on a Keras LSTM model using the Java SDK for Konduit Serving. ```java import ai.konduit.serving.InferenceConfiguration; import ai.konduit.serving.config.ParallelInferenceConfig; import ai.konduit.serving.config.ServingConfig; import ai.konduit.serving.configprovider.KonduitServingMain; import ai.konduit.serving.configprovider.KonduitServingMainArgs; import ai.konduit.serving.model.ModelConfig; import ai.konduit.serving.model.ModelConfigType; import ai.konduit.serving.pipeline.step.ModelStep; import ai.konduit.serving.verticles.inference.InferenceVerticle; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import org.apache.commons.io.FileUtils; import org.nd4j.linalg.api.ndarray.INDArray; import org.nd4j.linalg.factory.Nd4j; import org.nd4j.linalg.io.ClassPathResource; import org.nd4j.serde.binary.BinarySerde; ``` ## Saving models in Keras HDF5 (.h5) format Models can be saved using Python with the `.save()` method. Refer to the [TensorFlow documentation for Keras](https://www.tensorflow.org/guide/keras/save_and_serialize) for details. These saved models shall be loaded in Java. {% hint style="info" %} Keras model loading functionality in Konduit Serving converts Keras models to Deeplearning4J models. As a result, Keras models containing operations not supported in Deeplearning4J cannot be served in Konduit Serving. See [issue 8348](https://github.com/eclipse/deeplearning4j/issues/8348). {% endhint %} ## Overview Konduit Serving works by defining a series of **steps**. These include operations such as 1. Pre- or post-processing steps 2. One or more machine learning models 3. Transforming the output in a way that can be understood by humans If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single `ModelStep`. {% hint style="info" %} A reference Java project is provided in the Example repository ( ) with a Maven pom.xml dependencies file. If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of the InferenceModelStepKeras class. {% endhint %} ## Configure the step Define the Keras configuration as a `ModelConfig` object. * `modelConfigType`: This argument requires a `ModelConfigType` object. Specify `modelType` as `ModelConfig.ModelType.KERAS`, and `modelLoadingPath` to point to the location of Keras weights saved in the HDF5 file format. For the `ModelStep` object, the following parameters are specified: * `modelConfig`: pass the ModelConfig object here * `parallelInferenceConfig`: specify the number of workers to run in parallel. Here, we specify `workers = 1`. * `inputName`, `outputName`: names for the input and output nodes, as lists ```java String kerasmodelfilePath = new ClassPathResource("data/keras/embedding_lstm_tensorflow_2.h5").getFile().getAbsolutePath(); ModelConfig kerasModelConfig = ModelConfig.builder() .modelConfigType(ModelConfigType.builder() .modelLoadingPath(kerasmodelfilePath.toString()) .modelType(ModelConfig.ModelType.KERAS).build()) .build(); ModelStep kerasmodelStep = ModelStep.builder() .modelConfig(kerasModelConfig) .inputName("input") .outputName("lstm_1") .parallelInferenceConfig(ParallelInferenceConfig.builder().workers(1).build()) .build(); ``` {% hint style="info" %} Input and output names can be obtained by visualizing the graph in [Netron](https://github.com/lutzroeder/netron). {% endhint %} ## Configure the server In the `ServingConfig`, specify any port number that is not reserved. ```java int port = Util.randInt(1000, 65535); ServingConfig servingConfig = ServingConfig.builder().httpPort(port). build(); ``` The `ServingConfig` has to be passed to `Server` in addition to the steps as a list. In this case, there is a single step: `kerasmodelStep`. ```java InferenceConfiguration inferenceConfiguration = InferenceConfiguration.builder() .servingConfig(servingConfig) .step(kerasmodelStep) .build(); ``` The `inferenceConfiguration` is stored as a JSON File. Set the KonduitServingMainArgs with the saved **config.json** file path as `configPath` and other necessary server configuration arguments. ```java File configFile = new File("config.json"); FileUtils.write(configFile, inferenceConfiguration.toJson(), Charset.defaultCharset()); KonduitServingMainArgs args1 = KonduitServingMainArgs.builder() .configStoreType("file").ha(false) .multiThreaded(false).configPort(port) .verticleClassName(InferenceVerticle.class.getName()) .configPath(configFile.getAbsolutePath()) .build(); ``` Start server by calling KonduitServingMain with the configurations mentioned in the KonduitServingMainArgs using Callback Function(as per the code mentioned in the **Inference** Section below) ## Inference NDARRAY inputs to set ModelStep must be specified with a shape size. To configure the client, set the required URL to connect server and specify any port number that is not reserved (as used in server configuration). A Callback Function onSuccess is implemented in order to post the Client request and get the HttpResponse, only after the successful run of the KonduitServingMain Server. ```java INDArray arr = Nd4j.create(new float[]{1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f}, 1, 10); File file = new File("src/main/resources/data/test-input.zip"); System.out.println(file.getAbsolutePath()); BinarySerde.writeArrayToDisk(arr, file); KonduitServingMain.builder() .onSuccess(() -> { try { String response = Unirest.post(String.format("http://localhost:%s/raw/nd4j", port)) .field("input", file) .asString().getBody(); System.out.print(response); System.exit(0); } catch (UnirestException e) { e.printStackTrace(); System.exit(0); } }) .build() .runMain(args1.toArgs()); ``` ## Confirm the output After executing the above, in order to confirm the successful start of the Server, check for the below output text: ``` Jan 07, 2020 2:31:37 PM ai.konduit.serving.configprovider.KonduitServingMain INFO: Deployed verticle ai.konduit.serving.verticles.inference.InferenceVerticle ``` The Output of the program is as follows: ```java System.out.print(response) ``` ``` { "lstm_1" : { "batchId" : "61c32b20-20c1-42d9-909e-40519304f2ac", "ndArray" : { "dataType" : "FLOAT", "shape" : [ 1, 6, 10 ], "data" : [ -0.0022880966, -0.0042849067, -0.005983479, -0.0073982426, -0.008555864, -0.009488584, -0.010229964, -0.010812048, -0.011263946, -0.011611057, -0.0027362786, -0.005198746, -0.0072902446, -0.009000374, -0.010361125, -0.011421581, -0.012234278, -0.012848378, -0.013306688, -0.013644846, 8.9187745E-4, 0.0012898755, 0.0014061421, 0.0013690761, 0.0012552886, 0.00110957, 9.573803E-4, 8.1235886E-4, 6.812691E-4, 5.6666887E-4, -0.0029521661, -0.0049125706, -0.0062154937, -0.0070830043, -0.007662085, -0.008049844, -0.008310454, -0.008486291, -0.00860548, -0.008686648, -2.41272E-4, -2.1998871E-4, -3.9213814E-5, 2.2318505E-4, 5.1378313E-4, 7.9911196E-4, 0.0010602918, 0.0012885022, 0.0014812968, 0.00164004, 0.0029523545, 0.0050065047, 0.006426977, 0.007400234, 0.008058914, 0.008497588, 0.008783782, 0.00896551, 0.009076695, 0.009141108 ] } } ``` The complete inference configuration in JSON format is as follows: ```java System.out.println(inferenceConfiguration.toJson()); ``` ``` { "memMapConfig" : null, "servingConfig" : { "httpPort" : 62969, "listenHost" : "localhost", "logTimings" : false, "metricTypes" : [ "CLASS_LOADER", "JVM_MEMORY", "JVM_GC", "PROCESSOR", "JVM_THREAD", "LOGGING_METRICS", "NATIVE" ], "outputDataFormat" : "JSON", "uploadsDirectory" : "file-uploads/" }, "steps" : [ { "@type" : "ModelStep", "inputColumnNames" : { }, "inputNames" : [ "input" ], "inputSchemas" : { }, "modelConfig" : { "@type" : "ModelConfig", "modelConfigType" : { "modelLoadingPath" : "C:\\konduit-serving-examples\\java\\target\\classes\\data\\keras\\embedding_lstm_tensorflow_2.h5", "modelType" : "KERAS" }, "tensorDataTypesConfig" : null }, "normalizationConfig" : null, "outputColumnNames" : { }, "outputNames" : [ "lstm_1" ], "outputSchemas" : { }, "parallelInferenceConfig" : { "batchLimit" : 32, "inferenceMode" : "BATCHED", "maxTrainEpochs" : 1, "queueLimit" : 64, "vertxConfigJson" : null, "workers" : 1 } } ] } ``` # Monitoring with Grafana Prometheus and Grafana can be used for displaying metrics to assist with troubleshooting production systems. ## Concepts ### Konduit Serving `metrics` endpoint For monitoring, the REST API of a Konduit Serving instance exposes a `/metrics` endpoint that returns metrics in the Prometheus format. By default, metrics returned by the `metrics` endpoint include * average CPU load; * memory use; * I/O wait time; * GPU bandwidth device to device, bandwidth device to host, current load for device, current available memory for each GPU; and * CPU current load for device, current available memory. The metrics above are implemented by the [NativeMetrics class](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-core/src/main/java/ai/konduit/serving/metrics/NativeMetrics.java). The `metrics` endpoint also returns Micrometer JVM and system metrics via the `ClassLoaderMetrics`, `JvmMemoryMetrics`, `JvmGcMetrics`, `ProcessorMetrics` and `JvmThreadMetrics` binders. See the [Micrometer documentation](https://micrometer.io/docs/ref/jvm) for descriptions of these classes. Error, warning, info, debug and trace counts are monitored using Micrometer's [`LogbackMetrics` binder](https://github.com/micrometer-metrics/micrometer/blob/master/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/logging/LogbackMetrics.java#L36). ### Prometheus Prometheus is a widely used time series database for tracking system metrics used for debugging production systems. This includes common metrics used to troubleshoot problems with production applications such as: * Out of memory * Latency For machine learning, we may include other metrics to help debug things such as: * Compute time for a neural net * ETL creation (number of times it takes to convert raw data to a minibatch or NumPy ndarray) Prometheus works by pulling data from the specified sources. A Prometheus instance is configured by a YAML file such as: ```yaml # Global configurations global: scrape_interval: 5s # Set the scrape interval to every 5 seconds. evaluation_interval: 5s # Evaluate rules every 5 seconds. scrape_configs: - job_name: 'scrape' static_configs: - targets: [ 'localhost:1337'] ``` This YAML file contains a global configuration and a [`scrap_config`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) section. See [Prometheus's configuration documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) for details. The main component to configure is `targets`. `targets` is where you specify the source to pull data from. A Konduit Serving instance exposes metrics to be picked up by Prometheus from `http://:/metrics`. ### Grafana [Grafana](https://grafana.com) is a dashboard system for pulling data from different sources and displaying it in real time. It can be used to visualize output from Prometheus. Grafana allows you to declare a dashboard as a JSON file. An imported Grafana dashboard will show some pre-configured metrics. You can always extend/add more metrics in the Grafana GUI and re-export the configuration. ## Installation * **Konduit Serving**: Follow the [installation steps](/0.1.0-snapshot/installation) to build a Konduit Serving JAR file and install the `konduit` Python module. * **Prometheus**: Download a [precompiled Prometheus binary](https://prometheus.io/download) for your OS architecture and unzip to a location on your local drive. * **Grafana**: Install Grafana from Grafana's [Downloads](https://grafana.com/grafana/download) page. See the [Grafana installation documentation](https://grafana.com/docs/grafana/latest/installation/) for platform-specific instructions. ## Usage The following instructions assume that you're in the [monitoring/quickstart directory](https://github.com/KonduitAI/konduit-serving-examples/tree/master/monitoring/quickstart) of the [KonduitAI/konduit-serving-examples](https://github.com/KonduitAI/konduit-serving-examples/) repository. ### Start Konduit server In this folder, run the following in a command line ```bash konduit serve --config ../../yaml/simple.yaml ``` This creates a local Konduit Serving instance using the YAML configuration file [simple.yaml](https://app.gitbook.com/s/-LsGy_78jsGh0h_1MndS-2657041423/yaml/simple.yaml) at port 1337. ### Start Prometheus server In this example, we use Prometheus to monitor the Konduit Serving instance. Copy the `prometheus.yml` file in this directory to the location of your Prometheus binary. Then, run: ```bash ./prometheus --config.file=prometheus_quickstart.yml ``` Omit the `./` if you're running Prometheus on `cmd.exe`. The `./` suffix is required on PowerShell. By default, Prometheus runs on port 9090. ### Start Grafana server In this example, we use Grafana, which provides a dashboard to visualize data from the Prometheus instance. See the Grafana installation instructions for your platform ([Windows](https://grafana.com/docs/grafana/latest/installation/windows/), [macOS](https://grafana.com/docs/grafana/latest/installation/mac/), [Ubuntu / Debian](https://grafana.com/docs/grafana/latest/installation/debian/), [Centos / Redhat](https://grafana.com/docs/grafana/latest/installation/rpm/)) for instructions to start a Grafana service or, optionally, have Grafana initialize on startup. If you use the Windows installer to install Grafana, [NSSM](https://nssm.cc/) will run Grafana automatically at startup, and there is no need to initialize the Grafana server instance. In your browser, open`localhost:3000`. Login with the username `admin` and password `admin`. Next, add a Prometheus data source. Click on Add Data Source > Prometheus, then insert the HTTP URL in the following page. On the bar on the left, mouse over on the + button, then click on Import. ![](https://3414063056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fserving%2F-Lve8kJvRp81z8HC63NS%2F-Lve94r3qM-aR0lgtC2q%2Fdashboardimport.png?generation=1575886545081913\&alt=media) Copy and paste the JSON in [dashboard.json](https://app.gitbook.com/s/-LsGy_78jsGh0h_1MndS-2657041423/model-monitoring/quickstart/dashboard.json) into the import page as follows, then click the Load button: ![](https://3414063056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fserving%2F-Lve8kJvRp81z8HC63NS%2F-Lve94r6MpWepcGIJbIG%2Fjsonimportdashboard.png?generation=1575886545072557\&alt=media) On the next page, enter a name for your dashboard (such as **Pipeline Metrics**). Click the Import button: ![](https://3414063056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fserving%2F-Lve8kJvRp81z8HC63NS%2F-Lve94r4LbcN9tie4fyk%2Fdashboardimportfinish.png?generation=1575886545069603\&alt=media) Your Grafana dashboard will render on the next page. This dashboard contains metrics for system load and memory as well as timings for performing inference and ETL. ![](https://3414063056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fserving%2F-Lve8kJvRp81z8HC63NS%2F-Lve94r5Bu1n9qqKa_GX%2Fdashboardrender.png?generation=1575886544974867\&alt=media) ### Obtaining a prediction Use the `predict-numpy` command: ```bash konduit predict-numpy --config ../../yaml/simple.yaml --numpy_data ../../data/simple/input_arr.npy ``` ### Stop server Remember to stop the Konduit Serving instance with ```bash konduit stop-server --config ../../yaml/simple.yaml ``` ## References * Grafana support for Prometheus: # Data transformation pipeline steps {% hint style="info" %} This page is currently under construction. In the meantime, please refer to the [DataVec](/0.1.0-snapshot/examples/python/datavec) example. {% endhint %} # Image loading pipeline steps {% hint style="info" %} This page is currently under construction. In the meantime, please refer to [BasicConfigurationImage.java](https://github.com/KonduitAI/konduit-serving-examples/blob/master/java/src/main/java/ai/konduit/serving/examples/basic/BasicConfigurationImage.java). {% endhint %} # Python pipeline steps You can integrate Python into a Konduit Serving instance by defining PythonConfig objects as steps to PythonPipelineStep. Konduit Serving uses [JavaCPP Presets](https://github.com/bytedeco/javacpp-presets/) to execute Python scripts using the [CPython API](https://github.com/bytedeco/javacpp-presets/tree/master/cpython). This allows you to build custom Konduit Serving pipeline steps by writing Python scripts to be run within a Konduit Serving Java process. ![](https://3414063056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-LwGXMv-motlCpJTEGq2%2F-LwGXhXoQGKWShF0F-vJ%2Fimage.png?alt=media\&token=6e97fc95-72a4-4fa2-843d-19035e507a2f) ## **`PythonConfig`** Define the configuration for a `PythonStep`. {% tabs %} {% tab title="Python" %} A `PythonConfig` takes on the following parameters: * `python_path`: Optional. The search path for Python modules, as defined by [`sys.path`](https://docs.python.org/3/library/sys.html#sys.path). See the [Python modules](/0.1.0-snapshot/steps/python#pythonpath) section for details. * `python_code_path`: Optional. Specify the location of your Python script. * `python_code`: Optional. A string that contains Python commands. * `python_inputs`: A dictionary with input names as keys and corresponding value types as values. Value types should be specified as [one of the following strings](https://github.com/KonduitAI/konduit-serving/blob/71260366719840bcc2fd58698cebe471267de4bb/python/konduit/inference.py#L156-L162): `"INT"`, `"STR"`, `"FLOAT"`, `"BOOL"`, `"NDARRAY"`. * `python_outputs`: A dictionary with output names as keys and corresponding value types as values. Values types should be specified as per `python_inputs`. * `extra_inputs`: potential extra input variables. Specify value types as per `python_inputs`. * `return_all_inputs`: Boolean. Whether or not to return all inputs in addition to outputs. * `setup_and_run`: Boolean. Whether or not to use the setup-and-run schematics. Defaults to False. {% endtab %} {% tab title="Java" %} In Java, we can define a `PythonConfig` object using the Lombok builder API. A `PythonConfig` takes on the following parameters: * `pythonPath`: Optional. The search path for Python modules, as defined by [`sys.path`](https://docs.python.org/3/library/sys.html#sys.path). See the [Python modules](/0.1.0-snapshot/steps/python#pythonpath) section for details. * `pythonCodePath`: Optional. Specify the location of your Python script. * `pythonCode`: Optional. Takes a String that contains Python commands. * `pythonInput`: Specify the name of the input and the type of value, as defined by the `name()` method of a `PythonVariables.Type` enumeration, namely: `"INT"`, `"STR"`, `"FLOAT"`, `"BOOL"`, `"NDARRAY"`, `"LIST"`. * `pythonOutput`: Specify the name of the output and the type of value as per `pythonInput`. {% endtab %} {% endtabs %} {% hint style="warning" %} NumPy array subclasses and some NumPy array data types are not supported. Unsupported NumPy array data types are as follows:`np.uint8`, `np.uint16`, `np.uint32`, `np.uint64`, `np.uintp`, `np.complex64`, `np.complex128`, `np.int8`, `np.int16`, `np.bool`, `np.byte`, `np.ubyte`, `np.ushort`, `np.uintc`, `np.uint`, `np.ulonglong`, `np.half`, `np.csingle`, `np.cdouble`, `np.clongdouble`. For output type NDARRAY, convert your output to a regular NumPy array and supported data type using `np.array()/np.ndarray()` and/or the `astype()` method. Also, ensure that the output is a NumPy **array** and not a NumPy scalar: see the documentation for [`np.isscalar`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.isscalar.html) for details. {% endhint %} #### **Example 1: Using Python script** {% tabs %} {% tab title="Python" %} ```python python_config = PythonConfig( python_code_path="scripts/loadimage.py", python_inputs={"x": "STR"}, python_outputs={"y": "STR"} ) ``` {% endtab %} {% tab title="Java" %} ```java String pythonCodePath = new ClassPathResource("scripts/loadimage.py") .getFile() .getAbsolutePath(); PythonConfig pythonConfig = PythonConfig.builder() .pythonCodePath(pythonCodePath) .pythonInput("x", PythonVariables.Type.STR.name()) .pythonOutput("y", PythonVariables.Type.STR.name()) .build(); ``` {% endtab %} {% endtabs %} #### Example 2: Specifying a custom Python path {% tabs %} {% tab title="Python" %} ```java python_config = PythonConfig( python_path=pythonPath, python_code="y = x + 2", python_inputs={"x": "NDARRAY"}, python_outputs={"y": "NDARRAY"} ) ``` {% endtab %} {% tab title="Java" %} ```java PythonConfig pythonConfig = PythonConfig.builder() .pythonPath(pythonPath) .pythonCode("y = x + 2") .pythonInput("x", PythonVariables.Type.NDARRAY.name()) .pythonOutput("y", PythonVariables.Type.NDARRAY.name()) .build(); ``` {% endtab %} {% endtabs %} ## `PythonStep` For most use cases,`PythonStep` can be set up as follows: {% tabs %} {% tab title="Python" %} ```java python_step = PythonStep().step(python_config) ``` Note that by default, the default name for each step is `default`. You will need this when specifying your data inputs via the `.predict()` method of the `Client` class. {% endtab %} {% tab title="Java" %} The `step` method of the`PythonStep` class is used to define the Python configuration. ```java PythonStep pythonStep = new PythonStep() .step(pythonConfig); ``` {% endtab %} {% endtabs %} Finally, the `PythonStep` object can be passed to an `InferenceConfiguration` object, which is used to configure the Konduit Serving instance: {% tabs %} {% tab title="Python" %} ```java inference_config = InferenceConfiguration( serving_config=serving_config, pipeline_steps=[python_step] ) ``` {% endtab %} {% tab title="Java" %} ```java InferenceConfiguration config = InferenceConfiguration.builder() .pipelineStep(pythonStep) .build(); ``` To test a `PythonStep` without starting a Konduit Serving instance, the output of the Python pipeline step can be retrieved as a `Writable[][]` object. Apply the`getRunner()` method to the `PythonStep` object, which gets the runner for the configuration, followed by the `transform()` method, which applies the transformations defined by the `PythonConfig` object. For example, ```java Writable[][] output = pythonStep.getRunner().transform(imagePath); ``` {% endtab %} {% endtabs %} Some models may require the server to transform more than one set of inputs. For instance, to serve object detection models, annotations and images may have to be transformed in a single `PythonStep`. This requires a unique name to be specified for each `PythonConfig`: {% tabs %} {% tab title="Python" %} ```java python_config_1 = PythonConfig( python_path=pythonPath, python_code="y = x + 2", python_inputs={"x": "INT"}, python_outputs={"y": "INT"} ) python_config_2 = PythonConfig( python_path=pythonPath, python_code="b = a + 3", python_inputs={"a": "INT"}, python_outputs={"b": "INT"} ) python_step = (PythonStep() .step("stepOne", python_config_1) .step("stepTwo", python_config_2)) ``` {% endtab %} {% tab title="Java" %} ```java PythonConfig pythonConfig1 = PythonConfig.builder() .pythonPath(pythonPath) .pythonCode("y = x + 2") .pythonInput("x", PythonVariables.Type.INT.name()) .pythonOutput("y", PythonVariables.Type.INT.name()) .build(); PythonConfig pythonConfig2 = PythonConfig.builder() .pythonPath(pythonPath) .pythonCode("b = a + 3") .pythonInput("a", PythonVariables.Type.INT.name()) .pythonOutput("b", PythonVariables.Type.INT.name()) .build(); PythonPipelineStep pythonPipelineStep = new PythonPipelineStep() .step("stepOne", pythonConfig1) .step("stepTwo", pythonConfig2); Writable[][] output = pythonPipelineStep .getRunner() .transform( new Object[] {3}, new Object[] {3} ); System.out.println(Arrays.deepToString(output)); ``` {% endtab %} {% endtabs %} ## YAML configuration Python steps can take any argument that can be passed to `PythonConfig`.The following is a basic example of specifying a Python step in a YAML configuration: ```yaml steps: python_step: type: PYTHON python_code_path: simple.py ``` * `type`: specify this as `PYTHON`. * `python_code`: if you want to specify your Python code directly in your YAML file. The following [documentation](http://blogs.perl.org/users/tinita/2018/03/strings-in-yaml---to-quote-or-not-to-quote.html) may be helpful for specifying multi-line Python code, specifically the section on literal block scalars. * `python_code_path`: specify the path of a Python `.py` script. * `python_inputs`: name-value pairs specifying the data types for each of the inputs referenced in the script. Data types should be one of the following: `INT`, `STR`, `FLOAT`, `BOOL`, `NDARRAY`. * `python_outputs`: name-value pairs specifying the data types for each of the outputs referenced in the script. Data types should be one of the following: `INT`, `STR`, `FLOAT`, `BOOL`, `NDARRAY`. * `extra_inputs`: potential extra input variables. Specify value types as per `python_inputs`. * `return_all_inputs`: Boolean. Whether or not to return all inputs in addition to output. * `setup_and_run`: Boolean. Whether or not to use the setup-and-run schematics. Defaults to `False`. * `python_path`: location of the Python modules. Generally, if your script only requires NumPy, setting a custom `python_path` is not necessary. Refer to the [Python modules](https://serving.oss.konduit.ai/python#python-modules-and-the-pythonpath-argument) documentation on setting a custom Python path with additional modules. The names referenced in `python_inputs` and `python_outputs` correspond with `inputColumnNames` and `outputColumnNames`. Modifying `python_inputs` and `python_outputs` does not modify the input and output name of the step. `input_names` and `output_names` are arguments to `PythonStep` which cannot be accessed through the YAML configuration, and default to the name `default`. ## Python modules and the `pythonPath` argument If the `pythonPath` is not specified, you will still be able to import modules cached for NumPy by [JavaCPP Presets](https://github.com/bytedeco/javacpp-presets/) in your Python script(s). In Java, you can find the location of the default modules by printing ```java Arrays.toString(cachePackages()) ``` where `cachePackages` is imported as a static variable: ```java import static org.bytedeco.numpy.presets.numpy.cachePackages; ``` If you require additional modules, you can set a custom`pythonPath` by running the following command in your Python environment and setting the output as your `pythonPath`: ```java import os from konduit.utils import default_python_path work_dir = os.path.abspath('.') default_python_path(work_dir) ``` Custom `pythonPath` follows the format defined by `sys.path.`The first element is the location of the script used to invoke the Python interpreter, and the remaining elements specify where Python should search for modules. {% hint style="info" %} To list the modules that you can access, run `help("modules")` in your Python interpreter. {% endhint %} # Java pipeline steps Implement your own PipelineStep via the CustomPipelineStep and associated PipelineStepRunner in Java. {% hint style="info" %} This page is currently under construction. {% endhint %} # Model pipeline steps {% hint style="info" %} This page is currently under construction. In the meantime, please refer to the examples. {% endhint %} # Server A Konduit Serving instance is configured by a Server object with a fully configured list of pipeline steps. After the Server object is configured, you can use the `.start()` and `.stop()` methods to initialize and stop the Serving instance. ```python from konduit.server import Server server = Server( inference_config=None, serving_config=None, steps=None, extra_start_args="-Xmx8g", config_path="config.json", jar_path=None, pid_file_path="konduit-serving.pid", start_timeout=120, ); ``` ## Configuring a Server object There are two options for configuring a Server object: ### Directly define a ServingConfig and a list of steps to Server ```python server = Server( serving_config=ServingConfig(http_port=port), steps=[preprocessing_step, onnx_step] ) ``` ### Create an InferenceConfig object ```python inference_config = InferenceConfig( serving_config=ServingConfig(http_port=port), steps=[preprocessing_step, onnx_step] ) server = Server( inference_config=inference_config ) ``` Configurations are stored as dictionaries. You can access a server's configuration as a Dictionary object using the `server.config.as_dict()` method. ## Additional arguments * `extra_start_args`: Java Virtual Machine (JVM) arguments. In this case, `-Xmx8g` specifies that the maximum memory allocation for the JVM is 8GB. * `config_path`: path to write the config object to (as json) * `jar_path`: path to the konduit uberjar. If `None`, defaults to the `KONDUIT_JAR_PATH` environment variable, or `~/.konduit/konduit-serving` if `KONDUIT_JAR_PATH` is not available. * `pid_file_path`: path to write the process ID to, as a text file. * `start_timeout`: time to wait for the server to timeout when starting the server instance. ## ServingConfig For most configurations, the following arguments are sufficient: * `http_port`: HTTP port of the Konduit Serving instance. * `listen_host`: Host of the Konduit Serving instance. Defaults to `'localhost'`. * `input_data_format`: Input data format: one of `'NUMPY'`, `'JSON'`, `'ND4J'`, `'IMAGE'`or `'ARROW'`. Defaults to `NUMPY`. * `output_data_format`: Output data format: one of `'NUMPY'`, `'JSON'`, `'ND4J'`, or `'ARROW'`. Defaults to `NUMPY`. The following arguments are optional: * `prediction_type`: Prediction type. This argument determines which "output adapter" is used to transform the output. Choose one of `'CLASSIFICATION'`, `'YOLO'`, `'SSD'`, `'RCNN'`, `'RAW'`, `'REGRESSION'`. The default prediction type is `'RAW'`: that is, no adapter is applied to the output. * `uploads_directory`: Directory to store file uploads. Defaults to `'file-uploads/'`. * `log_timings`: Whether to log timings for this config. Defaults to False * `metric_types`: The types of metrics logged for your `ServingConfig` can currently only be configured and extended from Java. Don't modify this property. ## `server.start()` The `start` method initializes a Konduit Serving instance, and ends any previously started server that is still running. Set the `kill_existing_server` argument to `False` to change this behaviour. ## `server.stop()` The `stop` method ends the Konduit Serving process defined by the `Server` object. ## YAML configuration Refer to the [YAML configuration page](/0.1.0-snapshot/yaml-configurations#serving). # Python client configuration The Client class allows a client to obtain outputs from a Konduit Serving instance given a named set of inputs via the predict() method. ```python from konduit.client import Client Client( timeout=60, input_data_format='NUMPY', output_data_format='NUMPY', input_names=['default'], output_names=['default'], host='http://localhost', port=None, prediction_type=None ) ``` ### Data formats Data formats define how data is transported between the Client and the Konduit Serving instance. In addition to `JSON`, Konduit Serving also supports `NUMPY`, `ARROW`, `RAW` and `IMAGE`data formats. Specify data formats as strings. * `input_data_format` defines the data format of inputs sent to the server via the `predict()` method. * `output_data_format`defines the data format returned by the API endpoint. If you want to convert the result of your endpoint to another data format on return, change the `convert_to_format` attribute of the `Client` object: ```python client = Client(port=3226) client.convert_to_format = "ARROW" ``` ### Input and output names Both the `input_names` and `output_names` arguments accept a list of strings. Input and output names are defined by the inputs and outputs to the first and last pipeline steps in the Konduit Serving pipeline configuration respectively. For `ModelStep`, input and output names should be configured when defining the model for training, or may need to be obtained by inspecting the model file. See the examples for details. For `PythonStep`, input names are defined in the `step()` method of a `PythonStep` object. ### Other arguments * `timeout`: Integer. Defaults to 60 (seconds). * `host`: String. If the model is hosted locally, the host should be specified as `http://localhost` (the default argument) * `port`: Integer. The arguments `output_data_format`, `input_data_format` and `prediction_type` are obtained from the server when the Client object is initialized. Refer to the [Server](/0.1.0-snapshot/server/inference) documentation for details. ### `.predict()`method The `.predict()`method takes a dictionary with`input_names` as keys and the data inputs as values. ## Example Assume a Server object has been fully configured as `server`. Start the server: ```python server.start() ``` Define a Client: ```python client = Client( input_data_format='NUMPY', output_data_format="NUMPY", return_data_output_format='NUMPY', input_names=["input1"], host='http://localhost', port=port ) ``` Request for a prediction: ```python client.predict({"input1": np.ones(5)}) server.stop() ``` ## YAML configuration Refer to the [YAML configuration page](/0.1.0-snapshot/yaml-configurations#client). # Introduction Konduit-Serving is a framework focused on deploying machine learning pipelines to production. ## Overview Konduit-Serving provides building blocks for developers to write their own production machine learning pipelines from pre-processing to model serving, exposable as a simple REST API. It also allows embedding Python/Java code (pre/post processing, custom models). It primarily focuses on server and edge deployments using REST and gRPC endpoints. Pipelines are deployed and defined using JSON/YAML or a command line interface. The core abstraction is an idea called a **pipeline step**. A pipeline step performs a task such as: 1. Pre-processing steps 2. Running one or more machine learning models 3. Post-processing steps: transforming the output in a way that can be understood by humans, such as labels in a classification example, as part of using a machine learning model in a deployment scenario. For instance, `TensorflowStep, KerasStep and Dl4jStep` performs inference on TensorFlow, Keras, Deeplearning4j (DL4J), respectively. Similarly, there are multiple steps defined in Konduit-Serving for setting up pre/post-processing in the machine learning pipeline. A custom pipeline step can be built using a `PythonStep`. This allows you to embed pre/post-processing steps into your machine learning pipeline, or to serve models built in frameworks that do not have built-in model steps such as **scikit-learn** and **PyTorch**. ### **High-Level Architecture** Konduit-Serving utilizes many popular and performant libraries. Out of them the major ones are [Vert.x](https://vertx.io/) and [Deeplearning4J](https://deeplearning4j.org/). #### **Vert.x Library** Vert. x is an open source, reactive and polyglot software development toolkit and has great support for Java language. For Konduit-Serving, Vert.x is used for building and managing CLI, Web-servers and implementing data transmission backends through multiple protocols like [gRPC](https://en.wikipedia.org/wiki/GRPC), [MQTT](https://en.wikipedia.org/wiki/MQTT) and [HTTP](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol). It also implements a [Kafka](https://kafka.apache.org/) client setup and can link with a Kafka backend server. Vert.x also allows us to scale our Konduit-Serving deployments very efficiently on multiple nodes. Although this feature hasn’t been tested thoroughly, Konduit-Serving has the ability to do it. The following stack shows how the different components in the stack work together. ![](https://docs.google.com/drawings/u/2/d/sWyHSpGIxUFD7erZvfIL7nw/image?w=477\&h=327\&rev=199\&ac=1\&parent=1pghi_Njn8fb-rcy9nOwCivozE5-CUEkbvq0vyDYCibc) ### **DL4J Support** Konduit-Serving runs dl4j through an interface agnostic architecture. This means Konduit-Serving is not aware of which backend it’s running. When compiling Konduit-Serving, you can use the command line to build a specific spin of Konduit-Serving optimized for a particular use case. ### Aurora Vector Engine Support In prior phases, we have focused on benchmarks, running tensorflow models on aurora [vector engines](https://www.nec.com/en/global/solutions/hpc/sx/vector_engine.html). We have also added support for running DL4J on Aurora from Java. This means running Aurora math workloads from java via compiled libnd4j c++ that uses NCC. We’ve managed to get DL4J compiled on Aurora running via JavaCpp. An up to date link can be found [here](https://github.com/KonduitAI/deeplearning4j/wiki/Current-Build-Status-On-Aurora). A build of Konduit-Serving that runs applications on aurora is pending, but possible based on the current work that’s been done. In our case, when running Konduit-Serving in production a particular spin can be used for certain operating systems and hardware allowing for testing on 1 platform, but deployment on another. The principal way a user would use aurora on Konduit-Serving and dl4j is the samediff framework. Aurora is accessible in a transparent way just by setting up a jar file containing the backend for aurora. GPUs are accessible in a similar way. The SameDiff pipeline step is the component to use in order to make aurora accessible to Konduit-Serving. SameDiffStep configuration looks like this from java: ```java Pipeline p = SequencePipeline.builder() .add(SameDiffStep.builder() .modelUri(f.toURI().toString()) .outputNames(Collections.singletonList("out")) .build()) .build(); ``` The configuration above represents a standalone model that runs a model specified as a file. A SameDiff model is saved as a [flatbuffers](https://google.github.io/flatbuffers/) file. This file is a descriptor containing the graph layout and associated weights for each ndarray embedded in the samediff graph. In order to input variables in to a graph, graphs in different frameworks (including tensorflow and pytorch) rely on a concept called PlaceHolders. Placeholders are what you pass in to a graph where an input variable is passed in and replaces a stub element in the graph. A SameDiffStep allows you to specify the input and output placeholders to get different outputs out of the graph accessible by name. All pipeline steps that use a graph based framework require these input and output steps. Sometimes, these inputs and outputs can be automatically inferred, but it’s generally a good idea. Specifying less outputs allows a user to specify only what outputs they want rather than everything. ### Supported Frameworks The following main model pipeline step execution framework are supported by Konduit-Serving: #### **Models Supported Via Onnx** 1. Pytorch 2. MXNet 3. Others which can be converted into ONNX) 4. PMML (Experimental) 5. Predictive Model Markup Language also a platform interchange format for mostly all of the traditional ML models such as random forest. #### Models supported via PMML 1. Scikit-learn 2. SparkML 3. XGBoost 4. Others with supports PMML conversion 5. Custom models 6. Through custom Python or Java code #### **Custom models** Through custom Python or Java code ### **Source Code** Konduit-serving source code can be found [here](https://github.com/KonduitAI/konduit-serving). There’s also a few demos and use cases on the repo link [here](https://github.com/KonduitAI/nec-sra-workshop). The examples are based on a Docker image implementation for both CPU and GPU backends which is a great source to start learning more about Konduit-Serving. To get started with Konduit-Serving, check out the [Quickstart](/quickstart) page. # Components Describes the components that make up Konduit-Serving ## **Components** Konduit-Serving has many internal components that work together to achieve the desired result. Following are the main concepts and components that make up Konduit-Serving: 1. CLI 2. Jar Package 3. Pipelines 4. Pipeline Step 5. Inference Configuration This page will go through each one of them and explain what they are used for. ### **CLI Interface** Konduit-Serving comes with a CLI interface (with a `konduit` alias) that's responsible for taking care of most aspects of the application. The help command will describe most of what we are able to do with Konduit-Serving. Executing `konduit --help` command will show us the following output: ```bash $ konduit --help --------------------------------------------------------------------------- Usage: konduit [COMMAND] [OPTIONS] [arg...] Commands: build Command line interface for performing Konduit Serving builds. config A helper command for creating boiler plate json/yaml for inference configuration inspect Inspect the details of a particular konduit server. list Lists the running konduit servers. logs View the logs of a particular konduit server predict Run inference on konduit servers using given inputs profile Command to List, view, edit, create and delete konduit serving run profiles. pythonpaths A utility command to manage system installed and manually registered python binaries. serve Start a konduit server application stop Stop a running konduit server version Displays konduit-serving version. Run 'konduit COMMAND --help' for more information on a command. --------------------------------------------------------------------------- ``` Each command describes its shorthand description right in front of it. If you want to look at an individual command in detail, you can use the corresponding --help command with them. For example, the help menu for the logs command can be seen by executing, konduit logs --help: ```bash $ konduit logs --help --------------------------------------------------------------------------- Usage: konduit logs [-f] [-l ] server-id View the logs of a particular konduit server View the logs of a particular konduit server given an id. Example usages: -------------- - Outputs the log file contents of server with an id of 'inf_server': $ konduit logs inf_server - Outputs and tail the log file contents of server with an id of 'inf_server': $ konduit logs inf_server -f - Outputs and tail the log file contents of server with an id of 'inf_server' from the last 10 lines: $ konduit logs inf_server -l 10 -f -------------- Options and Arguments: -f,--follow Follow the logs output. -l,--lines Sets the number of lines to be printed. Default is '10'. Use -1 for outputting everything. Konduit server id --------------------------------------------------------------------------- ``` As can be seen, the --help command for an individual help command describes its functionality in detail along with some explicit examples and use cases. It also describes each individual optional/non-optional argument that can be used with it. This can come in very handy while learning about konduit-serving for the first time and is a useful starting place to play around with a specific command. You can do the same for the rest of the commands. Which are: * build * config * inspect * list * logs * predict * metrics * profile * pythonpaths * serve * stop * version ### **Jar File Package** Each Konduit-Serving distribution whether it is for Windows, Linux or MacOS comes contained in a JAR file. So, you'll need a Java Virtual Machine present in the system where you're using Konduit-Serving as a Model Pipeline Server. The CLI itself is linked with the jar file and utilizes a java runtime internally to interact with the Konduit-Serving package. If you look at the konduit serving distribution, you'll see the following folder architecture in the root folder:
![](https://docs.google.com/drawings/u/2/d/sU9Kk3p6mTwO5DcFFatUAnA/image?w=397\&h=320\&rev=189\&ac=1\&parent=1pghi_Njn8fb-rcy9nOwCivozE5-CUEkbvq0vyDYCibc) The main CLI logic is places under bin/konduit file, which contains the following content: ```bash #!/usr/bin/env bash SCRIPT_DIR="$(dirname "$0")" . ${SCRIPT_DIR}/../conf/konduit-serving-env.sh java -jar -Dvertx.cli.usage.prefix=konduit ${SCRIPT_DIR}/../konduit.jar "$@" ``` As you can see, it uses the java command which is available through a Java runtime environment. The java command itself uses the konduit.jar file which is the main application package inside a Konduit-Serving distribution. This JAR file will be used as a Java application dependency while creating custom endpoints logic for a Konduit-Serving pipeline. We'll get to how we can do that later in this notebook. ### **Introduction to Pipelines** Throughout this document the term "Model Serving Pipeline" has been used. This refers to how Machine Learning or Deep Learning models get served on an application server. Machine/Deep Learning models work on n-dimensional arrays (also known as ND-Arrays). They don't know how to convert a JPEG or PNG image into numbers directly. Instead, they expect pre-processed data in the form of a multidimensional array. Also, any other form of data, be it text, audio or video, gets converted into numbers ND-Arrays before getting fed into a machine learning model. The process during which the data is converted from one form to another is called pre-processing and is done just before it's fed as a model input. So, in a sense you can see this as being Lego blocks fitting into each other. One part takes input in a specific form and outputs it into another form, which in turn gets fed into the next part. This chaining of processes creates a series of steps which have specific jobs to perform before the next step and the end result is a machine learning Pipeline. The typical flow of the pipeline looks like the following: ![](https://docs.google.com/drawings/u/2/d/sSSQ69On15sDrkay14FWOCQ/image?w=624\&h=112\&rev=70\&ac=1\&parent=1pghi_Njn8fb-rcy9nOwCivozE5-CUEkbvq0vyDYCibc) A pipeline can also be in the form of a directed acyclic graph or DAG where data can flow into the graph and can give multiple outputs. In Konduit-Serving a Pipeline graph can also contain optional graph branches and can also concatenate outputs from multiple graph nodes. For the sake of the current goal (BMI Model Serving) we'll stick to a Sequential Pipeline which only has one input and one output. ### **Pipeline Steps** Inside Konduit-Serving, a pipeline can be broken down into steps, where each step is responsible for performing a specific function. A pipeline step is the smallest component of a whole pipeline and can be used for a whole list of operations. To see the list of available pipeline steps you can use the config command in Konduit-Serving CLI. ```bash $ konduit config --help --------------------------------------------------------------------------- Usage: konduit config [-m] [-o ] -p [-pr ] [-y] A helper command for creating boilerplate json/yaml for inference configuration This command is a utility to create boilerplate json/yaml configurations that can be conveniently modified to start konduit servers. Example usages: -------------- -- FOR SEQUENCE PIPELINES-- - Prints 'logging -> tensorflow -> logging' config in pretty format: $ konduit config -p logging,tensorflow,logging - Prints 'logging -> tensorflow -> logging' config with gRPC protocol in pretty format: $ konduit config -p logging,tensorflow,logging -pr grpc - Prints 'dl4j -> logging' config in minified format: $ konduit config -p dl4j,logging -m - Saves 'dl4j -> logging' config in a 'config.json' file: $ konduit config -p dl4j,logging -o config.json - Saves 'dl4j -> logging' config in a 'config.yaml' file: $ konduit config -p dl4j,logging -y -o config.json -- FOR GRAPH PIPELINES -- - Generates a config that logs the input(1) then flow them through two tensorflow models(2,3) and merges the output(4): $ konduit config -p 1=logging(input),2=tensorflow(1),3=tensorflow(1),4=merge(2,3) - Generates a config that logs the input(1) then channels(2) them through one of the two tensorflow models(3,4) and then selects the output(5) based on the value of the selection integer field 'select' $ konduit config -p 1=logging(input),[2_1,2_2]=switch(int,select,1),3=tensorflow(2_1),4=tensorflow(2_2),5=any(3,4) - Generates a config that logs the input(1) then channels(2) them through one of the two tensorflow models(3,4) and then selects the output(5) based on the value of the selection string field 'select' in the selection map (x:0,y:1). $ konduit config -p 1=logging(input),[2_1,2_2]=switch(string,select,x:0,y:1,1),3=tensorflow(2_1),4=tensorflow(2_2),5=any(3,4) -------------- Options and Arguments: -m,--minified If set, the output json will be printed in a single line, without indentations. (Ignored for yaml) -o,--output Optional: If set, the generated json/yaml will be saved here. Otherwise, it's printed on the console. -p,--pipeline A comma-separated list of sequence/graph pipeline steps to create boilerplate configuration from. For sequences, allowed values are: [crop_grid, crop_fixed_grid, dl4j, keras, Draw_bounding_box, draw_fixed_grid, draw_grid, draw_segmentation,extract_bounding_box, Camera_frame_capture, video_frame_capture, Image_to_ndarray, logging, ssd_to_bounding_box, samediff, show_image, tensorflow, nd4jtensorflow, python, onnx]. For graphs, the list item should be in the format '=()' or '[outputs]=switch()' for switches. The pre-defined root input is named, 'input'. Examples are ==> Pipeline step: 'a=tensorflow(input),b=dl4j(input)' Merge Step: 'c=merge(a,b)' Switch Step (int): '[d1,d2,d3]=switch(int,select,input)' Switch Step (string): '[d1,d2,d3]=switch(string,select,x:1,y:2,z:3,input)' 'Any Step: 'e=any(d1,d2,d3)' See the examples above for more usage information. -pr,--protocol Protocol to use with the server. Allowed values are [http, grpc, mqtt] -y,--yaml Set if you want the output to be a yaml configuration. --------------------------------------------------------------------------- ``` config command can take a pipeline pattern string and creates a pipeline configuration as well as server configuration. The server configuration combined with pipeline step configuration is collectively called Inference Configuration in Konduit-Serving. This is the final output from the config command. The following pipeline step types can be used to configure a pipeline configuration. | **#** | **Pipeline Step Type Name** | **Description** | | ------ | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **01** | **CROP\_GRID** | **A pipeline step that crops sub images out of a larger image, based on a grid.** | | **02** | **CROP\_FIXED\_GRID** | **This step is similar to CROP\_GRID with the difference that the x/y location values are hardcoded into the configuration, instead of coming dynamically from the input Data instance.** | | **03** | **DL4J** | **A model pipelinestep that serves a DL4J model.** | | **04** | **KERAS** | **A model pipeline step that serves a Keras model.** | | **05** | **DRAW\_BOUNDING\_BOX** | **A pipeline step that configures how to draw a bounding box onto an image. The bounding box data, that's to be drawn, is taken from the previous step's data instance.** | | **06** | **DRAW\_GRID** | **Draw a grid on the specified image, based on the x/y coordinates of the corners, and the number of segments within the grid in both directions.** | | **07** | **DRAW\_FIXED\_GRID** | **A pipeline step that draws a grid on an image. This is similar to DRAW\_GRID but the corner x/y location values are hardcoded into the configuration (via points), instead of coming dynamically from the input Data instance.** | | **08** | **DRAW\_SEGMENTATION** | **A pipeline step that configures how to draw a segmentation mask, optionally on an image.** | | **09** | **EXTRACT\_BOUNDING\_BOX** | **A pipeline step that extracts sub-images from an input image, based on the locations of input bounding boxes.** | | **10** | **CAMERA\_FRAME\_CAPTURE** | **A pipeline step that specifies an input that's taken from a camera feed.** | | **11** | **VIDEO\_FRAME\_CAPTURE** | **A pipeline step that configures how to extract a single frame from a video each time inference is called. The video path is hardcoded, mainly used for testing/demo purposes.** | | **12** | **IMAGE\_TO\_NDARRAY** | **A PipelineStep for converting images to n-dimensional arrays.** | | **13** | **LOGGING** | **A step that logs the key and value pairs coming from the previous steps.** | | **14** | **SSD\_TO\_BOUNDING\_BOX** | **A pipeline step that configures extraction of bounding boxes from an SSD model output.** | | **15** | **SAMEDIFF** | **A model pipeline step that serves a SameDiff model.** | | **16** | **SHOW\_IMAGE** | **A pipeline step that configures how to show/render an image from a previous step in an application frame. Usually only used for testing and debugging locally, not when serving from HTTP/GRPC etc endpoints.** | | **17** | **TENSORFLOW** | **A model pipeline step that serves a TensorFlow model using Tensorflow Java API. This is packaged into Konduit-Serving through JavaCPP.** | | **18** | **ND4JTENSORFLOW** | **A pipeline step that configures a TensorFlow model that is to be executed based on an ND4J graph runner. This has performance benefits over native TensorFlow Java distribution.** | | **19** | **PYTHON** | **This pipeline step can take an arbitrary python script and serve that through Konduit-Serving.** | | **20** | **ONNX** | **A model pipeline step that serves a ONNX model.** | ### **Inference Configuration** Inference configuration contains all the details about how the pipeline server should be setup, which protocol it’s supposed to use for sending in/out data, whether we have a sequential or graph pipeline and finally how the pipeline is configured. The config command outputs inference configuration for us that can be directly used with a serve command after necessary changes are made. An Example of inference configuration through config command is as follows: ```bash $ konduit config --pipeline logging --yaml --------------------------------------------------------------------------- --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "LOGGING" logLevel: "INFO" log: "KEYS_AND_VALUES" --------------------------------------------------------------------------- ``` The above command creates a yaml configuration for a logging pipeline step. Logging pipeline step just takes in an input from the previous pipeline step and outputs (and logs on console) the same information. This is useful for debugging issues in the running servers. This configuration can be saved inside a file by executing the following: ```bash $ konduit config --pipeline logging --yaml --output server-conf.yaml --------------------------------------------------------------------------- Config file created successfully at /root/server-conf.yaml --------------------------------------------------------------------------- ``` Afterwards, this configuration can be used with the serve command to start a Konduit-Serving server. Example is following: ```bash $ konduit serve --config server-conf.yaml -id server --background --runWithoutManifest --------------------------------------------------------------------------- Starting konduit server... Using classpath: /root/konduit/bin/../konduit.jar INFO: Running command /root/miniconda/jre/bin/java -Dkonduit.logs.file.path=/root/.konduit-serving/command_logs/server.log -Dlogback.configurationFile=/tmp/logback-run_command_4b1acb956d0d436c.xml -jar /root/konduit/bin/../konduit.jar run --instances 1 -s inference -c server-conf.yaml -Dserving.id=server For server status, execute: 'konduit list' For logs, execute: 'konduit logs server' --------------------------------------------------------------------------- ``` # Quickstart Quickstart guide to start using Konduit-Serving Konduit-Serving is a framework-agnostic model serving solution focused on deploying machine learning pipelines to production. The Python SDK allows data scientists to quickly test machine learning deployment scenarios, bridging the gap between data science teams and DevOps. Before running these commands, set up Konduit Serving according to the installation instructions on the [Building From Source](/building-from-source) and [Installation](/installation) page: ## Configuration Files Konduit Serving configuration files consist of `serving`, `steps` and `client` components. Save the configuration below as a text file named `hello-world.yaml` in your current directory: ```yaml serving: http_port: 1337 input_data_format: NUMPY output_data_format: NUMPY steps: python_step: type: PYTHON python_code: | first += 2 second = first python_inputs: first: NDARRAY python_outputs: second: NDARRAY client: port: 1337 ``` The pages in this section show you how to start and interact with a Konduit Serving instance. For these examples, the Konduit Serving instance and client are on the same machine. For quick experimentation, check out the quickstart for the command line interface (CLI): {% content-ref url="/pages/-MUvZF32kjXyst1DHu7q" %} [Using CLI](/quickstart/using-cli) {% endcontent-ref %} To access additional options, you will want to configure Konduit Serving instances with the Python SDK. Start with the Python quickstart: {% content-ref url="/pages/-MUvZD79ScH92vh7C6z6" %} [Using Python SDK](/quickstart/using-python-sdk) {% endcontent-ref %} # Using Docker Guide to start using Konduit-Serving with Docker In this section, we provide guidance on how to demonstrate Konduit-Serving with Docker. Konduit-Serving is a platform to serve the ML/DL models directly through a single line of code. We'll start building and installing the Konduit-Serving from docker image and follow by deploying the trained model in Konduit-Serving. ### Prerequisites You will need following prerequisites to follow along * [Docker 19.03.14](https://docs.docker.com/) * [Docker-Compose 1.27.4](https://docs.docker.com/compose/install/) To ensure Konduit-Serving works properly, install these prerequisites. We’ll be using Konduit-Serving to deploy machine/deep learning pipelines easier using a few lines of code. We've prepared a Github repository to simplify showcasing Konduit-Serving. ### Introduction to Repository The repository contains simple examples of deploying a pipeline using different model types. To clone the repository, run the following command: ```bash git clone https://github.com/ShamsUlAzeem/konduit-serving-demo ``` Build the CPU version of docker image by running the following command in the root directory: ```bash bash build.sh CPU ``` After a successful build, run docker image with docker-compose at current working directory: ``` docker-compose up ``` Now, open [JupyterLab](http://localhost:8889/) in the browser. ### Explore Konduit-Serving in Repository Let's take a look inside the [demos](https://github.com/ShamsUlAzeem/konduit-serving-demo/tree/master/demos) directory from [konduit-serving-demo](https://github.com/ShamsUlAzeem/konduit-serving-demo). Each folder inside the demos folder demonstrate serving a different kind of model, using a configuration file in either JSON or YAML, through the Konduit-Serving CLI. The examples use different frameworks, including Keras, Tensorflow, Pytorch, and DL4J. These examples can be run in IPython Notebook (.ipynb) with Java-based kernel. Konduit-Serving provides a platform for users to take advantage of models using `konduit` CLI such as `serve`, `list`, `logs`, `predict` and `stop`. You can also build your model and start using Konduit-Serving. Let's look at training and serving a model with Konduit-Serving. ### Build your model These are steps to train a model for Keras and DL4J from scratch: {% tabs %} {% tab title="Keras" %} * First, import library that will be use: ```python import keras from keras.models import Sequential from keras.layers import Dense from keras.optimizers import Adam from sklearn.preprocessing import LabelEncoder from sklearn.model_selection import train_test_split from sklearn.metrics import confusion_matrix import seaborn as sns import pandas as pd import numpy as np ``` * Load Iris data set: ```python dataset = sns.load_dataset("iris") print(dataset) ``` * Display distribution of data set: ```python sns.set(style="ticks") sns.set_palette("husl") sns.pairplot(dataset.iloc[:,0:6], hue="species") ``` * Apply data pre-processing and split data into training and testing: ```python X = dataset.iloc[:,0:4].values y = dataset.iloc[:,4].values encoder = LabelEncoder() y1 = encoder.fit_transform(y) Y = pd.get_dummies(y1).values X_train,X_test, y_train,y_test = train_test_split(X,Y,test_size=0.3,random_state=0) ``` * Configure the model and print summary: ```python model = Sequential() model.add(Dense(4,input_shape=(4,),activation='relu', name="input")) model.add(Dense(3,activation='softmax', name='output')) model.compile(Adam(lr=0.01),'categorical_crossentropy',metrics=['accuracy']) model.summary() ``` * Train the model by training data with 800 epochs: ```python model.fit(X_train,y_train,epochs=800) ``` * Test the model by predicting testing data: ```python y_pred = model.predict(X_test) y_test_class = np.argmax(y_test,axis=1) y_pred_class = np.argmax(y_pred,axis=1) ``` * Display confusion matrix to see more details of model prediction between actual and predicted result (if satisfied can save the model): ```python cm = confusion_matrix(y_test_class, y_pred_class) print(cm) ``` * Also, you can try to predict by using your value in the trained model: ```python X_test2 = np.array([[5, 3.9, 2, 0.5],[5,2.5,3,1],[8,3.5,6,2]])#setosa, versicolor, virginica y_pred2 = model.predict(X_test2) print(y_pred2) ``` * Print the result of classification: ```python print(np.argmax(y_pred2,axis=1)) #0 = setosa 1 = versicolor 2 = virginica ``` * Then, save the trained model in HDF5 (.h5) format which will be used in Konduit-Serving later: ```python model.save_weights("model.h5") print("Saved model to disk") ``` **The model is now ready to be deployed.** {% endtab %} {% tab title="DL4J" %} * First, import library that will be use (auto generated if you are using IntelliJ - proceed to next step) : ```java import org.datavec.api.records.reader.RecordReader; import org.datavec.api.records.reader.impl.csv.CSVRecordReader; import org.datavec.api.split.FileSplit; import org.deeplearning4j.api.storage.StatsStorage; import org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator; import org.deeplearning4j.nn.conf.BackpropType; import org.deeplearning4j.nn.conf.MultiLayerConfiguration; import org.deeplearning4j.nn.conf.NeuralNetConfiguration; import org.deeplearning4j.nn.conf.layers.DenseLayer; import org.deeplearning4j.nn.conf.layers.OutputLayer; import org.deeplearning4j.nn.multilayer.MultiLayerNetwork; import org.deeplearning4j.nn.weights.WeightInit; import org.deeplearning4j.ui.api.UIServer; import org.deeplearning4j.ui.stats.StatsListener; import org.deeplearning4j.ui.storage.InMemoryStatsStorage; import org.deeplearning4j.util.ModelSerializer; import org.nd4j.evaluation.classification.Evaluation; import org.nd4j.linalg.activations.Activation; import org.nd4j.linalg.dataset.DataSet; import org.nd4j.linalg.dataset.SplitTestAndTrain; import org.nd4j.linalg.dataset.api.iterator.DataSetIterator; import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization; import org.nd4j.linalg.dataset.api.preprocessor.NormalizerStandardize; import org.nd4j.linalg.io.ClassPathResource; import org.nd4j.linalg.learning.config.Nesterovs; import org.nd4j.linalg.lossfunctions.LossFunctions; import java.io.File; import java.util.Arrays; ``` * Declare the variables that will be used in the training process (this code starts under main body): ```java int numLinesToSkip = 0; char delimiter = ','; int batchSize = 150; // Iris data set: 150 examples total. We are loading all of them into one DataSet (not recommended for large data sets) int labelIndex = 4; // index of label/class column int numClasses = 3; int seed = 1234; int epochs = 800; double learningRate = 0.1; int nHidden = 4; ``` * Load Iris data set from resources file: ```java File inputFile = new ClassPathResource("datavec/iris.txt").getFile(); FileSplit fileSplit = new FileSplit(inputFile); ``` * Get data set using record reader (to handle loading or parsing): ```java RecordReader recordReader = new CSVRecordReader(numLinesToSkip, delimiter); recordReader.initialize(fileSplit); ``` * Create iterator from record reader: ```java DataSetIterator iterator = new RecordReaderDataSetIterator(recordReader, batchSize, labelIndex, numClasses); DataSet allData = iterator.next(); ``` * Shuffling the arrangement of data and splitting into training and testing: ```java allData.shuffle(seed); SplitTestAndTrain testAndTrain = allData.splitTestAndTrain(0.7); DataSet trainingData = testAndTrain.getTrain(); DataSet testData = testAndTrain.getTest(); ``` * Apply data pre-processing by normalization: ```java DataNormalization normalizer = new NormalizerStandardize(); normalizer.fit(trainingData); normalizer.transform(trainingData); normalizer.transform(testData); ``` * Configure and initiate the model that will be used: ```java MultiLayerConfiguration config = new NeuralNetConfiguration.Builder() .seed(seed) .weightInit(WeightInit.XAVIER) .activation(Activation.TANH) .updater(new Nesterovs(learningRate, Nesterovs.DEFAULT_NESTEROV_MOMENTUM)) .l2(1e-4) .list() .layer(0, new DenseLayer.Builder() .nIn(labelIndex) .nOut(nHidden) .build()) .layer(1, new DenseLayer.Builder() .nOut(nHidden) .build()) .layer(2, new OutputLayer.Builder(LossFunctions.LossFunction.MCXENT) .activation(Activation.SOFTMAX) .nOut(numClasses) .name("output") .build()) .backpropType(BackpropType.Standard) .build(); MultiLayerNetwork model = new MultiLayerNetwork(config); model.init(); ``` * Display in UI of training process which can be seen when open in browser (optional): ```java StatsStorage storage = new InMemoryStatsStorage(); UIServer server = UIServer.getInstance(); server.attach(storage); model.setListeners(new StatsListener(storage, 10)); ``` * Train the model using training data: ```java for (int i=0; i < epochs; i++) { model.fit(trainingData); } ``` * Evaluate the model by using testing data (save the model if satisfied): ```java Evaluation eval = new Evaluation(3); eval.eval(model.output(testData.getFeatures()),testData.getLabels()); System.out.println(eval.stats()); ``` * Then, save the model at the location you want to keep with the name in Zip format. The model is ready to be used (for example, the model saved in the current working directory): ```java File locationToSave = new File("./dl4j_iris_model.zip"); boolean saveUpdater = true; ModelSerializer.writeModel(model,locationToSave,saveUpdater); System.out.println("******PROGRAM IS FINISHED******"); ``` **The model is now ready to be deployed.** {% endtab %} {% endtabs %} ### Deploy your model in Konduit-Serving Now, you are ready to deploy a model in Konduit-Serving by using `konduit` CLI. This step needs a saved model file (h5 or zip file) and JSON/YAML configuration file. Let's begin with: * Create a new folder in the demos directory (for example, 10-iris-model): ![](https://215936813-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-MVPJhje7EOlcDFCylA7%2F-MVPJnFfzJAtxCsbD_MD%2F10-iris-model.png?alt=media\&token=741c7548-9778-4de1-aa04-2db9db3b738b) * Drag the model file into the demos directory folder and create an IPython Notebook file with Java kernel (for example, iris-model.ipynb). In this notebook, we will use `konduit` CLI. * Check the version of Konduit-Serving, and either is installed or not: ``` %%bash konduit -version ``` * Create the JSON/YAML configuration file by using `konduit config` command: {% tabs %} {% tab title="Keras" %} YAML configuration: ```bash %%bash konduit config -p keras -o iris-keras.yaml -y ``` Or, you can try this command to get a JSON configuration file: ``` %%bash konduit config -p keras -o iris-keras.json ``` {% endtab %} {% tab title="DL4J" %} YAML configuration: ``` %%bash konduit config -p dl4j -o iris-dl4j.yaml -y ``` Or, you can try this command to get a JSON configuration file: ``` %%bash konduit config -p dl4j -o iris-dl4j.json ``` {% endtab %} {% endtabs %} * In the configuration file, you need to edit the YAML/JSON file in the pipeline section (for this example, we will use YAML with DL4J): {% tabs %} {% tab title="Before" %} ```yaml --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "DEEPLEARNING4J" modelUri: "" inputNames: - "1" - "2" outputNames: - "11" - "22" ``` {% endtab %} {% tab title="After" %} ```yaml --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "DEEPLEARNING4J" modelUri: "dl4j_iris_model.zip" inputNames: - "input" outputNames: - "output" ``` {% endtab %} {% endtabs %} * To determine the name of input and output, you can use [Netron](https://netron.app/) to read ML/DL model, for example: {% tabs %} {% tab title="Keras" %} In node properties, use the name's value of first weights as `"inputNames"`. ![](https://215936813-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-MVPJhje7EOlcDFCylA7%2F-MVPJwCLxvSO3DpsLHyt%2Fkeras-input-h5.png?alt=media\&token=6f2368b9-f9e9-419a-b4f8-c52a38100252) In node properties, use the name's value of last weights as `"outputname"`. ![](https://215936813-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-MVPJhje7EOlcDFCylA7%2F-MVPK-XsM15l84BXfGLZ%2Fkeras-output-h5.png?alt=media\&token=d996a261-15ec-4d29-8b3d-a18da1d62746) {% endtab %} {% tab title="DL4J" %} In model properties, use the name's value of input for `"inputNames"` and output for `"outputNames"`. ![](https://215936813-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-MVPJhje7EOlcDFCylA7%2F-MVPK2jTPTkK-L2yezwT%2Fdl4j-zip.png?alt=media\&token=e164c70e-9602-4503-82ee-9316417bf87c) {% endtab %} {% endtabs %} * Start the server by using `konduit serve` command and give the id's name based on your own: ``` %%bash konduit serve -id dl4j-iris -c iris-dl4j.yaml -rwm -b ``` * Listing the active server in Konduit-Serving, `konduit list`: ``` %%bash konduit list ``` * Show the log of the selected server’s id for 100 lines by `konduit logs`: ``` %%bash konduit logs dl4j-iris -l 100 ``` * Test the prediction of ML/DL model by `konduit predict` in the Konduit-Serving at the selected id (in this example: dl4j-iris): ``` %%bash konduit predict dl4j-iris "{\"input\":[[1,2,3,4]]}" ``` * You can test again with another input value to get another result: ``` %%bash konduit predict dl4j-iris "{\"input\":[[5.1,3.5,1.4,0.2]]}" ``` * And, the result will be like this: ![](https://215936813-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-MVPJhje7EOlcDFCylA7%2F-MVPK6QD1Wihjln9vL8R%2Fresult-1.png?alt=media\&token=b6afa1ae-c377-4906-9515-75b77acef589) * For more interactive result, you can edit the JSON/YAML file in the pipeline section as below: ```yaml --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "DEEPLEARNING4J" modelUri: "dl4j_iris_model.zip" inputNames: - "input" outputNames: - "output" - '@type': "CLASSIFIER_OUTPUT" input_names: "layer2" labels: - Sentosa - Versicolor - Virginica ``` * So, you will get the result of classification straightforward with prediction's label: ![](https://215936813-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-MVPJhje7EOlcDFCylA7%2F-MVPK9cGBYacsjQ2OJ8K%2Fresult-2.png?alt=media\&token=b875cc2e-6e0b-4c8a-bf59-5877ce969cc1) * Lastly but not least, use `konduit stop` to terminate selected id in Konduit-Serving: ``` %%bash konduit stop dl4j-iris ``` **Congratulation! You have deployed Konduit-Serving on your own. What's next?** # Using Java SDK Guide to start using Konduit-Serving with Java SDK This quickstart article will show you to begin your project in the Java environment. Konduit-Serving provides Java SDK, a developer tool that enable you to write the code with more ease, effectiveness and efficiency. Let's start building and installing Konduit-Serving from the source and take a look to demonstrate the examples of Konduit-Serving in Java. ### Prerequisites You will need following prerequisites to follow along * Maven 3.x * JDK 8 * Git * [IntelliJ](https://www.jetbrains.com/idea/) ## Installation from Sources The following section explains how to clone and build Konduit-Serving from sources. To build from source, follow the guide below: {% content-ref url="/pages/-LwHP0JVDZsAP328VRoA" %} [Building from source](/building-from-source) {% endcontent-ref %} Once you've installed the Konduit-Serving to your local maven repository, you can now include it in your build tool's dependencies. Follow the instructions below for an example of Konduit-Serving with Java SDK. ## Java SDK Let's look at the examples how to use Konduit-Serving in Java ### Cloning Examlpe Repository Let's clone the `konduit-serving-examples` repository: ``` $ git clone https://github.com/KonduitAI/konduit-serving-examples ``` You'll see the following files in `konduit-serving-examples`: ``` konduit-serving-examples/ ├── data ├── java ├── monitoring ├── notebooks ├── python ├── quickstart ├── README.md ├── utils └── yaml ``` You'll need to open the "java" file in IntelliJ as a project, and you'll find two examples under the `./src/main/java` subfolder. Let's look at the examples of Konduit-Serving to create a configuration and deploy a server. ### Example Let’s start by defining the inference configuration and sequence pipeline that define the configuration of the server: ```java InferenceConfiguration inferenceConfiguration = new InferenceConfiguration(); SequencePipeline sequencePipeline = SequencePipeline .builder() .add(new LoggingStep().log(LoggingStep.Log.KEYS_AND_VALUES)) .build(); ``` The inference configuration should have a pipeline to deploy the server, include pipeline with: ```java inferenceConfiguration.pipeline(sequencePipeline); System.out.format(inferenceConfiguration.toYaml()); ``` You'll see the printed server configuration as a YAML: ```yaml --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "LOGGING" logLevel: "INFO" log: "KEYS_AND_VALUES" ``` Let's deploy the server by using `DeplotKonduitServing.deploy()` includes created inference configuration with the pipeline into the deployment: ```java DeployKonduitServing.deploy( new VertxOptions(), new DeploymentOptions(), inferenceConfiguration, handler -> { if (handler.succeeded()) { InferenceDeploymentResult inferenceDeploymentResult = handler.result(); int runningPort = inferenceDeploymentResult.getActualPort(); String deploymentId = inferenceDeploymentResult.getDeploymentId(); System.out.format("%nPort is %s and Deployment Id is %s.%n", runningPort, deploymentId); try { String result = Unirest.post(String.format("http://localhost:%s/predict", runningPort)) .header("Content-Type", "application/json") .header("Accept", "application/json") .body(new JSONObject().put("input_key", "input_value")) .asString().getBody(); System.out.format("Result: %s%n", result); System.exit(0); } catch (UnirestException e) { e.printStackTrace(); System.exit(1); } } else { System.out.println(handler.cause().getMessage()); System.exit(1); } } ); ``` The handler expression is a callback after a successful or failed server deployment which can be implemented inside the handler block, as shown: ```aspnet Port is 38019 and Deployment Id is cc5e2081-81e4-4d3e-9734-3201af512641. Result: { "input_key" : "input_value" } Process finished with exit code 0 ``` Congratulation! You've deployed Konduit-Serving using Java SDK. # Using Python SDK Coming soon ... # Using CLI Guide to start using Konduit-Serving with CLI This document will demonstrate using Konduit-Serving using mainly CLI tools. You can deploy ML/DL models to production using minimal effort using Konduit-Serving. Let's look at the process of building and installing Konduit-Serving from source and how to deploy a model using a simple configuration. ### Prerequisite You will need following prerequisites to follow along * Maven 3.x * JDK 8 * Git ## Installation from Sources The following two sections explains how to clone, build and install Konduit-Serving from sources. To build from source, follow the guide below {% content-ref url="/pages/-LwHP0JVDZsAP328VRoA" %} [Building from source](/building-from-source) {% endcontent-ref %} To install the respective built binaries you can navigate to the section below {% content-ref url="/pages/-LtTMd5ReUOXuuHoM\_rH" %} [Installing Binaries](/installation) {% endcontent-ref %} After you've installed Konduit-Serving in your local machine you can switch to a terminal and verify the installation by running ``` konduit --version ``` You'll see an output similar to the one below ``` $ konduit --version ------------------------------------------------ Version: 0.1.0-SNAPSHOT Commit hash: 3dd38832 Commit time: 01.03.2021 @ 03:37:08 MYT Build time: 07.03.2021 @ 16:57:51 MYT ``` ## Deploying Models Let's look at how to deploy a dl4j/keras model using Konduit-Serving ### Cloning Examples Repo Let's clone the `konduit-serving-examples` repo ``` git clone https://github.com/KonduitAI/konduit-serving-examples.git ``` and navigate to the `quickstart` folder ``` cd konduit-serving-demo/quickstart ``` The examples we want to run are under the folders `3-keras-mnist` and `5-dl4j-mnist`. Let's follow a basic workflow for both models using the Konduit-Serving CLI. {% tabs %} {% tab title="Keras" %} Navigate to `3-keras-mnist` ``` cd 3-keras-mnist ``` Here, you'll find the following files: ``` . ├── keras-mnist.ipynb | A supplementary jupyter/beakerx notebook ├── keras.h5 | Model file we want to serve ├── keras.json | Konduit-Serving configuration ├── test-image.jpg | Test input for predictions └── train.py | Script for creating 'keras.h5' model file ``` The `keras.json` contains the configuration file for running an MNIST dataset trained model in Keras. To serve the model, execute the following command ```bash konduit serve --config keras.json -id keras-server ``` You'll be able to see a similar output like the following ```bash . . . 15:00:08.575 [vert.x-worker-thread-0] INFO a.k.s.m.d.step.DL4JRunner - 15:00:08.576 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### 15:00:08.576 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - Pending server start, please wait... . . . . 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 40987 with 4 pipeline steps ``` The last line will show you the details about which URL the server is serving the models at. Press `Ctrl + C`, or execute `konduit stop keras-server` to kill the server. To run the server in the background, you can run the same command with the `--background` or `-b` flag. ``` konduit serve --config keras.json -id keras-server --background ``` You'll see something similar to ``` Starting konduit server... Expected classpath: /Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../konduit.jar INFO: Running command /Users/konduit/opt/miniconda3/jre/bin/java -Dkonduit.logs.file.path=/Users/konduit/.konduit-serving/command_logs/keras-server.log -Dlogback.configurationFile=/Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../conf/logback-run_command.xml -cp /Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../konduit.jar ai.konduit.serving.cli.launcher.KonduitServingLauncher run --instances 1 -s inference -c keras.json -Dserving.id=keras-server For server status, execute: 'konduit list' For logs, execute: 'konduit logs keras-server' ``` To list the server, simply run ``` konduit list ``` You'll see the running servers as a list ``` Listing konduit servers... # | ID | TYPE | URL | PID | STATUS 1 | keras-server | inference | localhost:1000 | 1200 | Started ``` To view the logs, you can run the following command ``` konduit logs keras-server --lines 2 ``` The `--lines` or `-l` flag shows the specified number of last lines. By executing the above command you'll see the following ``` 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 1000 with 4 pipeline steps ``` Now finally, let's look at running predictions with Konduit-Serving by sending an image file to the server. ``` konduit predict keras-server -it multipart 'image=@test-image.jpg' ``` It will convert the image into an n-dimensional array and then send the input to the keras model and you'll see the following output ``` { "output_layer" : [ [ 9.0376153E-7, 1.0595608E-8, 1.3115231E-5, 0.44657645, 6.748624E-12, 0.5524258, 1.848306E-7, 2.7652052E-9, 9.76023E-4, 7.5933513E-6 ] ], "prob" : 0.5524258017539978, "index" : 5, "label" : "5" } ``` {% endtab %} {% tab title="DL4J" %} Navigate to `5-dl4j-mnist` ```bash cd 5-dl4j-mnist ``` Here, you'll find the following files: ``` . ├── dl4j-mnist.ipynb | A supplementary jupyter/beakerx notebook ├── dl4j-mnist.zip | Model file we want to serve ├── dl4j.json | Konduit-Serving configuration └── test-image.jpg | Test input for predictions ``` The `dl4j.json` contains the configuration file for running an MNIST dataset trained model in DL4J. To serve the model, execute the following command ```bash konduit serve --config dl4j.json -id dl4j-server ``` You'll be able to see a similar output like the following ```bash . . . 15:00:08.575 [vert.x-worker-thread-0] INFO a.k.s.m.d.step.DL4JRunner - 15:00:08.576 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### 15:00:08.576 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - Pending server start, please wait... . . . . 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 40987 with 4 pipeline steps ``` The last line will show you the details about which URL the server is serving the models at. Press `Ctrl + C`, or execute `konduit stop keras-server` to kill the server. To run the server in the background, you can run the same command with the `--background` or `-b` flag. ``` konduit serve --config dl4j.json -id dl4j-server --background ``` You'll see something similar to ``` Starting konduit server... Expected classpath: /Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../konduit.jar INFO: Running command /Users/konduit/opt/miniconda3/jre/bin/java -Dkonduit.logs.file.path=/Users/konduit/.konduit-serving/command_logs/dl4j-server.log -Dlogback.configurationFile=/Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../conf/logback-run_command.xml -cp /Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../konduit.jar ai.konduit.serving.cli.launcher.KonduitServingLauncher run --instances 1 -s inference -c dl4j.json -Dserving.id=dl4j-server For server status, execute: 'konduit list' For logs, execute: 'konduit logs dl4j-server' ``` To list the server, simply run ``` konduit list ``` You'll see the running servers as a list ``` Listing konduit servers... # | ID | TYPE | URL | PID | STATUS 1 | dl4j-server | inference | localhost:1000 | 1200 | Started ``` To view the logs, you can run the following command ``` konduit logs dl4j-server --lines 2 ``` The `--lines` or `-l` flag shows the specified number of last lines. By executing the above command you'll see the following ``` 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 1000 with 4 pipeline steps ``` Now finally, let's look at running predictions with Konduit-Serving by sending an image file to the server. ``` konduit predict dl4j-server -it multipart 'image=@test-image.jpg' ``` It will convert the image into an n-dimensional array and then send the input to the DL4J model and you'll see the following output ``` { "layer5" : [ [ 1.845163E-5, 1.8346094E-6, 0.31436875, 0.43937472, 2.6101702E-8, 0.24587035, 5.9430695E-6, 3.3270408E-4, 6.3698195E-8, 2.708706E-5 ] ], "prob" : 0.439374715089798, "index" : 3, "label" : "3" } ``` {% endtab %} {% endtabs %} Congratulations! You've learned the basic workflow for Konduit-Serving using the Command Line Interface. # Building from source Instructions for building Konduit-Serving binaries from source ## Pre-requisites * JDK 1.8+ * Maven 3+ * Git ## Cloning the repository Konduit Serving sources are hosted on GitHub. If you have [git](https://git-scm.com/) installed, clone the [konduit-serving repository](https://github.com/KonduitAI/konduit-serving) using the `git clone` command: ``` git clone https://github.com/KonduitAI/konduit-serving.git ``` ## Using the Build Script After cloning the repository, run `./build.sh --help` to see the available options: ``` $ ./build.sh --help ------------------------------------------------------------------- A command line utility for building konduit-serving distro packages. Usage: bash build.sh [CPU|GPU] [linux|windows|macosx] [tar|zip|exe|rpm|deb] Example: bash build.sh GPU linux tar,deb ------------------------------------------------------------------- ``` You can create CPU/GPU builds for available platforms by executing their respective commands. ## Example An example of creating **Ubuntu (deb)** build is as follows: ``` $ ./build.sh CPU linux deb ------------------------------------------------------------------- Building project version: 0.1.0-SNAPSHOT Building a konduit-serving distributable JAR file... Selecting CHIP=CPU Building CPU version of konduit-serving for linux with distro types: (deb) ... Running command: mvn clean install -Dmaven.test.skip=true -Denforcer.skip=true -Djavacpp.platform=linux-x86_64 -Ppython,uberjar,tar,deb -Ddevice=CPU [INFO] Scanning for projects... . . . . . [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 08:10 min [INFO] Finished at: 2021-01-08T13:40:01+05:00 [INFO] ------------------------------------------------------------------------ ---------------------------------------- DEB distro is available at: konduit-serving-deb/target/konduit-serving-custom-CPU_0.1.0-SNAPSHOT.deb ------------------------------------------------------------------- ``` # Installing Binaries Instructions for installing and setting up built binaries ## System requirements ### **OPERATING SYSTEMS** Konduit Serving is supported on * Linux * MacOS * Windows ### **DEPENDENCIES** * Ensure that you have JDK 8.0 installed * Install Python 3.7.x to be used with the Python Step ### **HARDWARE REQUIREMENTS** Binaries are provided for * Intel/x86 architectures * ARM ### **GPU SUPPORT** Hardware acceleration with * CUDA version 11.0 ## INSTALLATION INSTRUCTIONS Installation for each type of distribution is as follows: ### TAR/ZIP After extracting, just put the `bin` folder into the `PATH` environment variable. ### DEB ``` dpkg -i konduit-serving-deb/target/**/*.deb ``` ### RPM ``` rpm -i konduit-serving-rpm/target/**/*.rpm ``` ### EXE Just put the folder containing the `konduit.exe` file into the `PATH` environment variable. ## CONFIGURATIONS ### LINUX/MACOS The configuration file is present inside the `conf/konduit-serving-env.sh` file for the `TAR/ZIP` distro. For Konduit-Serving distro installed from the `DEB` and `RPM` packages, the configuration file is present inside the location `/opt/konduit/conf/konduit-serving-env.sh`. Just uncomment the variables you want to set and specify the value you like. ### WINDOWS The configuration file for the `TAR/ZIP` distro is present inside the `conf/konduit-serving-env.cmd` file. Just uncomment the variables you want to set and specify the value you like. # Configurations Konduit-Serving supports defining server configurations as JSON or YAML files. The configuration is essential to serve a Machine Learning or Deep Learning model in Konduit-Serving. The complete format contains inference configuration and pipeline steps to deploy the server to serve the model in details. A Konduit-Serving configuration file has two top-level items: 1. Inference configuration 2. Pipeline ## Inference Configuration A sample below is the inference configuration, which contains many setups for the server to run on Konduit-Serving. For explanation purpose, the YAML format is used like the following. ``` --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] ``` As the setting on the above, there are a lot of keys used in the inference configuration. The inference configuration takes the following arguments: * `host` : specify the port number * `port` : the host of the Konduit-Serving. Default is 'localhost'. * `use_ssl` : Enable SSL for internet connection security data protection. Default is 'false'. * `protocol` : protocol use with the server. Default is 'HTTP'. * `static_content_root` : root directory use to search for a folder in serving static content * `static_content_url` : URL that will link to the static contents specified in the `static_content_root` key. * `static_content_index_page` : index file name. Default is 'index.html'. * `kafka_configuration` : Configuration for Kafka message queue when `protocol` key is `KAFKA` * `mqtt_configuration` : configuration if 'MQTT' is used as protocol. * `custom_endpoints` : Custom created endpoints by implementing the `ai.konduit.serving.endpoint.HttpEndpoints` interface. This setting can be generated using the `CLI` command, and generally, this configuration file is a must to deploy any Pipeline Steps on Konduit-Serving. ## Pipeline A pipeline consists of steps on how the server should treat the data and the model. We can use many steps in the pipeline to serve the model, including the input pre-processing step and output post-processing step. The simple application makes Konduit-Serving more convenient for all level experience to use our platform. Below is the example of Pipeline Steps which only consists of serving the model and do post-processing by classifying the output product from the last layer of the model. ``` pipeline: steps: - '@type': "DEEPLEARNING4J" modelUri: "dl4j_iris_model.zip" inputNames: - "input" outputNames: - "output" - '@type': "CLASSIFIER_OUTPUT" input_names: "layer2" labels: - Sentosa - Versicolor - Virginica ``` These steps can be added as many as possible based on the requirement for custom endpoints. Among the steps that can be used in this pipeline are: * Sequences pipeline steps: * crop\_grid * crop\_fixed*\_*grip * dl4j (use in above example) * keras * draw\_bounding\_box * draw\_fixed*\_*grid * draw\_segmentation * extract\_bounding\_box * camera\_frame\_capture * video\_frame\_capture * image\_to\_ndarray * logging * ssd\_to\_bounding\_box * samediff * show\_image * tensorflow * nd4jtensorflow * python * onnx * classifier\_output (use in above example) * Graphs pipeline steps: * pipeline steps * merge step * switch step * any step The complete inference configuration files can be in: {% content-ref url="/pages/-MUvZYhUr4Ay5z\_23NZG" %} [JSON](/configurations/json) {% endcontent-ref %} or {% content-ref url="/pages/-MUvqLE1MpEkO3Rf7-lb" %} [YAML](/configurations/yaml) {% endcontent-ref %} # JSON In JSON format Below is the example of a default configuration file in JSON format that needs to serve the model in Konduit-Serving. This configuration file may need simple editing to set up your setup, especially in Pipeline Steps. Example default Inference Configuration with Sequence Pipeline Steps: ``` { "host" : "localhost", "port" : 0, "useSsl" : false, "protocol" : "HTTP", "staticContentRoot" : "static-content", "staticContentUrl" : "/static-content", "staticContentIndexPage" : "/index.html", "kafkaConfiguration" : { "startHttpServerForKafka" : true, "httpKafkaHost" : "localhost", "httpKafkaPort" : 0, "consumerTopicName" : "inference-in", "consumerKeyDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerValueDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerGroupId" : "konduit-serving-consumer-group", "consumerAutoOffsetReset" : "earliest", "consumerAutoCommit" : "true", "producerTopicName" : "inference-out", "producerKeySerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerValueSerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerAcks" : "1" }, "mqttConfiguration" : { }, "customEndpoints" : [ ], "pipeline" : { "steps" : [ { "@type" : "DEEPLEARNING4J", "modelUri" : "", "inputNames" : [ "1", "2" ], "outputNames" : [ "11", "22" ] }, { "@type" : "LOGGING", "logLevel" : "INFO", "log" : "KEYS_AND_VALUES" } ] } } ``` Example default Inference Configuration with Graph Pipeline Steps: ``` { "host" : "localhost", "port" : 0, "useSsl" : false, "protocol" : "HTTP", "staticContentRoot" : "static-content", "staticContentUrl" : "/static-content", "staticContentIndexPage" : "/index.html", "kafkaConfiguration" : { "startHttpServerForKafka" : true, "httpKafkaHost" : "localhost", "httpKafkaPort" : 0, "consumerTopicName" : "inference-in", "consumerKeyDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerValueDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerGroupId" : "konduit-serving-consumer-group", "consumerAutoOffsetReset" : "earliest", "consumerAutoCommit" : "true", "producerTopicName" : "inference-out", "producerKeySerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerValueSerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerAcks" : "1" }, "mqttConfiguration" : { }, "customEndpoints" : [ ], "pipeline" : { "outputStep" : "4", "steps" : { "1" : { "@type" : "LOGGING", "@input" : "input", "logLevel" : "INFO", "log" : "KEYS_AND_VALUES" }, "2" : { "@type" : "TENSORFLOW", "@input" : "1", "inputNames" : [ "1", "2" ], "outputNames" : [ "11", "22" ], "modelUri" : "" }, "3" : { "@type" : "DEEPLEARNING4J", "@input" : "1", "modelUri" : "", "inputNames" : [ "1", "2" ], "outputNames" : [ "11", "22" ] }, "4" : { "@type" : "MERGE", "@input" : [ "2", "3" ] } } } } ``` For more details on how to create the configuration file, please refer to the examples: {% content-ref url="/pages/-MUvuMrejKwhl7k4P1Un" %} [Creating a Sequence Pipeline](/examples/cli/use-cases/creating-a-sequence-pipeline) {% endcontent-ref %} {% content-ref url="/pages/-MUvuDm7HID4Bj9vrJkQ" %} [Creating a Graph Pipeline](/examples/cli/use-cases/creating-a-graph-pipeline) {% endcontent-ref %} # YAML In YAML format Apart from JSON, the configuration file also can be in YAML format. It is identical to JSON but more human-readable data representation and pretty straightforward. Difference to JSON, YAML's hierarchy is denoted by using double space characters as an example below. ``` --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "DEEPLEARNING4J" modelUri: "" inputNames: - "1" - "2" outputNames: - "11" - "22" - '@type': "LOGGING" logLevel: "INFO" log: "KEYS_AND_VALUES" ``` Example of default YAML configuration file with custom Graph Pipeline Steps: ``` --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: outputStep: "4" steps: "1": '@type': "LOGGING" '@input': "input" logLevel: "INFO" log: "KEYS_AND_VALUES" "2": '@type': "TENSORFLOW" '@input': "1" input_names: - "1" - "2" output_names: - "11" - "22" model_uri: "" "3": '@type': "DEEPLEARNING4J" '@input': "1" modelUri: "" inputNames: - "1" - "2" outputNames: - "11" - "22" "4": '@type': "MERGE" '@input': - "2" - "3" ``` For more details on how to create the configuration file, please refer to the examples: {% content-ref url="/pages/-MUvuMrejKwhl7k4P1Un" %} [Creating a Sequence Pipeline](/examples/cli/use-cases/creating-a-sequence-pipeline) {% endcontent-ref %} {% content-ref url="/pages/-MUvuDm7HID4Bj9vrJkQ" %} [Creating a Graph Pipeline](/examples/cli/use-cases/creating-a-graph-pipeline) {% endcontent-ref %} # Java Example of Konduit-Serving with Java SDK In this example, you'll create a configuration and deploy a server via Konduit-Serving. These basic can be used to replicate on other platforms with the same process to demonstrate in advance of Konduit-Serving. Let's look at the examples! {% hint style="info" %} You can get the `java` examples by cloning from the repository: `$ git clone`[`https://github.com/KonduitAI/konduit-serving-examples`](https://github.com/KonduitAI/konduit-serving-examples.git) If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of example class. {% endhint %} Here are the articles in this section: {% content-ref url="/pages/-MVp4qkhd3GoGCaQ6OUp" %} [Server](/examples/java/server) {% endcontent-ref %} {% content-ref url="/pages/-MUvtHi9BctRIFlferCR" %} [Pipeline Steps](/examples/java/server/pipeline-steps) {% endcontent-ref %} {% content-ref url="/pages/-MUvtJhsmA8uPTXlGt4W" %} [Sequence Pipeline](/examples/java/server/sequence-pipeline) {% endcontent-ref %} # Server Simple example to deploy a server with Konduit-Serving In this example, we'll deploy a server with Konduit-Serving. * First, let's start with creating a complete configuration of the server. ```java InferenceConfiguration inferenceConfiguration = new InferenceConfiguration(); inferenceConfiguration.pipeline( SequencePipeline .builder() .add(new LoggingStep().log(LoggingStep.Log.KEYS_AND_VALUES)) .build() ); ``` * Let's deploy the server with the configuration made above. The successful server deployment will give the port number and host of the server: ```java DeployKonduitServing.deploy( new VertxOptions(), // Default vertx options new DeploymentOptions(), // Default deployment options inferenceConfiguration, // Inference configuration with logging step handler -> { // this block will be called when server finishes the deployment if (handler.succeeded()) { // If the server is sucessfully running // Getting the result of the deployment InferenceDeploymentResult inferenceDeploymentResult = handler.result(); int runnningPort = inferenceDeploymentResult.getActualPort(); String deploymentId = inferenceDeploymentResult.getDeploymentId(); System.out.format("The server is running on port %s with deployment id of %s%n", runnningPort, deploymentId); try { String result = Unirest.post(String.format("http://localhost:%s/predict", runnningPort)) .header("Content-Type", "application/json") .header("Accept", "application/json") .body(new JSONObject().put("input_key", "input_value")) .asString().getBody(); System.out.format("Result from server : %s%n", result); System.exit(0); } catch (UnirestException e) { e.printStackTrace(); System.exit(1); } } else { // If the server failed to run System.out.println(handler.cause().getMessage()); System.exit(1); } }); ``` You'll be able to see the output similar to this once the server successfully deployed: ```aspnet The server is running on port 37663 with deployment id of 59d5d475-be83-4348-8983-4d3e7328e71d Result from server : { "input_key" : "input_value" } Process finished with exit code 0 ``` # Pipeline Steps Konduit-Serving works by defining a series of steps. These include operation such as: 1. Pre-processing steps 2. Running one or more ML/DL models 3. Post-processing steps, transforming the output in a way that humans can understand {% hint style="info" %} A reference Java project is provided in the Example repository from with a Maven pom.xml dependencies file. If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of each of example class. {% endhint %} Here are the articles in this section: {% content-ref url="/pages/-MUvtOOU7ibpjiCjW\_ij" %} [Image To NDArray Step](/examples/java/server/pipeline-steps/image-to-ndarray-step) {% endcontent-ref %} {% content-ref url="/pages/-MUvtbQs0ZSgA5xrI6Am" %} [DL4J Step](/examples/java/server/pipeline-steps/dl4j-step) {% endcontent-ref %} {% content-ref url="/pages/-MX5wpK3hhKMeK94Ujja" %} [Keras Step](/examples/java/server/pipeline-steps/keras-step) {% endcontent-ref %} {% content-ref url="/pages/-MX5xn6oX27JvqOGRoEo" %} [ONNX Step](/examples/java/server/pipeline-steps/onnx-step) {% endcontent-ref %} {% content-ref url="/pages/-MX5xqqhpFoc425AtEDD" %} [Tensorflow Step](/examples/java/server/pipeline-steps/tensorflow-step) {% endcontent-ref %} # Image To NDArray Step As pre-processing step Image To NDArray Step is used as a pre-processing step in Pipeline Step to manipulate or convert the image into an *n-*D array based on the model requirement input layer or the next pipeline step. We can include `ImageToNDArrayStep()` in pipeline into inference configuration. ```java InferenceConfiguration inferenceConfiguration = new InferenceConfiguration(); inferenceConfiguration.pipeline(SequencePipeline.builder() .add(new ImageToNDArrayStep() //add ImageToNDArrayStep() into pipeline to set image to NDArray for input .config(new ImageToNDArrayConfig() //image configuration .width(28) .height(28) .includeMinibatchDim(true) .channelLayout(NDChannelLayout.GRAYSCALE) .format(NDFormat.CHANNELS_LAST) ) .keys("image") .outputNames("input_layer")) ).build() ); ``` From `ImageToNDArrayConfig()` in the above, the input image will have 28 by 28 shape size and convert to a 3-D array. Set mini batch dimension to true, and the channel has a depth of 1 for grayscale, put at last of a list such \[1, 28, 28, 1]. {% hint style="info" %} The shape array such \[minibatch\_dim, width, height, channels] if format is *CHANNELS\_LAST* . {% endhint %} ```java inferenceConfiguration.pipeline(SequencePipeline.builder() .add(new ImageToNDArrayStep() //add ImageToNDArrayStep() into pipeline to set image to NDArray for input .config(new ImageToNDArrayConfig() //image configuration .width(28) .height(28) .includeMinibatchDim(true) .channelLayout(NDChannelLayout.GRAYSCALE) .format(NDFormat.CHANNELS_FIRST) .normalization(ImageNormalization.builder().type(ImageNormalization.Type.SCALE).build()) ) .keys("image") .outputNames("input_layer") ).build() ); ``` We'll be able to add data normalization and change channel element position to first which will give the input of \[1, 1, 28, 28] to the model. Failure to give an image with characteristics mentioned in configuration will affect the deployment of model in server and return an error. You can see the similar configuration in [Keras Step](/examples/java/server/pipeline-steps/keras-step) and [Tensorflow Step](/examples/java/server/pipeline-steps/tensorflow-step). # Python Step Coming soon... # DL4J Step Example of applying DL4J Step This example splits into two parts which are configuring the inference configuration and running the server. ```java import ai.konduit.serving.examples.utils.Train; import ai.konduit.serving.models.deeplearning4j.step.DL4JStep; import ai.konduit.serving.pipeline.impl.pipeline.SequencePipeline; import ai.konduit.serving.vertx.api.DeployKonduitServing; import ai.konduit.serving.vertx.config.InferenceConfiguration; import ai.konduit.serving.vertx.config.InferenceDeploymentResult; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import io.vertx.core.DeploymentOptions; import io.vertx.core.VertxOptions; import org.json.JSONArray; import org.json.JSONObject; import java.util.Arrays; ``` {% hint style="info" %} A reference Java project is provided in the Example repository from with a Maven pom.xml dependencies file. If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of Example\_1\_Dl4jStep class. {% endhint %} ### Configure the step Let's start from the main function by getting the trained model. ```java //get the file of trained model Train.ModelTrainResult modelTrainResult = Train.dl4jIrisModel(); ``` Create an inference configuration by default. ```java //a default Inference Configuration InferenceConfiguration inferenceConfiguration = new InferenceConfiguration(); ``` We'll need to include `DL4JStep` into the pipeline and bind with the inference configuration. Specify the following: * `modelUri` : the model file path * `inputNames` : names for model's input layer * `outputNames` : names for model's output layer ```java //include pipeline step into the Inference Configuration inferenceConfiguration.pipeline(SequencePipeline.builder() .add(new DL4JStep() //add DL4JStep into pipeline .modelUri(modelTrainResult.modelPath()) .inputNames(modelTrainResult.inputNames()) .outputNames(modelTrainResult.outputNames()) ).build() ); ``` ### Deploy the server Let's deploy the model in the server by calling `DeployKonduitServing` with the configuration made before. The handler, a callback function, is implemented to capture a successful or failed server deployment state. ```java //deploy the model in server DeployKonduitServing.deploy(new VertxOptions(), new DeploymentOptions(), inferenceConfiguration, handler -> { if (handler.succeeded()) { // If the server is sucessfully running // Getting the result of the deployment InferenceDeploymentResult inferenceDeploymentResult = handler.result(); int runnningPort = inferenceDeploymentResult.getActualPort(); String deploymentId = inferenceDeploymentResult.getDeploymentId(); System.out.format("The server is running on port %s with deployment id of %s%n", runnningPort, deploymentId); try { String result = Unirest.post(String.format("http://localhost:%s/predict", runnningPort)) .header("Content-Type", "application/json") .header("Accept", "application/json") .body(new JSONObject().put("layer0", new JSONArray().put(Arrays.asList(1.0, 1.0, 1.0, 1.0))) ) .asString().getBody(); System.out.format("Result from server : %s%n", result); System.exit(0); } catch (UnirestException e) { e.printStackTrace(); System.exit(1); } } else { // If the server failed to run System.out.println(handler.cause().getMessage()); System.exit(1); } }); ``` Note that we consider only one test input array in this example for inference to show the model's deployment in Konduit-Serving. After execution, the successful server deployment gives below output text. ```aspnet The server is running on port 39615 with deployment id of 2fe69a2d-1276-4a1d-b5af-50191640019f Result from server : { "layer2" : [ [ 5.287693E-4, 0.02540398, 0.9740673 ] ] } Process finished with exit code 0 ``` The complete inference configuration in YAML format is as follows. ```java System.out.format(inferenceConfiguration.toYaml()); ``` ```aspnet --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "DEEPLEARNING4J" modelUri: "/tmp/model6027458615639981189zip" inputNames: - "layer0" outputNames: - "layer2" ``` # Keras Step Example of applying Keras Step The example starts with configuring the pipeline step in the inference configuration and then deploying the server using the Keras model with Konduit-Serving. ```java import ai.konduit.serving.data.image.convert.ImageToNDArrayConfig; import ai.konduit.serving.data.image.convert.config.NDChannelLayout; import ai.konduit.serving.data.image.convert.config.NDFormat; import ai.konduit.serving.data.image.step.ndarray.ImageToNDArrayStep; import ai.konduit.serving.examples.utils.Train; import ai.konduit.serving.models.deeplearning4j.step.keras.KerasStep; import ai.konduit.serving.pipeline.impl.pipeline.SequencePipeline; import ai.konduit.serving.vertx.api.DeployKonduitServing; import ai.konduit.serving.vertx.config.InferenceConfiguration; import ai.konduit.serving.vertx.config.InferenceDeploymentResult; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import io.vertx.core.DeploymentOptions; import io.vertx.core.VertxOptions; import org.nd4j.common.io.ClassPathResource; ``` ### Configure the step Let's start from the main function by getting the trained model. ```java //get the file of trained model Train.ModelTrainResult modelTrainResult = Train.kerasMnistModel(); ``` Create a default inference configuration that the server will use. ```java //a default Inference Configuration InferenceConfiguration inferenceConfiguration = new InferenceConfiguration(); ``` Add `ImageToNDArrayStep()` which are pre-processing step into the pipeline as a need to convert an input image to an array and must specified with a shape size. We'll also need to include `KerasStep` into the pipeline of the inference configuration. Specify the following: * `modelUri` : the model file path * `inputNames` : names for model's input layer * `outputNames` : names for model's output layer ```java //include pipeline step into the Inference Configuration inferenceConfiguration.pipeline(SequencePipeline.builder() .add(new ImageToNDArrayStep() //add ImageToNDArrayStep() into pipeline to set image to NDArray for input .config(new ImageToNDArrayConfig() //image configuration .width(28) .height(28) .includeMinibatchDim(true) .channelLayout(NDChannelLayout.GRAYSCALE) .format(NDFormat.CHANNELS_LAST) ) .keys("image") .outputNames("input_layer")) .add(new KerasStep() //add KerasStep into pipeline .modelUri(modelTrainResult.modelPath()) .inputNames(modelTrainResult.inputNames()) .outputNames(modelTrainResult.outputNames()) ).build() ); ``` ### Deploy the server Let's deploy the model in the server by calling `DeployKonduitServing` with the configuration made before. A callback function is implemented to get a response only after a successful or failed server deployment inside the handler block. ```java //deploy the model in server DeployKonduitServing.deploy(new VertxOptions(), new DeploymentOptions(), inferenceConfiguration, handler -> { if (handler.succeeded()) { // If the server is sucessfully running // Getting the result of the deployment InferenceDeploymentResult inferenceDeploymentResult = handler.result(); int runnningPort = inferenceDeploymentResult.getActualPort(); String deploymentId = inferenceDeploymentResult.getDeploymentId(); System.out.format("The server is running on port %s with deployment id of %s%n", runnningPort, deploymentId); try { String result; try { result = Unirest.post(String.format("http://localhost:%s/predict", runnningPort)) .header("Accept", "application/json") .field("image", new ClassPathResource("inputs/mnist-image-2.jpg").getFile(), "image/jpg") .asString().getBody(); System.out.format("Result from server : %s%n", result); System.exit(0); } catch (IOException e) { e.printStackTrace(); System.exit(1); } } catch (UnirestException e) { e.printStackTrace(); System.exit(1); } } else { // If the server failed to run System.out.println(handler.cause().getMessage()); System.exit(1); } }); ``` Note that we consider only one test input image in this example for inference to show the model's deployment in Konduit-Serving. After the above execution, you can check for the below output to confirm the successful server deployment. ```aspnet The server is running on port 46233 with deployment id of f9b7a616-2d54-4814-86ac-1f888052ef34 Result from server : { "output_layer" : [ [ 6.086365E-10, 6.585195E-11, 7.845706E-7, 1.8983503E-6, 2.3600207E-11, 1.6447022E-8, 4.0799048E-11, 2.2013203E-12, 0.99999714, 6.1216234E-8 ] ] } Process finished with exit code 0 ``` The complete inference configuration in JSON format is as follows. ```java System.out.format(inferenceConfiguration.toJson()); ``` ```aspnet { "host" : "localhost", "port" : 0, "useSsl" : false, "protocol" : "HTTP", "staticContentRoot" : "static-content", "staticContentUrl" : "/static-content", "staticContentIndexPage" : "/index.html", "kafkaConfiguration" : { "startHttpServerForKafka" : true, "httpKafkaHost" : "localhost", "httpKafkaPort" : 0, "consumerTopicName" : "inference-in", "consumerKeyDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerValueDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerGroupId" : "konduit-serving-consumer-group", "consumerAutoOffsetReset" : "earliest", "consumerAutoCommit" : "true", "producerTopicName" : "inference-out", "producerKeySerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerValueSerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerAcks" : "1" }, "mqttConfiguration" : { }, "customEndpoints" : [ ], "pipeline" : { "steps" : [ { "@type" : "IMAGE_TO_NDARRAY", "config" : { "height" : 28, "width" : 28, "dataType" : "FLOAT", "includeMinibatchDim" : true, "aspectRatioHandling" : "CENTER_CROP", "format" : "CHANNELS_LAST", "channelLayout" : "GRAYSCALE", "normalization" : { "type" : "SCALE" }, "listHandling" : "NONE" }, "keys" : [ "image" ], "outputNames" : [ "input_layer" ], "keepOtherValues" : true, "metadata" : false, "metadataKey" : "@ImageToNDArrayStepMetadata" }, { "@type" : "KERAS", "modelUri" : "/home/zulfadzli/KS2/konduit-serving-examples/java/target/classes/models/keras/mnist/keras-mnist.h5", "inputNames" : [ "input_layer" ], "outputNames" : [ "output_layer" ] } ] } } ``` # ONNX Step Example of applying ONNX Step This page provides a Java example of deploying a built-in model Python with Open Neural Network Exchange (ONNX) platform. The ONNX format is supported by other deep learning frameworks such Tensorflow, Pytorch, etc. In this example, the ONNX model is used to deploy the Iris model in the server. ```java import ai.konduit.serving.examples.utils.Train; import ai.konduit.serving.models.onnx.step.ONNXStep; import ai.konduit.serving.pipeline.impl.pipeline.SequencePipeline; import ai.konduit.serving.vertx.api.DeployKonduitServing; import ai.konduit.serving.vertx.config.InferenceConfiguration; import ai.konduit.serving.vertx.config.InferenceDeploymentResult; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import io.vertx.core.DeploymentOptions; import io.vertx.core.VertxOptions; import org.json.JSONArray; import org.json.JSONObject; import java.io.IOException; import java.util.Arrays; ``` ### Configure the step Let's start from the main function by getting the trained model. ```java //get the file of trained model Train.ModelTrainResult modelTrainResult = Train.onnxIrisModel(); ``` Create an inference configuration by default. ```java //a default Inference Configuration InferenceConfiguration inferenceConfiguration = new InferenceConfiguration(); ``` We'll need to include `ONNXStep` into the pipeline and specify the following: * `modelUri` : the model file path * `inputNames` : names for model's input layer * `outputNames` : names for model's output layer ```java //include pipeline step into the Inference Configuration inferenceConfiguration.pipeline(SequencePipeline.builder() .add(new ONNXStep() //add ONNXStep into pipeline .modelUri(modelTrainResult.modelPath()) .inputNames(modelTrainResult.inputNames()) .outputNames(modelTrainResult.outputNames()) ).build() ); ``` ### Deploy the server Let's deploy the model in the server by calling `DeployKonduitServing` with the configuration made before. A callback function is used to respond only after a successful or failed server deployment inside the handler block. ```java //deploy the model in server DeployKonduitServing.deploy(new VertxOptions(), new DeploymentOptions(), inferenceConfiguration, handler -> { if (handler.succeeded()) { // If the server is sucessfully running // Getting the result of the deployment InferenceDeploymentResult inferenceDeploymentResult = handler.result(); int runnningPort = inferenceDeploymentResult.getActualPort(); String deploymentId = inferenceDeploymentResult.getDeploymentId(); System.out.format("The server is running on port %s with deployment id of %s%n", runnningPort, deploymentId); try { String result = Unirest.post(String.format("http://localhost:%s/predict", runnningPort)) .header("Content-Type", "application/json") .header("Accept", "application/json") .body(new JSONObject().put("input", new JSONArray().put(Arrays.asList(1.0, 1.0, 1.0, 1.0))) ) .asString().getBody(); System.out.format("Result from server : %s%n", result); System.exit(0); } catch (UnirestException e) { e.printStackTrace(); System.exit(1); } } else { // If the server failed to run System.out.println(handler.cause().getMessage()); System.exit(1); } }); ``` Note that we consider only one test input array in this example for inference to show the model's deployment in Konduit-Serving. After execution, the successful server deployment gives below output text. ```aspnet The server is running on port 44301 with deployment id of 775bfbd3-2d18-435b-86c6-e9fbe7303cad Result from server : { "output" : [ [ 0.035723433, 0.27029678, 0.69397974 ] ] } Process finished with exit code 0 ``` The complete inference configuration in YAML format is as follows. ```java System.out.format(inferenceConfiguration.toYaml()); ``` ```aspnet --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "ONNX" modelUri: "/home/zulfadzli/KS2/konduit-serving-examples/java/target/classes/models/onnx/iris/iris.onnx" inputNames: - "input" outputNames: - "output" ``` # Tensorflow Step Example of applying Tensorflow Step In this example, we include a series of steps in the core abstraction, a pipeline step. The pipeline step performs a task such as: * Pre-processing step * Running a model * Post-processing by transforming the output in a way that humans can understand Once the pipeline is set up, the server can deploy the model. ```java import ai.konduit.serving.data.image.convert.ImageToNDArrayConfig; import ai.konduit.serving.data.image.convert.config.AspectRatioHandling; import ai.konduit.serving.data.image.convert.config.ImageNormalization; import ai.konduit.serving.data.image.convert.config.NDChannelLayout; import ai.konduit.serving.data.image.convert.config.NDFormat; import ai.konduit.serving.data.image.step.ndarray.ImageToNDArrayStep; import ai.konduit.serving.examples.utils.Train; import ai.konduit.serving.models.nd4j.tensorflow.step.Nd4jTensorFlowStep; import ai.konduit.serving.pipeline.api.data.NDArrayType; import ai.konduit.serving.pipeline.impl.pipeline.SequencePipeline; import ai.konduit.serving.pipeline.impl.step.ml.classifier.ClassifierOutputStep; import ai.konduit.serving.vertx.api.DeployKonduitServing; import ai.konduit.serving.vertx.config.InferenceConfiguration; import ai.konduit.serving.vertx.config.InferenceDeploymentResult; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import io.vertx.core.DeploymentOptions; import io.vertx.core.VertxOptions; import org.nd4j.common.io.ClassPathResource; import java.io.IOException; import java.util.Arrays; ``` ### Configure the pipeline step Let's start from the main function by declare the variable of labels that will be used by `ClassifierOutputStep()` and getting the trained model. ```java String[] labels = {"0","1","2","3","4","5","6","7","8","9"}; //get the file of trained model Train.ModelTrainResult modelTrainResult = Train.tensorflowMnistModel(); ``` Create an inference configuration by default. ```java //a default Inference Configuration InferenceConfiguration inferenceConfiguration = new InferenceConfiguration(); ``` We'll need to include pre-processing step using `ImageToNDArrayStep()` to convert the input image into an array and specify all characteristics of the input image. To run a model, add `Nd4jTensorFlowStep()` into the pipeline and specify `modelUri`, `inputNames` and `outputNames`. `ClassifierOutputStep()` can be included for transforming the output in a way that humans can understand and set the following: * `inputName` : names for model's output layer * `labels` : list of output labels classifier * `allProbabilities` : false ```java //include pipeline step into the Inference Configuration inferenceConfiguration.pipeline(SequencePipeline.builder() .add(new ImageToNDArrayStep() //add ImageToNDArrayStep() into pipeline to set image to NDArray for input .config(new ImageToNDArrayConfig() //image configuration .width(28) .height(28) .dataType(NDArrayType.FLOAT) .aspectRatioHandling(AspectRatioHandling.CENTER_CROP) .includeMinibatchDim(true) .channelLayout(NDChannelLayout.GRAYSCALE) .format(NDFormat.CHANNELS_FIRST) .normalization(ImageNormalization.builder().type(ImageNormalization.Type.SCALE).build()) ) .keys("image") .outputNames("input_layer") .keepOtherValues(true) .metadata(false) .metadataKey(ImageToNDArrayStep.DEFAULT_METADATA_KEY)) .add(new Nd4jTensorFlowStep() //add Nd4jTensorFlowStep into pipeline .modelUri(modelTrainResult.modelPath()) .inputNames(modelTrainResult.inputNames()) .outputNames(modelTrainResult.outputNames()) ).add(new ClassifierOutputStep() .inputName(modelTrainResult.outputNames().get(0)) .labels(Arrays.asList(labels.clone())) .allProbabilities(false) ).build() ); ``` ### Deploy the server Let's deploy the model in the server by calling `DeployKonduitServing` with the configuration made before. The handler, a callback function, is applied, only after a successful or failed server deployment inside the handler block, as shown. ```java //deploy the model in server DeployKonduitServing.deploy(new VertxOptions(), new DeploymentOptions(), inferenceConfiguration, handler -> { if (handler.succeeded()) { // If the server is successfully running // Getting the result of the deployment InferenceDeploymentResult inferenceDeploymentResult = handler.result(); int runnningPort = inferenceDeploymentResult.getActualPort(); //get server's port String deploymentId = inferenceDeploymentResult.getDeploymentId(); //get server's deployment id System.out.format("The server is running on port %s with deployment id of %s%n", runnningPort, deploymentId); try { String result; try { result = Unirest.post(String.format("http://localhost:%s/predict", runnningPort)) .header("Accept", "application/json") .field("image", new ClassPathResource("inputs/test_files/test_input_number_2.png").getFile(), "image/png") .asString().getBody(); System.out.format("Result from server : %s%n", result); System.exit(0); } catch (IOException e) { e.printStackTrace(); System.exit(1); } } catch (UnirestException e) { e.printStackTrace(); System.exit(1); } } else { // If the server failed to run System.out.println(handler.cause().getMessage()); System.exit(1); } }); ``` Note that we consider only one test input image in this example for inference to show the model's deployment in Konduit-Serving. After implementation, the successful server deployment gives below output text. ```aspnet The server is running on port 40521 with deployment id of 4b7d8bc5-a711-499f-ad9f-9ccd82c3e142 Result from server : { "output_layer/Softmax" : [ [ 1.8688768E-8, 0.0962552, 0.7753802, 1.7737559E-8, 0.122773424, 4.7498935E-13, 3.0434896E-6, 0.005588151, 7.329317E-12, 3.4533626E-10 ] ], "prob" : 0.7753801941871643, "index" : 2, "label" : "2" } Process finished with exit code 0 ``` The complete inference configuration in YAML format is as follows. ```java System.out.format(inferenceConfiguration.toYaml()); ``` ```aspnet --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "IMAGE_TO_NDARRAY" config: height: 28 width: 28 dataType: "FLOAT" includeMinibatchDim: true aspectRatioHandling: "CENTER_CROP" format: "CHANNELS_FIRST" channelLayout: "GRAYSCALE" normalization: type: "SCALE" listHandling: "NONE" keys: - "image" outputNames: - "input_layer" keepOtherValues: true metadata: false metadataKey: "@ImageToNDArrayStepMetadata" - '@type': "ND4JTENSORFLOW" input_names: - "input_layer" output_names: - "output_layer/Softmax" constants: {} model_uri: "/home/zulfadzli/KS2/konduit-serving-examples/java/target/classes/models/tensorflow/mnist/tensorflow.pb" - '@type': "CLASSIFIER_OUTPUT" input_name: "output_layer/Softmax" return_label: true return_index: true return_prob: true label_name: "label" index_name: "index" prob_name: "prob" labels: - "0" - "1" - "2" - "3" - "4" - "5" - "6" - "7" - "8" - "9" all_probabilities: false ``` # Sequence Pipeline Example of sequence pipeline In this example, you'll create the configuration for a server. The same as the previous example, you'll print the configuration to demonstrate the step in details that will help you see the difference and notice the configuration contents. * Let's create a configuration by adding logging step in a sequence pipeline and print the output to JSON: ```java SequencePipeline sequencePipelineWithLoggingStep = SequencePipeline .builder() .add(new LoggingStep()) //add logging step into pipeline .build(); System.out.format("----------%n" + "Pipeline with a Logging step output%n" + "------------%n" + "%s%n" + "------------%n%n", sequencePipelineWithLoggingStep.toJson()); ``` * Call for default or empty inference configuration and print the output to JSON: ```java InferenceConfiguration defaultInferenceConfiguration = new InferenceConfiguration(); System.out.format("----------%n" + "Default inference configuration%n" + "------------%n" + "%s%n" + "------------%n%n", defaultInferenceConfiguration.toJson()); ``` * Combine both to create complete configuration for a server: ```java InferenceConfiguration inferenceConfigurationWithPipeline = new InferenceConfiguration(); inferenceConfigurationWithPipeline.pipeline(sequencePipelineWithLoggingStep); // Printing InferenceConfiguration in YAML System.out.format("----------%n" + "Inference Configuration in YAML%n" + "------------%n" + "%s%n" + "------------%n%n", inferenceConfigurationWithPipeline.toYaml()); ``` You'll see the output similar to: ```aspnet ---------- Pipeline with a Logging step output ------------ { "steps" : [ { "@type" : "LOGGING", "logLevel" : "INFO", "log" : "KEYS" } ] } ------------ ---------- Default inference configuration ------------ { "host" : "localhost", "port" : 0, "useSsl" : false, "protocol" : "HTTP", "staticContentRoot" : "static-content", "staticContentUrl" : "/static-content", "staticContentIndexPage" : "/index.html", "kafkaConfiguration" : { "startHttpServerForKafka" : true, "httpKafkaHost" : "localhost", "httpKafkaPort" : 0, "consumerTopicName" : "inference-in", "consumerKeyDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerValueDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerGroupId" : "konduit-serving-consumer-group", "consumerAutoOffsetReset" : "earliest", "consumerAutoCommit" : "true", "producerTopicName" : "inference-out", "producerKeySerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerValueSerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerAcks" : "1" }, "mqttConfiguration" : { }, "customEndpoints" : [ ] } ------------ ---------- Inference Configuration in YAML ------------ --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "LOGGING" logLevel: "INFO" log: "KEYS" ------------ Process finished with exit code 0 ``` # Graph Pipeline Coming soon... # Client Coming soon... # Running Predictions Coming soon... # Inspecting a Server Coming soon... # Python Coming soon... # Server # Pipeline Steps # Image To NDArray Step # Python Step # DL4J Step # Sequence Pipeline # Graph Pipeline # Client # Running Predictions # Inspecting a Server # IPython Notebook This document is a simple example of deploying Konduit-Serving with Jupyter Notebook based on a model that you can find on open source. This demonstrates the basics of Konduit-Serving, but you can treat it in advance as you understand the state of the art of Konduit-Serving.
{% hint style="info" %} We provide repository which you be able to clone by:\ `$ git clone` [`https://github.com/ShamsUlAzeem/konduit-serving-demo`](https://github.com/ShamsUlAzeem/konduit-serving-demo) {% endhint %} **Follow** [**Quickstart**](/quickstart) **guide for more information on how to run from the repository !** # Basic Simple example to demonstrate Konduit-Serving In the first example, we’ll use normal operations as a model, but it is deploying on Konduit-Serving. You can give input and get the return of output from the server. This will make your model more straightforward to understand by humans as it can provide direct results. ### Viewing directory structure Let’s run cells with bash in a sub-process by using cell magic command and view files in the current directory that will be used in this demonstration. ```bash %%bash echo "Current directory $(pwd)" && tree ``` The following files are present in our simple python script demo. ``` Current directory /root/konduit/demos/0-python-simple . ├── init_script.py ├── python-simple.ipynb ├── python.yaml └── run_script.py 0 directories, 4 files ``` ### Viewing Python script content The scripts contain a simple initialization script for an add function which loads the main function in the `init_script.py` and executes the incoming input through `run_script.py`. ```bash %%bash less init_script.py ``` You’ll be able to see the following. ``` def add_function(x, y): return x + y ``` Once again, let’s browse through the python script for the calling function from `init_script.py`. ```bash %%bash less run_script.py ``` You’ll notice the script only has a line of code. ```aspnet c = add_function(a, b) ``` ### Viewing the main configuration file The main configuration should define the inputs as `a` and `b` and the output as `c`, just as we've showed in the `run_script.py`. ```bash %%bash less python.yaml ``` The YAML script file is as follows. ```aspnet --- host: "0.0.0.0" pipeline: steps: - '@type': "PYTHON" python_config: append_type: "BEFORE" extra_inputs: {} import_code_path: "init_script.py" python_code_path: "run_script.py" io_inputs: a: python_type: "float" secondary_type: "NONE" type: "DOUBLE" b: python_type: "float" secondary_type: "NONE" type: "DOUBLE" io_outputs: c: python_type: "float" secondary_type: "NONE" type: "DOUBLE" job_suffix: "konduit_job" python_config_type: "CONDA" python_path: "1" environment_name: "base" python_path_resolution: "STATIC" python_inputs: {} python_outputs: {} return_all_inputs: false setup_and_run: false port: 8082 protocol: "HTTP" ``` ### Using the configuration to start a server Now we can use the `konduit serve` command to start the server in background with the given files and configurations. ```bash %%bash konduit serve -rwm --config python.yaml -id server --background ``` You’ll get the message like this. ```aspnet Starting konduit server... Expected classpath: /root/konduit/bin/../konduit.jar INFO: Running command /root/miniconda/jre/bin/java -Dkonduit.logs.file.path=/root/.konduit-serving/command_logs/server.log -Dlogback.configurationFile=/tmp/logback-run_command_13ccd5e27dfe43b1.xml -cp /root/konduit/bin/../konduit.jar ai.konduit.serving.cli.launcher.KonduitServingLauncher run --instances 1 -s inference -c python.yaml -Dserving.id=server For server status, execute: 'konduit list' For logs, execute: 'konduit logs server' ``` ### Listing the servers We can list the created servers with `konduit list` command ```bash %%bash konduit list ``` The ID’s server lists like below, giving the status of the server. ```aspnet Listing konduit servers... # | ID | TYPE | URL | PID | STATUS 1 | server | inference | 0.0.0.0:8082 | 421 | started ``` ### Viewing logs Logs can be viewed for the server with an ID of `server` through running `konduit logs server ..` command. ```bash %%bash konduit logs server --lines 1000 ``` Logs output of started server: ```aspnet 09:44:17.852 [main] INFO a.k.s.c.l.command.KonduitRunCommand - Processing configuration: /root/konduit/demos/0-python-simple/python.yaml . . . 09:44:19.436 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### 09:44:19.436 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - Pending server start, please wait... . . . 09:44:19.589 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: '0.0.0.0' 09:44:19.589 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 8082 with 1 pipeline steps ``` ### Sending inputs Now we’ll be able to send the inputs for inferring the output. ```bash %%bash konduit predict server '{"a":1,"b":2}' ``` The output result of the function deployed from the server. ```aspnet { "c" : 3.0 } ``` ### Stopping the server Stop the server by giving the ID’s we want to terminate. ```bash %%bash konduit stop server ``` Status of the server will be printed out as below. ```aspnet Stopping konduit server 'server' Application 'server' terminated with status 0 ``` As you can see from this example, we only use a simple function to deploy in Konduit-Serving. Next, we'll deploy the model in Konduit-Serving. # ONNX Running classifier through CUSTOM endpoints Here are some documents used the Open Neural Network Exchange (ONNX) framework: {% content-ref url="/pages/-MXWU\_wegf7LqcQ8tML5" %} [Pytorch (IRIS)](/examples/ipython-notebook/onnx/onnx-pytorch-iris) {% endcontent-ref %} {% content-ref url="/pages/-MXWV35slQn-UAJ6CIWe" %} [Pytorch (MNIST)](/examples/ipython-notebook/onnx/onnx-pytorch-mnist) {% endcontent-ref %} # Pytorch (IRIS) Running and IRIS dataset classifier through CUSTOM endpoints ### Overview This documentation shows Konduit-Serving can serve a custom model and include post-processing in the pipeline to give a direct output label understood by a human. Iris model is used in this example to deploy on the server as a classifier through custom endpoints. ### Adding package to the classpath First of all we need to add the main package to the classpath so that the notebook can load all the necessary libraries from Konduit-Serving into the Jupyter Notebook kernel. Classpaths can be considered similar to `site-packages` in the python ecosystem where each library that's to be imported to your code is loaded from. We package almost everything you need to get started with the `konduit.jar` package so you can just start working on the actual code, without having to care about any boilerplate configuration. ```bash %classpath add jar ../../konduit.jar ``` Let's ensure the working directory is correct and list all the file available in the directory. ```bash %%bash echo "Current directory $(pwd)" && tree ``` You'll be able to view as the following. ``` Current directory /root/konduit/demos/1-pytorch-onnx-iris . ├── dataset │ └── iris.csv ├── iris.onnx ├── onnx-iris.ipynb ├── onnx.yaml └── train.py 1 directory, 5 files ``` ### Main model script code We're creating a Pytorch model from scratch here and then converting that into ONNX format. ```bash %%bash less train.py ``` You'll be able to browse the source code how the training takes places. ``` import pandas as pd from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score, precision_score, recall_score import torch import torch.nn as nn import torch.nn.functional as F from torch.autograd import Variable class Net(nn.Module): # define nn def __init__(self): super(Net, self).__init__() self.fc1 = nn.Linear(4, 100) self.fc2 = nn.Linear(100, 100) self.fc3 = nn.Linear(100, 3) self.softmax = nn.Softmax(dim=1) def forward(self, X): X = F.relu(self.fc1(X)) X = self.fc2(X) X = self.fc3(X) X = self.softmax(X) return X # load IRIS dataset dataset = pd.read_csv('dataset/iris.csv') # transform species to numerics dataset.loc[dataset.species == 'Iris-setosa', 'species'] = 0 dataset.loc[dataset.species == 'Iris-versicolor', 'species'] = 1 dataset.loc[dataset.species == 'Iris-virginica', 'species'] = 2 train_X, test_X, train_y, test_y = train_test_split(dataset[dataset.columns[0:4]].values, dataset.species.values, test_size=0.8) # wrap up with Variable in pytorch train_X = Variable(torch.Tensor(train_X).float()) test_X = Variable(torch.Tensor(test_X).float()) train_y = Variable(torch.Tensor(train_y).long()) test_y = Variable(torch.Tensor(test_y).long()) net = Net() criterion = nn.CrossEntropyLoss() # cross entropy loss optimizer = torch.optim.SGD(net.parameters(), lr=0.01) for epoch in range(1000): optimizer.zero_grad() out = net(train_X) loss = criterion(out, train_y) loss.backward() optimizer.step() if epoch % 100 == 0: print('number of epoch', epoch, 'loss', loss.item()) predict_out = net(test_X) _, predict_y = torch.max(predict_out, 1) print('prediction accuracy', accuracy_score(test_y.data, predict_y.data)) print('macro precision', precision_score(test_y.data, predict_y.data, average='macro')) print('micro precision', precision_score(test_y.data, predict_y.data, average='micro')) print('macro recall', recall_score(test_y.data, predict_y.data, average='macro')) print('micro recall', recall_score(test_y.data, predict_y.data, average='micro')) # Input to the model x = torch.randn(1, 4, requires_grad=True) # Export the model torch.onnx.export(net, # model being run x, # model input (or a tuple for multiple inputs) "iris.onnx", # where to save the model (can be a file or file-like object) export_params=True, # store the trained parameter weights inside the model file opset_version=10, # the ONNX version to export the model to do_constant_folding=True, # whether to execute constant folding for optimization input_names=['input'], # the model's input names output_names=['output'], # the model's output names dynamic_axes={'input': {0: 'batch_size'}, # variable length axes 'output': {0: 'batch_size'}}) ``` ### Viewing the configuration file The configuration for the custom endpoint is as follow: ```bash %%bash less onnx.yaml ``` The output shows configurations in YAML, in which you can see two steps in the pipeline, serving a model and post-processing to make it directly understood by a human. ``` --- host: "localhost" port: 0 protocol: "HTTP" pipeline: steps: - '@type': "ONNX" modelUri: "iris.onnx" inputNames: - "input" outputNames: - "output" - '@type': "CLASSIFIER_OUTPUT" input_name: "output" labels: - Setosa - Versicolor - Virginica ``` ### Starting the server `konduit serve` can be used together with ID's name (use any name) and configuration file to start the server. ```bash %%bash konduit serve -id onnx-iris -c onnx.yaml -rwm -b ``` You'll get the following message. ``` Starting konduit server... Using classpath: /root/konduit/bin/../konduit.jar INFO: Running command /root/miniconda/jre/bin/java -Dkonduit.logs.file.path=/root/.konduit-serving/command_logs/onnx-iris.log -Dlogback.configurationFile=/tmp/logback-run_command_80a3902b721c4c3f.xml -jar /root/konduit/bin/../konduit.jar run --instances 1 -s inference -c onnx.yaml -Dserving.id=onnx-iris For server status, execute: 'konduit list' For logs, execute: 'konduit logs onnx-iris' ``` To view the logs of the running server, use `konduit logs` commands. ```bash %%bash konduit logs onnx-iris -l 100 ``` You'll be able to see the following from logging. ``` 15:01:50.334 [main] INFO a.k.s.c.l.command.KonduitRunCommand - Processing configuration: /root/konduit/demos/1-pytorch-onnx-iris/onnx.yaml . . . #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### . . . 15:01:50.919 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:01:50.919 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 35761 with 2 pipeline steps ``` ### Sending inputs Now we can send our inputs through `cURL` for inference ``` %%bash konduit predict onnx-iris "{\"input\":[[5.1,3.5,1.4,0.2]]}" ``` So, the server will print out the output with a label. ``` { "output" : [ [ 0.99312085, 0.0068791825, 6.1220806E-9 ] ], "prob" : 0.9931208491325378, "index" : 0, "label" : "Setosa" } ``` Try once again with `--input-type` flag. ``` %%bash konduit predict onnx-iris --input-type multipart "input=[[5.1,3.5,1.4,0.2]]" ``` You'll see the output of the prediction. ``` { "output" : [ [ 0.99312085, 0.0068791825, 6.1220806E-9 ] ], "prob" : 0.9931208491325378, "index" : 0, "label" : "Setosa" } ``` ### Stopping the server Now after we're done with the server, we can stop it through the `konduit stop` command ``` %%bash konduit stop onnx-iris ``` You'll receive this once the server is terminated. ``` Stopping konduit server 'onnx-iris' Application 'onnx-iris' terminated with status 0 ``` Let's take a look at the following example, where we are going to give an image input and doing pre-processing step before fetching it into the model. # Pytorch (MNIST) Running and MNIST dataset classifier through CUSTOM image endpoints ### Overview This example shows a complete step in the pipeline through custom endpoints: 1. Pre-processing step 2. Serve a model step 3. Post-processing step ### Adding package to the classpath We need to add the main package to the classpath so that the notebook can load all the necessary libraries from Konduit-Serving into the Jupyter Notebook kernel. ``` %classpath add jar ../../konduit.jar ``` {% hint style="info" %} Classpaths can be considered similar to `site-packages` in the python ecosystem where each library that's to be imported to your code is loaded from. {% endhint %} ### Viewing the configuration file To view the configuration file contents, use the following command and select the JSON file you want to view. ``` %%bash less config.json ``` You'll be able to view the following. ``` { "host" : "localhost", "port" : 0, "protocol" : "HTTP", "pipeline" : { "steps" : [ { "@type" : "IMAGE_TO_NDARRAY", "config" : { "height" : 28, "width" : 28, "dataType" : "FLOAT", "includeMinibatchDim" : true, "aspectRatioHandling" : "CENTER_CROP", "format" : "CHANNELS_FIRST", "channelLayout" : "GRAYSCALE", "normalization" : { "type" : "SCALE" }, "listHandling" : "NONE" }, "keys" : [ "image" ], "outputNames" : [ "Input3" ], "keepOtherValues" : true, "metadata" : false, "metadataKey" : "@ImageToNDArrayStepMetadata" }, { "@type" : "LOGGING", "logLevel" : "INFO", "log" : "KEYS_AND_VALUES" }, { "@type" : "ONNX", "modelUri" : "mnist.onnx", "inputNames" : [ "Input3" ], "outputNames" : [ "Plus214_Output_0" ] }, { "@type" : "CLASSIFIER_OUTPUT", "inputName" : "Plus214_Output_0", "labels" : [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" ], "allProbabilities" : false } ] } } ``` ### Starting a server Starts a server in the background with an id of `onnx-mnist` using `config.json` as configuration file without creating the manifest jar file before launching the server. ``` %%bash konduit serve -id onnx-mnist -c config.json -rwm -b ``` You'll get the following message once the server starts in the background. ``` Starting konduit server... Using classpath: /root/konduit/bin/../konduit.jar INFO: Running command /root/miniconda/jre/bin/java -Dkonduit.logs.file.path=/root/.konduit-serving/command_logs/onnx-mnist.log -Dlogback.configurationFile=/tmp/logback-run_command_2ead2d4d1b15431d.xml -jar /root/konduit/bin/../konduit.jar run --instances 1 -s inference -c config.json -Dserving.id=onnx-mnist For server status, execute: 'konduit list' For logs, execute: 'konduit logs onnx-mnist' ``` We use `konduit list` command to view the list of activated server ``` %%bash konduit list ``` The list of the activated server is below. ``` Listing konduit servers... # | ID | TYPE | URL | PID | STATUS 1 | onnx-mnist | inference | localhost:33895 | 888 | started ``` To view the logs of the running server, use `konduit logs` commands. ``` %%bash konduit logs onnx-mnist --lines 100 ``` The output of server logging. ``` 14:59:45.188 [main] INFO a.k.s.c.l.command.KonduitRunCommand - Processing configuration: /root/konduit/demos/2-pytorch-onnx-mnist/config.json . . . #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### 14:59:45.601 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - Pending server start, please wait... . . . 14:59:45.733 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 14:59:45.733 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 33895 with 4 pipeline steps ``` ### Making a prediction Let's display the test image before feeding it as an input into the model for the classification. ``` %%html title ``` Here is the test image used in the prediction for this example: ![](https://215936813-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-MX_XSxQdp5o18ffk4Qd%2F-MX_XiuG3hnZWKK28zco%2Ftest-image.jpg?alt=media\&token=b536cee0-1c47-4864-901d-0cc60c950454) `konduit predict` command is used to classify the image based on the model served in Konduit-Serving with the id given before. ``` %%bash konduit predict onnx-mnist --input-type multipart 'image=@test-image.jpg' ``` You'll be able to get the output similar to the following. ``` { "Plus214_Output_0" : [ [ -1.7924803, -9.652266, 11.478509, 5.148998, -7.9367347, 9.756878, 0.544513, -7.6820283, 9.234719, -6.431969 ] ], "prob" : 11.478508949279785, "index" : 2, "label" : "2" } ``` ### Stopping the server After we're finished with the server, we can terminate it through the `konduit stop` command. ``` %%bash konduit stop onnx-mnist ``` You'll receive this message once the server is terminated. ``` Stopping konduit server 'onnx-mnist' Application 'onnx-mnist' terminated with status 0 ``` # Keras Example of Keras framework with CUSTOM endpoints ### Adding package to the classpath Firstly, we require to include the main package to the classpath so that the notebook can load every one of the important libraries from Konduit-Serving into the Jupyter notebook kernel. ``` %classpath add jar ../../konduit.jar ``` {% hint style="info" %} Classpaths can be considered similar to `site-packages` in the python ecosystem where each library that's to be imported to your code is loaded from. {% endhint %} ### Viewing Python script code We're creating a Keras model from scratch here and then converting that into .h5 (HDF5) format. ``` %%bash less train.py ``` You can view and follow through the code to get more information on how the model is trained. ``` import tensorflow as tf from keras.datasets import mnist tensorflow_version = tf.__version__ print(tensorflow_version) # Load data train_data, test_data = mnist.load_data() x_train, y_train = train_data x_test, y_test = test_data # Normalize x_train = x_train / 255.0 x_test = x_test / 255.0 def get_model(): inputs = tf.keras.layers.Input(shape=(28, 28), name="input_layer") x = tf.keras.layers.Flatten()(inputs) x = tf.keras.layers.Dense(200, activation="relu")(x) x = tf.keras.layers.Dense(100, activation="relu")(x) x = tf.keras.layers.Dense(60, activation="relu")(x) x = tf.keras.layers.Dense(30, activation="relu")(x) outputs = tf.keras.layers.Dense(10, activation="softmax", name="output_layer")(x) model = tf.keras.Model(inputs=inputs, outputs=outputs) model.compile( optimizer='sgd', loss='sparse_categorical_crossentropy', metrics=['accuracy'] ) return model def train(epochs=8): model = get_model() model.fit(x_train, y_train, epochs=epochs) model.summary() print("\n\n---\n" "Inputs: {}".format(model.inputs)) print("Outputs: {}\n---".format(model.outputs)) return model train(8).save("keras.h5", save_format="h5") ``` ### Starting a server Let's start to serve the model in Konduit-Serving. ``` %%bash konduit serve -id keras-mnist -c keras.json -rwm -b ``` You'll be able to see a similar output like below. ``` Starting konduit server... Expected classpath: /root/konduit/bin/../konduit.jar INFO: Running command /root/miniconda/jre/bin/java -Dkonduit.logs.file.path=/root/.konduit-serving/command_logs/keras-mnist.log -Dlogback.configurationFile=/tmp/logback-run_command_4c3da934e0334efc.xml -cp /root/konduit/bin/../konduit.jar ai.konduit.serving.cli.launcher.KonduitServingLauncher run --instances 1 -s inference -c keras.json -Dserving.id=keras-mnist For server status, execute: 'konduit list' For logs, execute: 'konduit logs keras-mnist' ``` List the active servers available by using `konduit list` command. ``` %%bash konduit list ``` You'll see the following list of the active Konduit servers. ``` Listing konduit servers... # | ID | TYPE | URL | PID | STATUS 1 | keras-mnist | inference | localhost:33997 | 24142 | started ``` View the logs for the last 100 lines for a given id by using the `konduit logs` command. ``` %%bash konduit logs keras-mnist --lines 100 ``` Logging is printed on the notebook once you run the above command. ``` . . . #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### . . . 15:15:13.074 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:15:13.074 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 33997 with 4 pipeline steps ``` ### Feeding an input to test the model View the test image before testing the model ``` %%html title ``` We're going to use available test image in this directory. ![](https://215936813-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-MX_XSxQdp5o18ffk4Qd%2F-MX_XiuG3hnZWKK28zco%2Ftest-image.jpg?alt=media\&token=b536cee0-1c47-4864-901d-0cc60c950454) Let's predict the output from the server with the above input image. ``` %%bash konduit predict keras-mnist -it multipart 'image=@test-image.jpg' ``` The output of classification: * `output_layer` : probabilities of possible outputs * `prob` : highest probability * `index` : the location of an item in an array * `label` : label of image classification ``` { "output_layer" : [ [ 9.0376153E-7, 1.0595608E-8, 1.3115231E-5, 0.44657645, 6.748624E-12, 0.5524258, 1.848306E-7, 2.7652052E-9, 9.76023E-4, 7.5933513E-6 ] ], "prob" : 0.5524258017539978, "index" : 5, "label" : "5" } ``` ### Stopping the server Once we're finished with the server, we can stop using the `konduit stop` command following the id's server. ``` %%bash konduit stop keras-mnist ``` You'll be able to see the following message. ``` Stopping konduit server 'keras-mnist' Application 'keras-mnist' terminated with status 0 ``` # Tensorflow Example of Tensorflow framework with CUSTOM endpoints ### Overview In this example, we demonstrate Konduit-Serving with a complete pipeline step consist of : 1. Pre-processing step 2. Running a deep learning model 3. Post-processing step, expressing the output in a way human can understand ### Adding package to the classpaths Let's add the main package of Konduit-Serving so that the notebook can load all the required libraries that need to be used by Jupyter Notebook kernel. ``` %classpath add jar ../../konduit.jar ``` {% hint style="info" %} Classpaths can be considered similar to `site-packages` in the python ecosystem where each library that's to be imported to your code is loaded from. {% endhint %} ### Starting a server Before starting a server, let's check if there is a running server with id `tensorflow-mnist` and stop it. This command may use once the server finished. ``` %%bash konduit stop tensorflow-mnist ``` You'll get the following message if there is no server running with mentioned id. ``` No konduit server exists with an id: 'tensorflow-mnist'. ``` Or, if you have a running server you'll received as following. ``` Stopping konduit server 'tensorflow-mnist' Application 'tensorflow-mnist' terminated with status 0 ``` Now, let's start the server with an id of `tensorflow-mnist` using `tensorflow.json` as a configuration file in the background without creating the manifest jar file before launching the server. ``` %%bash konduit serve -id tensorflow-mnist -c tensorflow.json -rwm --background ``` You'll be able to view a similar message like below. ``` Starting konduit server... Using classpath: /root/konduit/bin/../konduit.jar INFO: Running command /root/miniconda/jre/bin/java -Dkonduit.logs.file.path=/root/.konduit-serving/command_logs/tensorflow-mnist.log -Dlogback.configurationFile=/tmp/logback-run_command_5fd1b7c309d448ea.xml -jar /root/konduit/bin/../konduit.jar run --instances 1 -s inference -c tensorflow.json -Dserving.id=tensorflow-mnist For server status, execute: 'konduit list' For logs, execute: 'konduit logs tensorflow-mnist' ``` View the logs for the last 1000 lines `-l` for a given id by using the `konduit logs` command. ``` %%bash konduit logs tensorflow-mnist -l 1000 ``` The output of log is similar as following. ``` 08:23:16.423 [main] INFO a.k.s.c.l.command.KonduitRunCommand - Processing configuration: /root/konduit/demos/4-tensorflow-mnist/tensorflow.json . . . #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### 08:23:17.982 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - Pending server start, please wait... . . . 08:23:18.145 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: '0.0.0.0' 08:23:18.145 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 9008 with 4 pipeline steps ``` ### Sending an input to served model We can display all the available image for the inference result of the model. You'll be able to see the picture from zero to nine. ``` %%html
``` Let's take one of the testing images and send it to the served model in Konduit-Serving. With the help of the pipeline considered in the configuration, we could translate the image into an array and feed it into the model. ``` %%bash konduit predict tensorflow-mnist --input-type multipart "image=@test_files/test_input_number_9.png" ``` Thus, giving a result straight forward with the label of number classification based on prediction probabilities. ``` { "output_layer/Softmax" : [ [ 3.0811898E-7, 6.085964E-6, 1.1470697E-4, 1.5436264E-9, 0.0023717284, 1.7763212E-12, 6.587209E-11, 0.99487466, 4.904844E-11, 0.0026325122 ] ], "prob" : 0.9948746562004089, "index" : 7, "label" : "7" } ``` # DL4J Example of DL4J framework with CUSTOM endpoints ### Including package to the classpath Before starting to serve the model, let's add the main package to the classpath to load the whole necessary libraries to Jupyter Notebook kernel from Konduit-Serving. ``` %classpath add jar ../../konduit.jar ``` {% hint style="info" %} Classpaths can be considered similar to `site-packages` in the python ecosystem. It is loaded from each library that's to be imported to your code. {% endhint %} ### Starting a server Let's start a server with an id of `dl4j-mnist` and use `dl4j.json` as the configuration file. ``` %%bash konduit serve -id dl4j-mnist -c dl4j.json -rwm -b ``` You'll notice with the following message indicating the server is starting. ``` Starting konduit server... Using classpath: /root/konduit/bin/../konduit.jar INFO: Running command /root/miniconda/jre/bin/java -Dkonduit.logs.file.path=/root/.konduit-serving/command_logs/dl4j-mnist.log -Dlogback.configurationFile=/tmp/logback-run_command_a6000ad26ed94583.xml -jar /root/konduit/bin/../konduit.jar run --instances 1 -s inference -c dl4j.json -Dserving.id=dl4j-mnist For server status, execute: 'konduit list' For logs, execute: 'konduit logs dl4j-mnist' ``` {% hint style="info" %} The DL4J model is taken from the dl4j-example here: {% endhint %} Use `konduit logs` to get the logs of served model. ``` %%bash konduit logs dl4j-mnist -l 100 ``` The output of logging is similar to the below. ``` . . . 15:00:54.683 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### 15:00:54.683 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - Pending server start, please wait... 15:00:54.703 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - MetricsProvider implementation detected, adding endpoint /metrics 15:00:54.718 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - No GPU binaries found. Selecting and scraping only CPU metrics. 15:00:54.861 [vert.x-eventloop-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - Writing inspection data at '/root/.konduit-serving/servers/1517.data' with configuration: . . . 15:00:54.862 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:00:54.862 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 39487 with 4 pipeline steps ``` We'll be able to use `konduit list` command to view all active servers. ``` %%bash konduit list ``` These are examples of active servers if the previous one is still in use. ``` Listing konduit servers... # | ID | TYPE | URL | PID | STATUS 1 | keras-mnist | inference | localhost:33387 | 31757 | started 2 | dl4j-mnist | inference | localhost:35921 | 31893 | started ``` ### Sending an input to served model We're going to display the test image first before feeding it into the model. ``` %%html ``` The previous image is used as the testing image for this deployed model: ![](https://215936813-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsGy_78jsGh0h_1MndS%2F-MX_XSxQdp5o18ffk4Qd%2F-MX_XiuG3hnZWKK28zco%2Ftest-image.jpg?alt=media\&token=b536cee0-1c47-4864-901d-0cc60c950454) Now, let's predict the input by using the test image above. ``` %%bash konduit predict dl4j-mnist -it multipart "image=@test-image.jpg" ``` You'll see the following output with the label of classification. ``` { "layer5" : [ [ 1.845163E-5, 1.8346094E-6, 0.31436875, 0.43937472, 2.6101702E-8, 0.24587035, 5.9430695E-6, 3.3270408E-4, 6.3698195E-8, 2.708706E-5 ] ], "prob" : 0.439374715089798, "index" : 3, "label" : "3" } ``` ### Stopping the server We can stop the running server by using `konduit stop` command. ``` %%bash konduit stop dl4j-mnist ``` You'll see this output once the mentioned id's server is terminated. ``` Stopping konduit server 'dl4j-mnist' Application 'dl4j-mnist' terminated with status 0 ``` # CLI # Use-Cases In this use-cases, we'll apply simple application using `CLI` command of Konduit-Serving. Don't worry, these simple example be able to follow by any level experienced user. If you are not install **Konduit-Serving** yet, you can follow [Quickstart](/quickstart) guide. Here are the articles in this section: {% content-ref url="/pages/-MUvuMrejKwhl7k4P1Un" %} [Creating a Sequence Pipeline](/examples/cli/use-cases/creating-a-sequence-pipeline) {% endcontent-ref %} {% content-ref url="/pages/-MUvuDm7HID4Bj9vrJkQ" %} [Creating a Graph Pipeline](/examples/cli/use-cases/creating-a-graph-pipeline) {% endcontent-ref %} # Creating a Sequence Pipeline To create boilerplate configurations A Sequence Pipeline is used to treat the data and Machine Learning or Deep Learning model in a series of steps from pre-processing to model serving and post-processing on the output product. In this example, the `CLI` command specifies on `konduit config` is used to configure the configuration file to serve the models on `Konduit-Serving`. You'll be able to follow this example on your local terminal on any directory. If deploying the model does not need pre- nor post-processing, only one step, a deep learning model is needed. This configuration is defined using a single Step to serve a model, and the command for creating the configuration file is like the following. ``` $ konduit config --pipeline dl4j --output config_dl4j.yaml --yaml ``` The YAML configuration is as follows. ``` --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: steps: - '@type': "DEEPLEARNING4J" modelUri: "" inputNames: - "1" - "2" outputNames: - "11" - "22" ``` The Steps are included in the `--pipeline` based on the model's requirement and how output should represent. For example, the model fetches an image input, so the `image_to_ndarray` should be pre-processing step to convert the image into an array. The table below shows all steps that can be used in the Sequence Pipeline. | Pre-processing Step | Model/Python Step | Post-processing Step | Logging | | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | |
  • image\_to\_ndarray
|

  • dl4j
  • keras
  • tensorflow
  • nd4jtensorflow
  • onnx
  • samediff
  • python
|
  • crop\_grid
  • crop\_fixed*grip
  • draw\_bounding\_box
  • draw\_fixed*grid
  • draw\_segmentation
  • extract\_bounding\_box
  • camera\_frame\_capture
  • video\_frame\_capture
  • ssd\_to\_bounding\_box
  • show\_image
  • classifier\_output
|

  • logging
| Here is another example with a series of step in Sequence Pipeline (`image_to_ndarray` to `nd4jtensorflow` to `classifier_output`). The input image needs to convert into an *n-*D array before feeding into the model and produce the classification output. A command likes below: ``` $ konduit config -p image_to_ndarray,nd4jtensorflow,classifier_output -o config.json ``` The command would give the configuration file in JSON with the complete Pipeline Steps. ``` { "host" : "localhost", "port" : 0, "useSsl" : false, "protocol" : "HTTP", "staticContentRoot" : "static-content", "staticContentUrl" : "/static-content", "staticContentIndexPage" : "/index.html", "kafkaConfiguration" : { "startHttpServerForKafka" : true, "httpKafkaHost" : "localhost", "httpKafkaPort" : 0, "consumerTopicName" : "inference-in", "consumerKeyDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerValueDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerGroupId" : "konduit-serving-consumer-group", "consumerAutoOffsetReset" : "earliest", "consumerAutoCommit" : "true", "producerTopicName" : "inference-out", "producerKeySerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerValueSerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerAcks" : "1" }, "mqttConfiguration" : { }, "customEndpoints" : [ ], "pipeline" : { "steps" : [ { "@type" : "IMAGE_TO_NDARRAY", "config" : { "height" : 100, "width" : 100, "dataType" : "FLOAT", "includeMinibatchDim" : true, "aspectRatioHandling" : "CENTER_CROP", "format" : "CHANNELS_FIRST", "channelLayout" : "RGB", "normalization" : { "type" : "SCALE" }, "listHandling" : "NONE" }, "keys" : [ "key1", "key2" ], "outputNames" : [ "output1", "output2" ], "keepOtherValues" : true, "metadata" : false, "metadataKey" : "@ImageToNDArrayStepMetadata" }, { "@type" : "ND4JTENSORFLOW", "inputNames" : [ "1", "2" ], "outputNames" : [ "11", "22" ], "modelUri" : "" }, { "@type" : "CLASSIFIER_OUTPUT", "inputName" : "inputName (optional)", "returnLabel" : true, "returnIndex" : true, "returnProb" : true, "labelName" : "label", "indexName" : "index", "probName" : "prob", "labels" : [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" ], "allProbabilities" : false } ] } } ``` {% hint style="info" %} Here is the example which use almost similar steps. You can find the JSON file on {% endhint %} Every Step in the Pipeline needs to modify based on the input characteristics, model configurations and how output should looks like in the end. Using a configuration file allows you to serve the model with Konduit-Serving. # Creating a Graph Pipeline Unlike Sequence Pipeline, a Graph Pipeline gives us more control in managing the pipeline's flow in parallel of steps. Same as the previous example, we'll use the `konduit config` command to configure graph pipeline steps and show the configuration file. In this example, we only focus on the steps with default inference configuration to demonstrate the basis of managing steps. There are five type of steps in configuring the Graph Pipeline: 1. Pipeline steps 2. Switch step (string) 3. Switch step (int) 4. Merge step 5. Any step Konduit-Serving provides flexibility to use multiple models in a single Graph Pipeline. The pipeline should be in single quote format such `'=()'` or `'[output]=()'` for both switches step. The input must be continuously related to the previous output assigned before. If not, the Switches step is used to channel the input through the separate models. Let's generate a configuration that logs the input(1), then flow them through two different frameworks models(2 for tensorflow, 3 for dl4j) and merge the output(4). The configuration file then saves to the JSON format. ``` $ konduit config --pipeline '1=logging(input),2=tensorflow(1),3=dl4j(1),4=merge(2,3)' --output config.json ``` The configuration is as the following. ``` { "host" : "localhost", "port" : 0, "useSsl" : false, "protocol" : "HTTP", "staticContentRoot" : "static-content", "staticContentUrl" : "/static-content", "staticContentIndexPage" : "/index.html", "kafkaConfiguration" : { "startHttpServerForKafka" : true, "httpKafkaHost" : "localhost", "httpKafkaPort" : 0, "consumerTopicName" : "inference-in", "consumerKeyDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerValueDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerGroupId" : "konduit-serving-consumer-group", "consumerAutoOffsetReset" : "earliest", "consumerAutoCommit" : "true", "producerTopicName" : "inference-out", "producerKeySerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerValueSerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerAcks" : "1" }, "mqttConfiguration" : { }, "customEndpoints" : [ ], "pipeline" : { "outputStep" : "4", "steps" : { "1" : { "@type" : "LOGGING", "@input" : "input", "logLevel" : "INFO", "log" : "KEYS_AND_VALUES" }, "2" : { "@type" : "TENSORFLOW", "@input" : "1", "inputNames" : [ "1", "2" ], "outputNames" : [ "11", "22" ], "modelUri" : "" }, "3" : { "@type" : "DEEPLEARNING4J", "@input" : "1", "modelUri" : "", "inputNames" : [ "1", "2" ], "outputNames" : [ "11", "22" ] }, "4" : { "@type" : "MERGE", "@input" : [ "2", "3" ] } } } } ``` Same as the previous example, all the steps still need to configure the setting based on the input, model configuration like input and output layer name and the path to the model's directory. You'll need to modify the configuration based on your setting. Let's try another command example for multiple integer inputs for different kinds of models in YAML configuration file format. In this case, we'll use Switch Step (int) to separate and channel the inputs to TensorFlow model and DL4J model. ``` $ konduit config --pipeline '1=logging(input),[2_1,2_2]=switch(int,select,1),3=tensorflow(2_1),4=dl4j(2_2),5=any(3,4)' --output graph_config.yaml --yaml ``` The YAML configuration file generated from above command. ``` --- host: "localhost" port: 0 use_ssl: false protocol: "HTTP" static_content_root: "static-content" static_content_url: "/static-content" static_content_index_page: "/index.html" kafka_configuration: start_http_server_for_kafka: true http_kafka_host: "localhost" http_kafka_port: 0 consumer_topic_name: "inference-in" consumer_key_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_value_deserializer_class: "io.vertx.kafka.client.serialization.JsonObjectDeserializer" consumer_group_id: "konduit-serving-consumer-group" consumer_auto_offset_reset: "earliest" consumer_auto_commit: "true" producer_topic_name: "inference-out" producer_key_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_value_serializer_class: "io.vertx.kafka.client.serialization.JsonObjectSerializer" producer_acks: "1" mqtt_configuration: {} custom_endpoints: [] pipeline: outputStep: "5" steps: "1": '@type': "LOGGING" '@input': "input" logLevel: "INFO" log: "KEYS_AND_VALUES" "3": '@type': "TENSORFLOW" '@input': "2_1" input_names: - "1" - "2" output_names: - "11" - "22" model_uri: "" "2_1": '@type': "SWITCH_OUTPUT" '@input': "1_switch_9ed8d534" outputNum: 0 "4": '@type': "DEEPLEARNING4J" '@input': "2_2" modelUri: "" inputNames: - "1" - "2" outputNames: - "11" - "22" "2_2": '@type': "SWITCH_OUTPUT" '@input': "1_switch_9ed8d534" outputNum: 1 "5": '@type': "ANY" '@input': - "3" - "4" "1_switch_9ed8d534": '@type': "SWITCH" '@input': "1" switchFn: '@type': "INT_SWITCH" numOutputs: 2 fieldName: "select" ``` If the input is string type,we can change `[2_1,2_2]=switch(int,select,1)` to `[2_1,2_2]=switch(string,select,x:0,y:1,1)` in the command (for Switch step (string). Same goes to three string elements such `[2_1,2_2,2_3]=switch(string,select,x:1,y:2,z:3,1)` ,then the input can passed to another three models. The Graph Pipeline helps manage the inputs and multiple models that can be run in parallel, producing the output in instanced via Konduit-Serving. # Create Server URL with Inspection Queries Coming soon... # Adding Extra Classpaths Coming soon... # Multiple Instances of a Server Coming soon... # Commands # Serve Command Examples of CLI with serve command The`serve` command is used to deploy a Konduit-Serving application which must be followed by configuration file either in JSON or YAML. There are a few other options to use with `serve` command which can be seen through the `konduit serve --help` command. #### Examples The server identifies with an id that can be set using the `--serving-id` or `-id` option, for example: ```bash $ konduit serve -id inf_server -c config.json ``` You'll be able to see the following output (trimmed for brevity): ```bash . . . 16:52:08.812 [vert.x-worker-thread-0] INFO o.d.nn.multilayer.MultiLayerNetwork - Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE] 16:52:08.838 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### 16:52:08.838 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - Pending server start, please wait... . . . . 16:52:09.052 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 16:52:09.052 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 42823 with 4 pipeline steps ``` To start with a specific profile you can run the following command which starts a server in the foreground with an id of 'inf\_server' using 'config.json' as configuration file and GPU profile: ```bash $ konduit serve -id inf_server -c config.json -p GPU ``` To learn more about profiles navigate to the following section: {% content-ref url="/pages/-MVu1Lned7kqUxaiJdzh" %} [Profile Command](/examples/cli/commands/profile-command) {% endcontent-ref %} You’ll see output like this, although the version number, etc. may be different based on your local machine: ```bash Starting konduit server... . . . 16:06:49.704 [vert.x-worker-thread-0] INFO org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for linear algebra: 32 16:06:49.722 [vert.x-worker-thread-0] INFO o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CUDA]; OS: [Linux] 16:06:49.722 [vert.x-worker-thread-0] INFO o.n.l.a.o.e.DefaultOpExecutioner - Cores: [12]; Memory: [5.2GB]; 16:06:49.722 [vert.x-worker-thread-0] INFO o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [CUBLAS] 16:06:49.729 [vert.x-worker-thread-0] INFO o.nd4j.linalg.jcublas.JCublasBackend - ND4J CUDA build version: 11.0.221 16:06:49.730 [vert.x-worker-thread-0] INFO o.nd4j.linalg.jcublas.JCublasBackend - CUDA device 0: [GeForce RTX 2060]; cc: [7.5]; Total memory: [6222839808] 16:06:49.731 [vert.x-worker-thread-0] INFO o.nd4j.linalg.jcublas.JCublasBackend - Backend build information: GCC: "9.3.0" STD version: 201402L CUDA: 11.0.221 DEFAULT_ENGINE: samediff::ENGINE_CUDA HAVE_FLATBUFFERS . . ``` Starts a server in the background with an id of 'inf\_server' using 'config.yaml' as configuration file without creating the manifest jar file before launching the server: ```bash $ konduit serve -id inf_server -c config.yaml -b -rwm ``` The output will be like this showing the server is running in background: ```bash Starting konduit server... Expected classpath: /opt/konduit/bin/../konduit.jar INFO: Running command /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Dkonduit.logs.file.path=/home/zulfadzli/.konduit-serving/command_logs/inf_server.log -Dlogback.configurationFile=/opt/konduit/bin/../conf/logback-run_command.xml -cp /opt/konduit/bin/../konduit.jar ai.konduit.serving.cli.launcher.KonduitServingLauncher run --instances 1 -s inference -c config.json -Dserving.id=inf_server For server status, execute: 'konduit list' For logs, execute: 'konduit logs inf_server' ``` # Logs Command Examples of CLI with logs command The`logs` command can be used to view the logs of a particular Konduit Server, given a server id. There are a couple other options to use with the `logs` command. #### Examples The following command outputs the log file contents of server with an id of 'inf\_server': ```bash $ konduit logs inf_server ``` The output of log file will print the last 10 lines by default: ```bash "labelName" : "label", "indexName" : "index", "probName" : "prob", "labels" : [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" ], "allProbabilities" : false } ] } } 17:18:15.938 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 17:18:15.938 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 42849 with 4 pipeline steps ``` You can also output and tail the log file contents of server with an id of 'inf\_server' by adding `-f` or `--follow` option: ```bash $ konduit logs inf_server --follow ``` You'll notice the output is similar to `konduit logs inf_server` but the text cursor is still in the tail of the printed logs. You can press CTRL + C to exit. To view the last 50 lines of the server logs, run the following command: ```bash $ konduit logs inf_server --lines 50 ``` You'll be able to see the last 50 lines of output ```bash "producerAcks" : "1" }, "mqttConfiguration" : { }, "customEndpoints" : [ ], "pipeline" : { "steps" : [ { "@type" : "IMAGE_TO_NDARRAY", "config" : { "height" : 28, "width" : 28, "dataType" : "FLOAT", "includeMinibatchDim" : true, "aspectRatioHandling" : "CENTER_CROP", "format" : "CHANNELS_FIRST", "channelLayout" : "GRAYSCALE", "normalization" : { "type" : "SCALE" }, "listHandling" : "NONE" }, "keys" : [ "image" ], "outputNames" : [ "layer0" ], "keepOtherValues" : true, "metadata" : false, "metadataKey" : "@ImageToNDArrayStepMetadata" }, { "@type" : "LOGGING", "logLevel" : "INFO", "log" : "KEYS_AND_VALUES" }, { "@type" : "DEEPLEARNING4J", "modelUri" : "dl4j-mnist.zip", "inputNames" : [ "layer0" ], "outputNames" : [ "layer5" ] }, { "@type" : "CLASSIFIER_OUTPUT", "inputName" : "layer5", "returnLabel" : true, "returnIndex" : true, "returnProb" : true, "labelName" : "label", "indexName" : "index", "probName" : "prob", "labels" : [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" ], "allProbabilities" : false } ] } } 17:18:15.938 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 17:18:15.938 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 42849 with 4 pipeline steps ``` # Inspect Command Examples of CLI with inspect command The `inspect` command can be used to inspect the details of a particular Konduit Server based on given the server's id. This command helps in getting the details of a server configuration which can be further filter and formatted through a query string. You can specify the query string with either the `--query` or `-q` option. #### Examples The following command will inspect the whole configuration of server with an id of 'inf\_server': ```bash $ konduit inspect inf_server ``` The command will let you inspect the whole configuration setting based on your JSON/YAML file: ```bash { "host" : "localhost", "port" : 42849, "useSsl" : false, "protocol" : "HTTP", "staticContentRoot" : "static-content", "staticContentUrl" : "/static-content", "staticContentIndexPage" : "/index.html", "kafkaConfiguration" : { "startHttpServerForKafka" : true, "httpKafkaHost" : "localhost", "httpKafkaPort" : 0, "consumerTopicName" : "inference-in", "consumerKeyDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerValueDeserializerClass" : "io.vertx.kafka.client.serialization.JsonObjectDeserializer", "consumerGroupId" : "konduit-serving-consumer-group", "consumerAutoOffsetReset" : "earliest", "consumerAutoCommit" : "true", "producerTopicName" : "inference-out", "producerKeySerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerValueSerializerClass" : "io.vertx.kafka.client.serialization.JsonObjectSerializer", "producerAcks" : "1" }, "mqttConfiguration" : { }, "customEndpoints" : [ ], "pipeline" : { "steps" : [ { "@type" : "IMAGE_TO_NDARRAY", "config" : { "height" : 28, "width" : 28, "dataType" : "FLOAT", "includeMinibatchDim" : true, "aspectRatioHandling" : "CENTER_CROP", "format" : "CHANNELS_FIRST", "channelLayout" : "GRAYSCALE", "normalization" : { "type" : "SCALE" }, "listHandling" : "NONE" }, "keys" : [ "image" ], "outputNames" : [ "layer0" ], "keepOtherValues" : true, "metadata" : false, "metadataKey" : "@ImageToNDArrayStepMetadata" }, { "@type" : "LOGGING", "logLevel" : "INFO", "log" : "KEYS_AND_VALUES" }, { "@type" : "DEEPLEARNING4J", "modelUri" : "dl4j-mnist.zip", "inputNames" : [ "layer0" ], "outputNames" : [ "layer5" ] }, { "@type" : "CLASSIFIER_OUTPUT", "inputName" : "layer5", "returnLabel" : true, "returnIndex" : true, "returnProb" : true, "labelName" : "label", "indexName" : "index", "probName" : "prob", "labels" : [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" ], "allProbabilities" : false } ] } } ``` You can use `--query` command flag to get specific fields of the server configuration. For example, the following command will print the host and port of the server: ```bash $ konduit inspect inf_server --query {host}:{port} ``` You'll get the output based on what you have specified: ```bash localhost:42849 ``` You can also use same command flag to get pipeline details, for example: ```bash $ konduit inspect inf_server --query {host}:{port}-{pipeline} ``` You'll be able to see similar output including pipeline details like this: ```bash localhost:42849-{"steps":[{"@type":"IMAGE_TO_NDARRAY","config":{"height":28,"width":28,"dataType":"FLOAT","includeMinibatchDim":true,"aspectRatioHandling":"CENTER_CROP","format":"CHANNELS_FIRST","channelLayout":"GRAYSCALE","normalization":{"type":"SCALE"},"listHandling":"NONE"},"keys":["image"],"outputNames":["layer0"],"keepOtherValues":true,"metadata":false,"metadataKey":"@ImageToNDArrayStepMetadata"},{"@type":"LOGGING","logLevel":"INFO","log":"KEYS_AND_VALUES"},{"@type":"DEEPLEARNING4J","modelUri":"dl4j-mnist.zip","inputNames":["layer0"],"outputNames":["layer5"]},{"@type":"CLASSIFIER_OUTPUT","inputName":"layer5","returnLabel":true,"returnIndex":true,"returnProb":true,"labelName":"label","indexName":"index","probName":"prob","labels":["0","1","2","3","4","5","6","7","8","9"],"allProbabilities":false}]} ``` # Profile Command Coming soon ... # Serving a BMI Model Custom model in Konduit-Serving with HTML content ### Introduction Body Mass Index (BMI) is a well-used measure to describe weight status based on the ratio between an individual's height and weight. BMI is used to classify an individual's weight status as underweight, normal weight, overweight or obese. Even though this is a simple application to be used, it is necessary to monitor healthcare by providing weight status generally. Findings from [Pursey et al.](https://pubmed.ncbi.nlm.nih.gov/24398335/) and [Stommel et al.](https://bmcpublichealth.biomedcentral.com/articles/10.1186/1471-2458-9-421) showed that adults tend to self-report inaccurate BMI, overestimate their weight, and underestimate the weight. Time-consuming in measuring weight and height also one of the factors that contribute to this issue. Thus, the BMI model is introduced as a new approach for estimating BMI using facial images that contain facial features. The model is deployed in **Konduit-Serving** through the pipeline server from pre-processing step of input until producing output that a human can understand. The backend server used for taking image data and providing BMI values is served using Konduit-Serving, a high-performance model pipeline server. The main workflow we'll look at in this document is how to serve a model that can see at a person's face and respond with a BMI value through REST API. We'll also be setting up a web server through Konduit-Serving "custom endpoints" that will make use of a webcam and label the canvas with the detected face along with the corresponding estimated BMI value. Note that gathering, preparing dataset and model training are out of the scope of this document. Assuming you're on notebook environment and opened `bmi-onnx-pytorch.ipynb`. {% hint style="info" %} The notebook is ready to use from . Please follow [Quickstart](/quickstart/using-docker) guide to run the notebook. {% endhint %} ### Start the server Let's serve the model via Konduit-Serving with provided configuration file and python scripts. The `serve` command is as following, serving id name (can be any other preferred name) is `bmi-onnx-pytorch` and configuration file is `bmi-onnx-pytorch.yaml`. ``` %%bash nohup konduit serve --serving-id bmi-onnx-pytorch --config bmi-onnx-pytorch.yaml & ``` Once the command committed, the server is started in Konduit-Serving. We are using Python Step in Pipeline from the configuration file,, where the step run by python scripts `init_script.py` and `run_script.py`. The step calls the model, `version-RFB-320.onnx` from the model directory, `konduit-serving-demo/demos/6-bmi-oonx-pytorch/models`and perform the BMI classification based on facial feature captured on the test image. You can view any scripts, either configuration or python, by adding the cell on the notebook and run, ``` %%bash less bmi-onnx-pytorch.yaml less run_script.py ``` ### View the logs The command to view the logs from the server is `konduit logs` command, and we'll view the last 300 lines by using `--lines` flag with the command. By default, the konduit logs command will only preview ten lines of the logs. The command is like the following: ``` %%bash konduit logs bmi-onnx-pytorch --lines 300 ``` You can view the logs output similar to below. ``` 06:53:46.317 [main] INFO a.k.s.c.l.command.KonduitRunCommand - Processing configuration: /root/konduit/demos/6-bmi-onnx-pytorch/bmi-onnx-pytorch.yaml . . . #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### . . . 06:53:48.703 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: '0.0.0.0' 06:53:48.703 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 9009 with 2 pipeline steps ``` ### View a list of server The `konduit list` command is used to check the running server from Konduit-Serving. If you serve another model on the server, it will show more than one ID based on the specified given name. Run this command to view the list of the servers. ``` %%bash konduit list ``` ### Make an inference We can display the test image for the inference result of the model. You'll be able to see the picture we provided as the testing image by running the following. ``` %%html ``` Now, we want to make the inference of the testing image. The `konduit predict` command is used to classify the BMI of the person in the testing image and classify the output based on the label of highest probability. ``` %%bash konduit predict bmi-onnx-pytorch --input-type multipart "image=@image_me.jpg" ``` If we are viewing the configuration file, the post-processing (`CLASSIFIER_OUTPUT`) is taking place to classify the output based on labels rendered from the output layer of the served model. You'll get a result similar to the following. ``` { "bmi_value" : 22.18, "boxes" : [ 447.0, 174.0, 636.0, 470.0 ], "predictions" : [ 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "prob" : 1.0, "index" : 1, "bmi_class" : "Normal_Range" } ``` The test image is not limited to provide one, and you can use your testing image to make the inference. Let's do something interesting by following the next step. ### Before stopping the server You can demonstrate the served model with HTML content to view a dashboard and test the model with your camera. Let's move to the sub-section to find more. {% content-ref url="/pages/-MUvqVE0U-1xurtHNOp\_" %} [With HTML Content](/how-to-guides/serving-a-bmi-model/with-html-content) {% endcontent-ref %} ### Stop the server Let's stop the server once finished to serve on Konduit-Serving. ``` %%bash konduit stop bmi-onnx-pytorch ``` As the server is stopped, you'll see a message similar to the following. ``` Stopping konduit server 'bmi-onnx-pytorch' Application 'bmi-onnx-pytorch' terminated with status 0 ``` # With HTML Content Visualizing the deployment with HTML In this section, we're implementing visualization via HTML for: 1. Metrics 2. Web Application ## Metrics The metrics are data related to the process of every execution on Konduit-Serving. The REST API handles the metrics endpoint of Konduit-Serving and returns to the Prometheus. Several metrics return to the metrics endpoint by default, which are: * Time taken for request and execution * CPU usage and current available memory * GPU usage, GPU temperature and GPU current used memory * Result of execution Prometheus and Grafana are used to store the data and visualize the metrics onto the dashboard. ### Prometheus Prometheus is an open-source system monitoring and alerting toolkit widely used in time series database for tracking system metrics used for debugging production systems. The Prometheus collects all the metrics from metrics endpoint and formatting into something that Prometheus can read, present in the form of time series data for every time it polls the server for metrics scraping. A Konduit-Serving instance exposes metrics to be picked up by Prometheus and shows the metrics on `localhost:9090`. ### Grafana Grafana is a dashboard system for pulling data from different sources and displaying it in real-time. It can take the data from multiple sources. One of those is that Grafana takes the data from Konduit-Serving through Prometheus and visualizes all the data on the dashboard. Let's run the HTML code like the following (assuming you're still running the notebook from the demo file). ``` %%html
``` This web application is configured on `static_content` in the YAML configuration file, `bmi-onnx-pytorch.yaml` to implement via HTML. The directory of the HTML file is `web-app/index.html`, which is executing video streaming from the local machine in a small window of running cell and predict BMI from the image captured when the start button is clicked. You'll notice that the dashboard is updating the metrics for each time the prediction is made continuously on the running stream video feed and producing the time series data for all available metrics until the stop button is clicked. # Performing Object Detection # RPA Use-Case # Showing Metrics # Prometheus # Grafana # Pipeline Steps Various type of steps is available in the Pipeline of Konduit-Serving. Konduit-Serving provides various types of steps included in Pipeline from pre-processing, serving Machine Learning or Deep Learning model and post-processing. This section offers all the configurations with descriptions that may help set your Pipeline Steps. The example of `PipelineSteps`: ``` inferenceConfiguration.pipeline(SequencePipeline.builder() .add(new ImageToNDArrayStep() //add ImageToNDArrayStep() into pipeline to set image to NDArray for input .config(new ImageToNDArrayConfig() //image configuration .width(28) .height(28) .dataType(NDArrayType.FLOAT) .aspectRatioHandling(AspectRatioHandling.CENTER_CROP) .includeMinibatchDim(true) .channelLayout(NDChannelLayout.GRAYSCALE) .format(NDFormat.CHANNELS_FIRST) .normalization(ImageNormalization.builder().type(ImageNormalization.Type.SCALE).build()) ) .keys("image") .outputNames("input_layer") .keepOtherValues(true) .metadata(false) .metadataKey(ImageToNDArrayStep.DEFAULT_METADATA_KEY)) .add(new Nd4jTensorFlowStep() //add Nd4jTensorFlowStep into pipeline .modelUri(modelTrainResult.modelPath()) .inputNames(modelTrainResult.inputNames()) .outputNames(modelTrainResult.outputNames()) ).add(new ClassifierOutputStep() .inputName(modelTrainResult.outputNames().get(0)) .labels(Arrays.asList(labels.clone())) .allProbabilities(false) ).build() ); ``` Here are the references in this section: # IMAGE\_TO\_NDARRAY `ImageToNDArrayStep` is a `PipelineStep` for converting images to n-dimensional arrays. The exact way that images are converted is highly configurable (formats, channels, output sizes, normalization, etc). | Configs | Descriptions | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | config | Configuration for how conversion should be performed. | | keys | May be null. If non-null, these are the names of images in the Data instance to convert. | | outputNames | May be null. If non-null, the input images are renamed to this in the output Data instance after conversion to n-dimensional array. | | keepOtherValues | True by default. If true, copy all the other (non-converted/non-image) entries in the input data to the output data. | | metadata | False by default. If true, include metadata about the images in the output data. For example, if/how it was cropped, and the original input size. | | metadataKey | Sets the key that the metadata will be stored under. Not relevant if metadata == `false`. Default is `@ImageToNDArrayStepMetadata` | `ImageToNDArrayConfig` is configuration for converting an image into n-dimensional array. This configuration is used in `config` from `ImageToNDArrayStep`, for example: ``` inferenceConfiguration.pipeline(SequencePipeline.builder() .add(new ImageToNDArrayStep() //add ImageToNDArrayStep() into pipeline to set image to NDArray for input .config(new ImageToNDArrayConfig() //image configuration .width(28) .height(28) .dataType(NDArrayType.FLOAT) .aspectRatioHandling(AspectRatioHandling.CENTER_CROP) .includeMinibatchDim(true) .channelLayout(NDChannelLayout.GRAYSCALE) .format(NDFormat.CHANNELS_FIRST) .normalization(ImageNormalization.builder().type(ImageNormalization.Type.SCALE).build()) ) .keys("image") .outputNames("input_layer") .keepOtherValues(true) .metadata(false) .metadataKey(ImageToNDArrayStep.DEFAULT_METADATA_KEY)) .build() ``` | Configs | Descriptions | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | height | Output array image height. Leave null to convert to the same size as the image height. | | width | Output array image width. Leave null to convert to the same size as the image width. | | dataType | Data type of the n-dimensional array. Default value is `FLOAT`. | | includeMinibatchDim | If true, the output array will contain an extra dimension for the minibatch number. This will look like (1, Channels, Height, Width) instead of (Channels, Height, Width) for format == `CHANNELS_FIRST` or (1, Height, Width, Channels) instead of (Height, Width, Channels) for format == `CHANNELS_LAST`. Default is `true`. | | aspectRatioHandling |

An enum to Handle the situation where the input image and output NDArray have different aspect ratios.

CENTER\_CROP (crop larger dimension then resize if necessary), PAD (pad smaller dimension then resize if necessary), STRETCH (simply resize, distorting if necessary). Default is CENTER\_CROP.

| | format | The format to be used when converting an Image to an NDArray. Default is `CHANNEL_FIRST`. Another option is `CHANNEL_LAST`. | | channelLayout | An enum that represents the type (and order) of the color channels for an image after it has been converted to an NDArray. For example, `RGB` vs. `BGR` etc, default value is `RGB`. | | normalization | Configuration that specifies the normalization type of an image array values. | | listHandling | An enum to specify how to handle a list of input images. Default is `NONE`. | # IMAGE\_CROP `ImageCropStep` is used to crop an image to the specified rectangular region. The crop region may be specified in one of two ways: 1. Via a bounding box, or 2. Via a list of points of length 2, containing the top-left and bottom-right crop locations. These may be specified statically (i.e., fixed crop region) via `cropBox` or `cropPoints` property, or dynamically via `cropName` (which may specify a bounding box or list of point in the input data instance). Furthermore, the bounding box and corner point coordinates may be specified in terms of either pixels or fraction of image - specified via the `coordsArePixels` property. Note that if the crop region falls partly outside the input image region, black padding will be added as necessary to keep the requested output size. | Configs | Descriptions | | --------------- | ------------------------------------------------------------------------------------------------------ | | imageName | Name of the `Image` or in the list, `List` field to crop. | | cropName | Name of the input Data field used for dynamic cropping. May be a `BoundingBox` or `List`. | | cropPoints | Static crop region defined as a list of point, `List`. | | cropBox | Static crop region defined as a bounding box, `BoundingBox`. | | coordsArePixels | Weather the crop region (`BoundingBox` / `List`) are specified in pixels, or fraction of image. | # IMAGE\_RESIZE `ImageResizeStep` is a pipeline step that resizes an image, scaling up or down as needed to comply with the specified output height/width. Usually, both height and width are specified. However, if only one is specified, the other value is calculated based on the aspect ration of the input image. When both height and width are specified, and the aspect ratios of the input doesn't match the aspect ratio of the output, (for example, 100x200 in, 200x200 out) the `aspectRatioHandling` setting is used to determine how to handle this situation. Note that the names of the inputs data fields to resize may or may not be specified. If no value is provided for `inputNames` configuration, all input images fields in the input Data instance will be resized, regardless of name. If `inputNames` is specified, only those fields with those names will be resized. | Configs | Descriptions | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | inputNames | Name of the input keys whose values contain images from the previous step. | | height | Resize height. | | width | Resize width. | | aspectRatioHandling | An enum to define how to handle the aspect ratio when the aspect ratio doesn't match with that of the input image. Default value is `STRETCH`. | --- [Next Page](/llms-full.txt/1)