# Using CLI This document will demonstrate using Konduit-Serving using mainly CLI tools. You can deploy ML/DL models to production using minimal effort using Konduit-Serving. Let's look at the process of building and installing Konduit-Serving from source and how to deploy a model using a simple configuration. ### Prerequisite You will need following prerequisites to follow along * Maven 3.x * JDK 8 * Git ## Installation from Sources The following two sections explains how to clone, build and install Konduit-Serving from sources. To build from source, follow the guide below {% content-ref url="/pages/-LwHP0JVDZsAP328VRoA" %} [Building from source](/building-from-source.md) {% endcontent-ref %} To install the respective built binaries you can navigate to the section below {% content-ref url="/pages/-LtTMd5ReUOXuuHoM\_rH" %} [Installing Binaries](/installation.md) {% endcontent-ref %} After you've installed Konduit-Serving in your local machine you can switch to a terminal and verify the installation by running ``` konduit --version ``` You'll see an output similar to the one below ``` $ konduit --version ------------------------------------------------ Version: 0.1.0-SNAPSHOT Commit hash: 3dd38832 Commit time: 01.03.2021 @ 03:37:08 MYT Build time: 07.03.2021 @ 16:57:51 MYT ``` ## Deploying Models Let's look at how to deploy a dl4j/keras model using Konduit-Serving ### Cloning Examples Repo Let's clone the `konduit-serving-examples` repo ``` git clone https://github.com/KonduitAI/konduit-serving-examples.git ``` and navigate to the `quickstart` folder ``` cd konduit-serving-demo/quickstart ``` The examples we want to run are under the folders `3-keras-mnist` and `5-dl4j-mnist`. Let's follow a basic workflow for both models using the Konduit-Serving CLI. {% tabs %} {% tab title="Keras" %} Navigate to `3-keras-mnist` ``` cd 3-keras-mnist ``` Here, you'll find the following files: ``` . ├── keras-mnist.ipynb | A supplementary jupyter/beakerx notebook ├── keras.h5 | Model file we want to serve ├── keras.json | Konduit-Serving configuration ├── test-image.jpg | Test input for predictions └── train.py | Script for creating 'keras.h5' model file ``` The `keras.json` contains the configuration file for running an MNIST dataset trained model in Keras. To serve the model, execute the following command ```bash konduit serve --config keras.json -id keras-server ``` You'll be able to see a similar output like the following ```bash . . . 15:00:08.575 [vert.x-worker-thread-0] INFO a.k.s.m.d.step.DL4JRunner - 15:00:08.576 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### 15:00:08.576 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - Pending server start, please wait... . . . . 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 40987 with 4 pipeline steps ``` The last line will show you the details about which URL the server is serving the models at. Press `Ctrl + C`, or execute `konduit stop keras-server` to kill the server. To run the server in the background, you can run the same command with the `--background` or `-b` flag. ``` konduit serve --config keras.json -id keras-server --background ``` You'll see something similar to ``` Starting konduit server... Expected classpath: /Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../konduit.jar INFO: Running command /Users/konduit/opt/miniconda3/jre/bin/java -Dkonduit.logs.file.path=/Users/konduit/.konduit-serving/command_logs/keras-server.log -Dlogback.configurationFile=/Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../conf/logback-run_command.xml -cp /Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../konduit.jar ai.konduit.serving.cli.launcher.KonduitServingLauncher run --instances 1 -s inference -c keras.json -Dserving.id=keras-server For server status, execute: 'konduit list' For logs, execute: 'konduit logs keras-server' ``` To list the server, simply run ``` konduit list ``` You'll see the running servers as a list ``` Listing konduit servers... # | ID | TYPE | URL | PID | STATUS 1 | keras-server | inference | localhost:1000 | 1200 | Started ``` To view the logs, you can run the following command ``` konduit logs keras-server --lines 2 ``` The `--lines` or `-l` flag shows the specified number of last lines. By executing the above command you'll see the following ``` 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 1000 with 4 pipeline steps ``` Now finally, let's look at running predictions with Konduit-Serving by sending an image file to the server. ``` konduit predict keras-server -it multipart 'image=@test-image.jpg' ``` It will convert the image into an n-dimensional array and then send the input to the keras model and you'll see the following output ``` { "output_layer" : [ [ 9.0376153E-7, 1.0595608E-8, 1.3115231E-5, 0.44657645, 6.748624E-12, 0.5524258, 1.848306E-7, 2.7652052E-9, 9.76023E-4, 7.5933513E-6 ] ], "prob" : 0.5524258017539978, "index" : 5, "label" : "5" } ``` {% endtab %} {% tab title="DL4J" %} Navigate to `5-dl4j-mnist` ```bash cd 5-dl4j-mnist ``` Here, you'll find the following files: ``` . ├── dl4j-mnist.ipynb | A supplementary jupyter/beakerx notebook ├── dl4j-mnist.zip | Model file we want to serve ├── dl4j.json | Konduit-Serving configuration └── test-image.jpg | Test input for predictions ``` The `dl4j.json` contains the configuration file for running an MNIST dataset trained model in DL4J. To serve the model, execute the following command ```bash konduit serve --config dl4j.json -id dl4j-server ``` You'll be able to see a similar output like the following ```bash . . . 15:00:08.575 [vert.x-worker-thread-0] INFO a.k.s.m.d.step.DL4JRunner - 15:00:08.576 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - #################################################################### # # # | / _ \ \ | _ \ | | _ _| __ __| | / | / # # . < ( | . | | | | | | | . < . < # # _|\_\ \___/ _|\_| ___/ \__/ ___| _| _|\_\ _) _|\_\ _) # # # #################################################################### 15:00:08.576 [vert.x-worker-thread-0] INFO a.k.s.v.verticle.InferenceVerticle - Pending server start, please wait... . . . . 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 40987 with 4 pipeline steps ``` The last line will show you the details about which URL the server is serving the models at. Press `Ctrl + C`, or execute `konduit stop keras-server` to kill the server. To run the server in the background, you can run the same command with the `--background` or `-b` flag. ``` konduit serve --config dl4j.json -id dl4j-server --background ``` You'll see something similar to ``` Starting konduit server... Expected classpath: /Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../konduit.jar INFO: Running command /Users/konduit/opt/miniconda3/jre/bin/java -Dkonduit.logs.file.path=/Users/konduit/.konduit-serving/command_logs/dl4j-server.log -Dlogback.configurationFile=/Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../conf/logback-run_command.xml -cp /Users/konduit/Projects/Konduit/konduit-serving/konduit-serving-tar/target/konduit-serving-tar-0.1.0-SNAPSHOT-dist/bin/../konduit.jar ai.konduit.serving.cli.launcher.KonduitServingLauncher run --instances 1 -s inference -c dl4j.json -Dserving.id=dl4j-server For server status, execute: 'konduit list' For logs, execute: 'konduit logs dl4j-server' ``` To list the server, simply run ``` konduit list ``` You'll see the running servers as a list ``` Listing konduit servers... # | ID | TYPE | URL | PID | STATUS 1 | dl4j-server | inference | localhost:1000 | 1200 | Started ``` To view the logs, you can run the following command ``` konduit logs dl4j-server --lines 2 ``` The `--lines` or `-l` flag shows the specified number of last lines. By executing the above command you'll see the following ``` 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server is listening on host: 'localhost' 15:00:08.752 [vert.x-eventloop-thread-0] INFO a.k.s.v.p.h.v.InferenceVerticleHttp - Inference HTTP server started on port 1000 with 4 pipeline steps ``` Now finally, let's look at running predictions with Konduit-Serving by sending an image file to the server. ``` konduit predict dl4j-server -it multipart 'image=@test-image.jpg' ``` It will convert the image into an n-dimensional array and then send the input to the DL4J model and you'll see the following output ``` { "layer5" : [ [ 1.845163E-5, 1.8346094E-6, 0.31436875, 0.43937472, 2.6101702E-8, 0.24587035, 5.9430695E-6, 3.3270408E-4, 6.3698195E-8, 2.708706E-5 ] ], "prob" : 0.439374715089798, "index" : 3, "label" : "3" } ``` {% endtab %} {% endtabs %} Congratulations! You've learned the basic workflow for Konduit-Serving using the Command Line Interface. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://serving.konduit.ai/quickstart/using-cli.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.