# Keras (TensorFlow 2.0) ```java import ai.konduit.serving.InferenceConfiguration; import ai.konduit.serving.config.ParallelInferenceConfig; import ai.konduit.serving.config.ServingConfig; import ai.konduit.serving.configprovider.KonduitServingMain; import ai.konduit.serving.configprovider.KonduitServingMainArgs; import ai.konduit.serving.model.ModelConfig; import ai.konduit.serving.model.ModelConfigType; import ai.konduit.serving.pipeline.step.ModelStep; import ai.konduit.serving.verticles.inference.InferenceVerticle; import com.mashape.unirest.http.Unirest; import com.mashape.unirest.http.exceptions.UnirestException; import org.apache.commons.io.FileUtils; import org.nd4j.linalg.api.ndarray.INDArray; import org.nd4j.linalg.factory.Nd4j; import org.nd4j.linalg.io.ClassPathResource; import org.nd4j.serde.binary.BinarySerde; ``` ## Saving models in Keras HDF5 (.h5) format Models can be saved using Python with the `.save()` method. Refer to the [TensorFlow documentation for Keras](https://www.tensorflow.org/guide/keras/save_and_serialize) for details. These saved models shall be loaded in Java. {% hint style="info" %} Keras model loading functionality in Konduit Serving converts Keras models to Deeplearning4J models. As a result, Keras models containing operations not supported in Deeplearning4J cannot be served in Konduit Serving. See [issue 8348](https://github.com/eclipse/deeplearning4j/issues/8348). {% endhint %} ## Overview Konduit Serving works by defining a series of **steps**. These include operations such as 1. Pre- or post-processing steps 2. One or more machine learning models 3. Transforming the output in a way that can be understood by humans If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single `ModelStep`. {% hint style="info" %} A reference Java project is provided in the Example repository ( ) with a Maven pom.xml dependencies file. If using the IntelliJ IDEA IDE, open the java folder as a Maven project and run the main function of the InferenceModelStepKeras class. {% endhint %} ## Configure the step Define the Keras configuration as a `ModelConfig` object. * `modelConfigType`: This argument requires a `ModelConfigType` object. Specify `modelType` as `ModelConfig.ModelType.KERAS`, and `modelLoadingPath` to point to the location of Keras weights saved in the HDF5 file format. For the `ModelStep` object, the following parameters are specified: * `modelConfig`: pass the ModelConfig object here * `parallelInferenceConfig`: specify the number of workers to run in parallel. Here, we specify `workers = 1`. * `inputName`, `outputName`: names for the input and output nodes, as lists ```java String kerasmodelfilePath = new ClassPathResource("data/keras/embedding_lstm_tensorflow_2.h5").getFile().getAbsolutePath(); ModelConfig kerasModelConfig = ModelConfig.builder() .modelConfigType(ModelConfigType.builder() .modelLoadingPath(kerasmodelfilePath.toString()) .modelType(ModelConfig.ModelType.KERAS).build()) .build(); ModelStep kerasmodelStep = ModelStep.builder() .modelConfig(kerasModelConfig) .inputName("input") .outputName("lstm_1") .parallelInferenceConfig(ParallelInferenceConfig.builder().workers(1).build()) .build(); ``` {% hint style="info" %} Input and output names can be obtained by visualizing the graph in [Netron](https://github.com/lutzroeder/netron). {% endhint %} ## Configure the server In the `ServingConfig`, specify any port number that is not reserved. ```java int port = Util.randInt(1000, 65535); ServingConfig servingConfig = ServingConfig.builder().httpPort(port). build(); ``` The `ServingConfig` has to be passed to `Server` in addition to the steps as a list. In this case, there is a single step: `kerasmodelStep`. ```java InferenceConfiguration inferenceConfiguration = InferenceConfiguration.builder() .servingConfig(servingConfig) .step(kerasmodelStep) .build(); ``` The `inferenceConfiguration` is stored as a JSON File. Set the KonduitServingMainArgs with the saved **config.json** file path as `configPath` and other necessary server configuration arguments. ```java File configFile = new File("config.json"); FileUtils.write(configFile, inferenceConfiguration.toJson(), Charset.defaultCharset()); KonduitServingMainArgs args1 = KonduitServingMainArgs.builder() .configStoreType("file").ha(false) .multiThreaded(false).configPort(port) .verticleClassName(InferenceVerticle.class.getName()) .configPath(configFile.getAbsolutePath()) .build(); ``` Start server by calling KonduitServingMain with the configurations mentioned in the KonduitServingMainArgs using Callback Function(as per the code mentioned in the **Inference** Section below) ## Inference NDARRAY inputs to set ModelStep must be specified with a shape size. To configure the client, set the required URL to connect server and specify any port number that is not reserved (as used in server configuration). A Callback Function onSuccess is implemented in order to post the Client request and get the HttpResponse, only after the successful run of the KonduitServingMain Server. ```java INDArray arr = Nd4j.create(new float[]{1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f}, 1, 10); File file = new File("src/main/resources/data/test-input.zip"); System.out.println(file.getAbsolutePath()); BinarySerde.writeArrayToDisk(arr, file); KonduitServingMain.builder() .onSuccess(() -> { try { String response = Unirest.post(String.format("http://localhost:%s/raw/nd4j", port)) .field("input", file) .asString().getBody(); System.out.print(response); System.exit(0); } catch (UnirestException e) { e.printStackTrace(); System.exit(0); } }) .build() .runMain(args1.toArgs()); ``` ## Confirm the output After executing the above, in order to confirm the successful start of the Server, check for the below output text: ``` Jan 07, 2020 2:31:37 PM ai.konduit.serving.configprovider.KonduitServingMain INFO: Deployed verticle ai.konduit.serving.verticles.inference.InferenceVerticle ``` The Output of the program is as follows: ```java System.out.print(response) ``` ``` { "lstm_1" : { "batchId" : "61c32b20-20c1-42d9-909e-40519304f2ac", "ndArray" : { "dataType" : "FLOAT", "shape" : [ 1, 6, 10 ], "data" : [ -0.0022880966, -0.0042849067, -0.005983479, -0.0073982426, -0.008555864, -0.009488584, -0.010229964, -0.010812048, -0.011263946, -0.011611057, -0.0027362786, -0.005198746, -0.0072902446, -0.009000374, -0.010361125, -0.011421581, -0.012234278, -0.012848378, -0.013306688, -0.013644846, 8.9187745E-4, 0.0012898755, 0.0014061421, 0.0013690761, 0.0012552886, 0.00110957, 9.573803E-4, 8.1235886E-4, 6.812691E-4, 5.6666887E-4, -0.0029521661, -0.0049125706, -0.0062154937, -0.0070830043, -0.007662085, -0.008049844, -0.008310454, -0.008486291, -0.00860548, -0.008686648, -2.41272E-4, -2.1998871E-4, -3.9213814E-5, 2.2318505E-4, 5.1378313E-4, 7.9911196E-4, 0.0010602918, 0.0012885022, 0.0014812968, 0.00164004, 0.0029523545, 0.0050065047, 0.006426977, 0.007400234, 0.008058914, 0.008497588, 0.008783782, 0.00896551, 0.009076695, 0.009141108 ] } } ``` The complete inference configuration in JSON format is as follows: ```java System.out.println(inferenceConfiguration.toJson()); ``` ``` { "memMapConfig" : null, "servingConfig" : { "httpPort" : 62969, "listenHost" : "localhost", "logTimings" : false, "metricTypes" : [ "CLASS_LOADER", "JVM_MEMORY", "JVM_GC", "PROCESSOR", "JVM_THREAD", "LOGGING_METRICS", "NATIVE" ], "outputDataFormat" : "JSON", "uploadsDirectory" : "file-uploads/" }, "steps" : [ { "@type" : "ModelStep", "inputColumnNames" : { }, "inputNames" : [ "input" ], "inputSchemas" : { }, "modelConfig" : { "@type" : "ModelConfig", "modelConfigType" : { "modelLoadingPath" : "C:\\konduit-serving-examples\\java\\target\\classes\\data\\keras\\embedding_lstm_tensorflow_2.h5", "modelType" : "KERAS" }, "tensorDataTypesConfig" : null }, "normalizationConfig" : null, "outputColumnNames" : { }, "outputNames" : [ "lstm_1" ], "outputSchemas" : { }, "parallelInferenceConfig" : { "batchLimit" : 32, "inferenceMode" : "BATCHED", "maxTrainEpochs" : 1, "queueLimit" : 64, "vertxConfigJson" : null, "workers" : 1 } } ] } ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://serving.konduit.ai/0.1.0-snapshot/examples/java/keras_java.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.