29 April, 2020

Breaking changes

This page explains and demonstrate some of the breaking API/Configuration changes that were introduced in PR 285.

Configurations have been optimised for InferenceConfiguration by taking out a few levels of the config hierarchies and renaming of some schema keys.

1. API

  1. ModelConfig and its implementations have been removed. Instead they'll now all be first class citizens of PipelineStep. Each implementation of ModelConfig i.e <Framework>Config has been re-implemented into <Framework>Step. So, KerasConfig would now be KerasStep and similarly, DL4JConfig would now be Dl4jStep and so on. This will reduce the config levels that we had previously from ModelStep -> ModelConfig -> <config_properties> to just ModelStep -> <config_properties>

  2. TensorDataTypeConfig is taken out and its two internal properties inputDataTypes and outputDataTypes are now part of the ModelStep configuration. Now they can be directly specified in the ModelStep without having to create and extra object layer.

  3. ModelConfigType object has been removed which previously contained the model type under modelType variable and path of the model under modelLoadingPath variable.

  4. The name of the modelLoadingPath inside ModelConfig -> ModelConfigType has been renamed to path.

  5. PmmlConfig has been merged into PmmlStep.

Following are a few examples of how the new changes look like:

Example

Using ModelStep instead of ModelConfig, TensorDataTypeConfig and ModelConfigType. Notice how the modelLoadingPath is renamed to path.

Java

Java
Python
Java
// Before
ModelStep modelStep = ModelStep.builder()
.inputNames(Collections.singletonList("image_tensor"))
.outputNames(Collections.singletonList("detection_classes"))
.modelConfig(ModelConfig.builder()
.modelConfigType(ModelConfigType.builder()
.modelLoadingPath(path)
.modelType(ModelType.TENSORFLOW)
.build())
.tensorDataTypeConfig(TensorDataTypeConfig.builder()
.inputDataType("image_tensor", TensorDataType.INT64)
.build())
.build())
.build();
/* -------------------------------------------------- */
// After
ModelStep modelStep = TensorFlowStep.builder()
.inputNames(Collections.singletonList("image_tensor"))
.outputNames(Collections.singletonList("detection_classes"))
.path(path)
.inputDataType("image_tensor", TensorDataType.INT64)
.build();
Python
# Before
modelStep = ModelStep(
input_names=["image_tensor"],
output_names=["detection_classes"],
model_config=ModelConfig(
model_config_type=ModelConfigType(
model_loading_path=path,
model_type="TENSORFLOW")
),
tensor_data_type_config=TensorDataTypeConfig(
input_data_types={"image_tensor", "INT64"}
)
)
''' -------------------------------------------------- '''
# After
modelStep = TensorFlowStep(
input_names=["image_tensor"],
output_names=["detection_classes"],
path=path,
input_data_types={"image_tensor", "INT64"}
)

2. CLI and Deployment APIs

Due to the addition of the CLI wrapper class KonduitServingLauncher there was no need for the KonduitServingNodeConfigurer, KonduitServingMain and KonduitServingOrchestration classes. They are now refactored into DeployKonduitServing and DeployKonduitOrchestration class. The different between them in usage is as follows:

Before

//Inference Configuration
InferenceConfiguration inferenceConfiguration = InferenceConfiguration.builder()
.servingConfig(servingConfig)
.step(bertModelStep)
.build();
File configFile = new File("config.json");
FileUtils.write(configFile, inferenceConfiguration.toJson(), Charset.defaultCharset());
KonduitServingMainArgs args1 = KonduitServingMainArgs.builder()
.configStoreType("file").ha(false)
.multiThreaded(false).configPort(port)
.verticleClassName(InferenceVerticle.class.getName())
.configPath(configFile.getAbsolutePath())
.build();
KonduitServingMain.builder()
.eventHandler(handler -> {
if(handler.succeeded()) {
try {
//client config.
String response = Unirest.post(String.format("http://localhost:%s/raw/numpy", port))
.field("IteratorGetNext:0", input0)
.field("IteratorGetNext:1", input1)
.field("IteratorGetNext:4", input4)
.asString().getBody();
System.out.print(response);
System.exit(0);
} catch (UnirestException e) {
e.printStackTrace();
System.exit(0);
}
})
.build()
.runMain(args1.toArgs());
}

After

//Inference Configuration
InferenceConfiguration inferenceConfiguration = InferenceConfiguration.builder()
.servingConfig(servingConfig)
.step(bertModelStep)
.build();
DeployKonduitServing.deployInference(inferenceConfiguration, handler -> {
if(handler.succeeded()) {
try {
//client config.
String response = Unirest.post(String.format("http://localhost:%s/raw/numpy", port))
.field("IteratorGetNext:0", input0)
.field("IteratorGetNext:1", input1)
.field("IteratorGetNext:4", input4)
.asString().getBody();
System.out.print(response);
System.exit(0);
} catch (UnirestException e) {
e.printStackTrace();
System.exit(0);
}
})

3. Changes in JSON Configuration

JSON configurations has been reduced in many object layers, mostly due to the API changes described above in the API section. Following is an example that will show the difference between them.

Before

{
"servingConfig" : {
"createLoggingEndpoints" : false,
"httpPort" : 0,
"listenHost" : "localhost",
"logTimings" : false,
"metricTypes" : [ "CLASS_LOADER", "JVM_MEMORY", "JVM_GC", "PROCESSOR", "JVM_THREAD", "LOGGING_METRICS", "NATIVE" ],
"metricsConfigurations" : [ ],
"outputDataFormat" : "JSON",
"uploadsDirectory" : "/var/folders/2p/xl2c1gm94h54f0h76xc0xpbc0000gn/T"
},
"steps" : [ {
"@type" : "ModelStep",
"inputColumnNames" : { },
"inputNames" : [ "default" ],
"inputSchemas" : { },
"modelConfig" : {
"@type" : "TensorFlowConfig",
"modelConfigType" : {
"modelLoadingPath" : "<path-to-the-tensorflow-model>",
"modelType" : "TENSORFLOW"
},
"tensorDataTypesConfig" : {
"inputDataTypes" : {
"default" : "FLOAT"
},
"outputDataTypes" : {
"default" : "FLOAT"
}
}
},
"outputColumnNames" : { },
"outputNames" : [ "default" ],
"outputSchemas" : { },
"parallelInferenceConfig" : {
"batchLimit" : 32,
"inferenceMode" : "BATCHED",
"maxTrainEpochs" : 1,
"queueLimit" : 64,
"workers" : 1
}
} ]
}

After

{
"servingConfig" : {
"createLoggingEndpoints" : false,
"httpPort" : 0,
"listenHost" : "localhost",
"logTimings" : false,
"metricTypes" : [ "CLASS_LOADER", "JVM_MEMORY", "JVM_GC", "PROCESSOR", "JVM_THREAD", "LOGGING_METRICS", "NATIVE" ],
"metricsConfigurations" : [ ],
"outputDataFormat" : "JSON",
"uploadsDirectory" : "/var/folders/2p/xl2c1gm94h54f0h76xc0xpbc0000gn/T"
},
"steps" : [ {
"type" : "TENSORFLOW",
"inputColumnNames" : { },
"inputDataTypes" : {
"default" : "FLOAT"
},
"inputNames" : [ "default" ],
"inputSchemas" : { },
"outputColumnNames" : { },
"outputDataTypes" : {
"default" : "FLOAT"
},
"outputNames" : [ "default" ],
"outputSchemas" : { },
"parallelInferenceConfig" : {
"batchLimit" : 32,
"inferenceMode" : "BATCHED",
"maxTrainEpochs" : 1,
"queueLimit" : 64,
"workers" : 1
},
"path" : "<path-to-the-tensorflow-model>"
} ]
}

4. Changes in YAML Configuration

The YAML and JSON configurations now look the same. The following changes have been done so far:

  1. The steps are a list instead of a map.

  2. Python type config would be a map of PythonConfig instead of a single one. You can now specify more than one python configs in a python step.

  3. Both model_loading_path and path are valid for specifying model path for steps related to loading models.

  4. You can now ignore specifying input_names or output_names if you have already specified input_data_types or output_data_types.

  5. You can send in the same updated yaml configs to the Java CLI konduit serve -c config.yaml. Previously, it was only usable for the python konduit module.

Before

serving:
http_port: 1337
output_data_format: NUMPY
log_timings: True
steps:
python_step:
type: PYTHON
python_path: .
python_code_path: ./simple.py
python_inputs:
first: NDARRAY
second: NDARRAY
third: NDARRAY
python_outputs:
fourth: NDARRAY
fifth: NDARRAY
sixth: NDARRAY
tensorflow_step:
type: TENSORFLOW
model_loading_path: bert_mrpc_frozen.pb
input_names:
- IteratorGetNext:0
- IteratorGetNext:1
- IteratorGetNext:4
output_names:
- loss/Softmax
parallel_inference_config:
workers: 1
input_data_types:
IteratorGetNext:0: INT32
IteratorGetNext:1: INT32
IteratorGetNext:4: INT32

After

serving:
http_port: 1337
output_data_format: NUMPY
log_timings: True
steps:
- type: PYTHON
python_configs:
default:
python_path: .
python_code_path: ./simple.py
python_inputs:
first: NDARRAY
second: NDARRAY
third: NDARRAY
python_outputs:
fourth: NDARRAY
fifth: NDARRAY
sixth: NDARRAY
- type: TENSORFLOW
path: bert_mrpc_frozen.pb # model_loading_path is also valid
output_names:
- loss/Softmax
parallel_inference_config:
workers: 1
input_data_types:
IteratorGetNext:0: INT32
IteratorGetNext:1: INT32
IteratorGetNext:4: INT32