Parameter Usage in Infery

Optimization, benchmarking, deployment and other DL related actions often have a plethora of tunable knobs. In order to prevent huge method call sites and to clearly specify these knobs' defaults, Infery often uses parameters objects that may be passed to central methods like compile, benchmark and deploy.

In this quickstart we demonstrate the use of such parameters.

Compilation Params

Compilation parameters contain all fields applicable to a compilation/modification of a model from a framework (the source_framework) to a framework (the target_framework).

Instantiation and Inspection:

Most of Infery's compilation params are created dynamically and may not be directly imported. Thus, the easiest way to inspect specific parameters' fields is by instantiating them. To do so, use the create_compile_params function:

from infery import FrameworkType
from infery.compiler.params import create_compile_params

# Instantiating compilation params for PyTorch -> TFLite conversion
params = create_compile_params(FrameworkType.PYTORCH, FrameworkType.TFLITE)
print(params)

If you already have an instantiated Model object, you may use the compile_params method, passing the framework to which you'd like to compile your model along with any kwargs:

import infery
from infery import FrameworkType

# Instantiate a model (ONNX in this case)
model = infery.Model('/path/to/model.onnx')

# Get compilation params that may be used to convert your model to TFLite without GS simplification
params = model.compile_params(FrameworkType.TFLITE, simplify_with_gs=False)
print(params)

Usage

Armed with compilation params, you may now compile your model to the desired framework, altering fields prior to the compilation:

import infery
from infery import FrameworkType

# Instantiate a model (ONNX in this case)
model = infery.Model('/path/to/model.onnx')

# Get compilation params that may be used to convert your model to TFLite and bypass GraphSurgeon simplification
params = model.compile_params(FrameworkType.TFLITE)
params.simplify_with_gs = False

# Use the parameters to compile
model.compile(params=params)

Benchmark Params

Infery has a BenchmarkParams class that may be passed to model's benchmark method for more tight control over the nature of the benchmark. These parameters MUST be instantiated with a batch size that will be used for the benchmark:

import infery
from infery import BenchmarkParams

# Instantiate a model (ONNX in this case)
model = infery.Model('/path/to/model.onnx')

# Create benchmark parameters. Only batch_size is mandatory for instantiation.
benchmark_params = BenchmarkParams(batch_size=1, duration_secs=5, warmups=30)

# Use the parameters for benchmarking
benchmark_result = model.benchmark(benchmark_params=benchmark_params)

Notice that a model may also be benchmarked by passing the batch size and other parameter fields as kwargs directly to the benchmark method:

import infery

# Instantiate a model (ONNX in this case)
model = infery.Model('/path/to/model.onnx')

# Create benchmark parameters. Only batch_size is mandatory for instantiation.
benchmark_results = model.benchmark(batch_size=1, duration_secs=5, warmups=30)

Deployment Params

Soon to be released :)