Deci Inference Examples

A collection of demo-apps and inference scripts for various frameworks using infery.

infery is Deci's Inference Engine for Python with robust API for inference and benchmarks, on any hardware.

Basic Examples (Copy-Paste scripts)

Advanced Examples

Application Examples

Custom Hardware Examples

Getting Started

Infery
You can use the Docker Container for GPU installations.
Or, Install on your host machine
Git LFS
- We use Git LFS to store the example models checkpoints.

  # 1. Install dependencies: 
  python3 -m pip install infery[all]  #  or < infery-gpu[all] | infery[openvino] | infery-gpu[tensorrt_8.0] | ... > depending on your usage
  apt-get install git-lfs

  # 2. Clone this repo and download the example models
  git clone https://github.com/Deci-AI/infery-examples.git && cd infery-examples/ && git lfs fetch && git lfs checkout

  # 3. cd to the example's directory and run an example (The ONNX one, in this case).
  cd basic && python3 predict_onnx.py

Infery Documentation

Contributing

Feel free to request a feature, example or application.
Open an issue on GitHub and describe your desired usage.
Please format the code before opening a pull-request using ./scripts/lint.sh.

Support Matrix

	PyTorch	TensorFlow 2	OnnxRuntime	OpenVino	TensorRT	CoreML	TorchScript	TFLite	Keras
Supported Versions	torch 1.10.1	tensorflow ≥ 2.6.0	OnnxRunntime ≥ 1.10.*	openvino ≥ 2022.1.0	TensorRT 8.0.1.6	coremltools ≥ 5.2.0	torch 1.10.1	tensorflow ≥ 2.6.0, or tflite-runtime	tensorflow ≥ 2.6.0
CPU Support	✅	✅	✅	✅	❌	✅	✅	✅ (*)	✅ (*)
GPU Support	✅	✅ (*)	✅	❌	✅	❌	✅	✅ (*)	✅ (*)
Model Predict - Synchronous	✅	✅	✅	✅	✅	✅	✅	✅	✅
Model Predict - Asynchronous	❌	❌	❌	❌	✅	❌	❌	❌	❌
Model Benchmark	✅	✅	✅	✅	✅	✅	✅	✅	✅
Detect Model Input Dimensions Automatically	❌ - manual	✅	✅	✅	✅	❌	❌ - manual	✅	✅
Detect Model Batch Size Automatically	❌ - manual	✅	✅	✅	✅	❌	❌ - manual	✅	✅
Detect Model Input Data Type Automatically	❌ - manual	✅	✅	✅	✅	✅	❌ - manual	✅	✅
Dynamic Batch Size	✅	✅	✅	✅	✅	✅	✅	✅	✅
Open In Netron	✅	✅	✅	✅	❌	✅	✅	✅	✅
Multiple Inputs \ Outputs	✅	✅ (**)	✅	✅	✅	✅	✅	✅	❌
Dynamic Dimensions (in addition to Batch Size)	✅	❌	✅	❌	❌	❌	✅	❌	❌
Change Model Input Dimensions, Batch Size and Data Type	✅	❌	❌	❌	❌	❌	✅	❌	❌
Model Meta-Data (Legend)	Basic	Basic	Basic	Basic	Basic + TRT specific fields	Basic	Basic	Basic	Basic
Per-Layer Inference Profiling	❌	❌	✅	❌	✅	❌	❌	❌	❌
Benchmark Modes	synchronous	synchronous	synchronous	synchronous	synchronous with IO, synchronous without IO, Asynchronous	synchronous	synchronous	synchronous	synchronous

(*) will always use GPU if infery-gpu installed
(**) Limited multi-output support