Back to Console

Infery Examples On GitHub

Various example of how to use infery in your code.

Suggest Edits

Deci Inference Examples

A collection of demo-apps and inference scripts for various frameworks using infery.

infery is Deci's Inference Engine for Python with robust API for inference and benchmarks,
on any hardware.

Basic Examples (Copy-Paste scripts)

Advanced Examples

Applications

Custom Hardware Examples


Getting Started

  • Infery

    • You can use the Docker Container for GPU installations.
    • Or, Install on your host machine

  • Git LFS

    • We use Git LFS to store the example models checkpoints.
    # 1. Install dependencies: 
#   - python3 -m pip install <infery | infery-gpu | infery-openvino | infery-tensorrt | infery-onnx-gpu | infery-onnx-cpu | infery-tensorflow-gpu | infery-tensorflow-cpu | ... >
#   - apt-get install git-lfs

# 2. Clone this repo and download the example models
git clone https://github.com/Deci-AI/infery-examples.git && cd infery-examples/ && git lfs fetch && git lfs checkout

# 3. cd to the example's directory and run an example (The ONNX one, in this case).
cd basic && python3 predict_onnx.py

Infery Documentation

Contributing

  • Feel free to request a feature, example or application.
    • Open an issue on GitHub and describe your desired usage.
  • Please format the code before opening a pull-request using ./scripts/lint.sh.

Infery Support Matrix

PyTorchTensorFlow 2OnnxRuntimeOpenVinoTensorRTCoreMLTorchScriptTFLiteKeras
Supported Versionstorch 1.10.1tensorflow ≥ 2.6.0OnnxRunntime ≥ 1.10.*openvino ≥ 2022.1.0TensorRT 8.0.1.6coremltools ≥ 5.2.0torch 1.10.1tensorflow ≥ 2.6.0, or tflite-runtimetensorflow ≥ 2.6.0
CPU Support✅✅✅✅❌✅✅✅ (*)✅ (*)
GPU Support✅✅ (*)✅❌✅❌✅✅ (*)✅ (*)
Model Predict - Synchronous✅✅✅✅✅✅✅✅✅
Model Predict - Asynchronous❌❌❌❌✅❌❌❌❌
Model Benchmark✅✅✅✅✅✅✅✅✅
Detect Model Input Dimensions Automatically❌ - manual✅✅✅✅❌❌ - manual✅✅
Detect Model Batch Size Automatically❌ - manual✅✅✅✅❌❌ - manual✅✅
Detect Model Input Data Type Automatically❌ - manual✅✅✅✅✅❌ - manual✅✅
Dynamic Batch Size✅✅✅✅✅✅✅✅✅
Open In Netron✅✅✅✅❌✅✅✅✅
Multiple Inputs \ Outputs✅✅ (**)✅✅✅✅✅✅❌
Dynamic Dimensions (in addition to Batch Size)✅❌✅❌❌❌✅❌❌
Change Model Input Dimensions, Batch Size and Data Type✅❌❌❌❌❌✅❌❌
Model Meta-Data (Legend)BasicBasicBasicBasicBasic + TRT specific fieldsBasicBasicBasicBasic
Per-Layer Inference Profiling❌❌✅❌✅❌❌❌❌
Benchmark Modessynchronoussynchronoussynchronoussynchronoussynchronous with IO, synchronous without IO, Asynchronoussynchronoussynchronoussynchronoussynchronous
  • (*) will always use GPU if infery-gpu installed
  • (**) Limited multi-output support

👍

Missing an example for your use case? Let us know!

We thrive to implement new use cases to extend infery's support matrix.
You can let us know by opening an Issue in the GitHub repository, or by contating us through this guide.

Updated 3 months ago