Infery Examples On GitHub

Various example of how to use infery in your code.

Deci Inference Examples

A collection of demo-apps and inference scripts for various frameworks using infery.

infery is Deci's Inference Engine for Python with robust API for inference and benchmarks,
on any hardware.

Basic Examples (Copy-Paste scripts)

Advanced Examples

Applications

Custom Hardware Examples


Getting Started

  • Infery

    • You can use the Docker Container for GPU installations.
    • Or, Install on your host machine
  • Git LFS

    • We use Git LFS to store the example models checkpoints.
    # 1. Install dependencies: 
    #   - python3 -m pip install <infery | infery-gpu | infery-openvino | infery-tensorrt | infery-onnx-gpu | infery-onnx-cpu | infery-tensorflow-gpu | infery-tensorflow-cpu | ... >
    #   - apt-get install git-lfs
    
    # 2. Clone this repo and download the example models
    git clone https://github.com/Deci-AI/infery-examples.git && cd infery-examples/ && git lfs fetch && git lfs checkout
    
    # 3. cd to the example's directory and run an example (The ONNX one, in this case).
    cd basic && python3 predict_onnx.py
    

Infery Documentation

Contributing

  • Feel free to request a feature, example or application.
    • Open an issue on GitHub and describe your desired usage.
  • Please format the code before opening a pull-request using ./scripts/lint.sh.

Infery Support Matrix

PyTorchTensorFlow 2OnnxRuntimeOpenVinoTensorRTCoreMLTorchScriptTFLiteKeras
Supported Versionstorch 1.10.1tensorflow ≥ 2.6.0OnnxRunntime ≥ 1.10.*openvino ≥ 2022.1.0TensorRT 8.0.1.6coremltools ≥ 5.2.0torch 1.10.1tensorflow ≥ 2.6.0, or tflite-runtimetensorflow ≥ 2.6.0
CPU Support✅ (*)✅ (*)
GPU Support✅ (*)✅ (*)✅ (*)
Model Predict - Synchronous
Model Predict - Asynchronous
Model Benchmark
Detect Model Input Dimensions Automatically❌ - manual❌ - manual
Detect Model Batch Size Automatically❌ - manual❌ - manual
Detect Model Input Data Type Automatically❌ - manual❌ - manual
Dynamic Batch Size
Open In Netron
Multiple Inputs \ Outputs✅ (**)
Dynamic Dimensions (in addition to Batch Size)
Change Model Input Dimensions, Batch Size and Data Type
Model Meta-Data (Legend)BasicBasicBasicBasicBasic + TRT specific fieldsBasicBasicBasicBasic
Per-Layer Inference Profiling
Benchmark Modessynchronoussynchronoussynchronoussynchronoussynchronous with IO, synchronous without IO, Asynchronoussynchronoussynchronoussynchronoussynchronous
  • (*) will always use GPU if infery-gpu installed
  • (**) Limited multi-output support

👍

Missing an example for your use case? Let us know!

We thrive to implement new use cases to extend infery's support matrix.
You can let us know by opening an Issue in the GitHub repository, or by contating us through this guide.