Infery Examples On GitHub
Various example of how to use infery in your code.
Deci Inference Examples
A collection of demo-apps and inference scripts for various frameworks using infery.
infery
is Deci's Inference Engine for Python with robust API for inference and benchmarks,
on any hardware.
Basic Examples (Copy-Paste scripts)
Advanced Examples
Applications
Custom Hardware Examples
Getting Started
-
Infery
- You can use the Docker Container for GPU installations.
- Or, Install on your host machine
-
Git LFS
- We use Git LFS to store the example models checkpoints.
# 1. Install dependencies: # - python3 -m pip install <infery | infery-gpu | infery-openvino | infery-tensorrt | infery-onnx-gpu | infery-onnx-cpu | infery-tensorflow-gpu | infery-tensorflow-cpu | ... > # - apt-get install git-lfs # 2. Clone this repo and download the example models git clone https://github.com/Deci-AI/infery-examples.git && cd infery-examples/ && git lfs fetch && git lfs checkout # 3. cd to the example's directory and run an example (The ONNX one, in this case). cd basic && python3 predict_onnx.py
Infery Documentation
Contributing
- Feel free to request a feature, example or application.
- Open an issue on GitHub and describe your desired usage.
- Please format the code before opening a pull-request using
./scripts/lint.sh
.
Infery Support Matrix
PyTorch | TensorFlow 2 | OnnxRuntime | OpenVino | TensorRT | CoreML | TorchScript | TFLite | Keras | |
---|---|---|---|---|---|---|---|---|---|
Supported Versions | torch 1.10.1 | tensorflow ≥ 2.6.0 | OnnxRunntime ≥ 1.10.* | openvino ≥ 2022.1.0 | TensorRT 8.0.1.6 | coremltools ≥ 5.2.0 | torch 1.10.1 | tensorflow ≥ 2.6.0, or tflite-runtime | tensorflow ≥ 2.6.0 |
CPU Support | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (*) | ✅ (*) |
GPU Support | ✅ | ✅ (*) | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ (*) | ✅ (*) |
Model Predict - Synchronous | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
Model Predict - Asynchronous | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
Model Benchmark | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
Detect Model Input Dimensions Automatically | ❌ - manual | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ - manual | ✅ | ✅ |
Detect Model Batch Size Automatically | ❌ - manual | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ - manual | ✅ | ✅ |
Detect Model Input Data Type Automatically | ❌ - manual | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ - manual | ✅ | ✅ |
Dynamic Batch Size | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
Open In Netron | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |
Multiple Inputs \ Outputs | ✅ | ✅ (**) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
Dynamic Dimensions (in addition to Batch Size) | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
Change Model Input Dimensions, Batch Size and Data Type | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
Model Meta-Data (Legend) | Basic | Basic | Basic | Basic | Basic + TRT specific fields | Basic | Basic | Basic | Basic |
Per-Layer Inference Profiling | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
Benchmark Modes | synchronous | synchronous | synchronous | synchronous | synchronous with IO, synchronous without IO, Asynchronous | synchronous | synchronous | synchronous | synchronous |
- (*) will always use GPU if
infery-gpu
installed - (**) Limited multi-output support
Missing an example for your use case? Let us know!
We thrive to implement new use cases to extend infery's support matrix.
You can let us know by opening an Issue in the GitHub repository, or by contating us through this guide.
Updated 3 months ago