DocumentationAPI Reference
Back to ConsoleLog In

RTiC System Architecture

The following shows a high-level diagram of the RTiC architecture.

RTiC Container Architecture

  • Communication – Inference requests arrive at the RTiC server from a client-side application via HTTP/gRPC/IPC and are then routed to an inference scheduler.

  • Inference Scheduler – The engine configures multiple scheduling and batching algorithms in order to optimize resource utilization upon inference. For each model, RTiC optionally performs batching of inference requests and then passes the requests to the relevant inferencer that corresponds to the model type and framework.

  • Model Management – A file-system-based repository is used to manage all the models that RTiC made available for inferencing.

  • Inference Manager – Using the input provided with each request, the framework-based inferencers perform inferencing in order to generate the requested outputs. The output is then formatted to produce a response for the client application.

System Architecture – The RTiC Eco-System

  • Model Repository – The Deci model repository sits outside the RTiC engine, within the Deci platform. This is where the models are optimized for inference, and it is the location from which the user chooses the models to be registered in the container.

  • AutoNAC Optimization Engine – The Deci AutoNAC optimization engine leverage Deci's proprietary technology in order to optimize deep learning model's inference performance, for any hardware.


Did this page help you?