The OVERVIEW section of performance metrics is displayed by default and is described below. You can also display it by clicking the following button.
The expands to show the following. Simply scroll through the provided information or select to see Performance, Accuracy Metrics and Cost (coming soon) information.
The OVERVIEW section shows all the performance and accuracy metrics for a specific model on a specific target hardware environment and for a specific batch size. By default, it shows the comparison between any Optimized Model that you select and the baseline model.
Note – To see a comparison of the metrics of a specific model on all the hardware production environments and for all the batch sizes, select the BENCHMARK HW option.
The left side of this page shows a box for each of the models listed in the Lab. This includes the Baseline Model (shown below on the top left, which in our example is ResNet50) and a box for each of the Optimized Models (which in our example is named T4-Optimized v1.1 and T4-Optimized v1.2).
Clicking on one of these boxes displays the OVERVIEW section, which shows a comparison on the right side of the page of the selected Optimized Model with the baseline model.
The top right of the page indicates which two models are being compared, as shown below –
The details of each model are color-coded. In the example above, the baseline model is represented in purple and the selected optimized model is represented in turquoise.
The large number on the left shows the Deci Score (9.6) and Accuracy (76.3%) of the optimized model, while the small number on the right shows the same type of values for the unoptimized baseline model – 7.0 and 76.1% accordingly.
The radar graph in the center of the Insights page shows the comparative improvements in each metric of the optimized model’s score versus the baseline model. You can display additional detail for each metric by hovering over it in the radar graph – Accuracy Delta, Latency Improvement, Memory Footprint Reduction and Model Size Reduction.
In the example above, Throughput has increased by five times (5.8x).
Hovering over the Throughput shows a tooltip with more details –
The bar at the top of the page shows the target production environment and batch size of the displayed Optimized Model, as shown below –
This bar also enables you to select a different target production environment and batch size of the Optimized Model to be displayed.
The optimization of a model can be targeted at various batch sizes, which may affect its metrics, such as Throughput. For example, a larger batch size may have increased Throughput.
Choose the target batch size to display its measurements.
The following shows the accuracy and efficiency of the model when it is benchmarked on a Batch Size of 1, on a T4 GPU target environment.
Selecting 64 in this bar displays the benchmarks of the optimized model with a Batch Size of 64.
Note – When a baseline model is uploaded into Deci, its benchmark must be specified. For example, it might be 64. This means that selecting 32 in this bar displays this optimized model’s benchmark for a batch that is half the size of the baseline model.
The Select Target Hardware dropdown menu enables you to select a different target hardware environment in order to see the accuracy and efficiency of the model in that target environment. For example, the following shows the accuracy and efficiency of the model when it is benchmarked on a V100 target hardware environment.
The following describes how to display the absolute values of the optimized model instead of its comparison with the baseline model.
To only display the absolute values of the optimized model –
- Click the Stop Comparing icon in the top right of the page. The following displays showing the absolute accuracy and efficiency metrics for the optimized model.
For example, it shows that the throughput is 812.4fps, whereas previously it showed that throughput was 3.3X faster compared to the baseline model.
Compared to Baseline Model Absolute Values – Stop Comparing
To redisplay the comparative data –
- Click the Compare icon in the baseline model box.
Updated 9 months ago