Optimization

Asynchronous Inference with OpenVINO™

OpenVINO Runtime supports inference in either synchronous or asynchronous mode. The key advantage of the Async API is that when a device is busy with inference, the application can perform other tasks in parallel (for example, populating inputs or scheduling other requests) rather than wait for the current inference to complete first.

AsyncInferQueue

Asynchronous mode pipelines can be supported with the AsyncInferQueue wrapper class. This class automatically spawns the pool of InferRequest objects (also called “jobs”) and provides synchronization mechanisms to control the flow of the pipeline. It is a simpler way to manage the infer request queue in Asynchronous mode.

Implementation

Since our solution is devided into two steps and these steps have to be executed multiple times, we can use AsyncInferQueue which will create a stream of Infer-Requests, and one Infer-Request need not to wait until the previous one is completed.

We achived this by creating a pipeline which follow :

1. The main_loop will capture/read frame and pre-process it for detector, hence creating a detection InferRequest.
2. Once this detection InferRequest will complete it will call the detector_callback function which will further process the detection result and start a segmentation InferRequst.
3. And at last once the segmentation InferRequest will complete it will call the segmentor_callback function which will calculate reading and plot results.