WO2021078271A1

WO2021078271A1 - Method and device for accelerating image detection, and storage medium

Info

Publication number: WO2021078271A1
Application number: PCT/CN2020/123320
Authority: WO
Inventors: 李亚学
Original assignee: 深圳市道通智能航空技术有限公司
Priority date: 2019-10-25
Filing date: 2020-10-23
Publication date: 2021-04-29
Also published as: CN111047032A

Abstract

A method and device for accelerating image detection, and a storage medium. The method for accelerating image detection comprises: step S10, receiving a detection task; step S20, configuring a delay time; step S30, executing the detection task by means of a neural network inference engine (NNIE), and performing a first operation; step S40, according to a first operation result, continuing to execute the detection task by means of a central processing unit (CPU), and performing a second operation; and step S50: when the execution time of the detection task has reached the delay time, triggering the next NNIE to execute the next detection task. The method is used to improve real-time processing speed, improve detection performance while ensuring real-time performance, and improve the CPU utilization rate.

Description

Image detection acceleration method, device and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201911024692.2, and the application name is "Image detection acceleration method, device and storage medium" on October 25, 2019, the entire content of which is incorporated by reference In this application.

Technical field

The present invention relates to the field of target detection, and in particular to an acceleration method, device and storage medium for image detection.

Background technique

The forward reasoner NNIE used for acceleration in neural networks is the abbreviation of Neural Network Inference Engine. It is a hardware unit for acceleration processing of neural networks, especially deep learning convolutional neural networks. It is used for image/image detection tasks using NNIE At the same time, a single NNIE can only process one picture/image detection task at the same time. At the same time, the NNIE and the central processing unit CPU must be in serial operation mode, that is, after the NNIE processing is completed, they can be handed over to the CPU For the next step of calculation, the existing detection processing cannot meet the two problems of fast processing speed and good detection effect at the same time. The model with good detection effect has large input image resolution and large network model, but the processing speed is slow, which is difficult to meet. Real-time requirements; models with good real-time performance have low input image resolution and small network models, but the detection effect is poor. At the same time, when a large number of images and images are detected, there are problems of low detection efficiency and low CPU usage.

Summary of the invention

The main purpose of the present invention is to provide an acceleration method, device and storage medium for image detection, aiming to solve the problem that the prior art using NNIE for image detection cannot satisfy the real-time performance and detection effect.

In order to achieve the above objective, the present invention provides an acceleration method for image detection, which includes:

Step S10: receiving a detection task;

Step S20: Set the delay time;

Step S30: Execute the detection task through the neural network forward inference engine NNIE (Neural Network Inference Engine) to perform the first calculation;

Step S40: According to the first calculation result, continue to execute the detection task through a central processing unit (CPU) to perform a second calculation;

Step S50: When the execution time of the detection task reaches the delay time, trigger the next NNIE to execute the next detection task.

Further, the delay time is greater than 0 and less than the time required for the completion of a single detection task.

Preferably, the delay time is 40 milliseconds.

Further, a single detection task loop executes the first operation and the second operation in sequence until the current detection task is completed.

Further, after the last NNIE starts to receive and execute the detection task, the delay time also elapses, triggering the first NNIE to receive and execute the next detection task, and execute the detection task in turn.

Further, the detection task includes image detection in an image or video sequence.

Further, a neural network is used to detect images in an image or video frame sequence.

Preferably, the frame rate of the image is 30 fps (Frames Per Second, the number of frames transmitted per second).

To achieve the above objective, the present invention also provides an image detection acceleration device, characterized in that the image detection acceleration device includes a memory and a processor, and the memory stores an image that can run on the processor. A detection acceleration program, which implements the steps of the aforementioned image detection acceleration method when the image detection acceleration program is executed by the processor.

In addition, in order to achieve the above object, the present invention also provides a storage medium, characterized in that the storage medium is a computer-readable storage medium, the storage medium stores an image detection acceleration program, and the image detection acceleration The program may be executed by one or more processors to implement the steps of the acceleration method for image detection described above.

Compared with the prior art, the image detection acceleration method, device and storage medium provided by the present invention realize parallel processing of detection tasks, and through a delay trigger mechanism, it is ensured that each NNIE handles different detection tasks, and the detection speed is improved. , While ensuring fast real-time performance, it improves the detection effect and also improves the utilization rate of the central processing unit CPU.

Description of the drawings

FIG. 1 is a schematic flowchart of an image detection acceleration method provided by an embodiment of the present invention;

2 is a schematic diagram of the working principle of an acceleration method for image detection provided by an embodiment of the present invention;

Figure 3 is a schematic diagram of the existing single NNIE processing and detecting tasks;

4 is a schematic diagram of the internal structure of an image detection acceleration device provided by an embodiment of the present invention;

FIG. 5 is a schematic diagram of modules of an acceleration program for image detection in an image detection acceleration device provided by an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the present invention in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

Please refer to FIG. 1. FIG. 1 is an image detection acceleration method provided by an embodiment of the present invention. The method is used to accelerate image detection and improve the efficiency of image detection. The method can be executed by a device, and the device can be implemented by software and/or hardware. The acceleration method for image detection includes:

Step S10: Receive a detection task. Specifically, the detection task is: after acquiring the images in the video frame sequence, the neural network is used to detect the images in the video frame sequence. In more detail, in practical applications, the present invention collects video data through a camera to obtain an image with a sequence of video frames.

Step S20: Set a delay time; where the delay time is set to d, and the delay time d is greater than 0 and less than the detection time of a single detection task.

Step S30: Perform the detection task through a first neural network inference engine (NNIE) to perform a first calculation. Please refer to FIG. 2, specifically, the time end point for the first operation of the detection task by the first NNIE1 is T1; the first operation result is generated by the first NNIE1 after performing the first operation.

Step S40: According to the first calculation result, the central processing unit (CPU) executes the detection task to perform a second calculation. The end of time for the CPU to execute the second operation of the detection task is T2. Please refer to FIG. 2. The time point T1 is the starting point of the second operation, and the time point T2 is the second operation. The end point is also the starting point of the next first operation, and the first operation and the second operation are executed in a loop until the detection task is completed once.

Step S50: When the execution time of the detection task reaches the delay time, trigger the next NNIE to execute the next detection task. Specifically, when the detection task being executed reaches the delay time d, the second NNIE is triggered to execute the next detection task.

In more detail, by analogy, when there is an nth NNIEn executing the nth detection task, n delay times d are sequentially set, and the n+1th NNIE is triggered to execute the n+1th detection task.

Further, after the last NNIE executes the detection task, similarly, the delay time d is set, and when the first NNIE is in an idle state, the first NNIE is triggered to execute the new detection task again, and the execution is repeated in turn.

In the embodiment of the present invention, for each detection task received, it needs to be executed in two times: the first time is the first operation by the NNIE, and the second time is the second operation by the CPU. The second operation depends on the result of the first operation. When the first NNIE performs the first operation, the CPU is in an idle state.

Referring to FIG. 2, in a specific embodiment, the time required to execute a detection task is set to be 100 milliseconds (ms), and the delay time d is set to 40 ms, where the NNIE needs to perform the first operation The first calculation time of is 70ms, the second calculation time T2 required by the CPU for the second calculation is 30ms, and the frame rate of the image is 30fps (Frames Per Second, the number of frames transmitted per second), that is, the image is updated once in 33ms; Set the speed of a single NNIE to execute the detection task at 10fps. The task detection of a single image requires a single NNIE to be executed 3 times. After a single NNIE is executed, it is executed. At this time, the utilization rate of a single CPU is 30%.

In the present invention, different single NNIEs are triggered by a delay trigger mechanism. When the first NNIE starts to perform the detection task 40ms, the second NNIE is triggered to start the detection task, so that the input image of the second NNIE and the input image of the first NNIE are not the same frame, avoid unnecessary repetitive operations, and execute it in the first NNIE After 40ms, the second NNIE is triggered to start execution, ensuring that the input of the two NNIEs are different images, and the detection speed is also greatly improved. Assuming that within T time, a single NNIE can detect x pictures, then multiple NNIEs can be in T+ Detecting nx pictures in 40ms, performing detection tasks for a long time, the processing speed of multiple NNIEs is increased by n times, and the utilization rate of CPU is increased.

Refer to Figure 3, which is a schematic diagram of an existing single NNIE processing detection task. In the case of using this model, a single NNIE runs the detection task at a speed of 10fps, which has poor real-time performance, and most of the time is run under NNIE. At this time The CPU is in an idle state, and the CPU usage rate is low.

Compared with the prior art, the image detection acceleration method, device and storage medium provided by the present invention realize the parallel processing of detection tasks, and through the delay trigger mechanism, it is ensured that each NNIE handles different detection tasks in a longer period of time. In the detection task, the detection speed of the n NNIEs is increased by n times compared to the processing speed of the original NNIE. While ensuring fast real-time speed, the detection effect is improved, and the utilization rate of the central processing unit CPU is also improved.

The invention also provides an acceleration device for image detection. Refer to FIG. 4, which is a schematic diagram of the internal structure of an image detection acceleration device provided by an embodiment of the present invention.

In this embodiment, the acceleration device for image detection may be a PC (Personal Computer, personal computer), or a terminal device such as a smart phone, a tablet computer, or a portable computer. The image detection acceleration device includes at least a memory 11, a processor 12, a communication bus 13, and a network interface 14.

The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, and the like. In some embodiments, the memory 11 may be an internal storage unit of an image detection acceleration device, for example, a hard disk of the image detection acceleration device. In other embodiments, the memory 11 may also be an external storage device of the image detection acceleration device, such as a plug-in hard disk equipped on the image detection acceleration device, a smart memory card (Smart Media Card, SMC), and a secure digital (Secure Digital). Digital, SD) card, flash card (Flash Card), etc. Further, the memory 11 may also include both an internal storage unit of the image detection acceleration device and an external storage device. The memory 11 can be used not only to store application software and various data installed in the image detection acceleration device, such as the code of the image detection acceleration program, etc., but also to temporarily store data that has been output or will be output.

In some embodiments, the processor 12 may be a central processing unit (CPU), an inference engine (Neural Network Inference Engine, NNIE), a controller, a microcontroller, a microprocessor, or other data processing chips. The program code or processing data stored in the running memory 11, for example, an acceleration program for executing image detection, etc.

The communication bus 13 is used to realize the connection and communication between these components.

The network interface 14 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface), and is usually used to establish a communication connection between the image detection acceleration device and other electronic devices.

Optionally, the image detection acceleration device may further include a user interface, the user interface may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display can also be appropriately called a display screen or a display unit, which is used to display the information processed in the acceleration device for image detection and to display a visualized user interface.

FIG. 4 only shows an acceleration device for image detection with components 11-14 and an acceleration program for image detection. Those skilled in the art can understand that the structure shown in FIG. 4 does not constitute a limitation on the acceleration device for image detection It may include fewer or more components than shown, or combine some components, or different component arrangements.

In the embodiment of the image detection acceleration device shown in FIG. 4, the memory 11 stores an image detection acceleration program; the processor 12 implements the following steps when executing the image detection acceleration program stored in the memory 11:

Step S10: receiving a detection task;

Step S20: Set the delay time;

Step S30: Execute the detection task through NNIE to perform the first calculation;

Step S40: According to the first calculation result, continue to execute the detection task through the CPU to perform a second calculation;

5, which is a schematic diagram of the program modules of the acceleration program of image detection in an embodiment of the image detection acceleration device of the present invention. In this embodiment, the image detection acceleration program can be divided into a receiving module 10 and a first calculation. The module 20, the second arithmetic module 30, the setting module 40 and the triggering module 50 are exemplarily:

The receiving module 10 is used to receive detection tasks;

The first operation module 20, NNIE executes the detection task to perform the first operation;

The second calculation module 30 is configured to execute the detection task through the CPU according to the first calculation result to perform a second calculation;

The setting module 40 is used to set the delay time;

The triggering module 50, when the execution time of the detection task reaches the delay time, triggers the next NNIE to execute the next detection task.

The above-mentioned receiving module 10, the first computing module 20, the second computing module 30, the setting module 40, and the triggering module 50 realize the functions or operation steps when the program modules are executed, which are substantially the same as those in the above embodiment, and will not be repeated here.

In addition, an embodiment of the present invention also provides a storage medium, the storage medium is a computer-readable storage medium, the storage medium stores an acceleration program for image detection, and the acceleration program for image detection can be used by one or more The processor executes to achieve the following operations:

Step S10: receiving a detection task;

Step S20: Set the delay time;

The specific implementation of the storage medium of the present invention is basically the same as the foregoing embodiments of the image detection acceleration method and device, and will not be repeated here.

It should be noted that the sequence numbers of the above-mentioned embodiments of the present invention are only for description, and do not represent the advantages and disadvantages of the embodiments. And the terms "include", "include" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements not only includes those elements, but also includes those elements that are not explicitly included. The other elements listed may also include elements inherent to the process, device, article, or method. If there are no more restrictions, the element defined by the sentence "including one..." does not exclude the existence of other identical elements in the process, device, article, or method that includes the element.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of the present invention essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , Magnetic disk, optical disk), including several instructions to make a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the method described in each embodiment of the present invention.

Note that the above are only the preferred embodiments of the present invention and the applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made to those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention. The scope of is determined by the scope of the appended claims.

Claims

A method for accelerating image detection, characterized in that the method for accelerating image detection includes:

Step S10: receiving a detection task;

Step S20: Set the delay time;

Step S30: Execute the detection task through the neural network forward inference engine NNIE (Neural Network Inference Engine) to perform the first calculation;

Step S40: According to the first calculation result, continue to execute the detection task through a central processing unit (CPU) to perform a second calculation;

Step S50: When the execution time of the detection task reaches the delay time, trigger the next NNIE to execute the next detection task.
The method for accelerating image detection according to claim 1, wherein the delay time is greater than 0 and less than the time required for a single detection task to be completed.
The method for accelerating image detection according to claim 1 or 2, wherein the delay time is 40 milliseconds.
The method for accelerating image detection according to claim 1 or 2, wherein the first operation and the second operation are sequentially executed in a single cycle of the detection task until the current detection task is completed.
The method for accelerating image detection according to claim 1 or 2, wherein after the last NNIE starts to receive and execute the detection task, a delay time also elapses, triggering the first NNIE to receive and execute the next The detection tasks are executed sequentially and cyclically.
The method for accelerating image detection according to claim 1 or 2, wherein the detection task includes image detection in an image or video sequence.
The method for accelerating image detection according to claim 6, wherein a neural network is used to detect images in an image or a sequence of video frames.
The method for accelerating image detection according to claim 6, wherein the frame rate of the image is 30 fps (Frames Per Second, the number of frames transmitted per second).
An acceleration device for image detection, characterized in that the acceleration device for image detection includes a memory and a processor, and an acceleration program for image detection that can be run on the processor is stored in the memory, and the image detection When the acceleration program is executed by the processor, the steps of the image detection acceleration method according to any one of claims 1 to 8 are implemented.
A storage medium, characterized in that the storage medium is a computer-readable storage medium, and an acceleration program for image detection is stored on the storage medium, and the acceleration program for image detection can be executed by one or more processors, In order to realize the steps of the method for accelerating image detection according to any one of claims 1 to 8.