WO2020134408A1

WO2020134408A1 - Multi-path load-balanced asynchronous target detection method, storage medium, and processor

Info

Publication number: WO2020134408A1
Application number: PCT/CN2019/112485
Authority: WO
Inventors: 刘若鹏; 栾琳; 肖剑雄峰
Original assignee: 深圳光启空间技术有限公司
Priority date: 2018-12-29
Filing date: 2019-10-22
Publication date: 2020-07-02
Also published as: CN111382737A

Abstract

The present invention provides a multi-path load-balanced asynchronous target detection method, a storage medium, and a processor. The method comprises: S11, a dealer monitors a port occupied by a video sending module sending an image frame, so as to obtain an input image; S12, share the image frame by using a multi-path balanced load of ZMQ and a queue, and distribute the received image to a multi-process YOLO module started concurrently; S13, each YOLO process processes the distributed image frame and detects a target attribute in the obtained image; and S14, share a detection result by means of the multi-path balanced load of ZMQ and the queue, and return same to a downstream identification module. The multiple YOLO processes are started, process information is shared by means of a queue, a dictionary, and the like, multiple video inputs can be simultaneously processed in real time, hardware resources can be fully utilized by means of load balancing, and the method can be widely suitable for target identification in a YOLO monitoring system, facenet, tinyface, and the like.

Description

Multi-channel load balancing asynchronous target detection method, storage medium and processor

Technical field

The present invention relates to the field of target recognition technology, and more specifically, to a multi-path load balancing asynchronous target detection method, storage medium, and processor.

Background technique

Target detection can be used in many fields such as security, industry, and automobile assisted driving. For example, in the security field, the number of people in key areas can be counted to prevent crowded trampling incidents, and abnormal targets can be detected in sensitive areas to prevent regional intrusion. At the same time, target detection It is also the upstream input of visual technologies such as target recognition, instance segmentation, and morphological analysis. The quality of target detection directly determines the processing results of these more complex tasks.

The target detection technology based on computer vision is to give an image and give an understanding of the foreground and background of the image. We need to separate the target of interest from the background and determine the classification and position of the target; therefore, the target detection output Is a list, and each data group element of the list gives a classification and position of the detection target; it is easy for humans to locate and classify the target from a given image, but for computers, the "person" is obtained from the multi-channel pixel matrix ", "car" and other abstract concepts of target objects and determine their location, coupled with multiple targets overlapping, covering and other situations, target detection is more difficult.

technical problem

At present, the mainstream of target detection is based on deep learning, and a multi-process YOLO target detection scheme in PUSH/PULL mode that achieves a good balance between speed and accuracy.

YOLO is the pioneering work of the end-to-end model. At speed, it is named after only one process to get the target position and classification. Compared with the two-stage method, the real-time speed is impressive. The training process of YOLO_V3 is as follows:

1. Prepare the data: scale the picture to the specified size and input it to the network; the network that extracts the features will output 3 different scales [13*13, 26*26, 52*52] feature maps respectively [Only considering the area, not considering the location] Match to different anchors [yolo_v3 sets 3 anchors for each feature map, so there are 9 anchors in total];

2. Convolutional network: use Darknet-53 to extract features, each grid in the three-scale feature map predicts a conditional probability value for each category, and generates B boxes on each grid, each box prediction [num_anchors*(5+num_classes+num_properti es)] regression values, which are the output of x, y, width, height, configure score, classes scores of bounding boxes under different anchors. During the test, the score is calculated as follows:

The first term on the left side of the equation is predicted by the grid, and the last two terms are predicted by each box. The conditional probability is used to obtain the score of each box containing objects of different categories. Therefore, the number of prediction values output by the convolutional network is S×S×(B×5+C), where S is the number of grids, B is the number of boxes generated for each grid, and C is the number of categories;

When YOLO_V3 is doing forward prediction, the input is an image of any size. When the detection is finally output, NMS (Non-Maximum Suppression, non-maximum suppression) filtering is required to obtain the final prediction frame.

When using ZMQ's PUSH/PULL mode, one server is responsible for PUSH data, and multiple clients queue up for PULL data. A piece of data on the server PUSH will only be pulled by one of the clients, and either the server or the client will be disconnected. , Will not affect the other party;

When applied to the YOLO target detection system, in order to improve the real-time detection, the video providing module serves as the server, and the YOLO detection module serves as the client. Multiple YOLO detection processes can be started at the same time, which can detect the target faster.

This shows that its shortcomings are:

(1) When the YOLO module is working, it is only a more mechanical queue waiting, and there is no load balancing;

(2) When enabling multi-channel video detection, it is more troublesome to implement;

When PUSH messages on the server side are easy to accumulate, it is likely to cause data loss, which can never be allowed.

Technical solution

The technical problem to be solved by the present invention is to provide a multi-channel load balancing asynchronous target detection method, storage medium and processor, which can start multi-channel YOLO processes, and can process multi-channel video inputs in real time through queue, dictionary and other shared process information It can make full use of hardware resources through load balancing, and can be widely used in target recognition such as YOLO monitoring system, facenet, tinyface and so on.

In order to solve the above technical problems, on the one hand, an embodiment of the present invention provides a multi-path load balancing asynchronous target detection method, including:

S11. The dealer monitors the port occupied by the video sending module to send the image frame to obtain the input image;

S12. Use ZMQ's multi-channel balanced load and queue to share image frames, and distribute the received images to the multi-process YOLO module started concurrently;

S13. The multi-process YOLO module processes the distributed image frames and detects the target attributes in the images;

S14. The multi-channel balanced load through ZMQ and the result of queue sharing detection are returned to the downstream identification module.

Preferably, the dealer monitors the port occupied by the video sending module to send the image frame, and obtaining the input image includes:

The video sending module receives the video file from the camera, and processes the video file from the camera.

Preferably, the multi-channel balanced load and queue shared image frame using ZMQ includes:

The input image is shared in the multi-path balanced load and queue of ZMQ, and the multi-path balanced load and queue of ZMQ is to be processed.

Preferably, the processing of the distributed image frames by the multi-process YOLO module includes: the multi-process YOLO module receives the image frames distributed by the multi-path balanced load of ZMQ.

Preferably, the target attributes in the image detected by the multi-process YOLO module include: facial attributes.

Preferably, the target attributes in the image detected by the multi-process YOLO module include: license plate attributes.

Preferably, the facial attributes include: gender attributes, ethnic attributes, age attributes, and facial decorations.

Preferably, the license plate attributes include: model, year, and body color.

On the other hand, an embodiment of the present invention provides a storage medium, the storage medium includes a stored program, wherein, when the program runs, the above-described asynchronous target detection method for multiple load balancing is executed.

On the other hand, an embodiment of the present invention provides a processor, which is used to run a program, wherein the above-mentioned multi-path load balancing asynchronous target detection method is executed when the program is run.

Beneficial effect

Compared with the prior art, the above technical solution has the following advantages: open multiple YOLO processes, share process information through queues, dictionaries, etc., can simultaneously process multiple video inputs in real time, and fully utilize hardware resources through load balancing, which is generally applicable In YOLO monitoring system, facenet, tinyface and other target recognition.

BRIEF DESCRIPTION

In order to more clearly explain the technical solutions in the embodiments of the present invention, the drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without paying any creative labor, other drawings can also be obtained based on these drawings.

FIG. 1 is a schematic diagram of a load target detection method in the prior art.

FIG. 2 is a flowchart of the load balancing asynchronous target detection method of the present invention.

FIG. 3 is an analysis diagram of multi-path load balancing through an intermediate agent in ZMQ in the load balancing asynchronous target detection method of the present invention.

FIG. 4 is an embodiment of FIG. 3.

Fig. 5 is another embodiment of Fig. 3.

Embodiments of the invention

The technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present invention.

A

Example one

Target detection involves identifying targets first and then detecting them from many targets. Target recognition refers to the process of distinguishing a particular target (or one type of target) from other targets (or other types of targets). It includes both the recognition of two very similar targets and the recognition of one type of target and other types of targets. FIG. 1 is a schematic diagram of a multi-process target detection method in the prior art. The method shown in FIG. 1 is adopted in the background art.

FIG. 2 is a flowchart of the load balancing asynchronous target detection method of the present invention. As shown in Figure 2, an asynchronous target detection method for multi-channel load balancing includes: S11, the dealer monitors the port occupied by the video sending module to send the image frame to obtain the input image; S12, the multi-channel balanced load and queue sharing using ZMQ Image frames, distribute the received images to the multi-process YOLO module that is started concurrently; S13, the multi-process YOLO module processes the distributed image frames and detects the target attributes in the image; S14, multi-channel balanced load through ZMQ, and The queue sharing detection result is returned to the downstream identification module.

The dealer monitors the port occupied by the video sending module to send the image frame, and obtaining the input image includes: the video sending module receives the video file from the camera and processes the video file from the camera. ZMQ's multi-channel balanced load and queue-sharing image frames include: the input image is shared in ZMQ's multi-channel balanced load and queue, and ZMQ's multi-channel balanced load and queue are to be processed. The processing of the distributed image frames by the multi-process YOLO module includes: the multi-process YOLO module receives the image frames distributed by the multi-path balanced load of ZMQ. The multi-process YOLO module detects that the target attributes in the image include: facial attributes or license plate attributes. Facial attributes include but are not limited to: gender attributes, ethnic attributes, age attributes, facial decorations, etc. License plate attributes include but are not limited to: model, year, body color, etc. . The downstream facial recognition module performs feature extraction on the detected image, and then compares it with the face feature library to see how much the difference is. If the difference is less than a certain threshold, it is considered to be a person. The downstream license plate recognition module does OCR (Optical Character Recognition) recognition through deep learning, and then recognizes the license plate information.

FIG. 3 is an analysis diagram of multi-path load balancing through an intermediate agent in ZMQ in the load balancing asynchronous target detection method of the present invention. There are multiple video inputs, and multiple YOLO detection processes are also opened; the video input is the client and the YOLO detection process is the server; through ZMQ's multi-channel load balancing, reasonable and full use of hardware resources is achieved for asynchronous multi-process communication; The server and the client are not directly connected, but through ROUTER [router] and DEALER [dealer], all requests are queued fairly after arriving at the router, and then sent to the server by the dealer after load balancing, and the server answers the result The dealer and router then return to the client.

FIG. 4 is an embodiment of FIG. 3. Fig. 5 is another embodiment of Fig. 3. As shown in Figures 4 and 5, receive the image frame information of the DEALER port and detect the faces, license plates and other objects in it; DEALER and ROUTER communication modules: process the video transmission module, YOLO detection module and face recognition among other modules Asynchronous load balancing communication; send face information module: extract the face information from the detection result of the YOLO detection module, encode it into a format suitable for network transmission, and send it to the face recognition module; send the license plate recognition module: from the YOLO detection module In the detection result, the license plate information is extracted, encoded into a format suitable for network transmission, and sent to the license plate recognition module. The YOLO detection module, the send face information module, the DEALER and ROUTER communication modules, and the send vehicle information module interact with each other through queues and dictionaries.

A

Example 2

An embodiment of the present invention further provides a storage medium, the storage medium includes a stored program, wherein the above-mentioned multi-path load balancing asynchronous target detection method flow is executed when the above program runs.

Optionally, in this embodiment, the above storage medium may be set to store program code for performing the following face attribute recognition method flow:

Optionally, in this embodiment, the above storage medium may include, but is not limited to: a U disk, a read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory (referred to as RAM), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

It can be seen that by using the storage medium of the present invention, multiple YOLO processes are started, multiple process video inputs can be processed in real time through queues, dictionaries and other shared process information, and hardware resources can be fully utilized through load balancing, which can be generally applied to YOLO, Tinyface and other detection systems such as tinyface and other target recognition systems.

A

Example Three

An embodiment of the present invention further provides a processor, which is used to run a program, where the program executes the steps in the above-mentioned multi-path load balancing asynchronous target detection method when the program is running.

Optionally, in this embodiment, the above program is used to perform the following steps:

Optionally, for specific examples in this embodiment, reference may be made to the foregoing embodiments and the examples described in the specific implementation, and this embodiment will not be repeated here.

Industrial applicability

It can be seen that by using the processor of the present invention to start multiple YOLO processes, through the sharing of process information such as queues and dictionaries, multiple video inputs can be processed in real time, and the hardware resources can be fully utilized through load balancing. It can be generally applied to YOLO, Detection systems such as tinyface, and target recognition systems such as facenet.

As can be seen from the above description, using the multi-channel load balancing asynchronous target detection method, storage medium and processor according to the present invention, the multi-channel YOLO process is started, and the multi-channel video input can be simultaneously processed in real time through the shared process information such as queues and dictionaries. Load balancing can make full use of hardware resources, and can be universally applied to detection systems such as YOLO and facenet, and target recognition systems such as tinyface.

The embodiments of the present invention have been described in detail above, and specific examples have been used in this article to explain the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the method and core idea of the present invention; Those of ordinary skill in the art, according to the ideas of the present invention, will have changes in specific implementations and application scopes. In summary, the content of this specification should not be construed as limiting the present invention.

Claims

A multi-channel load balancing asynchronous target detection method, characterized in that it includes:

S11. The dealer monitors the port occupied by the video sending module to send the image frame to obtain the input image;

S12. Use ZMQ's multi-channel balanced load and queue to share image frames, and distribute the received images to the multi-process YOLO module started concurrently;

S13. The multi-process YOLO module processes the distributed image frames and detects the target attributes in the images;

S14. The multi-channel balanced load through ZMQ and the result of queue sharing detection are returned to the downstream identification module.
The multi-channel load balancing asynchronous target detection method according to claim 1, wherein the dealer monitors the port occupied by the video sending module to send the image frame, and obtaining the input image includes:

The video sending module receives the video file from the camera, and processes the video file from the camera.
The asynchronous target detection method of multi-path load balancing according to claim 1, wherein the multi-path load balancing and queue shared image frame using ZMQ comprises:

The input image is shared in the multi-path balanced load and queue of ZMQ, and the multi-path balanced load and queue of ZMQ is to be processed.
The asynchronous target detection method for multi-path load balancing according to claim 1, wherein the processing of the distributed image frames by the multi-process YOLO module comprises: the multi-process YOLO module receiving the image frames distributed by the multi-path load balancing of ZMQ .
The asynchronous target detection method of multi-channel load balancing according to claim 1, wherein the target attributes in the image detected by the multi-process YOLO module include: facial attributes.
The multi-path load balancing asynchronous target detection method according to claim 1, wherein the target attributes in the image detected by the multi-process YOLO module include: license plate attributes.
The asynchronous target detection method of multi-channel load balancing according to claim 5, wherein the facial attributes include: gender attributes, ethnic attributes, age attributes, and facial decorations.
The multi-channel load balancing asynchronous target detection method according to claim 6, wherein the license plate attributes include: vehicle model, year, and body color.
A storage medium, characterized in that the storage medium includes a stored program, wherein, when the program runs, the asynchronous target detection method for multi-channel load balancing according to any one of claims 1 to 8 is executed.
A processor, characterized in that the processor is used to run a program, wherein, when the program runs, the asynchronous target detection method for multi-channel load balancing according to any one of claims 1 to 8 is executed.