JP2021502645A - Target detection methods and devices, training methods, electronic devices and media - Google Patents

Abstract

The embodiments of the present disclosure disclose target detection methods and devices, training methods, electronic devices and media. The target detection method is an image of the detection frame and the detection frame of the target object, and the features of the template frame whose image size is smaller than the detection frame are extracted by the neural network, respectively, and the features of the template frame. Obtaining the classification weight and the regression weight of the local region detector based on the above, inputting the characteristics of the detection frame to the local region detector, and outputting from the local region detector of a plurality of candidate frames. This includes acquiring the classification result and the regression result, and acquiring the detection frame of the target object in the detection frame from the classification result and the regression result of a plurality of candidate frames output from the local region detector. .. The embodiments of the present disclosure can increase the speed and accuracy of target tracking.

Description

The present disclosure relates to computer vision technology, in particular to target detection methods and devices, training methods, electronic devices and media.
<Cross-reference of related applications>
This application is prioritized by the Chinese patent application filed with the Chinese Patent Office on November 12, 2017, with application number CN201711110587.1, and the title of the invention "Target detection method and device, training method, electronic device, program and medium". Claim the right and all of its disclosures are incorporated herein by reference.

Single-target tracking has become an important issue in the field of artificial intelligence, and is used for a series of tasks such as autonomous driving and multi-target tracking. The main task of single target tracking is to specify a target to be tracked in one frame of image within a segment of the video sequence, and to continuously track this specified target in subsequent frame images.

The embodiments of the present disclosure provide technical means for performing target tracking.

According to one side of the embodiments of the present disclosure
Extracting the features of the detection frame and the template frame, which is the image of the detection frame of the target object and the size of the image is smaller than the detection frame, by the neural network, respectively.
Obtaining the classification weight and regression weight of the local region detector based on the characteristics of the template frame,
The characteristics of the detection frame are input to the local area detector, and the classification result and the regression result of a plurality of candidate frames output from the local area detector are acquired.
Provided is a target tracking method including acquiring a detection frame of the target object in the detection frame based on a classification result and a regression result of a plurality of candidate frames output from the local region detector.

According to another aspect of the embodiments of the present disclosure,
Extracting the features of the detection frame and the template frame, which is the image of the detection frame of the target object and the size of the image is smaller than the detection frame, by the neural network, respectively.
The first convolution layer increases the channels of the features of the template frame, the acquired first features are the weight of the classification of the local region detector, and the second convolution layer provides the channels of the features of the template frame. The second feature obtained by increasing is to be the regression weight of the local region detector.
The characteristics of the detection frame are input to the local area detector, and the classification result and the regression result of a plurality of candidate frames output from the local area detector are acquired.
Obtaining the detection frame of the target object in the detection frame from the classification result and the regression result of the plurality of candidate frames output from the local region detector, and
The detection frame of the target object in the acquired detection frame is used as a prediction detection frame, and the neural network, the first convolution layer, and the second convolution are based on the labeling information of the detection frame and the prediction detection frame. It provides training methods for layer training and target detection networks, including.

According to another aspect of the embodiments of the present disclosure,
A neural network for extracting the features of the detection frame and the template frame, which is an image of the detection frame of the target object and whose image size is smaller than the detection frame.
A first convolution layer for increasing the channels of the features of the template frame and weighting the obtained first features to the classification of the local region detector.
A second convolution layer for increasing the channels of the features of the template frame and weighting the resulting second features to the regression of the local region detector.
A local region detector for outputting classification results and regression results of a plurality of candidate frames according to the characteristics of the detection frame, and
Provided is a target detection device including an acquisition unit for acquiring a detection frame of the target object in the detection frame based on a classification result and a regression result of a plurality of candidate frames output from the local region detector.

According to yet another aspect of the embodiments of the present disclosure, there is provided an electronic device comprising the target detection device according to any one of the embodiments of the present disclosure.

According to yet another aspect of the embodiments of the present disclosure,
Memory for storing executable commands and
Provided is another electronic device comprising a processor for completing the operation of the method according to any one of the embodiments of the present disclosure by communicating with the memory and executing the executable command. To do.

According to yet another aspect of the embodiments of the present disclosure, it is a computer storage medium for storing a computer-readable command, and when the command is executed, it becomes one of the embodiments of the present disclosure. Provided is a computer storage medium in which the operation of the above-described method is realized.

According to yet another aspect of the embodiments of the present disclosure, a computer program comprising a computer-readable command that, when the computer-readable command is activated on the device, tells the processor in the device of the embodiments of the present disclosure. Provided is a computer program that executes an executable command for realizing the step in the method according to any one of the above.

According to the above embodiment of the present disclosure, the characteristics of the template frame and the detection frame are extracted by the neural network, the classification weight of the local region detector and the regression weight are acquired based on the characteristics of the template frame, and the detection frame is obtained. Is input to the local area detector, the classification result and regression result of multiple candidate frames output from the local area detector are acquired, and the classification result and regression of multiple candidate frames output from the local area detector are acquired. Based on the result, the detection frame of the target object in the detection frame is acquired. In the embodiment of the present disclosure, similar features of the same target object can be better extracted by the same neural network or a neural network having the same configuration, the feature change of the target object extracted from different frames is small, and the detection frame. Contributes to improving the accuracy of the detection result of the target object in. In addition, by acquiring the classification weight and the regression weight of the local area detector based on the characteristics of the template frame, the local area detector acquires the classification result and the regression result of a plurality of candidate frames of the detection frame, and further. The detection frame of the target object in the detection frame can be acquired, the change in the position and size of the target object can be better estimated, the position of the target object in the detection frame can be determined more accurately, and the target can be determined. The tracking speed and accuracy are high, the tracking effect is excellent, and the speed is fast.

Hereinafter, the technical means of the present disclosure will be described in more detail with reference to the drawings and examples.

The drawings that form part of the specification are for explaining the embodiments of the present disclosure and for interpreting the principles of the present disclosure along with the description for the embodiments.
The following detailed description, accompanied by drawings, provides a clearer understanding of the present disclosure.
It is a flowchart of one Example of the target detection method of this disclosure. It is a flowchart of another embodiment of the target detection method of this disclosure. It is a flowchart of one Example of the training method of the target detection network of this disclosure. It is a flowchart of another embodiment of the training method of the target detection network of this disclosure. It is a block diagram of one Example of the target detection apparatus of this disclosure. It is a block diagram of another Example of the target detection apparatus of this disclosure. It is a block diagram of another Example of the target detection apparatus of this disclosure. It is a block diagram of one application example of the target detection apparatus of this disclosure. It is a block diagram of another application embodiment of the target detection apparatus of this disclosure. It is a block diagram of one application example of the electronic device of this disclosure.

Here, various exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. It should be noted that unless otherwise stated, the relative arrangements, formulas and values of members and steps described in these examples do not limit the scope of the present disclosure.

Further, it should be understood that in the embodiments of the present disclosure, "plurality" may refer to two or more, and "at least one" may refer to one, two or more. ..

Terms such as "first" and "second" in the embodiments of the present application are merely for distinguishing different steps, devices, modules, etc., and may represent a specific technical meaning or are inevitable. It is understandable to those skilled in the art that it does not represent a logical order.

Further, any member, data or structure referred to in this disclosure may generally be understood as one or more unless expressly limited or when the opposite is suggested before and after the specification. You should understand that.

Furthermore, it is understood that in the present disclosure, each embodiment will be described with emphasis on differences, and the same or similar points can be referred to each other, and therefore, for the sake of brevity, they will not be described repeatedly. Should.

At the same time, for convenience of explanation, it should be understood that the dimensions of each part shown in the drawings are not drawn according to the actual proportional relationship.

The description for at least one exemplary example below is merely for illustration purposes and does not limit the disclosure and its application or use in any way.

Techniques, methods and equipment known to those skilled in the art may not be described in detail, but in some cases said techniques, methods and equipment should be considered as reference as part of the specification.

It should be noted that in the drawings below, similar symbols and letters represent similar items, and if an item is defined in one drawing, it does not need to be further described in subsequent drawings.

The embodiments of the present disclosure are applicable to electronic devices such as terminal devices, computer systems, servers, etc., which can operate in many other general purpose or dedicated computing system environments or configurations. Examples of known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers include personal computer systems, server computer systems, thin clients, fat clients, etc. Handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, small computer systems, large computer systems and distributed cloud computing technology environments including any of the above systems. Including, but not limited to.

Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable commands (eg, program modules) executed by the computer system. In general, a program module may include routines, programs, objective programs, components, logics, data structures, etc. that perform a particular task or implement a particular abstract data type. The computer system / server may be implemented in a distributed cloud computing environment. In a decentralized cloud computing environment, tasks are performed by remote processing devices linked over a communication network. In a distributed cloud computing environment, the program module may reside on the storage medium of a local or remote computing system, including storage equipment.

FIG. 1 is a flowchart of an embodiment of the target detection method of the present disclosure. As shown in FIG. 1, the target detection method of the embodiment includes the following operations.

102. The features of the template frame and the detection frame are extracted by the neural network.
Here, the template frame is an image of the detection frame of the target object, the size of the image of the template frame is smaller than the detection frame, and the detection frame is the current frame for detecting the target object or the target object in the current frame. It is a possible area image of. When the detection frame is a region image in which the target object can exist in the current frame for detecting the target object, in one embodiment of each embodiment of the present disclosure, the size of the region image is the image of the template frame. Larger, for example, the region image may be 2-4 times larger than the size of the template frame image, with the center point of the image of the template frame as the center point.

In one embodiment of each embodiment of the present disclosure, the template frame is a frame in which the detection timing is located before the detection frame in the video sequence and the detection frame of the target object is specified, and the target tracking is performed in the video sequence. It may be the start frame to be performed, and the position of the start frame in the video frame sequence can be flexibly set, for example, the first frame in the video frame sequence or any intermediate frame. The detection frame is a frame for tracking the target, and after the detection frame of the target object is specified in the image of the detection frame, the image corresponding to the detection frame in the detection frame may be used as the image of the template frame of the next detection frame. ..

In one embodiment of each embodiment of the present disclosure, in the operation 102, the features of the template frame and the detection frame may be extracted by the same neural network, or the template frame and the template frame may be extracted by different neural networks having the same configuration. The characteristics of each detection frame may be extracted.

In one selectable example, the operation 102 may be executed by invoking a corresponding command stored in memory by the processor or by a neural network running on the processor.

104. Acquire the classification weight and the regression weight of the local region detector based on the characteristics of the template frame.

In one embodiment of each embodiment of the present disclosure, a convolution operation is performed on the features of the template frame by the first convolution layer, and the first feature acquired by the convolution operation is used as the weight of the classification of the local region detector. You may try to do it.

For example, in a selectable example, the first convolution layer increases the number of channels in the template frame feature so that the number of channels is 2k (k is an integer greater than 0) times the number of channels in the template frame feature. The weight of the classification of the local region detector can be obtained so as to obtain the first feature.

In one embodiment of each embodiment of the present disclosure, a second convolution layer performs a convolution operation on the features of the template frame, and the second feature acquired by the convolution operation is used as the regression weight of the local region detector. You may try to do it.

For example, in a selectable example, the second convolution layer increases the number of channels in the template frame feature so that the number of channels is 4k (k is an integer greater than 0) times the number of channels in the template frame feature. The regression weight of the local region detector can be obtained so as to obtain the second feature.

In a selectable example, the operation 104 may be executed by invoking a corresponding command stored in memory by the processor, or may be executed by a first convolution layer and a second convolution layer operated by the processor, respectively. Good.

106, the characteristics of the detection frame are input to the local area detector, and the classification result and the regression result of a plurality of candidate frames output from the local area detector are acquired.
Here, the classification result includes the probability value which is the detection frame of the target object of each candidate frame, and the regression result includes the deviation amount from the detection frame corresponding to the template frame of each candidate frame.

In a selectable example of each embodiment of the present disclosure, the plurality of candidate frames may include K candidate frames at each position of the detection frame. Here, K is a preset integer greater than 1. The ratio of length and width of K candidate frames is different, for example, the ratio of length and width of K candidate frames is 1: 1, 2: 1, 2: 1, 3: 1, 1: 1. 3 etc. may be included. The classification result is for expressing the probability value that the K candidate frames at each position are the detection frames of the target object.

In one selectable embodiment of the target detection method of the present disclosure, after the probability value that a plurality of candidate frames are the detection frames of the target object is acquired by the operation 106, the classification result is further normalized. Is performed, and the sum of the probability values in which each candidate frame is the detection frame of the target object is set to 1. In this way, it contributes to easily determining whether or not each candidate frame is a detection frame for the target object.

In a selectable example of each embodiment of the present disclosure, the regression result includes the amount of deviation of each of the K candidate frames at each position of the detection frame image from the detection frame of the target object in the template frame. The amount of deviation may include the amount of change in position and size, and this position may be the position of the center point, the position of the four vertices of the reference frame, or the like.

When the number of channels of the second feature is 4k times the number of channels of the feature of the template frame, the amount of deviation of each candidate frame from the detection frame of the target object in the template frame is, for example, the position of the center point. The abscissa deviation amount (dx), the vertical coordinate deviation amount (dy) of the position of the center point, the height change amount (dh), and the width change amount (dw) may be included.

In one embodiment of each embodiment of the present disclosure, in the operation 106, a convolution operation is performed on the features of the detection frame using the classification weights to acquire the classification results of a plurality of candidate frames, and regression. It may include performing a convolution operation on the feature of the detection frame using the weight of and acquiring the regression results of a plurality of candidate frames.

In one selectable example, the operation 106 may be executed by invoking the corresponding command stored in memory by the processor or by a local region detector operated by the processor.

108, The detection frame of the target object in the detection frame is acquired from the classification result and the regression result of the plurality of candidate frames output from the local area detector.

In a selectable example, the operation 108 may be executed by invoking a corresponding command stored in memory by the processor or by an acquisition unit operated by the processor.

According to the target detection method of the above embodiment of the present disclosure, the characteristics of the template frame and the detection frame are extracted by the neural network, and the classification weight and the regression weight of the local region detector are obtained based on the characteristics of the template frame. Then, the characteristics of the detection frame are input to the local area detector, the classification result and regression result of the multiple candidate frames output from the local area detector are acquired, and the multiple candidate frames output from the local area detector are acquired. The detection frame of the target object in the detection frame is acquired from the classification result and the regression result. In the embodiment of the present disclosure, similar features of the same target object can be better extracted by the same neural network or a neural network having the same configuration, the feature change of the target object extracted from different frames is small, and the detection frame. Contributes to improving the accuracy of the detection result of the target object in. In addition, by acquiring the classification weight and the regression weight of the local area detector based on the characteristics of the template frame, the local area detector acquires the classification result and the regression result of a plurality of candidate frames of the detection frame, and further. The detection frame of the target object in the detection frame can be acquired, the change in the position and size of the target object can be better estimated, the position of the target object in the detection frame can be determined more accurately, and the target can be determined. The tracking speed and accuracy are high, the tracking effect is excellent, and the speed is fast.

In the embodiment of the present disclosure, based on the template frame, the local region detector quickly generates a large number of candidate frames from the detection frame, and each of the K candidate frames at each position of the detection frame is a template frame. The amount of deviation from the detection frame of the target object can be obtained, the change in the position and size of the target object can be better estimated, and the position of the target object in the detection frame can be determined more accurately. The speed and accuracy of target tracking are high, the tracking effect is excellent, and the speed is fast.

In another embodiment of the target detection method of the present disclosure,
A neural network is used to extract the features of at least one other detection frame that chronologically follows the detection frame in the video sequence.
The features of the at least one other detection frame are input to the local region detector in order, and the plurality of candidate frames in the at least one other detection frame output from the local region detector and the classification result and regression of each candidate frame. Acquiring the results in sequence, that is, performing operation 106 on the features of at least one of the other detection frames in sequence,
Based on the classification result and the regression result of the plurality of candidate frames of the at least one other detection frame, the detection frames of the target object in the at least one other detection frame are sequentially acquired, that is, the at least one other detection frame is sequentially acquired. It may further include executing operation 108 on the classification result and the regression result of a plurality of candidate frames of the detection frame.

In another embodiment of the target detection method of the present disclosure, when the detection frame is a region image in which the target object can exist in the current frame for detecting the target object, the center point of the template frame is further set in advance. As the center point, a region image whose length and / or width is larger than the length and / or width of the image of the template frame, respectively, may be cut out from the current frame and used as a detection frame.

FIG. 2 is a flowchart of another embodiment of the target detection method of the present disclosure. As shown in FIG. 2, the target detection method of the embodiment includes the following operations.

202, the features of the template frame and the detection frame are extracted by the neural network.
Here, the template frame is an image of the detection frame of the target object, the image size of the template frame is smaller than the detection frame, and the detection frame is the current frame for detecting the target object or the target object in the current frame. It is a region image that can exist. The template frame is a frame in which the detection timing is positioned before the detection frame in the video sequence and the detection frame of the target object is specified.

In one embodiment of each embodiment of the present disclosure, in the operation 202, the features of the template frame and the detection frame may be extracted by the same neural network, or the template frame and the template frame may be extracted by different neural networks having the same configuration. The characteristics of each detection frame may be extracted.

In one selectable example, the operation 202 may be executed by invoking a corresponding command stored in memory by the processor or by a neural network running on the processor.

204, the third convolution layer performs a convolution operation on the features of the detection frame, the number of channels acquires the third feature similar to the number of channels of the features of the detection frame, and the fourth convolution layer of the detection frame. A convolution operation is performed on the feature, and a fourth feature whose number of channels is the same as the number of channels of the feature of the detection frame is acquired.

In a selectable example, the operation 204 may be executed by invoking a corresponding command stored in memory by the processor, or may be executed by a third convolution layer and a fourth convolution layer operated by the processor, respectively. Good.

206, Obtain the classification weight and regression weight of the local region detector based on the characteristics of the template frame.

In one embodiment of each embodiment of the present disclosure, a convolution operation is performed on the features of the template frame by the first convolution layer, and the first feature acquired by the convolution operation is used as the weight of the classification of the local region detector. You may try to do it.

In one embodiment of each embodiment of the present disclosure, a second convolution layer performs a convolution operation on the features of the template frame, and the second feature acquired by the convolution operation is used as the regression weight of the local region detector. You may try to do it.

Here, the operations 206 and 204 are not limited in the execution order, and may be executed at the same time, or may be executed in any first-after order.

In a selectable example, the operation 206 may be executed by invoking a corresponding command stored in memory by the processor, or may be executed by a first convolution layer and a second convolution layer operated by the processor, respectively. Good.

208, the convolution operation is performed on the third feature using the classification weights, the classification results of a plurality of candidate frames are acquired, and the convolution operation is performed on the fourth feature using the regression weights. Get the regression result of the candidate frame of.
Here, the classification result includes the probability value which is the detection frame of the target object of each candidate frame, and the regression result includes the deviation amount from the detection frame corresponding to the template frame of each candidate frame.

In one selectable example, the operation 208 may be executed by invoking the corresponding command stored in memory by the processor or by a local region detector operated by the processor.

210, The detection frame of the target object in the detection frame is acquired from the classification result and the regression result of the plurality of candidate frames output from the local area detector.

In one selectable example, the operation 210 may be executed by invoking a corresponding command stored in memory by the processor or by an acquisition unit operated by the processor.

In one embodiment of each embodiment of the present disclosure, in operation 108 or 210, one candidate frame is selected from a plurality of candidate frames based on the classification result and the regression result, and is selected according to the deviation amount of the selected candidate frames. It may include regressing the candidate frame and acquiring the detection frame of the target object in the detection frame.

In one selectable example, when one candidate frame is selected from a plurality of candidate frames based on the classification result and the regression result, one candidate frame is selected from the plurality of candidate frames based on the weighting coefficient of the classification result and the regression result. Based on the weighting coefficients of the classification result and the regression result, the overall score is calculated based on the sum of the product of the probability value of each candidate frame and the weighting coefficient of the classification result and the product of the deviation amount and the weighting coefficient of the regression result. Then, it can be realized that one candidate frame is selected from the plurality of candidate frames based on the total score of the plurality of candidate frames.

Another selectable example may include, after obtaining the regression results by each of the above embodiments, further adjusting the probability value of the candidate frame according to the amount of change in the position and size of the regression results. For example, the probability value of the candidate frame is adjusted according to the amount of change in the position and size of the regression result. For example, a penalty is given to the probability value of a candidate frame in which the amount of change in position is large (that is, the amount of position movement is large) and the amount of change in size is large (that is, the shape change is large), and the probability value is lowered. To do. Correspondingly, in this example, when one candidate frame is selected from a plurality of candidate frames based on the classification result and the regression result, one candidate frame is selected from the plurality of candidate frames based on the adjusted classification result, for example. It can be realized that the candidate frame having the highest probability value is selected from a plurality of candidate frames according to the adjusted probability value.

In one selectable example, the above operation of adjusting the probability value of the candidate frame according to the amount of change in the position and size of the regression result may be executed by calling the corresponding command stored in the memory by the processor, and the processor may execute the operation. It may be performed by an operating adjustment unit.

FIG. 3 is a flowchart of an embodiment of the training method of the target detection network of the present disclosure. The target detection network of the embodiments of the present disclosure includes the neural network of the embodiments of the present disclosure, a first convolution layer and a second convolution layer. As shown in FIG. 3, the training method of the embodiment includes the following operations.

302, the features of the template frame and the detection frame are extracted by the neural network.
Here, the template frame is an image of the detection frame of the target object, the image size of the template frame is smaller than the detection frame, and the detection frame is the current frame for detecting the target object or the target object in the current frame. It is a region image that can exist. The template frame is a frame in which the detection timing is positioned before the detection frame in the video sequence and the detection frame of the target object is specified.

In one embodiment of each embodiment of the present disclosure, in the operation 302, the features of the template frame and the detection frame may be extracted by the same neural network, or the template frame and the template frame may be extracted by different neural networks having the same configuration. The characteristics of each detection frame may be extracted.

In one selectable example, the operation 302 may be executed by invoking a corresponding command stored in memory by the processor or by a neural network running on the processor.

304, the first convolution layer performs a convolution operation on the features of the template frame, the first feature acquired by the convolution operation is used as the weight of the classification of the local area detector, and the second convolution layer of the template frame. A convolution operation is performed on the feature, and the second feature acquired by the convolution operation is used as the regression weight of the local region detector.

In a selectable example, the operation 304 may be executed by invoking a corresponding command stored in memory by the processor, or may be executed by a first convolution layer and a second convolution layer operated by the processor, respectively. Good.

306, The characteristics of the detection frame are input to the local area detector, and the classification result and the regression result of a plurality of candidate frames output from the local area detector are acquired.
Here, the classification result includes the probability value which is the detection frame of the target object of each candidate frame, and the regression result includes the deviation amount from the detection frame corresponding to the template frame of each candidate frame.

In one embodiment of each embodiment of the present disclosure, the operation 306 performs a convolution operation on the features of the detection frame using the classification weights to acquire the classification results of a plurality of candidate frames, and regresses. It may include performing a convolution operation on the feature of the detection frame using the weight and acquiring the regression result of a plurality of candidate frames.

In one selectable example, the operation 306 may be executed by invoking the corresponding command stored in memory by the processor or by a region detector operated by the processor.

308, The detection frame of the target object in the detection frame is acquired from the classification result and the regression result of a plurality of candidate frames output from the local area detector.

In one selectable example, the operation 308 may be executed by invoking a corresponding command stored in memory by the processor or by an acquisition unit operated by the processor.

310. Using the detection frame of the target object in the acquired detection frame as the prediction detection frame, the neural network, the first convolution layer, and the second convolution layer are trained based on the labeling information of the detection frame and the prediction detection frame.

In one selectable example, the operation 310 may be executed by invoking a corresponding command stored in memory by the processor or by a training unit operated by the processor.

According to the target detection network training method of the above-described embodiment of the present disclosure, the characteristics of the template frame and the detection frame are extracted by the neural network, and the weight and regression of the classification of the local region detector based on the characteristics of the template frame. The weights are acquired, the characteristics of the detection frame are input to the local area detector, the classification results and regression results of multiple candidate frames output from the local area detector are acquired, and multiple output from the local area detector. Based on the classification result and regression result of the candidate frame, the detection frame of the target object in the detection frame is acquired, and the target detection network is trained based on the labeling information of the detection frame and the prediction detection frame. According to the target detection network trained based on the examples of the present disclosure, similar features of the same target object can be better extracted by the same neural network or a neural network having the same configuration, and extracted from different frames. The characteristic change of the target object is small, which contributes to improving the accuracy of the detection result of the target object in the detection frame. In addition, by acquiring the classification weight and the regression weight of the local area detector based on the characteristics of the template frame, the local area detector acquires the classification result and the regression result of a plurality of candidate frames of the detection frame, and further. The detection frame of the target object in the detection frame can be acquired, the change in the position and size of the target object can be better estimated, the position of the target object in the detection frame can be determined more accurately, and the target can be determined. The tracking speed and accuracy are high, the tracking effect is excellent, and the speed is fast.

In another embodiment of the training method of the present disclosure, a neural network is used to extract features of at least one other detection frame that chronologically follows the detection frame in the video sequence.
The features of at least one other detection frame are input to the local region detector in order, and the classification result and regression result of a plurality of candidate frames and each candidate frame in at least one other detection frame output from the local region detector are input. Acquiring in sequence, i.e. performing operation 306 on features of at least one other detection frame in sequence.
Based on the classification result and the regression result of the plurality of candidate frames of at least one other detection frame, the detection frames of the target object in at least one other detection frame are sequentially acquired, that is, the detection frames of at least one other detection frame are sequentially acquired. The operation 308 may be further included for the classification result and the regression result of a plurality of candidate frames.

In another embodiment of the training method of the present disclosure, when the detection frame is a region image in which the target object can exist in the current frame for detecting the target object, the center point of the template frame is set as the center point in advance. It may further include cutting out a region image whose length and / or width is larger than the length and / or width of the image of the template frame, respectively, from the current frame to obtain a detection frame.

FIG. 4 is a flowchart of another embodiment of the training method of the target detection network of the present disclosure. The target detection network of the embodiments of the present disclosure includes the neural network of the embodiments of the present disclosure, a first convolution layer, a second convolution layer, a third convolution layer, and a fourth convolution layer. As shown in FIG. 4, the training method of the embodiment includes the following operations.

402, the features of the template frame and the detection frame are extracted by the neural network.
Here, the template frame is an image of the detection frame of the target object, the image size of the template frame is smaller than the detection frame, and the detection frame is the current frame for detecting the target object or the target object in the current frame. It is a region image that can exist. The template frame is a frame in which the detection timing is positioned before the detection frame in the video sequence and the detection frame of the target object is specified.

In one embodiment of each embodiment of the present disclosure, in the operation 402, the features of the template frame and the detection frame may be extracted by the same neural network, or the template frame and the template frame may be extracted by different neural networks having the same configuration. The characteristics of each detection frame may be extracted.

In one selectable example, the operation 402 may be executed by invoking a corresponding command stored in memory by the processor or by a neural network running on the processor.

404, the third convolution layer performs a convolution operation on the features of the detection frame, the number of channels acquires the third feature similar to the number of channels of the features of the detection frame, and the fourth convolution layer of the detection frame. A convolution operation is performed on the feature, and a fourth feature whose number of channels is the same as the number of channels of the feature of the detection frame is acquired.

In a selectable example, the operation 404 may be executed by invoking a corresponding command stored in memory by the processor, or may be executed by a third convolution layer and a fourth convolution layer operated by the processor, respectively. Good.

406, the first convolution layer performs a convolution operation on the features of the template frame, the first feature acquired by the convolution operation is used as the weight of the classification of the local area detector, and the second convolution layer of the template frame. A convolution operation is performed on the feature, and the second feature acquired by the convolution operation is used as the regression weight of the local region detector.

Here, the operations 406 and 404 are not limited in the execution order, and may be executed at the same time, or may be executed in any first-after order.

In a selectable example, the operation 406 may be executed by invoking a corresponding command stored in memory by the processor, or may be executed by a first convolution layer and a second convolution layer operated by the processor, respectively. Good.

408, the convolution operation is performed on the third feature using the classification weight, the classification results of a plurality of candidate frames are acquired, and the convolution operation is performed on the fourth feature using the regression weight. Get the regression result of the candidate frame of.
Here, the classification result includes the probability value which is the detection frame of the target object of each candidate frame, and the regression result includes the deviation amount from the detection frame corresponding to the template frame of each candidate frame.

In a selectable example, the operation 408 may be executed by invoking a corresponding command stored in memory by the processor or by a local region detector operated by the processor.

410, The detection frame of the target object in the detection frame is acquired from the classification result and the regression result of a plurality of candidate frames output from the local area detector.

In a selectable example, the operation 410 may be executed by invoking a corresponding command stored in memory by the processor, or may be executed by a first feature extraction unit 701 running on the processor.

412, the detection frame of the target object in the acquired detection frame is used as the prediction detection frame, and the difference between the position and size of the detection frame of the target object in the labeled detection frame and the position and size of the prediction detection frame Adjust the weight values of the neural network, the first convolution layer and the second convolution layer.

In one selectable example, the operation 412 may be executed by invoking a corresponding command stored in memory by the processor or by a training unit operated by the processor.

In one embodiment of each embodiment of the present disclosure, in operation 308 or 410, one candidate frame is selected from a plurality of candidate frames based on the classification result and the regression result, and is selected according to the deviation amount of the selected candidate frames. It may include regressing the candidate frame and acquiring the detection frame of the target object in the detection frame.

In one selectable example, when one candidate frame is selected from a plurality of candidate frames based on the classification result and the regression result, one candidate frame is selected from the plurality of candidate frames based on the weighting coefficient of the classification result and the regression result. Based on the weighting coefficients of the classification result and the regression result, the overall score is calculated based on the sum of the product of the probability value of each candidate frame and the weighting coefficient of the classification result and the product of the deviation amount and the weighting coefficient of the regression result. Then, it is possible to select a candidate frame having a high probability value and a small deviation amount from the plurality of candidate frames based on the total score of the plurality of candidate frames.

Another selectable example may include, after obtaining the regression results by each of the above embodiments, further adjusting the probability value of the candidate frame according to the amount of change in the position and size of the regression results. For example, the probability value of the candidate frame is adjusted according to the amount of change in the position and size of the regression result. Correspondingly, in this example, when one candidate frame is selected from a plurality of candidate frames based on the classification result and the regression result, one candidate frame is selected from the plurality of candidate frames based on the adjusted classification result, for example. It can be realized that the candidate frame having the highest probability value is selected from a plurality of candidate frames according to the adjusted probability value.

In one selectable example, the operation of adjusting the probability value of the candidate frame according to the amount of change in the position and size of the regression result may be executed by calling the corresponding command stored in the memory by the processor, and is operated by the processor. It may be performed by the adjusting unit.

In one embodiment of each embodiment of the present disclosure, in operation 308 or 410, one candidate frame is selected from a plurality of candidate frames based on the classification result and the regression result, and is selected according to the deviation amount of the selected candidate frames. It may include regressing the candidate frame and acquiring the detection frame of the target object in the detection frame.

In one selectable example, when one candidate frame is selected from a plurality of candidate frames based on the classification result and the regression result, one candidate frame is selected from the plurality of candidate frames based on the weighting coefficient of the classification result and the regression result. The overall score is calculated based on the probability value and the amount of deviation of each candidate frame based on the weighting coefficient of the classification result and the regression result, and one candidate frame from the plurality of candidate frames is calculated based on the overall score of the plurality of candidate frames. Can be realized to select.

Another selectable example may include, after obtaining the regression results by each of the above embodiments, further adjusting the probability value of the candidate frame according to the amount of change in the position and size of the regression results. For example, the probability value of the candidate frame is adjusted according to the amount of change in the position and size of the regression result. For example, a penalty is given to the probability value of a candidate frame in which the amount of change in position is large (that is, the amount of position movement is large) and the amount of change in size is large (that is, the shape change is large), and the probability value is lowered. To do. Correspondingly, in this example, when one candidate frame is selected from a plurality of candidate frames based on the classification result and the regression result, one candidate frame is selected from the plurality of candidate frames based on the adjusted classification result, for example. It can be realized that the candidate frame having the highest probability value is selected from a plurality of candidate frames according to the adjusted probability value.

In each embodiment of the present disclosure, the local region detector may include a third convolution layer, a fourth convolution layer and two convolution operation units. Here, the local region detector formed by combining the local region detector, the first convolution layer, and the second convolution layer may be referred to as a region proposal network (Region Proposal Network).

Any one of the target detection method and the target detection network training method provided in the embodiments of the present disclosure may be executed by any suitable device having data processing capability, and the device includes a terminal device, a server, and the like. However, it is not limited to them. Alternatively, any one of the target detection method and the target detection network training method provided in the embodiments of the present disclosure may be executed by the processor, for example, the processor calls the corresponding command stored in the memory. In, either one of the target detection method and the target detection network training method according to the embodiment of the present disclosure is executed. Hereinafter, detailed description will be omitted.

Those skilled in the art should understand that all or part of the steps to realize an embodiment of the above method can be completed by programmatically issuing instructions to the relevant hardware, said program being ROM, RAM, magnetic disk or A program code such as an optical disk can be stored in a computer-readable storage medium including various media capable of storing the program code, and when the program is executed, a step including an embodiment of the above method is executed.

FIG. 5 is a schematic configuration diagram of an embodiment of the target detection device of the present disclosure. The target detection device of each embodiment of the present disclosure can be used to realize the embodiment of each of the above-mentioned target detection methods of the present disclosure. As shown in FIG. 5, the target detector of the embodiment includes a neural network, a first convolution layer, a second convolution layer, a local region detector and an acquisition unit.

The neural network is used to extract the features of the detection frame and the image of the detection frame of the target object, and the size of the image is smaller than the detection frame, respectively. Here, the template frame is an image of the detection frame of the target object, the size of the image of the template frame is smaller than the detection frame, and the detection frame is the current frame for detecting the target object or the target object in the current frame. It is a possible area image of. The template frame is a frame in which the detection timing is positioned before the detection frame in the video sequence and the detection frame of the target object is specified. The neural network that extracts the features of the template frame and the detection frame may be the same neural network, or may be different neural networks having the same configuration.

The first convolution layer is used to perform a convolution operation on the features of the template frame, and to use the first features acquired by the convolution operation as the weight of the classification of the local region detector.

The second convolution layer is used to perform a convolution operation on the features of the template frame by the second convolution layer, and to use the second features acquired by the convolution operation as the regression weight of the local region detector. Used.

The local region detector is used to output the classification result and the regression result of a plurality of candidate frames according to the characteristics of the detection frame, and here, the classification result is the probability that the classification result is the detection frame of each target object of each candidate frame. The value is included, and the regression result includes the amount of deviation from the detection frame corresponding to the template frame of each candidate frame.

The acquisition unit is used to acquire the detection frame of the target object in the detection frame based on the classification result and the regression result of a plurality of candidate frames output from the local area detector.

According to the target detection device of the above embodiment of the present disclosure, the characteristics of the template frame and the detection frame are extracted by the neural network, and the classification weight and the regression weight of the local region detector are acquired based on the characteristics of the template frame. Then, the characteristics of the detection frame are input to the local area detector, the classification result and regression result of the multiple candidate frames output from the local area detector are acquired, and the multiple candidate frames output from the local area detector are acquired. The detection frame of the target object in the detection frame is acquired from the classification result and the regression result. In the embodiment of the present disclosure, similar features of the same target object can be better extracted by the same neural network or a neural network having the same configuration, the feature change of the target object extracted from different frames is small, and the detection frame. Contributes to improving the accuracy of the detection result of the target object in. In addition, by acquiring the classification weight and the regression weight of the local area detector based on the characteristics of the template frame, the local area detector acquires the classification result and the regression result of a plurality of candidate frames of the detection frame, and further. The detection frame of the target object in the detection frame can be acquired, the change in the position and size of the target object can be better estimated, the position of the target object in the detection frame can be determined more accurately, and the target can be determined. The tracking speed and accuracy are high, the tracking effect is excellent, and the speed is fast.

In one embodiment of each embodiment of the target detection apparatus of the present disclosure, the local region detector performs a convolution operation on the features of the detection frame using the classification weights to acquire the classification results of a plurality of candidate frames. It is also used to perform a convolution operation on the features of the detection frame using the regression weights and acquire the regression results of a plurality of candidate frames.

In another embodiment of the target detection device of the present disclosure, where the detection frame is an image of a region in which the target object can exist in the current frame for detecting the target object, the center point of the template frame is set as the center point. A preprocessing unit for cutting out a region image whose length and / or width is larger than the length and / or width of the image of the template frame, respectively, from the current frame and making it a detection frame may be included. As shown in FIG. 6, it is a block diagram of another embodiment of the target detection apparatus of this disclosure.

Further, referring to FIG. 6 again, in another embodiment of the target detection device of the present disclosure, a convolution operation is performed on the feature of the detection frame, and the number of channels is the same as the number of channels of the feature of the detection frame. A third convolution layer may be further included to obtain the characteristics of. Correspondingly, in this embodiment, the local region detector is used to perform a convolution operation on the third feature using the classification weights.

Further, referring to FIG. 6 again, in yet another embodiment of the target detection device of the present disclosure, a convolution operation is performed on the feature of the detection frame, and the number of channels is the same as the number of channels of the feature of the detection frame. A fourth convolutional layer for acquiring the characteristics of 4 may be further included. Correspondingly, in this embodiment, the local region detector is used to perform a convolution operation on the fourth feature using the regression weights.

In another embodiment of each embodiment of the target detection apparatus of the present disclosure, the acquisition unit selects one candidate frame from a plurality of candidate frames based on the classification result and the regression result, and selects by the deviation amount of the selected candidate frames. It is used to regress the candidate frame and acquire the detection frame of the target object in the detection frame.

Illustratively, when the acquisition unit selects one candidate frame from a plurality of candidate frames based on the classification result and the regression result, the acquisition unit selects one candidate frame from the plurality of candidate frames based on the weighting coefficient of the classification result and the regression result. Used for.

Also, with reference to FIG. 6 again, in yet another embodiment of the target detection apparatus of the present disclosure, an adjustment unit for adjusting the classification result based on the regression result may be further included. Correspondingly, the acquisition unit is used to select one candidate frame from a plurality of candidate frames according to the adjusted classification result when selecting one candidate frame from a plurality of candidate frames based on the classification result and the regression result. ..

FIG. 7 is a schematic configuration diagram of still another embodiment of the target detection device of the present disclosure. The target detection device of the embodiment can be used to realize any one of the examples of the training method of the target detection network of FIGS. 3 to 4 of the present disclosure. As shown in FIG. 7, as compared with the embodiment shown in FIG. 5 or 6, the target detection device of the embodiment uses the detection frame of the target object in the acquired detection frame as the prediction detection frame, and the detection frame of the detection frame. It further includes a neural network, a training unit for training the first convolution layer and the second convolution layer based on the labeling information and the predictive detection frame.

In one embodiment, the detection frame labeling information includes the position and size of the detection frame of the target object in the labeled detection frame. Correspondingly, in the embodiment, the training unit has a neural network, a first convolution layer and a second convolution layer due to the difference between the position and size of the labeled detection frame and the position and size of the predicted detection frame. It is used to adjust the weight value of.

According to the above embodiment of the present disclosure, the characteristics of the template frame and the detection frame are extracted by the neural network, the classification weight of the local region detector and the regression weight are acquired based on the characteristics of the template frame, and the detection frame is obtained. Is input to the local area detector, the classification result and regression result of multiple candidate frames output from the local area detector are acquired, and the classification result and regression of multiple candidate frames output from the local area detector are acquired. Based on the result, the detection frame of the target object in the detection frame is acquired, and the target detection network is trained based on the labeling information of the detection frame and the prediction detection frame. According to the target detection network trained based on the examples of the present disclosure, similar features of the same target object can be better extracted by the same neural network or a neural network having the same configuration, and extracted from different frames. The characteristic change of the target object is small, which contributes to improving the accuracy of the detection result of the target object in the detection frame. In addition, by acquiring the classification weight and the regression weight of the local area detector based on the characteristics of the template frame, the local area detector acquires the classification result and the regression result of a plurality of candidate frames of the detection frame, and further. The detection frame of the target object in the detection frame can be acquired, the change in the position and size of the target object can be better estimated, the position of the target object in the detection frame can be determined more accurately, and the target can be determined. The tracking speed and accuracy are high, the tracking effect is excellent, and the speed is fast.

FIG. 8 is a schematic configuration diagram of an application embodiment of the target detection device of the present disclosure. FIG. 9 is a schematic configuration diagram of another application embodiment of the target detection device of the present disclosure. In FIGS. 8 and 9, at LxMxN (eg, 256x20x20), L represents the number of channels and M and N represent the height (ie, length) and width, respectively.

The embodiments of the present disclosure further provide an electronic device that includes a target detector of any one of the above embodiments of the present disclosure.

The embodiments of the present disclosure include a memory for storing an executable command and a target detection method or target detection network of any one of the above embodiments of the present disclosure by communicating with the memory and executing the executable command. Further provides additional electronic equipment, including a processor for completing the operation of the training method.

FIG. 10 is a schematic configuration diagram of an application embodiment of the electronic device of the present disclosure. Hereinafter, FIG. 10 will be referred to, which shows a schematic configuration diagram of an electronic device suitable for realizing the terminal device or server according to the embodiment of the present application. As shown in FIG. 10, the electronic device includes one or more processors, a communication unit, and the like. The one or more processors are, for example, one or more central processing units (CPUs) and / or one or more image processors (GPUs), where the processors are in read-only memory (ROM). Various appropriate operations and processes can be realized by the stored executable command or the executable command loaded from the storage unit into the random access memory (RAM). The communication unit may include, but is not limited to, a network card, the network card may include, but is not limited to, an IB (Infiniband) network card. The processor communicates with a read-only memory and / or a random access memory to execute an executable command, is connected to a communication unit via a bus, and communicates with another target device via the communication unit, according to the present disclosure. The operation corresponding to any one of the methods provided in the examples may be completed. For example, the neural network extracts the features of the detection frame and the image of the detection frame of the target object, and the size of the image is smaller than the detection frame, respectively, and localizes based on the features of the template frame. The classification weight and the regression weight of the region detector are acquired, the characteristics of the detection frame are input to the local region detector, and the classification result and the regression result of a plurality of candidate frames output from the local region detector are input. The detection frame of the target object in the detection frame is acquired from the classification result and the regression result of the plurality of candidate frames output from the local area detector. Further, for example, the features of the detection frame and the image of the detection frame of the target object, which are smaller than the detection frame, are extracted by the neural network, and the first convolution layer is used to extract the features of the template frame. The channel of the feature of the template frame is increased, the acquired first feature is used as the weight of the classification of the local region detector, and the channel of the feature of the template frame is increased by the second convolution layer, and the obtained first feature is obtained. The feature 2 is used as the regression weight of the local region detector, the feature of the detection frame is input to the local region detector, and the classification result and the regression result of a plurality of candidate frames output from the local region detector are input. The detection frame of the target object in the detection frame is acquired based on the classification result and the regression result of the plurality of candidate frames acquired and output from the local region detector, and the target object in the acquired detection frame. The detection frame is used as a prediction detection frame, and the neural network, the first convolution layer, and the second convolution layer are trained based on the labeling information of the detection frame and the prediction detection frame.

Further, the RAM may store various programs and data necessary for the operation of the device. The CPU, ROM and RAM are connected to each other via a bus. If you have RAM, ROM is a selectable module. The RAM stores an executable command for causing the processor to perform an operation corresponding to any one of the above methods of the present disclosure, or writes the executable command to the ROM during operation. The input / output (I / O) interface is also connected to the bus. The communication unit may be installed in an integrated manner, may have a plurality of submodules (for example, a plurality of IB network cards), and may be installed so as to be linked via a bus.

An input unit including a keyboard, a mouse, etc., an output unit including a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, etc., a storage unit including a hard disk, etc., and a network interface such as a LAN card, a modem, etc. The communication unit including the card is connected to the I / O interface. The communication unit executes communication processing via a network such as the Internet. The drive is also connected to the I / O interface as needed. For example, a removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is mounted on a drive as needed, and a computer program read from the removable medium is installed in a storage unit as needed.

The architecture shown in FIG. 10 is only one selectable embodiment. In a specific practical process, the number and type of parts shown in FIG. 10 can be selected, deleted, added, or replaced as actually required. For the installation of parts with different functions, implementation methods such as individual installation or integrated installation can be adopted. For example, the GPU and CPU are installed separately or the GPU is integrated in the CPU, and the communication unit is individually installed. It can be installed, or it can be installed in a CPU or GPU in an integrated manner. All of these alternative embodiments are within the scope of protection of the present application.

Further, the embodiment of the present disclosure is a computer storage medium for storing a computer-readable command, and when the command is executed, the target detection method or the target detection network training method of the above-described embodiment of the present disclosure is executed. Further provided is a computer storage medium in which any one of the above operations is realized.

Further, the embodiment of the present disclosure is a computer program including a computer-readable command, and when the computer-readable command is activated on the device, the target detection method or the target of the above-described embodiment of the present disclosure is applied to the processor in the device. Further provided is a computer program that executes an executable command to implement any one of the detection network training methods.

The embodiments of the present disclosure can perform single target tracking. For example, in a multi-target tracking system, it is not necessary to perform target detection for each frame, and detection is performed once every 10 frames at a predetermined detection interval, and 9 frames in the meantime are detected in an intermediate frame by single target tracking. The position of the target may be specified. Due to the high speed of the algorithms in the embodiments of the present disclosure, the overall multi-target tracking system can complete the tracking faster and achieve better effects.

Those skilled in the art should understand that all or part of the steps to realize an embodiment of the above method can be completed by programmatically issuing instructions to the relevant hardware, said program being ROM, RAM, magnetic disk or A program code such as an optical disk can be stored in a computer-readable storage medium including various media capable of storing the program code, and when the program is executed, a step including an embodiment of the above method is executed.

The various examples herein have been described incrementally, with each example focused on the differences from the other examples, but the same or similar parts of each example are referred to each other. Just do it. Since the system embodiment basically corresponds to the method embodiment, it has been briefly described, but the related part may refer to a part of the description of the method embodiment.

The methods and devices of the present disclosure can be realized in various forms. For example, the methods and devices of the present disclosure can be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The order of the steps of the method is merely for illustration purposes, and the steps of the methods of the present disclosure are not limited to the specifically described order, unless specifically described in other embodiments. Also, in some embodiments, the disclosure may be programs stored on a recording medium, which include machine-readable commands to implement the methods of the disclosure. Accordingly, the present disclosure also includes a recording medium in which a program for executing the method of the present disclosure is stored.

The description of the present application is for illustration and explanation purposes only and is not intended to be complete or limited to the disclosed form. Many modifications and modifications are apparent to those skilled in the art. The purpose of selecting and explaining the examples is to better explain the principles and practical applications of the present application, to allow those skilled in the art to understand the present application, and to design various examples with various modifications suitable for a specific application. is there.

Claims (43)

Extracting the features of the detection frame and the template frame, which is the image of the detection frame of the target object and the size of the image is smaller than the detection frame, by the neural network, respectively.
Obtaining the classification weight and regression weight of the local region detector based on the characteristics of the template frame,
The characteristics of the detection frame are input to the local area detector, and the classification result and the regression result of a plurality of candidate frames output from the local area detector are acquired.
A target detection method comprising acquiring a detection frame for a target object in the detection frame based on a classification result and a regression result of a plurality of candidate frames output from the local region detector. The neural network is used to extract features of at least one other detection frame that are chronologically located after the detection frame in the video sequence.
The features of the at least one other detection frame are input to the local region detector in order, and a plurality of candidate frames and the classification result of each candidate frame in the at least one other detection frame output from the local region detector. And to get the regression results in order,
The feature is that the detection frames of the target object in the at least one other detection frame are sequentially acquired based on the classification result and the regression result of the plurality of candidate frames of the at least one other detection frame. The method according to claim 1. Extracting the characteristics of the template frame and the detection frame by the neural network is
Extracting the features of the template frame and the detection frame by the same neural network, or
The method according to claim 1 or 2, wherein the features of the template frame and the detection frame are extracted by separate neural networks having the same configuration. The template frame according to any one of claims 1 to 3, wherein the detection timing is located before the detection frame in the video sequence, and the detection frame of the target object is specified. the method of. The method according to any one of claims 1 to 4, wherein the detection frame is a current frame for detecting the target object or a region image in which the target object can exist in the current frame. .. When the detection frame is a region image in which the target object can exist in the current frame for detecting the target object,
It further includes cutting out a region image having a center point of the template frame as a center point and having a length and / or width larger than the length and / or width of the image of the template frame, respectively, from the current frame to obtain the detection frame. The method according to claim 5, wherein the method is characterized by the above. Obtaining the classification weights of the local region detector based on the characteristics of the template frame
A claim comprising performing a convolution operation on the features of the template frame by the first convolution layer and using the first features acquired by the convolution operation as weights for classification of the local region detector. Item 8. The method according to any one of Items 1 to 6. Obtaining the regression weights of the local region detector based on the characteristics of the template frame
A claim comprising performing a convolution operation on the features of the template frame by the second convolution layer and using the second features acquired by the convolution operation as the regression weight of the local region detector. Item 8. The method according to any one of Items 1 to 7. It is possible to input the characteristics of the detection frame into the local area detector and acquire the classification result and the regression result of a plurality of candidate frames output from the local area detector.
A convolution operation is performed on the features of the detection frame using the weights of the classification to acquire classification results of a plurality of candidate frames, and a convolution operation is performed on the features of the detection frame using the weights of the regression. The method according to any one of claims 1 to 8, wherein the method is to obtain the regression results of a plurality of candidate frames, and to include. After extracting the features of the detection frame, a convolution operation is performed on the features of the detection frame by the third convolution layer to acquire a third feature whose number of channels is the same as the number of channels of the features of the detection frame. Including that
Acquiring the classification results of a plurality of candidate frames by performing a convolution operation on the feature of the detection frame using the weight of the classification is a convolution operation on the feature of the third feature using the weight of the classification. The method according to claim 9, wherein the method includes obtaining the classification results of a plurality of candidate frames. After extracting the features of the template frame, a convolution operation is performed on the features of the detection frame by the fourth convolution layer to acquire a fourth feature whose number of channels is the same as the number of channels of the features of the detection frame. Including that
Acquiring the regression results of a plurality of candidate frames by performing a convolution operation on the feature of the detection frame using the weight of the regression is a convolution operation on the feature of the fourth feature using the weight of the regression. The method according to claim 9 or 10, wherein the method comprises performing regression results of a plurality of candidate frames. Acquiring the detection frame of the target object in the detection frame from the classification result and the regression result of the plurality of candidate frames output from the local region detector is possible.
One candidate frame is selected from the plurality of candidate frames based on the classification result and the regression result, the selected candidate frame is regressed according to the deviation amount of the selected candidate frame, and the target object in the detection frame is used. The method according to any one of claims 1 to 11, which comprises acquiring a detection frame. Selecting one candidate frame from the plurality of candidate frames based on the classification result and the regression result is not possible.
The method according to claim 12, wherein one candidate frame is selected from the plurality of candidate frames based on the weighting coefficient of the classification result and the regression result. Further including adjusting the classification result according to the regression result after obtaining the regression result.
The selection of one candidate frame from the plurality of candidate frames based on the classification result and the regression result includes selecting one candidate frame from the plurality of candidate frames based on the adjusted classification result. The method according to claim 12. Extracting the features of the detection frame and the template frame, which is the image of the detection frame of the target object and the size of the image is smaller than the detection frame, by the neural network, respectively.
A convolution operation is performed on the features of the template frame by the first convolution layer, the first feature acquired by the convolution operation is used as the classification weight of the local region detector, and the template frame is performed by the second convolution layer. The convolution operation is performed on the feature of, and the second feature acquired by the convolution operation is used as the regression weight of the local region detector.
The characteristics of the detection frame are input to the local area detector, and the classification result and the regression result of a plurality of candidate frames output from the local area detector are acquired.
Obtaining the detection frame of the target object in the detection frame from the classification result and the regression result of the plurality of candidate frames output from the local region detector, and
The detection frame of the target object in the acquired detection frame is used as a prediction detection frame, and the neural network, the first convolution layer, and the second convolution are based on the labeling information of the detection frame and the prediction detection frame. A method of training a target detection network, characterized by training and including layers. Extracting the features of at least one other detection frame located after the detection frame in time series in the video sequence by the neural network.
The features of the at least one other detection frame are input to the local region detector in order, and a plurality of candidate frames and the classification result of each candidate frame in the at least one other detection frame output from the local region detector. And to get the regression results in order,
The feature is that the detection frames of the target object in the at least one other detection frame are sequentially acquired based on the classification result and the regression result of the plurality of candidate frames of the at least one other detection frame. The method according to claim 15. Extracting the characteristics of the template frame and the detection frame by the neural network is
Extracting the features of the template frame and the detection frame by the same neural network, or
The method according to claim 15 or 16, wherein the features of the template frame and the detection frame are extracted by separate neural networks having the same configuration. The template frame according to any one of claims 15 to 17, wherein the detection timing is positioned before the detection frame in the video sequence, and the detection frame of the target object is specified. the method of. The method according to any one of claims 15 to 18, wherein the detection frame is a current frame for detecting the target object or a region image in which the target object can exist in the current frame. .. When the detection frame is a region image in which the target object can exist in the current frame for detecting the target object,
It further includes cutting out a region image having a center point of the template frame as a center point and having a length and / or width larger than the length and / or width of the image of the template frame, respectively, from the current frame to obtain the detection frame. The method according to claim 19, wherein the method is characterized by the above. It is possible to input the characteristics of the detection frame into the local area detector and acquire the classification result and the regression result of a plurality of candidate frames output from the local area detector.
A convolution operation is performed on the features of the detection frame using the weights of the classification to acquire classification results of a plurality of candidate frames, and a convolution operation is performed on the features of the detection frame using the weights of the regression. The method according to any one of claims 15 to 20, wherein the method is to obtain the regression results of a plurality of candidate frames, and to include. After extracting the characteristics of the detection frame,
Further including performing a convolution operation on the feature of the detection frame by the third convolution layer to acquire a third feature whose number of channels is similar to the number of channels of the feature of the detection frame.
Acquiring the classification results of a plurality of candidate frames by performing a convolution operation on the feature of the detection frame using the weight of the classification is a convolution operation on the feature of the third feature using the weight of the classification. The method according to claim 21, wherein the method includes obtaining the classification results of a plurality of candidate frames. After extracting the features of the template frame,
Further including performing a convolution operation on the feature of the detection frame by the fourth convolution layer to acquire a fourth feature whose number of channels is similar to the number of channels of the feature of the detection frame.
Acquiring the regression results of a plurality of candidate frames by performing a convolution operation on the feature of the detection frame using the weight of the regression is a convolution operation on the feature of the fourth feature using the weight of the regression. 21. The method according to claim 21, wherein the method includes performing regression results of a plurality of candidate frames. Acquiring the detection frame of the target object in the detection frame from the classification result and the regression result of the plurality of candidate frames output from the local region detector is possible.
One candidate frame is selected from the plurality of candidate frames based on the classification result and the regression result, the selected candidate frame is regressed according to the deviation amount of the selected candidate frame, and the target object in the detection frame is used. The method according to any one of claims 15 to 23, which comprises acquiring a detection frame. Selecting one candidate frame from the plurality of candidate frames based on the classification result and the regression result is not possible.
24. The method of claim 24, wherein one candidate frame is selected from the plurality of candidate frames based on the weighting coefficient of the classification result and the regression result. Further including adjusting the classification result according to the regression result after obtaining the regression result.
The selection of one candidate frame from the plurality of candidate frames based on the classification result and the regression result includes selecting one candidate frame from the plurality of candidate frames based on the adjusted classification result. 25. The method of claim 25. The labeling information of the detection frame includes the position and size of the detection frame of the target object in the labeled detection frame.
The detection frame of the target object in the acquired detection frame is used as a prediction detection frame, and based on the labeling information of the detection frame and the prediction detection frame, the neural network, the first convolution layer, and the second Training the convolutionary layer is
Including adjusting the weight values of the neural network, the first convolutional layer, and the second convolutional layer by the difference between the position and size of the labeled detection frame and the position and size of the predicted detection frame. The method according to any one of claims 15 to 26. A neural network for extracting the features of the detection frame and the template frame, which is an image of the detection frame of the target object and whose image size is smaller than the detection frame.
A first convolution layer for increasing the channels of the features of the template frame and weighting the obtained first features to the classification of the local region detector.
A second convolution layer for increasing the channels of the features of the template frame and weighting the resulting second features to the regression of the local region detector.
A local region detector for outputting classification results and regression results of a plurality of candidate frames according to the characteristics of the detection frame, and
A target detection characterized by including an acquisition unit for acquiring a detection frame of the target object in the detection frame based on a classification result and a regression result of a plurality of candidate frames output from the local region detector. apparatus. 28. The apparatus of claim 28, wherein the neural network has the same configuration and includes separate neural networks used to extract features of the template frame and the detection frame, respectively. The apparatus according to claim 28 or 29, wherein the template frame is a frame in which the detection timing is located before the detection frame in the video sequence and the detection frame of the target object is specified. The apparatus according to any one of claims 28 to 30, wherein the detection frame is a current frame for detecting the target object or a region image in which the target object can exist in the current frame. .. Preprocessing for cutting out a region image whose length and / or width is larger than the length and / or width of the image of the template frame, respectively, from the current frame with the center point of the template frame as the center point to obtain the detection frame. 31. The apparatus of claim 31, further comprising a unit. The local region detector performs a convolution operation on the features of the detection frame using the weights of the classification to acquire the classification results of a plurality of candidate frames, and uses the weights of the regression to obtain the detection frame. The apparatus according to any one of claims 28 to 32, wherein the convolution operation is performed on the feature of the above, and the device is used for acquiring regression results of a plurality of candidate frames. A third convolution layer for performing a convolution operation on the features of the detection frame and acquiring a third feature in which the number of channels is the same as the number of channels of the features of the detection frame.
33. The apparatus of claim 33, further comprising the local region detector for performing a convolution operation on the third feature using the weights of the classification. A fourth convolution layer for performing a convolution operation on the features of the detection frame and acquiring a fourth feature in which the number of channels is the same as the number of channels of the features of the detection frame.
33. The apparatus of claim 33, further comprising the local region detector for performing a convolution operation on the fourth feature using the weights of the regression. The acquisition unit selects one candidate frame from the plurality of candidate frames based on the classification result and the regression result, returns the selected candidate frame according to the deviation amount of the selected candidate frame, and returns the selected candidate frame in the detection frame. The apparatus according to any one of claims 28 to 35, which is used for acquiring a detection frame for the target object. When the acquisition unit selects one candidate frame from the plurality of candidate frames based on the classification result and the regression result, the acquisition unit selects one candidate frame from the plurality of candidate frames based on the weighting coefficient of the classification result and the regression result. 36. The device of claim 36, characterized in that it is used for selection. An adjustment unit for adjusting the classification result according to the regression result is further included.
The acquisition unit is used to select one candidate frame from the plurality of candidate frames based on the adjusted classification result when selecting one candidate frame from the plurality of candidate frames based on the classification result and the regression result. 36. The apparatus according to claim 36. The detection frame of the target object in the acquired detection frame is used as a prediction detection frame, and the neural network, the first convolution layer, and the second convolution are based on the labeling information of the detection frame and the prediction detection frame. The device according to any one of claims 28 to 38, further comprising a training unit for training layers. The labeling information of the detection frame includes the position and size of the detection frame of the target object in the labeled detection frame.
The training unit determines the weight values of the neural network, the first convolution layer, and the second convolution layer by the difference between the position and size of the labeled detection frame and the position and size of the prediction detection frame. 39. The apparatus of claim 39, characterized in that it is used for coordinating. An electronic device comprising the target detection device according to any one of claims 28 to 40. Memory for storing executable commands and
An electronic device comprising: a processor for completing the operation of the method according to any one of claims 1-27 by communicating with the memory and executing the executable command. A computer storage medium for storing a computer-readable command, wherein when the command is executed, the operation of the method according to any one of claims 1 to 27 is realized. Storage medium.