JP2022544893A - Network training method and apparatus, target detection method and apparatus, and electronic equipment - Google Patents

Abstract

Embodiments of the present application relate to network training methods and apparatus, target detection methods and apparatus, and electronic devices. The network training method is to input unlabeled sample images into a target detection network for processing to obtain target detection results, which include target image regions, feature information and classification probabilities. determining a category confidence of the target based on the classification probability of the target; and labeling a sample image in which the first target is located for the first target whose category confidence is greater than or equal to a threshold and adding to the training set, performing feature correlation mining on the second target for which the category confidence is less than the first threshold to determine a fourth target, and adding sample images in which is located to a training set; and training a target detection network based on the training set.

Description

(Cross reference to related applications)
This application claims priority from a Chinese patent application with application number 202010681178.2 filed on Jul. 15, 2020, the entire content of which is incorporated herein by reference.

The present application relates to the field of computer technology, and more particularly to network training method and apparatus, target detection method and apparatus, and electronic equipment.

Computer vision is an important trend in artificial intelligence technology, and in computer vision processing, it is generally necessary to perform detection on targets (eg, pedestrians, objects, etc.) in images or videos. Large-scale long-tail data target detection has important applications in many fields, such as abnormal object detection, abnormal behavior detection, and incident warning in urban surveillance. However, the data volume of the long tail data is huge, and the positive and negative samples are badly balanced, i.e., most of the data pictures are background pictures, and only a small part of the pictures are detected. Because of the inclusion of possible targets, it causes the target detection effect of long-tail data to be poor by target detection schemes in related arts.

Embodiments of the present application provide technical solutions for network training and target detection.

According to one aspect of embodiments of the present application, a network training method is provided. The method includes:
inputting an unlabeled first sample image into a target detection network and processing to obtain a target detection result of the first sample image, wherein the target detection result is the number of targets in the first sample image; determining a category confidence of the target based on the classification probability of the target; and category confidence of the targets is greater than or equal to a first threshold. For a first target, a first sample image in which the first target is located is a labeled second sample image and added to a training set, wherein the labeling information of the second sample image is: an image region of the first target and a category corresponding to the category confidence of the first target, wherein the training set includes labeled third sample images; For a second target whose category reliability is smaller than the first threshold, feature correlation mining is performed on the second target based on feature information of the third target in the third sample image, and feature correlation mining is performed. from the second target, determine a fourth target and a first sample image on which the fourth target is located, and add the first sample image on which the fourth target is located as a fourth sample image to the training set; and training the target detection network based on the labeling information of the fourth sample images, the second sample images, the third sample images and the fourth sample images in the training set.

In a possible implementation, training the target detection network based on the labeling information of the fourth sample image, the second sample image, the third sample image and the fourth sample image in the training set comprises:
each determining a first number sampled from each category of positive sample images based on the category of the target in the training set of positive sample images, wherein the positive sample images are samples whose images contain the target; and performing sampling on the positive sample images of each category to obtain a plurality of fifth sample images based on a first number sampled from the positive sample images of each category; sampling against a set of negative sample images to obtain a plurality of sixth sample images, wherein the negative sample images are sample images without targets in the images; and the fifth sample. training the target detection network based on the images and the sixth sample image.

In a possible implementation, feature correlation mining is performed on the second target based on the feature information of the third target in the third sample image, the feature correlation mining from the second target to the fourth target and Determining a first sample image in which the fourth target is located includes determining information entropy of the second target based on the classification probability of the second target; category confidence of the second target; selecting a fifth target from the second targets based on the information entropy; and based on the category of the third target in the third sample image and the total number of sample images waiting to be mined, each category of sample images waiting to be mined. and based on the feature information of the third target in the third sample image, the feature information of the fifth target, and the second number of sample images awaiting mining of each category, the fifth determining from the targets a fourth target and a first sample image in which the fourth target is located.

In a possible implementation, selecting a fifth target from the second target based on the categorical confidence and information entropy of the second target comprises: , ordering for the second target respectively, selecting a third number of the sixth target and a fourth number of the seventh target; performing a merge on the sixth and the seventh target; obtaining the fifth target.

In a possible implementation, determining a second number of awaiting mining sample images for each category based on a third target category and a total number of awaiting mining sample images in said third sample images respectively comprises: Determining the percentage of the third target in each category based on the category of the third target in the image; Determining the sampling weight of each category based on the percentage of the third target in each category; determining a second number of sample images to be mined for each category based on the sampling weights of the respective categories.

In a possible implementation, based on the feature information of the third target in the third sample image, the feature information of the fifth target and a second number of sample images to be mined for each category, from the fifth target, the fourth Determining a first sample image in which the target and the fourth target are located may be performed based on the distance between the feature information of the third target of the first category and the feature information of each fifth target of the first category. A third target having the shortest distance to each fifth target among the three targets is determined as the eighth target, and the first category is any one of the categories of the third targets. and determining a target with the longest distance among the eighth targets as the fourth target.

In a possible implementation, based on the feature information of the third target in the third sample image, the feature information of the fifth target and a second number of sample images to be mined for each category, from the fifth target, the fourth Determining a target and a first sample image in which the fourth target is located includes adding the determined fourth target to the third target in the first category and adding the determined fourth target to the unlabeled third target. 5 removing from the target.

In a possible implementation, the method further comprises inputting and processing the third sample image into the target detection network to obtain feature information of a third target in the third sample image.

In a possible implementation, prior to the step of inputting and processing an unlabeled first sample image into a target detection network to obtain a target detection result for said first sample image, said method comprises:
Further comprising pre-training the target detection network with labeled third sample images.

In a possible implementation, said first sample image comprises a long tail image.

According to one aspect of embodiments of the present application, a method of target detection is provided. The method includes inputting a pending image to a target detection network for processing to obtain a target detection result of the pending image, wherein the target detection result is a position and category of a target in the pending image. and wherein the target detection network has been trained by the network training method described above.

According to one aspect of embodiments of the present application, a network training device is provided. The device comprises:
A target detection unit configured to input and process an unlabeled first sample image into a target detection network to obtain a target detection result of the first sample image, wherein the target detection result is the a target detector, including target image regions, feature information and classification probabilities in the first sample image;
a confidence determiner configured to determine a category confidence of the target based on the classification probabilities of the target;
For a first target among the targets whose category reliability is greater than or equal to a first threshold, a first sample image in which the first target is located is defined as a labeled second sample image and added to the training set. wherein the labeling information of the second sample image includes an image region of the first target and a category corresponding to the category confidence of the first target, and the training set is labeled with a labeling portion containing a third sample image;
For a second target among the targets whose category reliability is smaller than the first threshold, feature correlation mining is performed on the second target based on the feature information of the third target in the third sample image. determining a fourth target and a first sample image in which the fourth target is located from the second target by feature correlation mining, and using the first sample image in which the fourth target is located as a fourth sample image; a feature mining unit configured to add to the training set;
a training unit configured to train the target detection network based on the labeling information of the fourth sample images, the second sample images, the third sample images and the fourth sample images in the training set. .

In a possible implementation, the training unit comprises a sampling number configured to respectively determine a first number to be sampled from each category of positive sample images based on target categories in the training set of positive sample images. a determining sub-unit, wherein the positive sample images are sample images whose images contain the target; a first sampling sub-unit configured to sample positive sample images of the training set to obtain a plurality of fifth sample images; and sample negative sample images of the training set to obtain a plurality of sixth samples. a second sampling sub-section configured to obtain an image, wherein the negative sample image is a sample image in which no target is included in the image; a training sub-unit configured to train the target detection network based on 6 sample images.

In a possible implementation, the feature mining unit comprises an information entropy determination sub-unit configured to determine the information entropy of the second target based on the classification probability of the second target; a target selection sub-unit configured to select a fifth target from the second target based on category confidence and information entropy; a mining number determination sub-unit configured to respectively determine a second number of sample images waiting to be mined for each category based on the total number; feature information of a third target in the third sample image; Targets and images configured to determine, from said fifth target, a fourth target and a first sample image in which said fourth target is located, based on feature information and a second number of sample images to be mined for each category. a decision sub-part.

In a possible implementation, the target selection sub-unit orders the second targets based on the category confidence and information entropy of the second target, respectively; selecting a seventh target of the number and performing a merge on said sixth target and said seventh target to obtain said fifth target.

In a possible implementation, the mining number determination sub-unit determines the percentage of third targets in each category based on the categories of third targets in the third sample image, and determines the percentage of third targets in each category. and determining a second number of sample images to be mined for each category based on the sampling weight of each category.

In a possible implementation, the target and image determination subunit selects among the third targets of the first category based on the distance between the characteristic information of the third target of the first category and the characteristic information of each fifth target. , a third target having the smallest distance from each fifth target is determined as an eighth target, the first category is any one of the categories of the third targets, and the eighth target is Among them, the target with the longest distance is determined as the fourth target.

In a possible implementation, the target and image determination sub-unit further adds the determined fourth target to the third target of the first category, and subtracts the determined fourth target from the unlabeled fifth target. configured to remove.

In a possible implementation, the apparatus comprises a feature extractor configured to input and process the third sample image into the target detection network to obtain feature information of a third target in the third sample image. Further prepare.

In a possible implementation, before the target detection unit, the device further comprises a pre-training unit configured to pre-train the target detection network with labeled third sample images.

In a possible implementation, said first sample image comprises a long tail image.

According to one aspect of embodiments of the present application, a target detection apparatus is provided. The apparatus is a detection processing unit configured to input a pending image to a target detection network for processing to obtain a target detection result of the pending image, wherein the target detection result is the pending image. The target detection network comprises a detection processing unit, including the location and category of the target in the image, which has been trained according to the network training method described above.

In a possible implementation, before the step of respectively determining the first number to be sampled from each category of positive sample images based on the target category in the training set of positive sample images, positive sample images in the training set and Sampling is performed on the negative sampled images to obtain positive sampled images and negative sampled images that are the same or close in number.

In a possible implementation, the total number of sample images awaiting mining is between 5% and 25% of the total number of first sample images.

In a possible implementation, merging on the sixth target and the seventh target to obtain the fifth target removes those of the sixth targets that are the same as the seventh target. , obtaining a surplus target different from the seventh target among the sixth targets, and setting the surplus target and the seventh target as the fifth target.

In a possible implementation, the method terminates feature correlation mining for the first category when the number of the fourth sample images of the first category reaches a second number of sample images to be mined of the first category. further comprising:

In a possible implementation, based on the distance between the feature information of the third target of the first category and the feature information of each fifth target, the distance to each fifth target among the third targets of the first category is determined. After determining each of the smallest third targets to be the eighth target, the method further comprises determining the number of the first sample images in which the fourth targets are located as a second number of to-be-mined sample images of the first category. is reached, terminating the determination for the eighth target.

In a possible implementation, based on the distance between the feature information of the third target of the first category and the feature information of each fifth target, the distance to each fifth target among the third targets of the first category is determined. After determining each of the smallest third targets to be the eighth target, the method further comprises determining the number of the first sample images in which the fourth targets are located as a second number of to-be-mined sample images of the first category. is not reached and the set storing feature information for the fifth target is null, terminating the determination for the eighth target.

In a possible implementation, inputting and processing the third sample image into the target detection network to obtain feature information of a third target in the third sample image comprises converting the third sample image into the target detection network. inputting to a network and obtaining a feature vector output from a hidden layer of the target detection network; and determining the feature vector as the feature information of the third target.

According to one aspect of an embodiment of the present application, an electronic device is provided. The electronic device comprises a processor and a memory storing instructions executable by the processor, the processor calling the instructions stored in the memory to perform the network training method or to train the target. Run the detection method.

According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided. The computer readable storage medium stores computer program instructions which, when executed by a processor, cause the processor to implement the network training method or implement the target detection method described above.

According to the embodiments of the present application, the target detection network obtains target detection results of unlabeled sample images, and based on the target detection results, performs pseudo-labeling and feature correlation mining, respectively, to obtain high-value sample images. Labeling and collecting, adding to the training set, training a target detection network based on the expanded training set, thereby expanding the number of positive sample data in the training set, and balancing the positive and negative samples It reduces the problem of not being taken and improves the training effectiveness of the target detection network.

It is to be understood that the general descriptions above and the detailed descriptions that follow are exemplary and explanatory only and are not restrictive of the embodiments of the present application. Other features and aspects of embodiments of the present application will become apparent with reference to the following detailed description of exemplary embodiments based on the drawings.

The drawings attached hereto are incorporated into the specification and constitute a part of the specification, and these drawings show the embodiments consistent with the present application, and together with the description, the technical solutions of the embodiments of the present application. Used for interpretation.
4 is a flowchart illustrating a network training method according to embodiments of the present application; Fig. 3 is a schematic diagram showing the processing process of the network training method of the embodiment of the present application; 1 is a block diagram of a network training device according to an embodiment of the present application; FIG. 1 is a block diagram illustrating an electronic device according to an embodiment of the present application; FIG. 1 is a block diagram illustrating an electronic device according to an embodiment of the present application; FIG.

Various exemplary implementations, features and aspects of embodiments of the present application are described in detail below with reference to the drawings. The same reference numerals in the drawings indicate elements having the same or similar functions. The drawings, which illustrate various aspects of the embodiments, are not necessarily drawn to scale unless specifically stated otherwise.

The term "exemplary" as used herein means "serving as an example, example, or for the purpose of explanation." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

As used herein, the term “and/or” is used to describe a related relationship between related objects, and indicates that there are three types of relationships. For example, A and/or B represents three cases: only A is present, A and B are present at the same time, and only B is present. Also, as used herein, the term "at least one" represents any one of the plurality or any combination of at least two of the plurality. For example, including at least one of A, B, and C means including any one or more elements selected from the set consisting of A, B, and C.

It is noted that many specific details are set forth in the specific embodiments below in order to better describe the embodiments of the present application. It should be understood by those skilled in the art that the present disclosure may be similarly practiced regardless of these specific details. In some instances, methods, means, devices, and circuits that are well known to those skilled in the art are not described in detail in order to avoid obscuring the subject matter of the embodiments of the present application.

FIG. 1 is a flowchart illustrating a network training method according to embodiments of the present application. As shown in FIG. 1, the network training method includes:

In step S11, an unlabeled first sample image is input to a target detection network and processed to obtain a target detection result of the first sample image; image regions, feature information and classification probabilities.

In step S12, the category reliability of the target is determined based on the classification probability of the target.

In step S13, for a first target among the targets whose category reliability is greater than or equal to a first threshold, a first sample image in which the first target is located is taken as a labeled second sample image, and a training set is obtained. In addition, wherein the labeling information of the second sample image includes an image region of the first target and a category corresponding to the category confidence of the first target, and the training set includes a labeled 3 sample images are included.

In step S14, for a second target having a category reliability smaller than the first threshold among the targets, based on the feature information of the third target in the third sample image, performing feature correlation mining, determining a fourth target and a first sample image in which the fourth target is located from the second target by feature correlation mining, and determining a first sample image in which the fourth target is located as a fourth target Take it as a sample image and add it to the training set.

In step S15, training the target detection network based on the labeling information of the fourth sample image, the second sample image, the third sample image and the fourth sample image in the training set.

In a possible implementation, the method may be performed by an electronic device such as a terminal device or a server, where the terminal device may be a User Equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless It may be a phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, and the like. The method may be implemented by invoking computer readable instructions stored in memory by a processor. Alternatively, the method can be performed by a server.

For example, the first sample image may be an image acquired by an image acquisition device (eg, camera). The first sample image may be a large, Long-tailed image, ie, most of the image is the background image and only a small portion of the image contains detectable targets. Detectable targets may include, for example, human bodies, faces, vehicles, objects, and the like. For example, in a security area, cameras may collect images of a geographic area. A small portion of the time someone may be passing through the geographic area. Thereby, most of the images collected are background images, and only a small portion of the images contain faces and/or human bodies. In this case, the multiple images collected can constitute a long-tail data set. Embodiments of the present application do not limit the acquisition method of the first sample image and the category of targets in the first sample image.

In a possible implementation, a target detection network may be preconfigured to detect the position (ie detection window) and category of the target in the image. The target detection network may be, for example, a convolutional neural network, and embodiments herein do not limit the network structure of the target detection network.

In a possible implementation, before step S11, the method further comprises pre-training the target detection network with labeled third sample images. That is, the training set may be preset. The training set includes labeled third sample images. The labeling information for the third sample image may include detection windows and categories of targets in the image. Based on the training set, a method in the related art can be used to pre-train a target detection network to make the target detection network have a certain detection accuracy.

However, due to the poor detection effectiveness of the pre-trained target detection network on large-scale long-tail images, it is recommended to further train the target detection network using the unlabeled first sample images in a manner of active learning. can be done.

In a possible implementation, in step S11, an unlabeled first sample image can be input to a target detection network for processing to obtain a target detection result for the first sample image. The target detection result may include an image region of the target in the first sample image, feature information and classification probability. The image region where the target is located may be a detection frame in the image, the feature information of the target may be, for example, a feature vector output from a hidden layer (e.g., convolutional layer) of the target detection network, The classification probabilities of a target can represent the classification posterior probabilities that the target belongs to each category.

In a possible implementation, a target in the first sample image may be called an instance. One or more targets may be detected from each first sample image. In actual processing, the number level of detected targets may be several times to tens of times the number level of images.

In a possible implementation, in step S12, based on the classification probabilities of the target, the maximum value of the classification probabilities can be determined as the category confidence of the target.

In a possible implementation, in step S13, for a target whose category confidence is greater than or equal to a first threshold (which may be referred to as a first target), a first sample image in which the first target is located is labeled. sample image (which may be referred to as the second sample image) and can be added to the training set. Let the image region of the first target be the image region to be labeled, and let the category corresponding to the category reliability of the first target be the labeling category of the first target. The same second sample image may be labeled multiple times with multiple first targets in the second sample image. Here, the first threshold is, for example, 0.99, and the embodiments of the present application do not limit the value of the first threshold.

In a possible implementation, the process of step S13 may be called pseudo-labeling. That is, an image in which a target with high reliability is located is taken as a high-value sample, and the inference result of target detection is directly taken as the labeling result of the target. This method can expand the number of positive sample data in the training set and solve the problem of difficulty in collecting positive samples.

In a possible implementation, in step S14, targets in the labeled third sample image in the training set (the second 3 targets), perform feature correlation mining on the second target, and mining a target (which may be referred to as a fourth target) that satisfies the requirements from the second target. can do. For example, the distance or the degree of correlation between the feature information of the third target and the feature information of the second target is calculated, a predetermined number of targets are selected based on the distance or the degree of correlation, and the selected predetermined number of targets are selected. can be a fourth target.

In a possible implementation, the first sample image where the mined fourth target is located is taken as the fourth sample image and added to the training set to complete the process of feature correlation mining. Such schemes can further expand the number of sample data in the training set.

In a possible implementation, the labeling information of the fourth sample image can be obtained by means of manual labeling. For example, manually determine the detection window and category of the target in the fourth sample image. Embodiments of the present application do not limit this.

In a possible implementation, in step S15, after obtaining the labeling information of the fourth sample image, training a target detection network based on the second, third and fourth sample images in the training set. can be done.

According to the embodiments of the present application, the target detection network obtains target detection results of unlabeled sample images, and based on the target detection results, performs pseudo-labeling and feature correlation mining, respectively, to obtain high-value sample images. Labeling and collecting, adding to the training set, training a target detection network based on the expanded training set, thereby expanding the number of positive sample data in the training set, and balancing the positive and negative samples It reduces the problem of not being taken and improves the training effectiveness of the target detection network.

In a possible implementation, the process of step S11 obtains the target detection result for each first sample image, and the process of S12 obtains the category confidence of the target in each first sample image. In step S13, the sample images in which the first targets with category confidence greater than or equal to the first threshold are located are added to the training set, and the labeled second sample images can be obtained by the pseudo-labeling method. In step S14, mining may be performed for a second target whose category confidence is less than the first threshold.

In a possible implementation, step S14 includes:
determining the information entropy of the second target based on the classification probability of the second target;
selecting a fifth target from the second target based on the category confidence and information entropy of the second target;
respectively determining a second number of sample images to be mined for each category based on a third target category and a total number of sample images to be mined in the third sample images;
Based on the feature information of the third target in the third sample image, the feature information of the fifth target, and the second number of sample images awaiting mining in each category, from the fifth target, the fourth target and the fourth target and determining a first sample image in which is located.

For example, the information entropy of the second target can be calculated based on the classification probability of the second target. The information entropy is intended to represent the degree of uncertainty of the second target. That is, the greater the information entropy of the second target, the greater the degree of uncertainty of the second target. Conversely, the smaller the information entropy of the second target, the smaller the degree of uncertainty of the second target. Embodiments of the present application do not limit the method of calculating information entropy.

In a possible implementation, a target (which may be referred to as a fifth target) that satisfies a certain condition may be selected from a plurality of second targets based on the category reliability and information entropy of the second target. can. For example, targets with high category reliability, targets with high information entropy, and the like are selected.

In a possible implementation, the step of selecting a fifth target from said second target based on the categorical confidence and information entropy of said second target comprises:
ordering the second targets based on their categorical confidence and information entropy, respectively, and selecting a third number of sixth targets and a fourth number of seventh targets;
performing a merge on the sixth target and the seventh target to obtain the fifth target.

That is, based on the category reliability of the second targets, the plurality of second targets are ordered, and based on the ordering result, a predetermined third number of targets (the sixth target and may be called). Similarly, based on the information entropy of the second target, a plurality of second targets are ordered, and based on the ordering result, a predetermined fourth number of targets (7th target and may be called). Here, the third number and the fourth number may each be 3K, K represents the number of sample images waiting to be mined, and the value of K is 10000, for example. In actual processing, the value of K may be 5% to 25% of the total number of unlabeled first sample images. The examples of the present application do not limit the value of K nor the quantitative relationship between K and the third and fourth numbers.

A person skilled in the art can set the values of the number K of sample images awaiting mining, the third number and the fourth number according to the actual situation, and the third number and the fourth number can be different. It should be understood that the examples do not limit this.

In a possible implementation, the 6th and 7th targets selected can be merged, and the merged multiple targets can be the 5th target, of which possible duplicate targets can be removed. In actual processing, about 6K fifth targets can be obtained.

The above processing scheme may be referred to as bootstrapping. In such a manner, a certain number of probable positive and negative samples can be selected simultaneously from the second target. This facilitates subsequent feature correlation mining, reduces the computational complexity of feature correlation mining, and improves processing efficiency.

In a possible implementation, determining a second number of sample images to be mined for each category based on a third target category and a total number of sample images to be mined in said third sample images respectively comprises:
determining a percentage of third targets in each category based on categories of third targets in the third sample image;
determining a sampling weight for each category based on the proportion of the third target for each category;
respectively determining a second number of sample images to be mined for each category based on the sampling weight of each category.

を算出することができる。

（１）

（２）
式（１）及び（２）において、ｒ_ｃは、カテゴリｃのサンプリング値を表し、ｔは、ハイパーパラメータであり、その値が例えば０．１であり、Ｃは、カテゴリの数を表し、ｒ_ｉは、Ｃ個のカテゴリのうちのｉ番目のカテゴリのサンプリング値を表す。 For example, based on the categories of tertiary targets in the existing labeled 3rd sample images in the training set, the proportion f _c of tertiary targets in each category can be determined. Based on the ratio f _c , the sampling weight of each category is calculated by the following formula

can be calculated.

(1)

(2)
In equations (1) and (2), r _c represents the sampling value of category c, t is a hyperparameter whose value is for example 0.1, C represents the number of categories, r _i represents the sampling value of the i-th category among the C categories.

The processing of equations (1) and (2) can increase the sampling weight corresponding to low-proportion categories and reduce the sampling weight corresponding to high-proportion categories, thereby reducing the number of samples in different categories. Alleviate the problem of imbalance and improve the training effect of the network.

及びマイニング待ちサンプル画像の総数（Ｋ個）に基づいて、各カテゴリのマイニング待ちサンプル画像の第２数を決定することができる。更に、第２数に基づいて、特徴相関マイニングを行うことができる。 Sampling weight for each category in possible implementations

and the total number of sample images to be mined (K), a second number of sample images to be mined for each category can be determined. Additionally, feature correlation mining can be performed based on the second number.

In a possible implementation, the method further comprises inputting and processing the third sample image into the target detection network to obtain feature information of a third target in the third sample image.

That is, inputting the labeled third sample image in the training set into the target detection network, and outputting feature information, e.g., a feature vector, of the third sample image from the hidden layer (e.g., convolutional layer) of the target detection network. can be done. With such a scheme, features of the third sample image can be obtained to contribute to subsequent feature correlation mining.

In a possible implementation, based on the feature information of the third target in the third sample image, the feature information of the fifth target and a second number of sample images to be mined for each category, from the fifth target, the fourth Determining a first sample image in which the target and the fourth target are located comprises:
Based on the distance between the feature information of the third target in the first category and the feature information of each fifth target, the third target with the smallest distance from each fifth target among the third targets in the first category is selected. each determining an eighth target, wherein the first category is any one of the categories of the third target;
determining a target having the longest distance among the eighth targets as a fourth target.

For example, after determining the second number of sample images waiting to be mined for each category, using a k-center method to mine the corresponding number of sample images from the sample images where the fifth target is located. can be done. For any one category (which may be called a first category) among the plurality of categories of the third target, the feature information of the third target of the first category and the feature information of each fifth target can be calculated. The distance may for example be the Euclidean distance. For any one fifth target, a third target having the smallest distance from the fifth target among the third targets in the first category can be determined, whereby the distance from each fifth target is can be determined, which may be referred to as the eighth target.

（３）
式（３）において、ｕは、特徴相関マイニングにより得られた第４ターゲットを表し、

は、ｊ番目の第５ターゲットの特徴情報

と第１カテゴリｃの１番目の第３ターゲットの特徴情報

との距離を表し、

は、第５ターゲットの特徴情報の集合を表し、

は、第１カテゴリｃの第３ターゲットの特徴情報の集合を表す。 In a possible implementation, from each eighth target, one target with the largest distance can be selected and determined as the fourth target obtained by this feature correlation mining. This is as shown in the following formula.

(3)
In equation (3), u represents the fourth target obtained by feature correlation mining,

is the feature information of the j-th fifth target

and feature information of the first third target of the first category c

represents the distance from

represents the set of feature information of the fifth target,

represents a set of feature information of the third target of the first category c.

In a possible implementation, the first sample image in which the fourth target is located can be determined and added to the training set to become the fourth sample image, thereby making the current feature correlation mining process complete.

In a possible implementation, based on the feature information of the third target in the third sample image, the feature information of the fifth target and a second number of sample images to be mined for each category, from the fifth target, the fourth Determining a first sample image in which the target and the fourth target are located comprises:
Adding the determined fourth target to the third target of the first category and removing the determined fourth target from the unlabeled fifth target.

に加え、第５ターゲットの特徴情報の集合

から除去することができる。このように、次回の特徴相関マイニングにおいて、式（３）により、更新後の２つの集合に対してマイニングを行い、上記プロセスを繰り返すことができる。 That is, the fourth target obtained by the current feature correlation mining is set as the labeled target, and the fourth target is removed from the unlabeled targets. In this case, the feature information of the fourth target is the set of feature information of the third target of the first category c.

In addition to the set of feature information of the fifth target

can be removed from Thus, in the next feature correlation mining, the two updated sets can be mined according to equation (3), and the above process can be repeated.

がヌルである）場合、該第１カテゴリの特徴相関マイニングを完了することができる。 In a possible implementation, the number of fourth sample images of the first category reaches or does not reach the second number of the first category and there is no fifth target (set

is null), then feature correlation mining for the first category can be completed.

With such a scheme, we can perform feature correlation mining for each category separately, and finally obtain a sufficient number of fourth sample images (generally, K sample images), so that We further expand the number of sample images in the training set to mitigate the imbalance between positive and negative samples.

In a possible implementation, human annotation can be performed on the mined fourth sample image to obtain the labeling information of the fourth sample image. The fourth sample image may have a positive sample image (i.e., the fourth sample image in which the image contains the target) and a negative sample image (i.e., the fourth sample image in which the image contains no target) at the same time. Therefore, the labeling information of the fourth sample image may include sample category information indicating whether the image is a positive sample image or a negative sample image, the image box in which the target is located in the positive sample image, and the category of the target.

In a possible implementation, after completing the manual labeling, in step S15, based on the labeling information of the fourth sample image, the second sample image, the third sample image and the fourth sample image in the training set, a target detection network can be trained.

wherein step S15 is respectively determining a first number to be sampled from each category of positive sample images based on target categories in the training set of positive sample images, wherein the positive sample images are: that the image is a sample image containing the target;
sampling the positive sample images of each category based on a first number sampled from the positive sample images of each category to obtain a plurality of fifth sample images;
sampling the negative sample images of the training set to obtain a plurality of sixth sample images, wherein the negative sample images are sample images in which the target is not included in the images;
training the target detection network based on the fifth sample image and the sixth sample image.

For example, the target detection network can be trained by a resampling scheme. Resampling increases the sampling frequency of data with low frequency of occurrence, improves the performance of the network on these data, and further improves the imbalance between positive and negative samples. .

In a possible implementation, each positive sample image and negative sample image in the training set (including the second sample image, the third sample image, and the fourth sample image) are sampled, and the number of positive sample images after sampling and the number of negative sample images can be the same or close.

（４） In a possible implementation, for positive sample images, a sampling total number of positive sample images may be preset. A first number to be sampled from each category of positive sample images is determined based on the target category in the positive sample images in the training set, respectively.
Similar to the above process, based on the categories of targets in the positive sample image, the percentage of targets in each category can be determined. Based on this ratio, the sampling weight of each category can be calculated by the following formula.

(4)

In equation (4), R _h represents the sampling weight of the positive sample image of the h-th category, q _h represents the target proportion of the h-th category, and t ₁ is a hyperparameter whose value is, for example, 0.1.

The processing of equation (4) can increase the sampling weight corresponding to categories with a small proportion and decrease the sampling weight corresponding to categories with a large proportion, thereby balancing the number of positive sample images in different categories. Reduce non-taken and improve the training effect of the network.

In a possible implementation, the first number of positive sample images in each category can be determined based on the sampling weight of positive sample images in each category and the total number of positive sample images sampled.

In a possible implementation, for any one category, randomly sample a first number of positive sample images from the positive sample images of the category based on the first number of the category, and generate a fifth sample image can be Sampling is performed on the positive sample images of each category to obtain the total number of sampled fifth sample images.

In a possible implementation, directly random sampling the negative sample images in the training set based on a preset sampling total for the negative sample images to obtain the sixth sample image of the sampling total. can be done. The sampling total number of negative sample images may be the same as or different from the sampling total number of positive sample images, and embodiments of the present application do not limit this.

In a possible implementation, a target detection network can be trained based on the fifth and sixth sample images. That is, the fifth and sixth sample images are input to the target detection network respectively, the target detection results of the fifth and sixth sample images are obtained, and the loss of the target detection network is determined based on the target detection results and the labeling information. and based on the loss, back-adjust the parameters of the target detection network, and by multiple iterations, obtain a trained target detection network if it meets a predetermined condition (eg, network convergence).

Such a scheme can significantly improve the detection effectiveness of the trained target detection network for long-tail images.

In a possible implementation, the step of pre-training the target detection network with labeled third sample images before step S11 may be performed by the resampling training scheme described above. Thereby improving the pre-training effect of the target detection network.

In practical application, the whole process in steps S11-S15 can be repeated to realize continuous incremental training. That is, when the unlabeled sample images are collected again, the target detection network after this training is taken as the initial target detection network, the training set after this expansion is taken as the initial training set, and pseudo-labeling - feature correlation mining - The process of resampling training can be repeated, thereby continuously improving the performance of the target detection network.

FIG. 2 is a schematic diagram showing the process of a network training method according to an embodiment of the present application. As shown in FIG. 2, the data source contains a large number of unlabeled first sample images 20 . A first sample image 20 is input to a target detection network and a predict is performed to obtain a target detection result 21 for each first sample image 20 . The target detection results include target image regions (not shown), feature vectors and classification probabilities in the first sample image.

As shown in FIG. 2, in this example, the target detection network may include a CNN backbone network 211, a feature map pyramid network (FPN) 212 and a fully connected network 213, where the fully connected network 213 is, for example, a bbox head. . After inputting the first sample image 20 into the target detection network, it is processed by the CNN backbone network 211 and the FPN 212 to obtain a feature map 214 of the first sample image. The feature map 214 is processed by the fully connected network 213 to obtain the target detection result 21 .

In such an example, a target's category confidence can be determined based on the target's classification probabilities. For first targets whose category confidence is greater than or equal to a first threshold (e.g., 0.99), a first sample image in which these first targets are located is determined to be a second sample image 22; Pseudo labeling is performed on the image 22 . That is, the category corresponding to the image area of the first target and the category reliability of the first target is used as the labeling information of the second sample image 22 . The labeled second sample images 22 are added to the training set 25, thereby achieving expansion of the positive samples in the training set.

In the example, for the second targets whose category confidence is less than the first threshold, bootstrapping selects a certain number of fifth targets to obtain the sample image 23 where the fifth targets are located. Based on the feature vector (not shown) of the third target in the labeled third sample image in the training set, perform feature correlation mining on the fifth target to determine the fourth target and the first target where the fourth target is located. A sample image is determined to be the fourth sample image 24 . Manual labeling is performed on the fourth sample image 24 and added to the training set 25, thereby realizing further extensions to the labeled images in the training set.

In the example, after two rounds of dilation, the training set 25 contains labeled second, third and fourth sample images. Resampling is performed on the training set 25 to balance the number of positive and negative samples and the number of positive samples of different categories to obtain a training set 26 after resampling. Further, a target detection network is trained based on the resampled training set 26, thereby completing the entire processing process.

Embodiments of the present application further provide a target detection method. The method comprises
inputting a to-be-processed image into a target detection network for processing to obtain a target detection result of the to-be-processed image, wherein the target detection result includes a position and category of a target in the to-be-processed image; The target detection network is trained according to the network training method described above.

That is, a target detection network trained according to the above method can be deployed to achieve target detection in pending images. A pending image may be, for example, an image captured by an image capture device (eg, a camera). Images may include targets waiting to be detected, such as human bodies, faces, vehicles, objects, and so on. Embodiments of the present application do not limit this.

In a possible implementation, a pending image can be input to a target detection network and processed to obtain a target detection result for said pending image. The target detection result includes the position and category of the target in the pending image, for example, the detection frame where the face is located in the pending image and the identity corresponding to the face.

With such a method, it is possible to improve the detection accuracy of target detection and realize target detection for large-scale long-tail image data.

According to the network training method of the embodiments of the present application, the active learning mining method is used to mine the latent unlabeled data, and the semi-supervised learning method is used to label the unlabeled data. , to expand the number of positive sample data, thereby solving the problem of large data scale and difficulty in collecting positive samples in large-scale long-tail detection, and to some extent the problem of unbalanced Effectively improve model performance in limited labeling and computing resource environments.

According to the network training method of the embodiments of the present application, a resampling scheme is used to train the target detection network so that the network training is not adversely affected by the unbalanced positive and negative samples. and reduce the adverse effects on network training of unbalanced positive samples of different categories, thereby effectively converging and improving the performance of the network when training a target detection network. can be done.

According to the network training methods of the embodiments of the present application, active learning methods can be used to mine potentially high-value samples that contribute to model performance improvement from a large amount of unlabeled data. , in a limited labeling and computing resource environment, it can effectively improve the performance of the model, and greatly save the labor and computing costs required for the deep learning model to be used in new business. . Utilizing resampling methods, target detection networks can be effectively trained when samples are unbalanced, without requiring too much involvement in manual tuning of parameters, allowing deep learning models to be novel. Labor costs required for business use can be saved.

The network training method of the embodiments of the present application can be applied in the fields of intelligent video analysis, crime prevention and so on. With limited man-made and computing resources, the method is used to perform online detection for potential targets in intelligent video analysis or intelligent monitoring, and rapid iterative improvements to the detection network used; With low labor and computing costs, the performance requirements required for business can be quickly achieved, and subsequently the performance of the network can be continuously improved.

The network training methods of the embodiments of the present application are applicable to online intelligent video analysis or intelligent monitoring. With limited man-made and computing resources, it can be rapidly iteratively improved online for potential target detection applications in intelligent video analysis or intelligent monitoring, with small labor and computing costs to meet the needs of the business. Performance requirements can be achieved quickly and subsequently the performance of the model can be continuously improved.

The embodiments of the above methods mentioned in the embodiments of the present application can be combined to form a combined embodiment without departing from the principle or logic. It should be understood that it is not explained step-by-step in Those skilled in the art should understand that the execution order of each step in the above method of a specific embodiment depends on its function and possible underlying logic.

In addition, the embodiments of the present application further provide a network training device, a target detection device, a computer-readable storage medium, and a program. All of the above are for implementing any one network training method or target detection method provided in the embodiments of the present application. For the corresponding technical solution and description, please refer to the description related to the method. Here, detailed description is omitted.

FIG. 3 is a block diagram illustrating a network training device according to an embodiment of the present application; Said apparatus comprises a processor (not shown in FIG. 3), said processor being arranged to execute a program part stored in a memory (not shown in FIG. 3), as shown in FIG. , the part of the program stored in memory is
A target detection unit 31 configured to input and process an unlabeled first sample image into a target detection network to obtain a target detection result of the first sample image, the target detection result comprising: a target detection unit 31, including target image regions, feature information and classification probabilities in the first sample image;
a confidence determiner 32 configured to determine a category confidence of the target based on the classification probabilities of the target;
For a first target among the targets whose category reliability is greater than or equal to a first threshold, a first sample image in which the first target is located is defined as a labeled second sample image and added to the training set. wherein the labeling information of the second sample image includes an image region of the first target and a category corresponding to the category confidence of the first target; a labeling unit 33, the collection of which includes the labeled third sample image;
For a second target among the targets whose category reliability is smaller than the first threshold, feature correlation mining is performed on the second target based on the feature information of the third target in the third sample image. determining a fourth target and a first sample image in which the fourth target is located from the second target by feature correlation mining, and using the first sample image in which the fourth target is located as a fourth sample image; a feature miner 34 configured to add to the training set;
a training unit 35 configured to train the target detection network based on the labeling information of the fourth sample images, the second sample images, the third sample images and the fourth sample images in the training set; Prepare.

In a possible implementation, the training unit comprises a sampling number configured to respectively determine a first number to be sampled from each category of positive sample images based on target categories in the training set of positive sample images. a determining sub-unit, wherein the positive sample images are sample images whose images contain the target; a first sampling sub-unit configured to sample positive sample images of the training set to obtain a plurality of fifth sample images; and sample negative sample images of the training set to obtain a plurality of sixth samples. a second sampling sub-section configured to obtain an image, wherein the negative sample image is a sample image in which no target is included in the image; a training sub-unit configured to train the target detection network based on 6 sample images.

In a possible implementation, the feature mining unit comprises an information entropy determination sub-unit configured to determine the information entropy of the second target based on the classification probability of the second target; a target selection sub-unit configured to select a fifth target from the second target based on category confidence and information entropy; a mining number determination sub-unit configured to respectively determine a second number of sample images waiting to be mined for each category based on the total number; feature information of a third target in the third sample image; Targets and images configured to determine, from said fifth target, a fourth target and a first sample image in which said fourth target is located, based on feature information and a second number of sample images to be mined for each category. a decision sub-part.

In a possible implementation, the target selection sub-unit orders the second targets based on the category confidence and information entropy of the second target, respectively; selecting a seventh target of the number and performing a merge on said sixth target and said seventh target to obtain said fifth target.

In a possible implementation, the mining number determination sub-unit determines the percentage of third targets in each category based on the categories of third targets in the third sample image, and determines the percentage of third targets in each category. and determining a second number of sample images to be mined for each category based on the sampling weight of each category.

In a possible implementation, the target and image determination subunit selects among the third targets of the first category based on the distance between the characteristic information of the third target of the first category and the characteristic information of each fifth target. , a third target having the smallest distance from each fifth target is determined as an eighth target, the first category is any one of the categories of the third targets, and the eighth target is Among them, the target with the longest distance is determined as the fourth target.

In a possible implementation, the target and image determination sub-unit further adds the determined fourth target to the third target of the first category, and subtracts the determined fourth target from the unlabeled fifth target. configured to remove.

In a possible implementation, the apparatus comprises a feature extractor configured to input and process the third sample image into the target detection network to obtain feature information of a third target in the third sample image. Further prepare.

In a possible implementation, the apparatus further comprises a pre-training unit configured to pre-train the target detection network with labeled third sample images.

In a possible implementation, said first sample image comprises a long tail image.

In a possible implementation, the number-of-sampling determination sub-unit further comprises, based on target categories in the positive sample images of the training set, prior to respectively determining the first number to be sampled from each category of positive sample images. Sampling is performed on the positive and negative sample images in the set to obtain positive and negative sample images that are equal or close in number.

In a possible implementation, the total number of sample images awaiting mining is between 5% and 25% of the total number of first sample images.

In a possible implementation, the target selection sub-unit further removes targets among the sixth targets that are the same as the seventh target, and removes surplus targets among the sixth targets that are different from the seventh target. and the surplus target and the seventh target are used as the fifth target.

In a possible implementation, the method includes determining each fifth target among the third targets of the first category based on the distance between the feature information of the third target of the first category and the feature information of each fifth target. After determining the third target with the smallest distance to and as the eighth target, the number of the first sample images in which the fourth target is located reaches the second number of the mining-waiting sample images of the first category. If yes, further comprising terminating the determination for the eighth target.

In a possible implementation, the target and image determination subunit further determines the third target of the first category based on the distance between the feature information of the third target of the first category and the feature information of each fifth target. Among them, the third target with the shortest distance from each fifth target is determined and set as the eighth target, and then the number of the first sample images in which the fourth target is located is the mining-waiting sample image of the first category. and the set storing feature information of the fifth target is null, the determination for the eighth target is terminated.

In a possible implementation, the feature extractor further inputs the third sample image into the target detection network, obtains a feature vector output from a hidden layer of the target detection network, converts the feature vector into the third 3 is configured to be determined as target feature information.

According to one aspect of embodiments of the present application, a target detection apparatus is provided. The apparatus is a detection processing unit configured to input a pending image to a target detection network for processing to obtain a target detection result of the pending image, wherein the target detection result is the pending image. The target detection network comprises a detection processing unit, including the location and category of the target in the image, which has been trained according to the network training method described above.

In some embodiments, the functions or parts comprising the apparatus provided in the embodiments of the present application may be configured to perform the methods described in the above method embodiments, the specific implementation of which is , the description of the embodiments of the above method can be referred to, and the detailed description is omitted here for the sake of brevity.

In some embodiments, a "part" may be part of a circuit, part of a processor, part of a program or software, etc., and may of course be a unit or even a module. may be non-modularized.

Embodiments of the present application further provide a computer-readable storage medium. The computer readable storage medium stores computer program instructions which, when executed by a processor, implement the method. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

Embodiments of the present application further provide an electronic device. The electronic device comprises a processor and a memory configured to store instructions executable by the processor, the processor calling the instructions stored in the memory to perform the method. Configured.

Embodiments of the present application further provide a computer program product. The computer program product comprises computer readable code, and when the computer readable code is executed in a device, a processor in the device causes a network training method or a target detection method provided in any one of the embodiments above to implement the network training method or the target detection method. execute the instructions of

Embodiments of the present application further provide another computer program product. The computer program product is configured to store computer readable instructions which, when executed, cause a computer to perform the operations of the network training method or target detection method provided in any one of the embodiments above. .

An electronic device may be provided as a terminal, server, or other form of device.

FIG. 5 is a block diagram illustrating electronic device 800 according to an embodiment of the present application. For example, electronic device 800 may be a terminal such as a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical equipment, fitness equipment, personal digital assistant, and the like.

Referring to FIG. 5, electronic device 800 includes processing component 802 , memory 804 , power component 806 , multimedia component 808 , audio component 810 , input/output (I/O) interface 812 , sensor component 814 and communication component 816 . may comprise one or more of

Processing component 802 generally controls the overall operation of electronic device 800 . For example, it controls operations related to display, phone calls, data communication, camera operation and recording operation. Processing component 802 may include one or more processors 820 for executing instructions. All or part of the steps of the above method are thereby performed. Note that processing component 802 may comprise one or more modules for interaction with other units. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802 .

Memory 804 is configured to support operations in electronic device 800 by storing various data. Examples of such data include instructions for any application programs or methods that operate on electronic device 800, contact data, phone book data, messages, images, videos, and the like. Memory 804 may be implemented by any type of volatile or non-volatile storage, or a combination thereof. For example, static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), electrically erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM). ), magnetic memory, flash memory, magnetic or optical disk.

Power supply component 806 provides power to various units of electronic device 800 . Power component 806 may comprise a power management system, one or more power sources, and other units related to power generation, management, and distribution for electronic device 800 .

A multimedia component 808 comprises a screen for providing an output interface between the electronic device 800 and a user. In some examples, the screen includes a liquid crystal display (LCD) and a touch panel (TP). When the screen includes a touch panel, it is implemented as a touch panel and receives input signals from the user. A touch panel comprises one or more touch sensors that sense touches, slides and gestures on the panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some examples, multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operating mode such as a shooting mode or a video mode, the front camera and/or the rear camera can receive multimedia data from the outside. Each front and rear camera may have a fixed optical lens system or focus and optical zoom capabilities.

Audio component 810 is configured to output and/or input audio signals. For example, audio component 810 comprises a microphone (MIC). When the electronic device 800 is in operating modes such as call mode, recording mode and voice recognition mode, the microphone is configured to receive audio signals from the outside. The received audio signal can be further stored in memory 804 or transmitted via communication component 816 . In some examples, audio component 810 further comprises a speaker configured to output an audio signal.

I/O interface 812 provides an interface between processing component 802 and peripheral interface modules. The peripheral interface modules may be keyboards, click wheels, buttons, and the like. These buttons include, but are not limited to, home button, volume button, start button and lock button.

Sensor component 814 comprises one or more sensors and is configured to perform various condition assessments for electronic device 800 . For example, the sensor component 814 can detect the on/off state of the electronic device 800, the relative positioning of the units. For example, the unit is the display and keypad of electronic device 800 . The sensor component 814 detects changes in the position of the electronic device 800 or a unit in the electronic device 800, whether there is contact between the user and the electronic device 800, the orientation or acceleration/deceleration of the electronic device 800, and changes in the temperature of the electronic device 800. You can also Sensor component 814 may comprise a proximity sensor and is configured to detect the presence of surrounding objects in the absence of any physical contact. The sensor component 814 may comprise an optical sensor such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor and is configured for imaging applications. In some examples, the sensor component 814 may comprise an accelerometer, gyro sensor, magnetic sensor, pressure sensor, or temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 may access a wireless network based on a communication standard such as wireless network (WiFi), second generation mobile technology (2G), third generation mobile technology (3G), or a combination thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from external broadcast channel management systems via broadcast channels. In one exemplary embodiment, the communication component 816 further comprises a Near Field Communication (NFC) module to facilitate near field communication. For example, NFC modules are implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 includes one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPDs), programmable logic devices (PLDs), field programmable It may be implemented by a gate array (FPGA), controller, microcontroller, microprocessor or other electronic device and configured to carry out the methods described above.

The illustrative embodiment further provides a non-volatile computer-readable storage medium, such as memory 804, containing computer program instructions. The computer program instructions are executed by processor 820 of electronic device 800 to complete the method.

FIG. 6 is a block diagram illustrating electronic device 1900 according to an embodiment of the present application. For example, electronic device 1900 may be provided as a server. Referring to FIG. 6, electronic device 1900 includes processing component 1922 . It further comprises one or more processors and memory resources represented by memory 1932 . The memory resources are for storing instructions to be executed by processing component 1922, such as application programs. An application program stored in memory 1932 may include one or more modules each corresponding to a set of instructions. It should be noted that processing component 1922 is configured to execute instructions to perform the methods described above.

The electronic device 1900 includes a power component 1926 configured to perform power management of the electronic device 1900; a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network; O) an interface 1958; Electronic device 1900 runs on Microsoft server operating system (Windows Server ^™ ), Apple's graphic user interface-based operating system (Mac OS X ^™ ), multi-user multi-process computer operating system (Unix ^™ ), free and open source code type An operating system stored in memory 1932 may be run, such as a Unix-like operating system (Linux ^™ ), an open source code-based Unix-like operating system (FreeBSD ^™ ), or the like.

Exemplary embodiments further provide a non-volatile computer-readable storage medium, such as memory 1932, which contains computer program instructions. The computer program instructions are executed by processing component 1922 of electronic device 1900 to complete the method.

Embodiments of the present application may be systems, methods and/or computer program products. A computer program product may comprise a computer readable storage medium having computer readable program instructions stored thereon for causing a processor to implement aspects of the present application.

A computer-readable storage medium may be a tangible device capable of holding or storing instructions for use in an instruction-executing device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination of the above. More specific examples (non-exhaustive list) of computer readable storage media are portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash) ), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, flexible disc, punched card in which instructions are stored, or protrusions in grooves and any suitable combination of the above. Computer-readable storage media, as used herein, include radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or through electrical wires. It should not be construed as being a transitory signal per se, such as a transmitted electrical signal.

The computer readable program instructions described herein can be downloaded to each computing/processing device from a computer readable storage medium or network such as the Internet, local area networks, wide area networks and/or wireless networks. can be downloaded to an external computer or external storage device via A network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. A network interface card or network interface at each computing/processing device receives computer-readable program instructions from the network, transfers the computer-readable program instructions for storage on a computer-readable storage medium at each computing/processing device.

Computer program instructions configured to perform the operations of this application may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or one or more It may be source code or target code written in a programming language. The programming languages include object-oriented programming languages such as Smalltalk, C++, etc., and traditional procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user computer, partially executed on the user computer, executed as an independent software package, or partially executed on the user computer. It may be executed partially on a remote computer as a remote computer, or it may be executed entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer or to an external computer through any type of network, including local area networks (LAN) or wide area networks (WAN). (eg connect over the Internet using an Internet Service Provider). In some embodiments, state information in computer readable program instructions is used to customize electronic circuits such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs). The electronic circuitry may implement aspects of the present application by executing computer readable program instructions.

Aspects of the present application are now described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products of embodiments of the present application. It is to be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus, thereby producing an apparatus, wherein these instructions, when executed by the processor of the computer or other programmable data processing apparatus, flow charts. and/or produce an apparatus that performs the functions/operations specified in one or more of the blocks in the block diagrams. These computer readable program instructions may be stored on a computer readable storage medium. These instructions cause computers, programmable data processing devices, and/or other devices to operate in specific manners. Accordingly, a computer-readable storage medium having instructions stored thereon comprises an article of manufacture containing instructions for each aspect of implementing the functions/operations specified in one or more blocks in the flowcharts and/or block diagrams.

The computer readable program instructions may be loaded into a computer, other programmable data processing device or other device. It causes a computer, other programmable data processing device, or other device to perform a series of operational steps to produce a computer-implemented process. As such, instructions executed by a computer, other programmable data processing device, or other apparatus implement the functions/operations defined in one or more blocks in the flowchart illustrations and/or block diagrams.

The flowcharts and block diagrams in the figures illustrate possible architectures, functionality, and operation of systems, methods and computer program products according to embodiments of the present application. In this regard, each block in a flowchart or block diagram can represent part of a module, program segment or instruction. Some of said modules, program segments or instructions comprise executable instructions for implementing one or more predetermined logic functions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed essentially in parallel, or possibly in the opposite order, as determined from the functionality involved. Each block in the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented by means of dedicated hardware-based systems, or dedicated hardware and computer instructions, to perform the specified functions or operations. Note that it can be realized by a combination of

The computer program product may be specifically implemented in hardware, software or a combination thereof. In alternative embodiments, the computer program product may be embodied as a computer storage medium. In another alternative embodiment, the computer program product is specifically embodied as a software product, such as, for example, a Software Development Kit (SDK).

While embodiments of the present application have been described above, the above description is intended to be illustrative, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will readily occur to those skilled in the art without departing from the scope and spirit of each described embodiment. The choice of terminology used herein is such that it best interprets the principles, practical applications, or improvements of the technology in the marketplace, or that others of ordinary skill in the art may understand each embodiment disclosed herein. The purpose is to be able to understand

Embodiments of the present application relate to network training methods and apparatus, target detection methods and apparatus, and electronic devices. The network training method is to input unlabeled sample images into a target detection network for processing to obtain target detection results, which include target image regions, feature information and classification probabilities. determining a category confidence of the target based on the classification probability of the target; and labeling a sample image in which the first target is located for the first target whose category confidence is greater than or equal to a threshold and add it to the training set, and perform feature correlation mining on the second target for the second target whose category confidence is smaller than the first threshold, and perform feature correlation mining on the second target to the fourth target and adding the sample image in which it is located to the training set; and training a target detection network based on the sample images in the training set. Embodiments of the present application can improve training effectiveness of target detection networks.

Claims (20)

A network training method comprising:
inputting an unlabeled first sample image into a target detection network and processing to obtain a target detection result of the first sample image, wherein the target detection result is the number of targets in the first sample image; including image regions, feature information and classification probabilities;
determining a category confidence for the target based on the target's classification probabilities;
For a first target among the targets whose category reliability is greater than or equal to a first threshold, a first sample image in which the first target is located is defined as a second labeled sample image and added to the training set. and the labeling information of the second sample images includes the image region of the first target and the category corresponding to the category confidence of the first target, and the training set includes labeled third sample images. and
For a second target among the targets whose category reliability is smaller than the first threshold, feature correlation mining is performed on the second target based on the feature information of the third target in the third sample image. determining a fourth target and a first sample image in which the fourth target is located from the second target by feature correlation mining, and using the first sample image in which the fourth target is located as a fourth sample image; adding to the training set;
training the target detection network based on labeling information of the fourth sample images, second sample images, third sample images and the fourth sample images in the training set. training the target detection network based on the labeling information of the fourth sample image, the second sample image, the third sample image and the fourth sample image in the training set;
each determining a first number sampled from each category of positive sample images based on the category of the target in the training set of positive sample images, wherein the positive sample images are samples whose images contain the target; that it is an image;
sampling the positive sample images of each category based on a first number sampled from the positive sample images of each category to obtain a plurality of fifth sample images;
sampling the negative sample images of the training set to obtain a plurality of sixth sample images, wherein the negative sample images are sample images in which the target is not included in the images;
2. The method of claim 1, comprising training the target detection network based on the fifth sample image and the sixth sample image. performing feature correlation mining on the second target based on the feature information of the third target in the third sample image, and by the feature correlation mining, the fourth target and the position of the fourth target are obtained from the second target; Determining the first sample image to
determining the information entropy of the second target based on the classification probability of the second target;
selecting a fifth target from the second target based on the category confidence and information entropy of the second target;
respectively determining a second number of sample images to be mined for each category based on a third target category and a total number of sample images to be mined in the third sample images;
Based on the feature information of the third target in the third sample image, the feature information of the fifth target, and the second number of sample images awaiting mining in each category, from the fifth target, the fourth target and the fourth target 3. A network training method according to claim 1 or 2, comprising determining the first sample image in which is located. selecting a fifth target from the second target based on the categorical confidence and information entropy of the second target;
ordering the second targets based on their categorical confidence and information entropy, respectively, and selecting a third number of sixth targets and a fourth number of seventh targets;
merging the sixth target and the seventh target to obtain the fifth target. Determining a second number of sample images to be mined for each category based on a third target category and the total number of sample images to be mined in the third sample images, respectively;
determining a percentage of third targets in each category based on categories of third targets in the third sample image;
determining a sampling weight for each category based on the proportion of the third target for each category;
Determining a second number of sample images to be mined for each category based on the sampling weight of each category, respectively. Based on the feature information of the third target in the third sample image, the feature information of the fifth target, and the second number of sample images awaiting mining in each category, from the fifth target, the fourth target and the fourth target Determining the first sample image in which
Based on the distance between the feature information of the third target in the first category and the feature information of each fifth target, the third target with the smallest distance from each fifth target among the third targets in the first category is selected. each determining an eighth target, wherein the first category is any one of the categories of the third target;
The network training method according to any one of claims 3 to 5, further comprising determining a target having the longest distance among the eighth targets as a fourth target. Based on the feature information of the third target in the third sample image, the feature information of the fifth target, and the second number of sample images awaiting mining in each category, from the fifth target, the fourth target and the fourth target Determining the first sample image in which
7. The method of claim 6, further comprising adding the determined fourth target to the third target of the first category and removing the determined fourth target from the unlabeled fifth target. network training method. The network training method comprises:
8. Further comprising inputting said third sample image into said target detection network for processing to obtain characteristic information of a third target in said third sample image. The network training method according to item 1. Prior to inputting the unlabeled first sample image into a target detection network for processing to obtain a target detection result for the first sample image, the network training method comprises:
A method for training a network according to any one of claims 1 to 8, further comprising pre-training the target detection network with labeled third sample images. A method for training a network according to any one of claims 1-9, wherein the first sample images comprise long-tail images. In a possible implementation, prior to the step of respectively determining a first number sampled from each category of positive sample images based on target categories in the training set of positive sample images, the network training method comprises:
3. The network training of claim 2, comprising sampling positive and negative sample images in a training set to obtain positive and negative sample images that are equal or close in number. Method. The network training method according to claim 3, wherein the total number of sample images waiting to be mined is 5% to 25% of the total number of the first sample images. merging the sixth target and the seventh target to obtain the fifth target,
removing a target among the sixth targets that is the same as the seventh target to obtain a surplus target that is different from the seventh target among the sixth targets;
5. The network training method according to claim 4, comprising setting the surplus target and the seventh target as the fifth target. Based on the distance between the feature information of the third target in the first category and the feature information of each fifth target, the third target with the smallest distance from each fifth target among the third targets in the first category is selected. After each determining and eighth targeting step, the network training method comprises:
terminating the determination for the eighth target when the number of first sample images in which the fourth target is located reaches a second number of to-be-mined sample images of the first category. 7. A network training method according to Item 6. Based on the distance between the feature information of the third target in the first category and the feature information of each fifth target, the third target with the smallest distance from each fifth target among the third targets in the first category is selected. After each determining and eighth targeting step, the network training method comprises:
the number of first sample images in which the fourth target is located does not reach the second number of mining-waiting sample images of the first category, and the set storing feature information of the fifth target is null; 8. The method of claim 7, further comprising terminating determination for the eighth target. Inputting the third sample image to the target detection network and processing to obtain feature information of a third target in the third sample image includes:
inputting the third sample image into the target detection network and obtaining a feature vector output from a hidden layer of the target detection network;
and determining the feature vector as the feature information of the third target. A target detection method comprising:
inputting a to-be-processed image into a target detection network for processing to obtain a target detection result of the to-be-processed image, the target-detection result including the position and category of the target in the to-be-processed image;
A target detection method, comprising: the target detection network being trained by a network training method according to any one of claims 1-10. A network training device comprising:
A target detection unit configured to input and process an unlabeled first sample image into a target detection network to obtain a target detection result of the first sample image, wherein the target detection result is the a target detector, including target image regions, feature information and classification probabilities in the first sample image;
a confidence determiner configured to determine a category confidence of the target based on the classification probabilities of the target;
For a first target among the targets whose category reliability is greater than or equal to a first threshold, a first sample image in which the first target is located is defined as a labeled second sample image and added to the training set. wherein the labeling information of the second sample image includes an image region of the first target and a category corresponding to the category confidence of the first target, and the training set is labeled with a labeling portion containing a third sample image;
For a second target among the targets whose category reliability is smaller than the first threshold, feature correlation mining is performed on the second target based on the feature information of the third target in the third sample image. determining a fourth target and a first sample image in which the fourth target is located from the second target by feature correlation mining, and using the first sample image in which the fourth target is located as a fourth sample image; a feature mining unit configured to add to the training set;
a training unit configured to train the target detection network based on the labeling information of the fourth sample images, the second sample images, the third sample images and the fourth sample images in the training set. , a network training device. a memory for storing instructions executable by a processor;
a processor for invoking the instructions stored in the memory to perform a network training method according to any one of claims 1 to 16 or a target detection method according to claim 17; An electronic device. Computer program instructions for, when executed by a processor, causing said processor to perform a network training method according to any one of claims 1 to 16 or to perform a target detection method according to claim 17. computer-readable storage medium.

Images