TW202205151A - Network training method, target detection method, electronic device and computer readable storage medium - Google Patents

Abstract

The disclosed embodiment relates to a network training method, a target detection method, an electronic device and a computer readable storage medium. The network training method includes: inputting the unlabeled sample image into the target detection network to obtain the target detection result, which includes the image region, feature information and classification probability of the target; according to the classification probability of the target, the category confidence of the target is determined; for the first target whose category confidence is greater than or equal to the threshold, the sample image of the first target is added to the training set as a labeled image; for the second target whose category confidence is less than the first threshold, the second target is extracted from feature correlation, and the fourth target is determined, the sample image of the target is added to the training set; the training target detection network is based on the training set.

Description

Network training method, target detection method, electronic device and computer-readable storage medium

The present invention relates to the field of computer technology, and in particular, to a network training method, a target detection method, an electronic device and a computer-readable storage medium.

Computer vision is an important direction of artificial intelligence technology. In computer vision processing, it is usually necessary to detect objects (such as pedestrians, objects, etc.) in images or videos. Target detection of large-scale long-tail data has important applications in many fields, such as abnormal object detection in urban surveillance, abnormal behavior detection and emergency alarm. However, due to the huge amount of long-tail data and the serious imbalance of positive and negative samples, that is, most of the data images are background images, and only a small part of the images contain detectable targets, resulting in the target detection method of related technologies. The target detection effect is poor.

The embodiment of the present invention provides a technical solution for network training and target detection.

According to an aspect of the embodiments of the present invention, a network training method is provided, including: Input the unlabeled first sample image into the target detection network for processing, and obtain the target detection result of the first sample image, and the target detection result includes the image of the target in the first sample image. image area, feature information and classification probability; according to the classification probability of the target, determine the category confidence of the target; for the first target whose category confidence is greater than or equal to the first threshold, the first target The first sample image where the target is located is taken as the marked second sample image and added to the training set, wherein the annotation information of the second sample image includes the image area of the first target and the The category corresponding to the category confidence of the first target, and the training set includes the labeled third sample image; for the second target whose category confidence in the target is less than the first threshold, according to the third sample The feature information of the third target in the image, the feature correlation mining is performed on the second target, and the fourth target and the first object where the fourth target is located are determined from the second target through the feature correlation mining. this image, and take the first sample image where the fourth target is located as the fourth sample image, and add it to the training set; according to the annotation information of the fourth sample image, the The second sample image, the third sample image and the fourth sample image are used to train the target detection network.

In a possible implementation manner, according to the annotation information of the fourth sample image, the second sample image, the third sample image and the fourth sample image in the training set, the training of the Object detection network, including: According to the category of the target in the positive sample images of the training set, respectively determine the first number of samples sampled from the positive sample images of each category, and the positive sample images are sample images including the target in the image; The first quantity sampled in the positive sample images of each category, sampling the positive sample images of each category to obtain a plurality of fifth sample images; sampling the negative sample images of the training set to obtain a plurality of A sixth sample image, the negative sample image is a sample image that does not include a target in the image; the target detection network is trained according to the fifth sample image and the sixth sample image.

In a possible implementation manner, the feature correlation mining is performed on the second target according to the feature information of the third target in the third sample image, and the feature correlation mining is performed from the second target. Determining the fourth object and the first sample image where the fourth object is located includes: determining the information entropy of the second object according to the classification probability of the second object; according to the category of the second object Confidence and information entropy, select the fifth target from the second target; according to the category of the third target in the third sample image and the total number of sample images to be mined, determine each category to be The second number of sample images to be mined; according to the feature information of the third target in the third sample image, the feature information of the fifth target and the second number of sample images to be mined in each category, from A fourth object and a first sample image where the fourth object is located are determined from the fifth object.

In a possible implementation manner, selecting a fifth target from the second target according to the category confidence and information entropy of the second target includes: according to the category confidence and information entropy of the second target , sort the second targets respectively, select the sixth target with the third quantity and the seventh target with the fourth quantity; combine the sixth target and the seventh target to obtain the fifth target .

In a possible implementation manner, according to the category of the third target in the third sample image and the total number of sample images to be mined, the second number of sample images to be mined for each category is determined respectively, including : determine the proportion of the third object of each category according to the category of the third object in the third sample image; determine the sampling proportion of each category according to the proportion of the third object of each category; determine the sampling proportion of each category according to the proportion of the third object of each category , respectively determine the second number of sample images to be mined in each category.

In a possible implementation manner, according to the feature information of the third target in the third sample image, the feature information of the fifth target and the second number of sample images to be mined in each category, from the Determining the fourth target and the first sample image where the fourth target is located in the fifth target includes: according to the distance between the characteristic information of the third target of the first category and the characteristic information of each fifth target, Determine the third target with the smallest distance from each fifth target among the third targets of the first category, and use it as the eighth target, and the first category is any one of the categories of the third targets; The target with the largest distance among the eight targets is determined as the fourth target.

In a possible implementation manner, according to the feature information of the third target in the third sample image, the feature information of the fifth target and the second number of sample images to be mined in each category, from the Determining the fourth object and the first sample image where the fourth object is located in the fifth object, further comprising: adding the determined fourth object to the third object of the first category, and adding the determined fourth object to the third object of the first category. The fourth target determined as described above is removed from the unmarked fifth target.

In a possible implementation manner, the method further includes: inputting the third sample image into the target detection network for processing to obtain feature information of the third target in the third sample image.

In a possible implementation manner, before the step of inputting the unlabeled first sample image into the target detection network for processing to obtain the target detection result of the first sample image, the method further include: The target detection network is pre-trained through the labeled third sample image.

In a possible implementation manner, the first sample image includes a long-tail image.

According to an aspect of the embodiments of the present invention, there is provided a target detection method, the method comprising: inputting a to-be-processed image into a target detection network for processing, and obtaining a target detection result of the to-be-processed image, the target detection result It includes the position and category of the target in the to-be-processed image, and the target detection network is trained according to the above-mentioned network training method.

According to an aspect of the embodiments of the present invention, a network training apparatus is provided, including: The target detection part is configured to input the unlabeled first sample image into the target detection network for processing, and obtain the target detection result of the first sample image, and the target detection result includes the first sample image The image area, feature information and classification probability of the target in this image; a confidence determination part, configured to determine the category confidence of the target according to the classification probability of the target; The labeling part is configured to take the first sample image where the first target is located as the labeled second sample image for the first target whose category confidence is greater than or equal to the first threshold in the target, and Add to the training set, wherein the annotation information of the second sample image includes the image area of the first target and the category corresponding to the category confidence of the first target, and the training set includes the marked first target. Three sample images; The feature mining part is configured to, for the second target whose category confidence is less than the first threshold in the target, perform a search on the second target according to the feature information of the third target in the third sample image. Feature correlation mining, through feature correlation mining, the fourth target and the first sample image where the fourth target is located are determined from the second target, and the first sample image where the fourth target is located is determined. image as the fourth sample image and added to the training set; The training part is configured to train the target detection network according to the annotation information of the fourth sample image, the second sample image, the third sample image and the fourth sample image in the training set .

In a possible implementation manner, the training part includes: a sampling quantity determination sub-part, configured to determine, according to the category of the target in the positive sample images of the training set, to sample samples from the positive sample images of each category respectively The first number of positive sample images is a sample image including the target in the image; the first sampling sub-section is configured to sample the positive sample images of each category according to the first number of samples of each category. The positive sample images are sampled to obtain a plurality of fifth sample images; the second sampling subsection is configured to sample the negative sample images of the training set to obtain a plurality of sixth sample images, and the negative sample images are obtained. The sample image is a sample image that does not include a target; the training subsection is configured to train the target detection network according to the fifth sample image and the sixth sample image.

In a possible implementation manner, the feature mining part includes: an information entropy determination sub-section, configured to determine the information entropy of the second target according to the classification probability of the second target; a target selection sub-section, configured by is configured to select a fifth target from the second target according to the category confidence and information entropy of the second target; the mining quantity determination subsection is configured to be configured according to the third sample image in the third sample image. The category of the target and the total number of sample images to be mined respectively determine the second number of sample images to be mined for each category; the target and image determination sub-section is configured to be based on the third sample image. The feature information of the third target, the feature information of the fifth target and the second number of sample images to be mined in each category, the fourth target and the fourth target where the fourth target is located are determined from the fifth target. a sample image.

In a possible implementation manner, the target selection sub-section is configured to: according to the category confidence and information entropy of the second target, sort the second targets respectively, and select a third number of sixth targets. A target and a seventh target of a fourth quantity; the sixth target and the seventh target are combined to obtain the fifth target.

In a possible implementation manner, the mining quantity determination subsection is configured to: determine the proportion of the third objects of each category according to the category of the third object in the third sample image; The proportion of the three targets determines the sampling proportion of each category; according to the sampling proportion of each category, the second quantity of sample images to be mined in each category is determined respectively.

In a possible implementation manner, the target and image determination subsections are configured to: determine the third target according to the distance between the characteristic information of the third target of the first category and the characteristic information of each fifth target. Among the third targets of a category, the third target with the smallest distance from each fifth target is used as the eighth target, and the first category is any one of the categories of the third targets; the distance among the eighth targets is the largest target, identified as the fourth target.

In a possible implementation manner, the target and image determination subsection is further configured to: add the determined fourth target to the third target of the first category, and add the determined fourth target to the third target of the first category. Four targets were removed from the unlabeled fifth target.

In a possible implementation manner, the apparatus further includes: a feature extraction part, configured to input the third sample image into the target detection network for processing, and obtain the third sample image in the third sample image for processing. Characteristic information of three targets.

In a possible implementation manner, before the target detection part, the apparatus further includes: a pre-training part, configured to pre-train the target detection network by using the labeled third sample images.

In a possible implementation manner, the first sample image includes a long-tail image.

According to an aspect of the embodiments of the present invention, a target detection apparatus is provided, the apparatus includes: a detection processing part configured to input an image to be processed into a target detection network for processing, and obtain a target of the to-be-processed image The detection result, the target detection result includes the position and category of the target in the to-be-processed image, and the target detection network is trained according to the above-mentioned network training method.

In a possible implementation manner, before the step of respectively determining the first number of samples sampled from the positive sample images of each category according to the category of the target in the positive sample images of the training set, the method includes: training the training set. The positive sample images and negative sample images in the set are sampled to obtain the same or similar number of positive sample images and negative sample images.

In a possible implementation manner, the total number of sample images to be mined is 5% to 25% of the total number of the first sample images.

In a possible implementation manner, the combining the sixth target and the seventh target to obtain the fifth target includes: removing the same target as the seventh target in the sixth target , obtain the remaining target that is different from the seventh target in the sixth target; take the remaining target and the seventh target as the fifth target.

In a possible implementation manner, the method further includes: when the number of the fourth sample images of the first category reaches a second number of sample images of the first category to be mined, ending Feature correlation mining for the first category.

In a possible implementation manner, according to the distance between the feature information of the third target of the first category and the feature information of each fifth target, determine the distance between the third target of the first category and the feature information of each fifth target, respectively. After the step of using the third target with the smallest distance from the five targets as the eighth target, the method further includes: when the number of the first sample images where the fourth target is located reaches the target of the first category to be mined When the second number of sample images is reached, the determination of the eighth target is ended.

In a possible implementation manner, according to the distance between the feature information of the third target of the first category and the feature information of each fifth target, determine the distance between the third target of the first category and the feature information of each fifth target, respectively. After the step of using the third target with the smallest distance between the five targets as the eighth target, the method further includes: when the number of the first sample images where the fourth target is located does not reach the number of the first category to be mined When the second number of sample images of , and the set storing the feature information of the fifth object is empty, the determination of the eighth object is ended.

In a possible implementation manner, the inputting the third sample image into the target detection network for processing to obtain feature information of the third target in the third sample image includes: The third sample image is input into the target detection network, and the feature vector output by the hidden layer of the target detection network is obtained; the feature vector is determined as the feature information of the third target.

According to an aspect of the embodiments of the present invention, an electronic device is provided, including: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory , to perform the above network training method, or to perform the above target detection method.

According to an aspect of the embodiments of the present invention, there is provided a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the network training method or the above target detection method is implemented.

According to the embodiment of the present invention, the target detection results of unlabeled sample images can be obtained through the target detection network; pseudo-labeling and feature correlation mining are respectively performed according to the target detection results, high-value sample images are marked and collected, and added to the training set ; Train the target detection network according to the expanded training set, thereby expanding the number of positive sample data in the training set, alleviating the imbalance between positive and negative samples, and improving the training effect of the target detection network.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of embodiments of the present invention. Other features and aspects of embodiments of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Various exemplary embodiments, features and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only a relationship to describe related objects, which means that there can be three relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the term "at least one" herein refers to any combination of any one of a plurality or at least two of a plurality, for example, including at least one of A, B, and C, and may mean including those composed of A, B, and C. Any one or more elements selected in the collection.

In addition, in order to better illustrate the embodiments of the present invention, numerous specific details are given in the following detailed description. It should be understood by those skilled in the art that the embodiments of the present invention can also be implemented without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as to obscure the subject matter of the embodiments of the present invention.

FIG. 1 shows a flowchart of a network training method according to an embodiment of the present invention. As shown in FIG. 1 , the network training method includes: In step S11, the unlabeled first sample image is input into a target detection network for processing to obtain a target detection result of the first sample image, where the target detection result includes the first sample image The image area, feature information and classification probability of the target in the image; In step S12, the category confidence of the target is determined according to the classification probability of the target; In step S13, for the first target whose category confidence is greater than or equal to the first threshold in the target, the first sample image where the first target is located is taken as the marked second sample image, and added In the training set, the annotation information of the second sample image includes the image area of the first target and the category corresponding to the category confidence of the first target, and the training set includes the labeled third sample image; In step S14, for a second target whose category confidence is less than the first threshold in the target, perform feature correlation on the second target according to the feature information of the third target in the third sample image Mining, through feature correlation mining, the fourth target and the first sample image where the fourth target is located are determined from the second target, and the first sample image where the fourth target is located is used as The fourth sample image is added to the training set; In step S15, the target detection network is trained according to the annotation information of the fourth sample image, the second sample image, the third sample image and the fourth sample image in the training set.

In a possible implementation manner, the method may be executed by an electronic device such as a terminal device or a server, and the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a wireless For telephones, personal digital assistants (PDAs), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc., the method can be implemented by the processor calling computer-readable instructions stored in the memory. Alternatively, the method may be performed by a server.

For example, the first sample image may be an image acquired by an image acquisition device (eg, a camera). The first sample image may include a large-scale long-tailed image, that is, most of the images are background images, and a small part of the images include detectable objects. Detectable targets may include, for example, human bodies, faces, vehicles, objects, and the like. For example, in the security field, images of a certain geographic area can be collected by cameras, and people may pass through the geographic area only a small part of the time, so most of the collected images are background images, and only a small part of the images Includes human face and/or human body. In this case, the multiple images collected can form a long-tail dataset. The embodiment of the present invention does not limit the acquisition method of the first sample image and the category of the target in the first sample image.

In a possible implementation manner, a target detection network may be preset to detect the position (that is, the detection frame) and category of the target in the image. The target detection network may be, for example, a convolutional neural network, and the embodiment of the present invention does not limit the network structure of the target detection network.

In a possible implementation manner, before step S11, the method further includes: pre-training the target detection network by using the labeled third sample image. That is to say, a training set may be preset, and the training set includes the labeled third sample images, and the labeling information of the third sample images may include the detection frame and category of the target in the image. According to the training set, the target detection network can be pre-trained by the method in the related art, so that the target detection network has a certain detection accuracy.

However, the pre-trained object detection network has poor detection effect on large-scale long-tail images. Therefore, the unlabeled first sample image can be used to further train the object detection network through active learning.

In a possible implementation manner, in step S11 , the unlabeled first sample image may be input into a target detection network for processing to obtain a target detection result of the first sample image. The target detection result may include the image area, feature information and classification probability of the target in the first sample image. The image area where the target is located can be the detection frame in the image; the feature information of the target can be, for example, the feature vector output by the hidden layer (such as the convolution layer) of the target detection network; the classification probability of the target can indicate that the target belongs to each category The classification posterior probability of .

In a possible implementation manner, the target in the first sample image may also be referred to as an instance, and one or more targets may be detected in each first sample image. In practical processing, the order of magnitude of detected objects may be several to dozens of times the order of magnitude of images.

In a possible implementation manner, in step S12, according to the classification probability of the target, the maximum value of the classification probability may be obtained and determined as the classification confidence level of the target.

In a possible implementation manner, in step S13 , for a target whose category confidence is greater than or equal to the first threshold (which may be referred to as a first target), the first sample image where the first target is located may be regarded as an existing target. Annotated sample images (which can be called second sample images) are added to the training set. The image area of the first target is taken as the marked image area, and the category corresponding to the category confidence of the first target is taken as the marked category of the first target. The same second sample image may be labeled multiple times by multiple first objects in the second sample image. The first threshold is, for example, 0.99, and the embodiment of the present invention does not limit the value of the first threshold.

In a possible implementation manner, the process of step S13 may be called pseudo-labeling. That is, the image where the target with higher confidence is located is regarded as a high-value sample, and the target detection inference result is directly used as the target annotation result. In this way, the number of positive samples in the training set can be expanded to solve the problem of difficulty in collecting positive samples.

In a possible implementation manner, in step S14, for the target whose category confidence is less than the first threshold (which may be referred to as the second target), the target in the third sample image that has been marked in the training set (which may be referred to as The feature information of the third target), carry out feature correlation mining on the second target, and dig out the target that meets the requirements from the second target (which can be called the fourth target). For example, the distance or correlation between the feature information of the third target and the feature information of the second target can be calculated, a preset number of targets can be selected according to the distance or the correlation degree, and the selected preset number of targets can be used as the first target. Four goals.

In a possible implementation manner, the first sample image where the mined fourth target is located may be taken as the fourth sample image and added to the training set, so as to complete the processing process of feature correlation mining. In this way, the number of sample data in the training set can be further expanded.

In a possible implementation manner, the annotation information of the fourth sample image may be obtained by manual annotation, for example, manually determining the detection frame and category of the target in the fourth sample image. This embodiment of the present invention does not limit this.

In a possible implementation manner, in step S15, after obtaining the annotation information of the fourth sample image, the training target can be trained according to the second sample image, the third sample image and the fourth sample image in the training set Check the network.

According to the embodiment of the present invention, the target detection results of unlabeled sample images can be obtained through the target detection network; pseudo-labeling and feature correlation mining are respectively performed according to the target detection results, high-value sample images are marked and collected, and added to the training set ; Train the target detection network according to the expanded training set, thereby expanding the number of positive sample data in the training set, alleviating the imbalance between positive and negative samples, and improving the training effect of the target detection network.

In a possible implementation manner, after the processing of step S11, the target detection results of each first sample image are obtained; after the processing of S12, the category confidence of the target in each of the first sample images is obtained. In step S13, the sample image of the first target whose category confidence is greater than or equal to the first threshold can be added to the training set, and the labeled second sample image can be obtained by pseudo-labeling; in step S14, the The second target whose class confidence is less than the first threshold is mined.

In a possible implementation manner, step S14 may include: determining the information entropy of the second target according to the classification probability of the second target; selecting a fifth target from the second targets according to the category confidence and information entropy of the second target; According to the category of the third target in the third sample image and the total number of sample images to be mined, respectively determine the second number of sample images to be mined in each category; According to the feature information of the third target in the third sample image, the feature information of the fifth target and the second number of sample images to be mined in each category, a fourth target is determined from the fifth target The target and the first sample image where the fourth target is located.

For example, according to the classification probability of the second target, the information entropy of the second target can be calculated to represent the degree of uncertainty of the second target, that is, the greater the information entropy of the second target, the higher the The greater the degree of uncertainty; on the contrary, the smaller the information entropy of the second target, the smaller the degree of uncertainty of the second target. The embodiment of the present invention does not limit the calculation method of the information entropy.

In a possible implementation manner, according to the category confidence and information entropy of the second target, a target (which may be referred to as a fifth target) that satisfies a certain condition may be selected from a plurality of second targets, for example, a category confidence may be selected. Targets with larger degrees, targets with larger information entropy, etc.

In a possible implementation manner, the step of selecting a fifth target from the second target according to the category confidence and information entropy of the second target may include: According to the category confidence and information entropy of the second targets, the second targets are sorted respectively, and the third number of sixth targets and the fourth number of seventh targets are selected; The sixth target and the seventh target are combined to obtain the fifth target.

That is, according to the category confidence of the second target, sort the multiple second targets; according to the sorting result, select a preset third number of targets (which may be referred to as the sixth target) from the plurality of second targets ). Similarly, according to the information entropy of the second targets, the plurality of second targets are sorted; according to the sorting result, a preset fourth number of targets (which may be referred to as seventh targets) are selected from the plurality of second targets. Wherein, the third number and the fourth number may be 3K respectively, K represents the number of sample images to be mined, and K is, for example, 10000. In actual processing, the value of K may be 5%~25% of the total number of unlabeled first sample images. The embodiment of the present invention does not limit the value of K and the quantitative relationship between the third quantity and the fourth quantity and K.

It should be understood that those skilled in the art can set the values of the number K, the third number and the fourth number of sample images to be excavated according to the actual situation, and the third number and the fourth number may be different. No restrictions apply.

In a possible implementation manner, the selected sixth target and the seventh target may be combined, and the combined multiple targets may be used as the fifth target, so as to remove possible duplicate targets therein. In actual processing, about 6K fifth objects are available.

The above processing method can be called bootstrapping. In this way, a certain number of positive samples and negative samples with high probability can be selected from the second target at the same time, so as to perform feature correlation mining in the future. Reduce the calculation amount of feature correlation mining and improve processing efficiency.

In a possible implementation manner, the step of respectively determining the second number of sample images to be mined in each category according to the category of the third target in the third sample image and the total number of sample images to be mined , which can include: According to the category of the third object in the third sample image, determine the proportion of the third object of each category; According to the proportion of the third target of each category, determine the sampling proportion of each category; According to the sampling proportion of each category, the second quantity of sample images to be mined in each category is determined respectively.

_c 。

（1）

（2） 在公式（1）和（2）中，r_c 表示類別c的抽樣值；t為超參數，取值例如為0.1；C表示類別數量；r_i 表示C個類別中第i個類別的抽樣值。For example, according to the category of the third object in the third sample image that has been marked in the training set, the proportion f _c of the third object in each category can be determined; according to the proportion f _c , the following formula can be used to calculate each Sampling weights for categories

_c .

(1)

(2) In formulas (1) and (2), rc represents the sampling value of category _c ; t is a hyperparameter, with a value of 0.1, for example; C represents the number of categories; ri represents the _ith category in the C categories sample value.

Through the processing of formulas (1) and (2), the sampling proportion corresponding to the category with a small proportion can be increased, and the sampling proportion corresponding to the category with a large proportion can be reduced, thereby alleviating the number imbalance between samples of different categories problem in order to improve the training effect of the network.

_c 以及待挖掘的樣本圖像的總數量（K個），可確定出各個類別待挖掘的樣本圖像的第二數量。進而可根據第二數量進行特徵相關挖掘。In one possible implementation, according to the sampling weight of each category

_c and the total number (K) of sample images to be mined, the second number of sample images to be mined for each category can be determined. Further, feature correlation mining may be performed according to the second quantity.

In a possible implementation manner, the method further includes: inputting the third sample image into the target detection network for processing to obtain feature information of the third target in the third sample image.

That is to say, the labeled third sample image in the training set can be input into the target detection network, and the feature information of the third sample image can be output by the hidden layer (such as the convolution layer) of the target detection network, such as Feature vector. In this way, the features of the third sample image can be obtained to facilitate subsequent feature correlation mining.

In a possible implementation manner, according to the feature information of the third target in the third sample image, the feature information of the fifth target and the second number of sample images to be mined in each category, from the In the fifth target, the fourth target and the first sample image where the fourth target is located are determined, including: According to the distance between the feature information of the third target of the first category and the feature information of each fifth target, the third target with the smallest distance from each fifth target among the third targets of the first category is respectively determined, and used as The eighth target, the first category is any one of the categories of the third target;

The target with the largest distance among the eighth targets is determined as the fourth target.

For example, after the second number of sample images to be mined in each category is determined, a k-center method may be used to mine a corresponding number of sample images from the sample images where the fifth target is located. For any one of the multiple categories of the third target (which may be referred to as the first category), the distance between the feature information of the third target of the first category and the feature information of each fifth target may be calculated, the distance It can be, for example, the Euclidean distance. For any fifth target, the third target with the smallest distance from the fifth target among the third targets in the first category can be determined, so that the third target with the smallest distance from each fifth target can be determined, which can be called the first target. Eight goals.

（3） 在公式（3）中，u表示特徵相關挖掘得到的第四目標；

表示第j個第五目標的特徵資訊

與第一類別c的第l個第三目標的特徵資訊

之間的距離；

表示第五目標的特徵資訊的集合；

表示第一類別c的第三目標的特徵資訊的集合。In a possible implementation manner, one target with the largest distance may be selected from each of the eighth targets, and determined as the fourth target obtained by this feature correlation mining. As shown in the following formula:

(3) In formula (3), u represents the fourth target obtained by feature correlation mining;

Indicates the feature information of the jth fifth target

The characteristic information of the lth third object with the first category c

the distance between;

A collection of feature information representing the fifth target;

A set of feature information representing the third object of the first category c.

In a possible implementation manner, the first sample image where the fourth target is located can be determined, and the sample image is added to the training set as the fourth sample image, thereby completing the feature correlation mining process this time.

In a possible implementation manner, according to the feature information of the third target in the third sample image, the feature information of the fifth target and the second number of sample images to be mined in each category, from the The step of determining the fourth target and the first sample image where the fourth target is located in the fifth target further includes: The determined fourth object is added to the third object of the first category, and the determined fourth object is removed from the unlabeled fifth object.

中，從第五目標的特徵資訊的集合

中移除。這樣，在下次的特徵相關挖掘中，可以通過公式（3）對更新後的兩個集合進行挖掘，重複上述過程。That is to say, the fourth target obtained by this feature correlation mining is regarded as the labeled target, and the fourth target is removed from the unlabeled target. In this case, the feature information of the fourth object can be added to the set of feature information of the third object of the first category c

, the collection of feature information from the fifth target

removed in. In this way, in the next feature correlation mining, the two updated sets can be mined by formula (3), and the above process can be repeated.

為空）時，可完成該第一類別的特徵相關挖掘。In one possible implementation, the number of the fourth sample images in the first category reaches the second number of the first category, or the second number is not reached and the fifth target is exhausted (the set

is empty), the feature correlation mining of the first category can be completed.

In this way, feature correlation mining can be performed on each category separately, and finally a sufficient number of fourth sample images (usually K sample images) can be obtained, so as to further expand the number of sample images in the training set and alleviate the positive and negative Imbalance between samples.

In a possible implementation manner, human annotation may be performed on the mined fourth sample image to obtain annotation information of the fourth sample image. Since there may be both a positive sample image (that is, the fourth sample image including the target in the image) and a negative sample image (that is, the fourth sample image that does not include the target) in the fourth sample image , therefore, the annotation information of the fourth sample image may include sample type information of whether the image is a positive sample image or a negative sample image, the image frame where the target is located in the positive sample image, and the target type.

In a possible implementation manner, after the manual annotation is completed, the second sample image, the third sample image and the fourth sample image in the training set may be selected according to the annotation information of the fourth sample image in step S15. Sample images to train the object detection network.

Wherein, step S15 may include: according to the category of the target in the positive sample images of the training set, respectively determining the first number of samples sampled from the positive sample images of each category, and the positive sample images are images that include a sample image of the target; Sampling the positive sample images of each category according to the first quantity sampled in the positive sample images of each category to obtain a plurality of fifth sample images; Sampling the negative sample images of the training set to obtain a plurality of sixth sample images, where the negative sample images are sample images that do not include the target in the image; The target detection network is trained according to the fifth sample image and the sixth sample image.

For example, the target detection network can be trained by resampling, and the sampling frequency of data that appears less frequently in the data can be increased by resampling, so as to improve the performance of the network for these data, and further improve the positive and negative Imbalance between samples.

In a possible implementation manner, the positive sample images and the negative sample images in the training set (including the second sample image, the third sample image, and the fourth sample image) may be sampled respectively, so that the sampling The number of post-positive sample images and negative sample images is the same or similar.

In a possible implementation manner, for the positive sample image, the total number of samples of the positive sample image may be preset. According to the categories of the objects in the positive sample images in the training set, the first number of samples sampled from the positive sample images of each category is determined respectively.

（4） 公式（4）中，R_h 表示第h個類別的正樣本圖像的抽樣比重；q_h 表示第h個類別的目標的比例；t₁ 為超參數，取值例如為0.1。Similar to the previous processing process, according to the category of the target in the positive sample image, the proportion of the target of each category can be determined; according to the proportion, the sampling proportion of each category can be calculated by the following formula:

(4) In formula (4), R _h represents the sampling proportion of positive sample images of the h th category; q _h represents the proportion of the h th category of targets; t ₁ is a hyperparameter, and the value is, for example, 0.1.

Through the processing of formula (4), the sampling proportion corresponding to the category with a small proportion can be increased, and the sampling proportion corresponding to the category with a large proportion can be reduced, thereby alleviating the number imbalance between positive sample images of different categories. In order to improve the training effect of the network.

In a possible implementation manner, the first number of positive sample images of each category may be determined according to the sampling proportion of positive sample images of each category and the total number of samples of positive sample images.

In a possible implementation manner, for any category, a first number of positive sample images may be randomly sampled from the positive sample images of the category according to the first number of the category, as the fifth sample image. The positive sample images of each category are sampled respectively, and the fifth sample image with the total number of samples can be obtained.

In a possible implementation manner, for the negative sample images, the negative sample images in the training set can be directly randomly sampled according to the preset total number of samples to obtain the sixth sample image with the total number of samples. The total number of samples of negative sample images may be the same as or different from the total number of samples of positive sample images, which is not limited in this embodiment of the present invention.

In a possible implementation manner, the target detection network can be trained according to the fifth sample image and the sixth sample image. That is, input the fifth and sixth sample images into the target detection network respectively to obtain the target detection results of the fifth and sixth sample images; determine the loss of the target detection network according to the target detection results and the annotation information; The loss adjusts the parameters of the target detection network in reverse; after multiple rounds of repeated operations, the trained target detection network is obtained when the preset conditions (such as network convergence) are met.

In this way, the detection effect of the trained object detection network for long-tail images can be significantly improved.

In a possible implementation manner, before step S11, the step of pre-training the target detection network by using the labeled third sample image can also be performed by the above-mentioned resampling training method, thereby improving target detection. The pre-training effect of the network.

In practical applications, the entire process of steps S11-S15 can be repeated to achieve continuous incremental training. That is to say, when the unlabeled sample images are collected again, the target detection network after this training can be used as the initial target detection network, and the expanded training set can be used as the initial training set, repeating Carry out the process of pseudo-annotation-feature correlation mining-resampling training, so as to continuously improve the performance of the target detection network.

FIG. 2 shows a schematic diagram of a processing procedure of a network training method according to an embodiment of the present invention. As shown in FIG. 2 , the data source includes a large number of unlabeled first sample images 20 , and the first sample images 20 are input into the target detection network for prediction (predict) to obtain each first sample image The target detection result 21 of 20 includes the image area (not shown), feature vector and classification probability of the target in the first sample image.

As shown in FIG. 2 , in this example, the target detection network may include a CNN backbone network 211 , a feature map pyramid network (FPN) 212 , and a fully connected network 213 . The fully connected network 213 is, for example, a bbox head. After the first sample image 20 is input into the target detection network, it is processed by the CNN backbone network 211 and the FPN 212 to obtain a feature map 214 of the first sample image, and the feature map 214 is processed by the fully connected network 213 to obtain the target Test results 21.

In this example, the category confidence of the target can be determined according to the classification probability of the target; for the first targets whose category confidence is greater than or equal to the first threshold (for example, 0.99), the first objects in which these first targets are located are determined. This image is taken as the second sample image 22, and the second sample image 22 is pseudo-labeled, that is, the image area of the first target and the category corresponding to the category confidence of the first target are taken as the second sample image Label information like 22. The labeled second sample image 22 is added to the training set 25, thereby realizing the expansion of the positive samples in the training set.

In this example, for the second target whose category confidence is less than the first threshold, a certain number of fifth targets are selected by the bootstrapping method, and the sample image 23 where the fifth target is located is obtained. According to the feature vector (not shown) of the third target in the marked third sample image in the training set, perform feature correlation mining on the fifth target, and determine the fourth target and the first sample image where the fourth target is located image, as the fourth sample image 24. The fourth sample image 24 is manually labeled and added to the training set 25, so as to further expand the labeled images in the training set.

In this example, after two expansions, the training set 25 includes the labeled second sample image, the third sample image, and the fourth sample image. Resampling the training set 25, balancing the number of positive and negative samples, as well as the number of positive samples of different categories, to obtain a resampled training set 26; and then train the target detection network according to the resampled training set 26, thereby completing the entire process.

According to an embodiment of the present invention, a target detection method is also provided, the method comprising: Input the to-be-processed image into the target detection network for processing, and obtain the target detection result of the to-be-processed image, where the target detection result includes the position and category of the target in the to-be-processed image, and the target detection network It is trained according to the above-mentioned network training method.

That is to say, the target detection network trained by the above method can be deployed to realize the target detection of the image to be processed. The image to be processed may be, for example, an image collected by an image collection device (eg, a camera), and the image may include a target to be detected, such as a human body, a face, a vehicle, an object, and the like. This embodiment of the present invention does not limit this.

In a possible implementation manner, the to-be-processed image may be input into a target detection network for processing to obtain a target detection result of the to-be-processed image. The target detection result includes the position and category of the target in the image to be processed, such as the detection frame where the face in the image to be processed is located and the identity corresponding to the face.

In this way, the detection accuracy of target detection can be improved, and target detection of large-scale long-tail image data can be realized.

According to the network training method of the embodiment of the present invention, the active learning mining method is used to mine potential unlabeled data, the semi-supervised learning method is used to label the auxiliary unlabeled data, and the number of positive sample data is expanded, thereby solving the problem of In large-scale long-tail detection, the problem of large data size and difficulty in collecting positive samples, and to a certain extent, alleviates the problem of imbalance between positive and negative samples. The model performance is effectively improved in the environment of limited annotation and computing resources.

According to the network training method of the embodiment of the present invention, the target detection network is trained by means of resampling, which can solve the negative impact of the imbalance of positive and negative samples on network training, and alleviate the imbalance between different categories of positive samples. The negative impact of the target detection network can effectively converge and improve the network performance during training.

According to the network training method of the embodiment of the present invention, using the active learning method, it is possible to mine potentially high-value samples for model improvement in a huge amount of unlabeled data, which can be effectively improved in the environment of limited labeling and computing resources Model performance, saving a lot of manpower and computing costs required for deep learning models to be applied to new businesses; using the resampling method, the target detection network can be effectively trained in the case of unbalanced samples, without too much manual parameter adjustment intervention, saving depth The labor cost required to apply the learning model to the new business.

The network training method according to the embodiment of the present invention can be applied to the fields of smart video analysis, security and other fields. With limited labor and computing resources, the method can be used to perform online training on potential targets in smart video analysis or smart monitoring. It can quickly and repeatedly improve the detection network of the application, and quickly meet the performance requirements required by the business with less labor and computing costs, and can continue to improve network performance in the future.

The network training method of the embodiment of the present invention can be applied to online smart video analysis or smart monitoring, so as to quickly perform online and online potential target detection applications in smart video analysis or smart monitoring under limited labor and computing resources Iterative computing improves, so that the performance requirements required by the business can be quickly achieved with less labor and computing costs, and the model performance can be continuously improved after that.

It can be understood that the foregoing method embodiments mentioned in the embodiments of the present invention can be combined with each other to form a combined embodiment without violating the principle and logic. Those skilled in the art can understand that, in the above method of the specific embodiment, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the embodiments of the present invention also provide a network training device, a target detection device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any network training method or target detection method provided by the embodiments of the present invention , the corresponding technical solutions and descriptions, and refer to the corresponding records in the method section, which will not be repeated.

FIG. 3 shows a block diagram of a network training apparatus including a processor (not shown in FIG. 3 ) for executing in a memory (not shown in FIG. 3 ) according to an embodiment of the present invention Stored program portion; as shown in Figure 3, the program portion stored in the memory includes: The target detection part 31 is configured to input the unlabeled first sample image into the target detection network for processing, and obtain the target detection result of the first sample image, and the target detection result includes the first sample image. The image area, feature information and classification probability of the target in the sample image; a confidence level determination part 32, configured to determine the category confidence level of the object according to the classification probability of the object; The labeling part 33 is configured to take the first sample image where the first target is located as the labeled second sample image for the first target whose category confidence is greater than or equal to the first threshold in the target, and added to the training set, wherein the annotation information of the second sample image includes the image area of the first target and the category corresponding to the category confidence of the first target, and the training set includes the marked the third sample image; The feature mining part 34 is configured to, for the second object whose category confidence is less than the first threshold in the object, according to the feature information of the third object in the third sample image, analyze the second object for the second object. Carry out feature correlation mining, through feature correlation mining, determine the fourth target and the first sample image where the fourth target is located from the second target, and use the first sample image where the fourth target is located. The image is used as the fourth sample image and added to the training set; The training part 35 is configured to train the target detection network according to the annotation information of the fourth sample image, the second sample image, the third sample image and the fourth sample image in the training set road.

In a possible implementation manner, the training part includes: a sampling quantity determination sub-part, configured to determine, according to the category of the target in the positive sample images of the training set, to sample samples from the positive sample images of each category respectively The first number of positive sample images is a sample image including the target in the image; the first sampling sub-section is configured to sample the positive sample images of each category according to the first number of samples of each category. The positive sample images are sampled to obtain a plurality of fifth sample images; the second sampling subsection is configured to sample the negative sample images of the training set to obtain a plurality of sixth sample images, and the negative sample images are obtained. The sample image is a sample image that does not include a target; the training subsection is configured to train the target detection network according to the fifth sample image and the sixth sample image.

In a possible implementation manner, the feature mining part includes: an information entropy determination sub-section, configured to determine the information entropy of the second target according to the classification probability of the second target; a target selection sub-section, configured by is configured to select a fifth target from the second target according to the category confidence and information entropy of the second target; the mining quantity determination subsection is configured to be configured according to the third sample image in the third sample image. The category of the target and the total number of sample images to be mined respectively determine the second number of sample images to be mined for each category; the target and image determination sub-section is configured to be based on the third sample image. The feature information of the third target, the feature information of the fifth target and the second number of sample images to be mined in each category, the fourth target and the fourth target where the fourth target is located are determined from the fifth target. a sample image.

In a possible implementation manner, the target selection sub-section is configured to: according to the category confidence and information entropy of the second target, sort the second targets respectively, and select a third number of sixth targets. A target and a seventh target of a fourth quantity; the sixth target and the seventh target are combined to obtain the fifth target.

In a possible implementation manner, the mining quantity determination subsection is configured to: determine the proportion of the third objects of each category according to the category of the third object in the third sample image; The proportion of the three targets determines the sampling proportion of each category; according to the sampling proportion of each category, the second quantity of sample images to be mined in each category is determined respectively.

In a possible implementation manner, the target and image determination subsections are configured to: determine the third target according to the distance between the characteristic information of the third target of the first category and the characteristic information of each fifth target. Among the third targets of a category, the third target with the smallest distance from each fifth target is used as the eighth target, and the first category is any one of the categories of the third targets; the distance among the eighth targets is the largest target, identified as the fourth target.

In a possible implementation manner, the target and image determination subsection is further configured to: add the determined fourth target to the third target of the first category, and add the determined fourth target to the third target of the first category. Four targets were removed from the unlabeled fifth target.

In a possible implementation manner, the apparatus further includes: a feature extraction part, configured to input the third sample image into the target detection network for processing, and obtain the third sample image in the third sample image for processing. Characteristic information of three targets.

In a possible implementation manner, the apparatus further includes: a pre-training part, configured to pre-train the target detection network by using the labeled third sample image.

In a possible implementation manner, the first sample image includes a long-tail image.

In a possible implementation manner, the sampling quantity determination sub-section is further configured to: in the category of the target according to the positive sample images of the training set, respectively determine the number of samples sampled from the positive sample images of each category. Before the first quantity, the positive sample images and the negative sample images in the training set are sampled to obtain the same or similar quantity of positive sample images and negative sample images.

In a possible implementation manner, the total number of sample images to be mined is 5% to 25% of the total number of the first sample images.

In a possible implementation manner, the target selection sub-section is further configured to: remove the same target as the seventh target from the sixth target to obtain the sixth target and the seventh target Different residual goals; take the residual goal and the seventh goal as the fifth goal.

In a possible implementation manner, the method further includes: according to the distance between the feature information of the third target of the first category and the feature information of each fifth target, respectively determining the first category of the first category After the third target with the smallest distance from each fifth target among the three targets is used as the eighth target, after the number of the first sample images where the fourth target is located reaches the sample images to be mined of the first category When the second number of images is reached, the determination of the eighth target is ended.

In a possible implementation manner, the target and image determination subsections are further configured to: the distance between the feature information of the third target according to the first category and the feature information of each fifth target, respectively After determining the third target with the smallest distance from each fifth target among the third targets of the first category and using it as the eighth target, the number of the first sample images where the fourth target is located does not reach the When the second quantity of sample images of the first category to be mined is empty, and the set of storing the feature information of the fifth target is empty, the determination of the eighth target is ended.

In a possible implementation manner, the feature extraction part is further configured to: input the third sample image into the target detection network to obtain features output by the hidden layer of the target detection network vector; determining the feature vector as feature information of the third target.

According to an aspect of the present invention, a target detection device is provided, the device includes: a detection processing part configured to input an image to be processed into a target detection network for processing, and obtain a target detection result of the to-be-processed image , the target detection result includes the position and category of the target in the to-be-processed image, and the target detection network is trained according to the above-mentioned network training method.

In some embodiments, the functions or included parts of the apparatus provided in the embodiments of the present invention may be configured to execute the methods described in the above method embodiments. For specific implementation, reference may be made to the above method embodiments. For brevity, I won't go into details here.

In some embodiments, a "part" can also be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, it can also be a unit, and it can also be a module or a non-module.

An embodiment of the present invention further provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present invention further provides an electronic device, including: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method .

An embodiment of the present invention also provides a computer program product, including computer-readable code, when the computer-readable code is run on a device, a processor in the device executes a network training method configured to implement the network training method provided in any of the above embodiments Or directives for object detection methods.

Embodiments of the present invention also provide another computer program product configured to store computer-readable instructions, and when the instructions are executed, the computer executes the operations of the network training method or the target detection method provided by any of the above embodiments.

The electronic device may be provided as a terminal, server or other form of device.

FIG. 4 shows a block diagram of an electronic device 800 according to an embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

4, an electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensing server component 814, and communication component 816.

The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

The memory 804 is configured to store various types of data to support the operation of the electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. Memory 804 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Disk or CD.

Power supply assembly 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when electronic device 800 is in operating modes, such as call mode, recording mode, and voice recognition mode. The received audio signal may be further stored in memory 804 or transmitted via communication component 816 . In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules. The peripheral interface modules may be keyboards, click wheels, buttons, and the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for electronic device 800 . For example, the sensor assembly 814 can detect the open/closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, the sensor assembly 814 can also detect the electronic device 800 or Changes in the position of a component of the electronic device 800 , presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor assembly 814 may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as wireless network (WiFi), second generation mobile communication technology (2G) or third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for performing the above method.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions executable by the processor 820 of the electronic device 800 to accomplish the above method.

FIG. 5 shows a block diagram of an electronic device 1900 according to an embodiment of the present invention. For example, the electronic device 1900 may be provided as a server. 5, electronic device 1900 includes processing component 1922, which further includes one or more processors, and a memory resource represented by memory 1932 configured to store instructions executable by processing component 1922, such as applications . An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) Interface 1958. The electronic device 1900 can operate an operating system based on the memory 1932, such as Microsoft Server Operating System (Windows Server ^TM ), a graphical user interface based operating system (Mac OS X ^TM ) introduced by Apple Inc. Process Computer Operating System (Unix ^™ ), Free and Open Source Unix-like Operating System (Linux ^™ ), Open Source Unix-like Operating System (FreeBSD ^™ ) or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions executable by the processing component 1922 of the electronic device 1900 to accomplish the above method.

Embodiments of the present invention may be systems, methods and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the embodiments of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable Design read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick , a floppy disk, a mechanically encoded device, such as a punched card or a raised structure in a groove with instructions stored thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or Electrical signals carried by wires.

The computer-readable program instructions described herein may be downloaded from computer-readable storage media to various computing/processing devices, or downloaded to external computers or external storage over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network device. Networks may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. A network interface card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for computer-readable storage stored in each computing/processing device in the medium.

The computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or any other information in one or more programming languages. Combining source or object code written in programming languages including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely remotely. run on a client computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network—including a local area network (LAN) or a wide area network (WAN)—or, can be connected to an external computer (for example, using the Internet road service provider to connect via the Internet). In some embodiments, electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer readable program instructions. Electronic circuits may execute computer readable program instructions to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to the processor of a general purpose computer, special purpose computer or other programmable data processing device to produce a machine for execution of the instructions by the processor of the computer or other programmable data processing device When, means are created that implement the functions/acts specified in one or more of the blocks in the flowchart and/or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium, the instructions causing the computer, programmable data processing device and/or other equipment to operate in a particular manner, so that the computer readable medium storing the instructions Included is an article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded into a computer, other programmable data processing device, or other equipment, so that a series of operational steps are performed on the computer, other programmable data processing device, or other equipment to generate a computer Processes of implementation such that instructions executing on a computer, other programmable data processing apparatus, or other device implement the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that contains one or more logic for implementing the specified logic Executable instructions for the function. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by dedicated hardware-based systems that perform the specified functions or actions. implementation, or may be implemented in a combination of special purpose hardware and computer instructions.

The computer program product can be implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Industrial Applicability Embodiments of the present invention relate to a network training method, a target detection method, an electronic device, and a computer-readable storage medium. The network training method includes: inputting unlabeled sample images into a target detection network for processing to obtain a target detection result, the result including the image area, feature information and classification probability of the target; determining the target according to the classification probability of the target For the first target whose category confidence is greater than or equal to the threshold, the sample image where the first target is located is taken as the marked image and added to the training set; for the second target whose category confidence is less than the first threshold, The feature correlation mining is performed on the second target, the fourth target is determined from the second target, and the sample image where it is located is added to the training set; the target detection network is trained according to the sample image in the training set. The embodiment of the present invention can improve the training effect of the target detection network.

20: First sample image 21: Target detection results 211: CNN backbone network 212: Feature Map Pyramid Network 213: Fully Connected Network 214: Feature Map 22: Second sample image 23: The sample image where the fifth target is located 24: Fourth sample image 25: Training set 26: Resampled training set 31: Target detection part 32: Confidence determination part 33: Labeling part 34: Feature Mining Section 35: Training Section 800: Electronics 802: Process component 804: memory 806: Power Components 808: Multimedia Components 810: Audio Components 812: Input/Output Interface 814: Sensor Assembly 816: Communication Components 820: Processor 1900: Electronic equipment 1922: Processing components 1926: Power Components 1932: Memory 1950: Web Interface 1958: Input and output interface S11~S15: Steps

The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, and these drawings illustrate embodiments consistent with the present invention, and together with the description, serve to explain the technical solutions of the embodiments of the present invention. FIG. 1 shows a flowchart of a network training method according to an embodiment of the present invention. FIG. 2 shows a schematic diagram of a processing procedure of a network training method according to an embodiment of the present invention. FIG. 3 shows a block diagram of a network training apparatus according to an embodiment of the present invention. FIG. 4 shows a block diagram of an electronic device according to an embodiment of the present invention. FIG. 5 shows a block diagram of an electronic device according to an embodiment of the present invention.

S11~S15: Steps

Claims (19)

A network training method, comprising: Input the unlabeled first sample image into the target detection network for processing, and obtain the target detection result of the first sample image, and the target detection result includes the image of the target in the first sample image. like regions, feature information, and classification probabilities; According to the classification probability of the target, determine the category confidence of the target; For the first target whose category confidence is greater than or equal to the first threshold in the target, the first sample image where the first target is located is taken as the marked second sample image, and added to the training set, wherein, The labeling information of the second sample image includes an image area of the first target and a class corresponding to the class confidence of the first target, and the training set includes the labelled third sample image; For the second target whose category confidence in the target is less than the first threshold, according to the feature information of the third target in the third sample image, feature correlation mining is performed on the second target. Digging, determining the fourth target and the first sample image where the fourth target is located from the second target, and using the first sample image where the fourth target is located as the fourth sample image , and join the training set; The target detection network is trained according to the annotation information of the fourth sample image, the second sample image, the third sample image and the fourth sample image in the training set. The method according to claim 1, wherein, according to the annotation information of the fourth sample image, the second sample image, the third sample image and the fourth sample image in the training set, Train the target detection network, including: According to the category of the target in the positive sample images of the training set, respectively determine the first number of samples sampled from the positive sample images of each category, and the positive sample images are sample images including the target in the image; Sampling the positive sample images of each category according to the first quantity sampled in the positive sample images of each category to obtain a plurality of fifth sample images; Sampling the negative sample images of the training set to obtain a plurality of sixth sample images, where the negative sample images are sample images that do not include the target in the image; The target detection network is trained according to the fifth sample image and the sixth sample image. The method according to claim 1 or 2, wherein the feature correlation mining is performed on the second target according to the feature information of the third target in the third sample image, and the feature correlation mining is performed from the Determine the fourth target and the first sample image where the fourth target is located in the second target, including: determining the information entropy of the second target according to the classification probability of the second target; selecting a fifth target from the second targets according to the category confidence and information entropy of the second target; According to the category of the third target in the third sample image and the total number of sample images to be mined, respectively determine the second number of sample images to be mined in each category; According to the feature information of the third target in the third sample image, the feature information of the fifth target and the second number of sample images to be mined in each category, a fourth target is determined from the fifth target The target and the first sample image where the fourth target is located. The method according to claim 3, wherein selecting a fifth target from the second target according to the category confidence and information entropy of the second target, comprising: According to the category confidence and information entropy of the second targets, the second targets are sorted respectively, and the third number of sixth targets and the fourth number of seventh targets are selected; The sixth target and the seventh target are combined to obtain the fifth target. The method according to claim 3, wherein according to the category of the third target in the third sample image and the total number of sample images to be mined, the second category of the sample images to be mined in each category is determined respectively. quantity, including: According to the category of the third object in the third sample image, determine the proportion of the third object of each category; According to the proportion of the third target of each category, determine the sampling proportion of each category; According to the sampling proportion of each category, the second quantity of sample images to be mined in each category is determined respectively. The method according to claim 3, wherein, according to the feature information of the third target in the third sample image, the feature information of the fifth target and the second number of sample images to be mined in each category, The fourth target and the first sample image where the fourth target is located are determined from the fifth target, including: According to the distance between the feature information of the third target of the first category and the feature information of each fifth target, the third target with the smallest distance from each fifth target among the third targets of the first category is respectively determined, and used as The eighth target, the first category is any one of the categories of the third target; The target with the largest distance among the eighth targets is determined as the fourth target. The method according to claim 6, wherein, according to the feature information of the third target in the third sample image, the feature information of the fifth target and the second number of sample images to be mined in each category, Determining a fourth target and a first sample image where the fourth target is located from the fifth target, further comprising: The determined fourth object is added to the third object of the first category, and the determined fourth object is removed from the unlabeled fifth object. The method according to claim 1 or 2, further comprising: The third sample image is input into the target detection network for processing to obtain feature information of the third target in the third sample image. The method according to claim 1 or 2, wherein before the step of inputting the unlabeled first sample image into a target detection network for processing to obtain the target detection result of the first sample image , the method also includes: The target detection network is pre-trained through the labeled third sample image. The method of claim 1 or 2, wherein the first sample image comprises a long-tail image. The method according to claim 2, wherein, before the step of respectively determining the first number of samples sampled from the positive sample images of each category according to the category of the target in the positive sample images of the training set, the The methods described include: Sampling the positive sample images and negative sample images in the training set to obtain the same or similar number of positive sample images and negative sample images. The method according to claim 3, wherein the total number of sample images to be mined is 5% to 25% of the total number of the first sample images. The method according to claim 4, wherein the combining the sixth target and the seventh target to obtain the fifth target includes: Remove the same target as the seventh target in the sixth target, and obtain the remaining target that is different from the seventh target in the sixth target; The remaining target and the seventh target are taken as the fifth target. The method according to claim 6, wherein according to the distance between the characteristic information of the third target of the first category and the characteristic information of each fifth target, the third targets of the first category are respectively determined After the third target with the smallest distance from each fifth target and serving as the eighth target, the method further includes: When the number of the first sample images where the fourth target is located reaches the second number of the sample images to be mined of the first category, the determination of the eighth target is ended. The method according to claim 7, wherein according to the distance between the characteristic information of the third target of the first category and the characteristic information of each fifth target, respectively determine the third targets of the first category After the third target with the smallest distance from each fifth target and serving as the eighth target, the method further includes: When the number of the first sample images where the fourth target is located does not reach the second number of sample images to be mined in the first category, and the set storing the feature information of the fifth target is empty , ending the determination of the eighth target. The method according to claim 8, wherein inputting the third sample image into the target detection network for processing to obtain feature information of the third target in the third sample image includes: Inputting the third sample image into the target detection network to obtain the feature vector output by the hidden layer of the target detection network; The feature vector is determined as feature information of the third target. A target detection method, the method comprising: Input the to-be-processed image into a target detection network for processing, and obtain a target detection result of the to-be-processed image, where the target detection result includes the position and category of the target in the to-be-processed image; The target detection network is trained according to the network training method of any one of request items 1 to 10. An electronic device comprising: processor; memory configured to store processor-executable instructions; Wherein, the processor is configured to call the instructions stored in the memory to execute the network training method described in any one of request items 1 to 16, or to execute the target detection method described in request item 17. A computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, implement the method network training method described in any one of claim items 1 to 16, or implement the method described in claim item 17. The target detection method described above.

Images