CN115439688A - Weak supervision object detection method based on surrounding area perception and association - Google Patents

Abstract

A weak supervision object detection method based on peripheral area perception and correlation relates to the technical field of object detection, and aims at solving the problems that in the prior art, weak supervision object detection is easy to converge on a local optimal solution, and the problem that in the weak supervision object detection method, the detection accuracy is low and converges on the local optimal solution due to the fact that only the most discriminative area of an object can be detected instead of all object areas, and the object positioning failure is caused, so that the detection accuracy is low. The invention belongs to basic technical research work of object detection in practical application scenes, promotes the landing of an object detection technology of artificial intelligence deep learning to a certain extent, and makes up the difference between weak supervision and full supervision object detection.

Description

A weakly supervised object detection method based on surrounding area perception and association

technical field

The invention relates to the technical field of object detection, in particular to a weakly supervised object detection method based on surrounding area perception and association.

Background technique

Weakly-supervised object detection is a technique to achieve object detection using only image-level labels, where the image-level labels indicate whether an object exists in an image or not. In the application of real scenes, fully supervised object detection cannot obtain instance-level annotations during the training process, while weakly supervised object detection technology uses image-level labels instead of instance-level labels of fully supervised object detection, which can greatly reduce the cost of fully supervised object detection. The need for instance-level labeled training data enables object detection under the premise of scarce labeled data. However, compared with fully supervised object detection, weakly supervised object detection rarely has modules designed to accurately locate object regions (full supervised detection has candidate area networks, feature pyramid networks, etc.). At the same time, the weakly supervised object detection task is usually regarded as a classification task of candidate regions, which in this case causes the weakly supervised object detector to converge to a local optimal solution, and the output result is the most discriminative region of the object. Based on the above, weakly supervised object detection is a challenging and promising technique.

At present, in order to bridge the gap between weak supervision and full supervision object detection and improve the phenomenon of local focus, the weak supervision detection method can be summarized as the following four representative methods. Based on the method of initializing high-quality candidate regions to ensure the recall rate index of object detection tasks, the class activation map and selective search algorithm are combined to generate high-quality candidate regions. The generated high-quality candidate regions are used as the input of the weakly supervised detector to ensure a high recall rate while improving the intersection ratio between the candidate region and the real object bounding box to achieve accurate detection results; the method based on iterative refinement strategy guides the detection The machine tends to the complete object area, and the high overlapping area should have the same category label as the prior knowledge of the training process, which provides supervision information for the next branch; the conversion method based on weak supervision and full supervision combines weak supervision (label information is easy to obtain) ) and the advantages of full supervision (strong regression ability), use the results of the output of the weakly supervised object detector to train the fully supervised detector, and the output of the fully supervised detector is used as the final detection result; based on the method of complete object search, using the class The activation map is used as the location prior of the object area, and the maximum score of the detection area and the minimum score of the surrounding area are searched to further locate the complete object area. However, the above-mentioned high-quality region generation methods, iterative refinement methods, fully supervised and weakly supervised transformation methods, and complete object search methods cannot fundamentally solve the phenomenon of local focus, and these methods do not have wide applicability and are only applicable to current One or a class of weakly supervised detection methods. Based on the above, the limitations of existing weakly supervised object detection methods can be summarized in two aspects: (1) Weakly supervised object detection tends to converge to a local optimal solution, and the intuitive performance can only detect objects with the most discriminative power area, not all object areas, resulting in object localization failure; (2) Fully supervised object detection In order to improve localization accuracy, well-designed modules can be integrated into any fully supervised object detector, such as: candidate region generation network, feature pyramid network. Compared with weakly supervised object detection, there are few general-purpose modules designed to improve positioning accuracy, and existing methods do not have wide applicability, and are only suitable for one or a class of weakly supervised detection methods.

Contents of the invention

The purpose of the present invention is: for weakly supervised object detection in the prior art, it is easy to converge to a local optimal solution, intuitively, it can only detect the most discriminative area of the object, not all object areas, resulting in failure of object positioning, and then To solve the problem of low detection accuracy, a weakly supervised object detection method based on surrounding area perception and association is proposed.

The technical scheme that the present invention takes in order to solve the problems of the technologies described above is:

A weakly supervised object detection method based on surrounding area perception and association, comprising the following steps:

Step 1: Obtain the image to be recognized, and use the weak supervision detector to predict the image to be recognized, and use the predicted object position as the most discriminative area;

Step 2: Expand the most discriminative area, and use the image block to crop the expanded area, and finally use the image block as the surrounding area;

Step 3: Perform feature extraction on the most discriminative region and surrounding regions, and cluster the obtained features, assign a cluster label to each region, and divide each region into different clusters through the cluster label;

Step 4: Obtain the surrounding area with the same label as the most discriminative area through the clustering label of each area, and fuse the surrounding area with the same label as the most discriminative area and the most discriminative area into a new object area;

Step 5: Carry out data amplification on the most discriminative region, and obtain two amplified most discriminative regions, namely q’ and q”;

Step 6: Extract features for q', q" and the surrounding area, and cluster the extracted features. If q' and q" are assigned to the same cluster during the clustering process, the clustering process is regarded as is the correct clustering, and execute step 7. If q' and q" are not assigned to the same cluster during the clustering process, then re-execute steps 3 to 6. If q' and q" are assigned this time into the same cluster, regard this clustering process as the correct clustering, and perform step 7, if q' and q" still cannot be assigned to the same cluster, then ignore the clustering result; then the last The discriminative region serves as the final object region;

Step 7: For the clustering process of assigning q' and q" to the same cluster, calculate the distance d ₁ and d ₂ between q' and q" to the center of the current cluster, if the distance difference |d ₁ -d ₂ | exceeds If the set threshold T _dis =0.1, the clustering result is ignored, otherwise, this clustering process is regarded as a correct clustering;

Step 8: Based on the correct clustering in step 7, obtain the surrounding area in the cluster containing both q' and q", and calculate the cosine similarity between the surrounding area and q' or q", if the cosine similarity value is greater than the threshold T _score = 0.95, then the most discriminative region in the cluster is fused with the surrounding region into the final object region;

Step 9: Use the most discriminative area and the surrounding area as input, and the final object area as output to train the neural network, and use the trained neural network for object detection.

Further, the extended range ratio α in the step 2 is greater than 1 times.

Further, the extended range ratio α in the second step is 1.2 times.

Further, the features are high-dimensional nonlinear features.

Further, the high-dimensional nonlinear features are extracted through ViT.

Further, the size of the image block is 32*32.

Further, the surrounding area in step 3 is 60% of the surrounding area in step 2.

Further, the data augmentation includes random color dithering, random grayscale, random Gaussian blur and random solarization.

Further, the neural network is a MoCov3 network.

Further, the specific steps of the training neural network are:

The training process is carried out by unsupervised contrastive learning, and a total of 100 epochs are trained;

(1) When rounds 0-29, the network input is the most discriminative area;

(2) When rounds 30, 35, 40...100, the network performs the fusion process, and at the same time, the final object area after fusion is used as the input of the network for training;

(3) When rounds 31-34, 36-39...96-99, the input to the network is the fused final object region and the unfused most discriminative region.

The beneficial effects of the present invention are:

This application solves the problems of low detection accuracy and convergence to local optimal solutions in the weak supervision object detection method, breaks through the limitation of weak supervision that there is no module for improving positioning accuracy, and reduces the need for expensive manual labeling in object detection technology. The present invention belongs to the basic technical research work of object detection in practical application scenarios, which promotes the implementation of artificial intelligence deep learning object detection technology to a certain extent, and bridges the gap between weak supervision and full supervision object detection.

Description of drawings

Figure 1 is an example diagram of the surrounding area perception and association module;

Fig. 2 is a structural diagram of a region feature extractor;

Fig. 3 is a regional association network structure diagram;

Figure 4 is a schematic diagram 1 of region fusion constraints;

Figure 5 is a schematic diagram 2 of region fusion constraints;

Figure 6 is a comparison chart a of weakly supervised object detection effect;

Figure 7 is a comparison chart b of weakly supervised object detection effects;

Figure 8 is a comparison chart c of weakly supervised object detection effects.

detailed description

It should be noted that, in the case of no conflict, various implementations disclosed in this application can be combined with each other.

Specific Embodiment 1: This embodiment is specifically described with reference to FIG. 1. A weakly supervised object detection method based on surrounding area perception and association described in this embodiment includes the following steps:

Step 1: Obtain the image to be recognized, and use the weak supervision detector to predict the image to be recognized, and use the predicted object position as the most discriminative area;

Step 2: Expand the most discriminative area, and use the image block to crop the expanded area, and finally use the image block as the surrounding area;

Step 3: Perform feature extraction on the most discriminative region and surrounding regions, and cluster the obtained features, assign a cluster label to each region, and divide each region into different clusters through the cluster label;

Step 4: Obtain the surrounding area with the same label as the most discriminative area through the clustering label of each area, and fuse the surrounding area with the same label as the most discriminative area and the most discriminative area into a new object area;

Step 5: Carry out data amplification on the most discriminative region, and obtain two amplified most discriminative regions, namely q’ and q”;

Step 6: Extract features for q', q" and the surrounding area, and cluster the extracted features. If q' and q" are assigned to the same cluster during the clustering process, the clustering process is regarded as is the correct clustering, and execute step 7. If q' and q" are not assigned to the same cluster during the clustering process, then re-execute steps 3 to 6. If q' and q" are assigned this time into the same cluster, regard this clustering process as the correct clustering, and perform step 7, if q' and q" still cannot be assigned to the same cluster, then ignore the clustering result; then the last The discriminative region serves as the final object region;

Step 7: For the clustering process of assigning q' and q" to the same cluster, calculate the distance d ₁ and d ₂ between q' and q" to the center of the current cluster, if the distance difference |d ₁ -d ₂ | exceeds If the set threshold T _dis =0.1, the clustering result is ignored, otherwise, this clustering process is regarded as a correct clustering;

Step 8: Based on the correct clustering in step 7, obtain the surrounding area in the cluster containing both q' and q", and calculate the cosine similarity between the surrounding area and q' or q", if the cosine similarity value is greater than the threshold T _score = 0.95, then the most discriminative region in the cluster is fused with the surrounding region into the final object region;

Step 9: Use the most discriminative area and the surrounding area as input, and the final object area as output to train the neural network, and use the trained neural network for object detection.

Aiming at the gap in positioning accuracy between weakly supervised and fully supervised object detectors, this application proposes a module that can be embedded into any weakly supervised detector, and forms an end-to-end learning framework with any weakly supervised detector, solving the problem that the training process converges to the local optimum The problem of optimal solution further shows the problem that weakly supervised object detection usually identifies the most characteristic regions of objects. In order to overcome the shortcomings of current weakly supervised object detectors that only identify the most discriminative regions of objects, the region association network proposed in this application dynamically queries the similarity between the surrounding regions and the object regions predicted by existing detectors based on the clustering method. According to the similarity result, the high similarity area is fused, so that the object area output by the weakly supervised object detector covers the complete area of the object. At the same time, the clustering process is introduced in the regional association network, which inevitably introduces the disadvantages of the clustering process, that is, the clustering process in the early training stage has instability and misclassification of regions with low confidence. The region fusion constraints proposed in this application can enhance the conditions of region fusion, hinder the wrong fusion process, further refine the rough object regions output from the region association network, and output accurate and complete detection results.

Specifically, an example of the surrounding area and association module proposed in this application is shown in Figure 1, which contains three components, namely, an area extractor, an area association network, and an area fusion constraint. The first component area extractor, for any weakly supervised detector, divides the most discriminative area and the surrounding area according to the detection results in an image, and uses different cropping ranges to define the surrounding area, improving the detection accuracy. At the same time, it reduces the execution time of the algorithm. The second component of the regional association network, the main function is to continuously perform comparative learning and clustering processes through the two types of regions output by the region extractor, extract good visual representations of the two types of regions, and input the obtained visual representations into the clustering process , specify a cluster label for each region, query the surrounding region with the same most discriminative label through the cluster label, and fuse into a new object region. The third component is the region fusion constraint. The execution region association network will output a new object region. The new object region may introduce unstable and low-confidence surrounding regions in the early stage of clustering. If such surrounding regions are combined with the most discriminative As the final result, region fusion affects the integrity and accuracy of the object. The region fusion constraint of this application is to remove such regions, refine the rough object regions output by the region association network and obtain accurate object regions .

The region extractor proposed in this application first considers the output of existing detectors as the most discriminative region. Then, the most discriminative area is expanded according to a certain ratio α, and the expanded most discriminative area is used as the clipping range. Within the cropping range, 32*32 patches are sequentially cropped as key regions. In the region extractor, each region is regarded as the most discriminative region query regions or surrounding region key regions, and the two types of regions are input into the region association network proposed in this application, and the comparative learning and clustering process is performed to discover the complete object area.

The regional association network proposed in this application combines contrastive learning and clustering processes. For contrastive learning, the region extractor outputs the most discriminative regions as query regions, and the surrounding regions as key regions, and implements RandomColorJitter random color jitter, RandomGrayScale random grayscale, RandomGaussianBlur random Gaussian blur and RandomSolarize random solarization in the query regions. The image enhancement strategy, complex image enhancement strategy helps to learn better feature representation of objects. The enhanced region (q', q") is input to the MoCov3 framework, and the unsupervised training strategy is used to extract the good visual representation of the query regions. By training the ViT feature extractor in the MoCov3 framework, the most discriminative region and surrounding As the input of the regional association network, the region extracts the features of the two types of regions and maps them to the high-dimensional nonlinear space. For the clustering process, the regional association network performs a clustering process on the high-dimensional nonlinear features of the above two types of regions, according to the high-dimensional space. The Euclidean distance assigns a cluster label to each region, and at the same time extracts the surrounding regions that are in the same cluster as the most discriminative region. Such surrounding regions are fused with the most discriminative region to form a new object region, which can completely cover the region of the real object. Part or most. In the regional association network of this application, the unsupervised training process does not require any label information.

For the region fusion constraints proposed in this application, the constraints include category sub-constraints and distance sub-constraints. Among them, the category sub-constraint indicates whether each query region in different viewing angles after image enhancement is assigned to the same cluster, and the distance sub-constraint is to calculate the query regions of different viewing angles in the corresponding cluster centroid under the premise that the category sub-constraint is satisfied The distance difference, the distance sub-constraint considers that the distance difference is less than the preset threshold is a successful distance process, otherwise it is considered that the clustering process fails, that is, the current clustering result is ignored, and the fusion of new object regions is not performed. Specifically, in the regional association network, query regions output q' and q" after image enhancement, and q' and q" are two data-enhanced regions under different viewing angles in the same region, which are input to the cluster together with the surrounding regions in process. During the clustering process, if q' and q" are in the same cluster, the clustering process is regarded as a successful clustering, otherwise, the category subconstraint ignores the clustering result, and in order to find the similarity with the most discriminative region Then, when q' and q" are in the same cluster, the distance sub-constraint calculates the distance between q' and q" to the center of the current cluster. If the distance is greater than the set threshold, Directly ignore the clustering results this time, proving that the original object area is complete and no further fusion is required.Similarly, if the distance is less than the set threshold, the category sub-constraint and the distance sub-constraint are satisfied at the same time, and at the same time The surrounding regions of the two sub-constraints are considered as candidate regions for the region to be fused.

This application takes the VOC2007/2012 dataset as the research object, and users can build corresponding databases according to actual application requirements. In this application, in order to better evaluate the weakly supervised object detection technology, the VOC data set widely used in the field of object detection is used, which contains 20 categories in actual scenes, including 9963 and 22531 image data respectively. VOC images are divided into VOC2007train/val, VOC07/12train/val and VOC2007test. Among them, VOC2007train/val and VOC07/12train/val are used to train the weakly supervised detector framework of this application, and VOC2007test is used to verify the performance of the weakly supervised detector framework of this application, and a wide range of object detection indicators mAP is used to evaluate the detection performance , that is, the intersection and union ratio IOU>0.5 of the detection instance and the true value detection instance is regarded as the correct detection result. After establishing the training database, firstly, according to the output results of the trained weakly supervised detectors, the region perception and association network proposed in this application is trained in an unsupervised end-to-end manner, and the final detection results output improve local focus and obtain complete object area.

In summary, this application proposes a novel weakly supervised object detection framework based on surrounding area perception and association. In the process of realizing weakly supervised object detection, the relationship between the area predicted by the existing detector and the surrounding area is directly considered. , using the similarity relationship between the two types of regions to update the original locally focused region to achieve complete object region detection. In the "Regional Association Network" of this application, when the most discriminative region is fused with the surrounding region to form a new object region, the new object region may be Coarse, that is, including the entire object area or most of the object area, although including the entire object area, may have a poor fit with the real object bounding box. In the "Regional Fusion Constraints" of this application, considering the influence of the early stage of clustering on the fusion process, the regional fusion constraints refine the rough object area, remove the unstable surrounding area, obtain the accurate object area, and use this object area as weak supervision The result of the detector output.

This application solves the local focusing phenomenon of existing weakly supervised object detectors, and greatly bridges the gap between weakly supervised detectors and fully supervised detectors. In the application of real scenes, it promotes the development of deep learning object detectors. It solves the scarcity or unavailability of training labels in practical applications, and further provides technical support for the implementation of artificial intelligence object detection technology.

This application proposes a novel surrounding area perception and association module, which can be integrated into any existing weakly supervised detector as an end-to-end training detection framework, which consists of three components, namely the area extractor, the area association network and Regional Fusion Constraints. Among them, in order to solve the limitation (1), this application directly focuses on the similarity between the surrounding area and the most discriminative area. According to the similarity results of the query, the surrounding area with high similarity will be regarded as the area to be fused, and the most discriminative area. The discriminative regions are fused into new object regions. Through the regional fusion constraints, the low confidence, high noise and unstable regions to be fused in the early training process are removed, the object regions output by the regional association network are refined, and accurate detection results containing complete objects are output. To address limitation (2), this application computes the loss on the most discriminative region predicted by a weakly supervised object detector during training, and our proposed method does not utilize any instance-level labels or image-level Label. Therefore, the surrounding area awareness and association module can be simply integrated with any weakly supervised object detector into an end-to-end unified framework with wide applicability.

Example:

This application uses the VOC2007/2012 dataset, which has extensively evaluated and verified the performance of weakly supervised detectors. Specifically, various types of images in the VOC image dataset are divided into VOC2007train/val, VOC07/12train/val and VOC2007test. Among them, VOC2007train/val and VOC07/12train/val are used to train the weakly supervised detector framework of this application, and VOC2007test is used to verify the performance of the weakly supervised detector framework of this application. After the training database is established, the area perception and association module proposed in this application is trained. First, the region extractor in this application generates the most discriminative region and surrounding regions, and uses two types of regions to train the region association network proposed in this application. According to the unsupervised training process and clustering results in the model, the query and the most discriminative The area is in the surrounding area in the same cluster, and the surrounding area and the most discriminative area are fused as a new object area, and then the area fusion constraint of this application is introduced. This constraint includes a category sub-constraint and a distance sub-constraint. The two sub-constraints shift In addition to the noisy surrounding areas and low confidence surrounding areas contained in the early unstable clustering process, the object area output from the regional association network is refined, and the refined accurate object area is output as the weak supervision detector. result. This application mainly solves the problem that the existing weakly supervised object detectors often locate the most characteristic areas of the object, and obtains the most local solution, which is manifested as local focus. It makes up for the gap between weakly supervised and fully supervised object detectors, and provides weak Module that supervises object detectors to improve localization accuracy and is easy to integrate.

Design region extractor. As shown in Figure 2, the main idea of the region extractor is to extract the object position predicted by the existing weakly supervised detector, and use the predicted object position as query regions. Then, the region extractor expands the object region with a certain range α, and crops the expanded object region with a fixed ratio of 32*32 patches as key regions. The coordinates of the initial object position are (x, y, w, h), and the extended object position is (x, y, αw, αh), where α>1. The query regions in an image are used as a collection, as shown in the formula:

where b _r denotes the object regions predicted by existing weakly supervised detectors in an image, and R denotes the total number of predicted regions in this image. Similarly, key regions are used as a collection, as shown in the formula:

Among them, b _rn represents the nth key regions corresponding to the rth query regions, and N represents the total number of keyregions. In different rn correspondences, the number of N changes, and the area is in the original predicted object area. initial size. It is worth noting that in the process of pruning key regions in this application, 60% of the key regions are used as the input of the regional association network. There are two reasons for the above. The first reason is that 60% of the keyregions area can realize an efficient regional regional association network. There is a clustering process in the regional association network. Reducing the number of regions helps to reduce the number of clustering and further reduce weak The training time of the supervised detection framework as well as the time of the inference process. The second reason is that 60% of the area includes part of the upper area, part of the left area, part of the right area, and part of the lower area of the original prediction area, and the fusion process after clustering can be realized by including the area in 4 directions. The semantic information near the most discriminative region will not be lost. Based on the above, 60% of the key regions are used to ensure that the algorithm detects the enhanced detection performance of the surrounding area and reduces the clustering time.

Design regional association networks. The regional association network proposed in this application combines the feature extraction ability of unsupervised learning and the clustering process. The network structure of the entire algorithm is shown in Figure 3, which is mainly composed of MoCov3 network and clustering network. Among them, the role of the MoCov3 network is the process of extracting query regions features based on the instance discrimination task using contrastive learning. It is worth noting that the training process of this application is divided into three stages. The first stage uses the process of contrastive learning to train the queryregions area. The second stage trains the fused new object area, which contains the semantic information of the surrounding area. The second stage The new object region after three-stage training fusion and the most discriminative region that has not been fused, according to the three-stage training strategy, enhance the feature extraction ability of ViT for query regions and key regions, in order to further query and query regions High Similar key regions and discover complete object regions. Specifically, for each input image, the feature extractor generates query regions and key regions, and uses rich data enhancement methods RandomColorJitter, RandomGrayScale, RandomGaussianBlur and RandomSolarize to obtain query regions (key regions) with different perspectives q' and q", this The application uses vision Transformer (ViT) as the basic encoder and momentum encoder. The basic encoder is used to extract the embedded features of the enhanced area. The output of the momentum encoder and the basic encoder are mutually predicted as a loss function to optimize the entire network. Specifically , the regional association network can be regarded as composed of the regional association algorithm (a) and the regional association algorithm (b). After the basic encoder branch, a mapping network is introduced. The mapping network consists of 3 fully connected layers (Linear layer) and 3 batches The normalization layer (Batch Normalization, BN) and 2 ReLU activation functions, the blue part is the fully connected layer, the green part is the batch normalization layer, and the orange part is the Relu activation function. The batch normalization layer makes the network more It is easy to converge, the model is more stable, and the Relu activation function makes the network input and output have a nonlinear relationship; a prediction network is added after the mapping head, and the composition of the prediction network is similar to that of the mapping network. Compared with the mapping network, the prediction network consists of 2 fully connected layers , 2 batch normalization layers and 1 Relu activation function. Compared with the basic encoder, the momentum encoder does not introduce a prediction network. The prediction vector generated by the prediction network of the basic encoder and the mapping vector generated by the momentum encoder use cross entropy The loss function optimizes the entire network, as shown in the formula:

L _predict ＝ctr(z' _d ,z″ _d )+ctr(z″ _d ,z' _d )

z' _d = pr(g(f _b (B' _d )))

z″ _d = g(f _m (B″ _d ))

Among them, ctr(*) represents a prediction loss function based on contrastive learning MoCov3 from A to B or B to A. z' _d , z" _d represent the high-dimensional nonlinear features extracted by the mapping network and prediction network for query regions or key regions. It is worth noting that the basic encoder branch consists of feature extraction network, mapping network and prediction network, while the momentum encoder The branch only includes the feature extraction network and the mapping network. At the same time, in order to build an encoder with more consistent features, the basic encoder uses a moving average to update the momentum encoder. In the early stage of training, ViT only focuses on the most discriminative area of the object, the most Although the discriminative power area does not contain the complete object area, it covers most of the object area and has rich semantic information of the object. Then, after using query regions to train ViT, the area association network performs the clustering process, and the area association algorithm ( The unsupervised training ViT in a) is used as a feature extractor to extract the embedded features of query regions and key regions, and extract the features of the two types of regions as the input for clustering in high-dimensional nonlinear space. Region association algorithm (b) respectively Perform k-means clustering on the features f _b (B _d ) and f _b (B _s ) of query regions and key regions, set the number of clusters to K, and the number of each cluster center is denoted as c _k , with With continuous dynamic clustering, each region is assigned a cluster label according to the Euclidean distance relationship in the embedding space, and the key regions will become positive key and negative key according to the clustering results, where the positive key represents the most discriminative region Grouped into a class of key regions, such key regions may become part of the object region, if such key regions and the most discriminative region query regions are fused, the original object region will be expanded and a more complete object region will be output. On the contrary , the negative key is used as the background area, or some areas of other instances in the same image. However, the regional association network assigns cluster labels to each area, and the experimental results found that the key is unstable, accompanied by noise, and low confidence Regions are assigned the same clustering labels as query regions, which leads to background key regions or key regions of other instances as positive keys, which inevitably affects the fusion results. In the early training process, the regional association network Only focus on query regions, ignoring the feature extraction capability of ViT on key regions. Therefore, this application introduces region fusion constraints on the basis of region association network.

Design area fusion constraints. The main role of the region fusion constraint is to remove low-confidence key regions, refine the object region output by the region association network and obtain accurate and complete object regions. As shown in Figure 4 and Figure 5, the regional fusion constraints include category sub-constraints and distance sub-constraints. This application considers the clustering labels of q' and q" output by the enhanced query regions and the clustering labels based on the regional association network. The distance relationship of the class center. Specifically, the category sub-constraint and the distance sub-constraint are designed to measure the accuracy of the clustering results. For the most discriminative region query regions, output q' and q" through data enhancement. In the regional association network, q', q" and key regions perform clustering operations after extracting high-dimensional nonlinear features through ViT. If the clustering process divides q' and q" into one cluster, the clustering process is considered correct clustering, otherwise, this application re-executes the region association algorithm to extract the features of the above-mentioned region q', q" and key regions, and re-executes the clustering process to query the accurate positive key as a part of the object region. When the category constraint is satisfied, This application calculates the distances d ₁ and d ₂ from q' and q" to the current cluster center. Similarly, if the distance difference |d ₁ -d ₂ | exceeds the set threshold T _dis =0.1, the distance sub-constraint is not satisfied, and the clustering result of this time will be ignored for the region fusion constraint. When the category sub-constraint and the distance sub-constraint are satisfied at the same time, then, calculate the cosine similarity between the remaining key regions of the above process and q' or q", if the cosine similarity value is greater than the threshold T _score , the remaining key regions will be the same as the query Regions are fused into the final object region and output the result of the weakly supervised detector.Region fusion constraints refine the object region from the region association network, and remove low-confidence key regions by re-executing the clustering process to find accurate and complete The object area, through the calculation of the distance difference, enhances the proximity relationship of the clustering results. The clustering process considers the similarity of the Euclidean distance, and at the same time, the cosine similarity considers the similarity in the direction. The above clustering process and the regional fusion constraints The schematic diagrams, as shown in Fig. 6, Fig. 7 and Fig. 8, include cases where the category sub-constraint is not satisfied, the distance sub-constraint is not satisfied, and the region fusion constraint is satisfied.

Train the weakly supervised object detection framework based on region awareness and association proposed in this application. Design a region extractor in the output of the existing weakly supervised detector, divide the most discriminative region and the surrounding region into two types of regions, design a region association network, and dynamically query the similarity between the surrounding region and the most discriminative region, And form a new object area, introduce area fusion constraints, refine the object area from the area association algorithm, and obtain an accurate and complete object area.

Specifically, in the region extractor, in order to define the clipping range of the surrounding region, α=1.2 is set, the size of the clipping patch is 32*32, and all regions are resized to 224*224 in size; in the region association algorithm, in order to perform aggregation Class process, considering the number of instances in an image is set to K=20, the entire model is iterated for 100 epochs, and the LARS optimizer optimizer is used. Set batch size=64, the initial learning rate is 1.5*10 ^-4 and change the initial learning rate according to the batch size; in the region fusion constraint, set the distance threshold T _dis =0.1 to measure the correctness of the clustering results, In order to express the similarity between query regions and key regions in the direction, set the cosine similarity threshold T _score =0.95, and set the momentum (Momentum) and weight decay to 0.9 and 1*10 ^-6 .

The weakly supervised object detection framework based on region awareness and association trained through the above steps has improved the existing weakly supervised detectors to only output the most discriminative regions of objects, making up for the gap between weakly supervised detectors and fully supervised detectors, and breaking through Weak supervision does not have the limitations of modules that improve localization accuracy, reducing the need for expensive manual annotation by object detection techniques. Experiments have proved that the "Weakly Supervised Object Detection Technology Based on Region Awareness and Association" of this application can detect complete and accurate object regions. Table 1 is the comparison data of the experimental results, and the proposed method is evaluated by using the standard evaluation index mAP in the field of object detection. From the comparative data, it can be seen that the "weakly supervised object detection technology based on area awareness and association" proposed in this application has a mAP improvement of 0.27% compared with the current state-of-the-art image super-resolution method Instance-aware. In addition, the weakly supervised detection framework proposed in this application is a single-stage model. Compared with other latest single-stage weakly supervised object detection methods, both have achieved the highest detection result of 55.17%. Moreover, compared with other multi-stage weakly supervised frameworks that introduce the FasterRCNN detector, it also achieves the highest detection results, which proves the effectiveness of the weakly supervised object detection framework based on region awareness and association proposed in this application. On the basis of VOC07trian/val as the training set, this application introduces an additional data set VOC07/12trian/val for training, and the test result is 58.22%, which still exceeds other methods using additional data sets, thus further proving the strength of this weakly supervised framework robustness and generalization. Figure 7 is a comparison of experimental results, where the green bounding box represents the real area of the object, the red bounding box represents the output result of the existing weakly supervised detector, and the blue bounding box represents the detection result based on the framework output proposed by this application . It can be seen from the figure that compared with other methods, the detection results output by the method proposed in this application contain complete object information, especially for the detection results of non-rigid objects, which improves the detection of only the most discriminative region. problem, achieving accurate and complete object detection results.

Table 1 VOC2007train/val as the quantification test results (mAP) of the training set

Table 2 Quantification test results (mAP) of VOC07/12train/val as a training set

It should be noted that the specific implementation is only an explanation and description of the technical solution of the present invention, and cannot limit the protection scope of rights. All changes made according to the claims and description of the present invention are only partial changes, and should still fall within the protection scope of the present invention.

Claims (10)

1. A weakly supervised object detection method based on surrounding area perception and association, characterized in that it comprises the following steps: Step 1: Obtain the image to be recognized, and use the weak supervision detector to predict the image to be recognized, and use the predicted object position as the most discriminative area; Step 2: Expand the most discriminative area, and use the image block to crop the expanded area, and finally use the image block as the surrounding area; Step 3: Perform feature extraction on the most discriminative region and surrounding regions, and cluster the obtained features, assign a cluster label to each region, and divide each region into different clusters through the cluster label; Step 4: Obtain the surrounding area with the same label as the most discriminative area through the clustering label of each area, and fuse the surrounding area with the same label as the most discriminative area and the most discriminative area into a new object area; Step 5: Carry out data amplification on the most discriminative region, and obtain two amplified most discriminative regions, namely q’ and q”; Step 6: Extract features for q', q" and the surrounding area, and cluster the extracted features. If q' and q" are assigned to the same cluster during the clustering process, the clustering process is regarded as is the correct clustering, and execute step 7. If q' and q" are not assigned to the same cluster during the clustering process, then re-execute steps 3 to 6. If q' and q" are assigned this time into the same cluster, regard this clustering process as the correct clustering, and perform step 7, if q' and q" still cannot be assigned to the same cluster, then ignore the clustering result; then the last The discriminative region serves as the final object region; Step 7: For the clustering process of assigning q' and q" to the same cluster, calculate the distance d ₁ and d ₂ between q' and q" to the center of the current cluster, if the distance difference |d ₁ -d ₂ | exceeds If the set threshold T _dis =0.1, the clustering result is ignored, otherwise, this clustering process is regarded as a correct clustering; Step 8: Based on the correct clustering in step 7, obtain the surrounding area in the cluster containing both q' and q", and calculate the cosine similarity between the surrounding area and q' or q", if the cosine similarity value is greater than the threshold T _score = 0.95, then the most discriminative region in the cluster is fused with the surrounding region into the final object region; Step 9: Use the most discriminative area and the surrounding area as input, and the final object area as output to train the neural network, and use the trained neural network for object detection. 2. A weakly supervised object detection method based on surrounding area perception and association according to claim 1, characterized in that the range ratio α expanded in step 2 is greater than 1 times. 3. A weakly supervised object detection method based on surrounding area perception and association according to claim 2, characterized in that the range ratio α expanded in step 2 is 1.2 times. 4. A weakly supervised object detection method based on surrounding area perception and association according to claim 1, wherein the feature is a high-dimensional nonlinear feature. 5. A weakly supervised object detection method based on surrounding area perception and association according to claim 4, characterized in that the high-dimensional nonlinear features are extracted by ViT. 6. A weakly supervised object detection method based on surrounding area perception and association according to claim 1, characterized in that the size of the image block is 32*32. 7. A weakly supervised object detection method based on surrounding area perception and association according to claim 1, characterized in that the surrounding area in step 3 is 60% of the surrounding area in step 2. 8. A weakly supervised object detection method based on surrounding area perception and association according to claim 1, characterized in that said data amplification includes random color dithering, random grayscale, random Gaussian blur and random solarization. 9. A kind of weakly supervised object detection method based on surrounding area perception and association according to claim 1, characterized in that said neural network is a MoCov3 network. 10. A kind of weakly supervised object detection method based on surrounding area perception and association according to claim 9, characterized in that the specific steps of said training neural network are: The training process is carried out by unsupervised contrastive learning, and a total of 100 epochs are trained; (1) When rounds 0-29, the network input is the most discriminative area; (2) When rounds 30, 35, 40...100, the network performs the fusion process, and at the same time, the final object area after fusion is used as the input of the network for training; (3) When rounds 31-34, 36-39...96-99, the input to the network is the fused final object region and the unfused most discriminative region.

Images