CN116091886A

CN116091886A - Semi-supervised target detection method and system based on teacher student model and strong and weak branches

Info

Publication number: CN116091886A
Application number: CN202211677840.2A
Authority: CN
Inventors: 蔡晓伟; 戚伟
Original assignee: Zhejiang University ZJU
Current assignee: Zhejiang University ZJU
Priority date: 2022-12-26
Filing date: 2022-12-26
Publication date: 2023-05-09

Abstract

The invention discloses a semi-supervised target detection method based on a teacher student model and strong and weak branches, which comprises a teacher model, a student model and a EMA (Exponential Moving Average) parameter updating system; from the internal structure of the teacher student model, each has a strong and weak double-branch structure and CBAM (Convolutional Block Attention Module) for enhancing the feature extraction capability. The invention is based on the classical dual-stage target detection framework FasterRCNN. The invention introduces a strong and weak double-branch structure to measure the quality of the pseudo tag, and trains the true value tag and the noisy pseudo tag more reasonably. For the teacher model, strong branch prediction results of the teacher model are used as pseudo tags, and weak branch prediction results assist in measuring the quality of the pseudo tags and filtering the pseudo tags. For student models, strong and weak branches can be better trained by using pseudo tag data and true tag data. At the same time, the convolution attention module CBAM is introduced to further enhance the feature extraction capability of the double-branch shared part.

Description

Semi-supervised target detection method and system based on teacher student model and strong and weak branches

Technical Field

The invention relates to the field of semi-supervised learning, in particular to a semi-supervised target detection method and system based on a teacher student model and strong and weak branches.

Background

Object detection is a classical task of computer vision, finding object locations and classifying in an image or video. The large-scale data set and massive computing resources enable the deep neural network to achieve powerful performance on a variety of visual tasks. Target detection has also made significant progress with the development of deep convolutional neural networks. However, training an accurate target detector requires a certain scale and labeling of good datasets. The target detection data set consists of accurate class labels and boundary box coordinates, and the construction of the high-quality data set depends on manual labeling, so that resources are seriously consumed. Based on the high cost of tagging data, one natural idea is to utilize rich unlabeled data to facilitate learning. How to obtain target detection performance as considerable as possible by using a small amount of marked data and a large amount of unmarked data is a problem worthy of research.

In recent years, semi-supervised learning (SSL, semi-Supervi sed Learning) has received attention because SSL can utilize unlabeled data to facilitate learning of limited labeled data. Currently, research on semi-supervised learning is mainly focused on image classification problems. The SSL method achieves many results on the task of image classification, but has little research in the field of target detection where accurate bounding box labeling is required. The existing work on object detection is mostly limited by the amount and accuracy of the annotation data, and only based on high quality and sufficient data sets, such as MS-COCO, can stronger and faster object detectors be trained. In view of the requirement of the target detection algorithm for massive high-precision labeling data, a semi-supervised learning algorithm suitable for the field of target detection needs to be researched.

Several achievements exist in the field of semi-supervised target detection. STAC is a self-training method. The method uses a pre-trained model to generate category pseudo labels and frame pseudo labels on unlabeled images, and trains a student model by the pseudo labels, thereby laying a semi-supervised target detection research paradigm. The Instant-training also generates pseudo tags and designs a teacher student model interactive learning algorithm. ISMT considers historical pseudo tags but performs poorly.

The semi-supervised target detection methods do not solve the problem of positioning quality measurement of the pseudo tags and the problem of reasonable training of the pseudo tags. The method is characterized in that the pseudo tag regression frame is selected to be discarded, and only the classification pseudo tag is used for training, or the classification confidence of the pseudo tag regression frame is used as a basis for judging regression quality. Both of the above methods cannot reasonably utilize regression information of the pseudo tag.

Disclosure of Invention

Aiming at the defects in the prior art, the invention aims to provide a semi-supervised target detection method based on a teacher student model and strong and weak branches, which solves the technical problems. In the invention, a STAC research paradigm is extended, a novel semi-supervised target detection algorithm is provided based on a classical dual-stage target detection framework FasterRCNN, pseudo tags are dynamically generated and filtered for a student model through a teacher, then the data characteristics of the pseudo tags are fully extracted by a convolution attention module, and the influence of low-quality pseudo tag noise on classification and regression is reduced by adopting strong and weak branch decoupling.

The technical scheme adopted for solving the technical problems is as follows:

a semi-supervised target detection method based on a teacher student model and strong and weak branches comprises the following steps:

a pre-training stage for training the student model by using the truth data; the student model is provided with a strong branch and a weak branch, the strong branch and the weak branch share the rest of networks except the ROI head, and all supervision data can pass through the strong branch and the weak branch;

copying the pre-trained student model parameters to a teacher model for semi-supervised training:

inputting the data with reinforced weak data into a teacher model, wherein the teacher model also has the same strong and weak branches, respectively reasoning to generate pseudo labels, and carrying out quality division by utilizing the results of the strong and weak branches so as to filter the pseudo labels;

and (3) inputting the truth value tag data and the pseudo tag data into the student model after strong data enhancement, training and updating parameters, and updating the parameters for the teacher model through the EMA.

The invention also provides a semi-supervised target detection system based on the teacher student model and strong and weak branches, which mainly comprises: a data enhancement system consisting of weak data enhancement and strong data enhancement; the network shares part of the channel and the spatial attention mechanism module; a whole system consisting of a teacher model, a student model and EMA; wherein: a pseudo tag generating and filtering system consisting of strong branches and weak branches of the teacher model; a pseudo tag training system consisting of strong branches and weak branches of the student model;

the combined utilization of multiple data enhancements plays a significant point raising role through the principles of consistency regularization. The picture fed into the teacher model is enhanced by weak data, which includes only random horizontal flipping. The data fed into the student model takes the form of strong data enhancements, including single picture enhancements and enhancements between pictures. Single picture data enhancement is handled in only one image, including random horizontal flipping, photometric distortion, random gaussian blurring, and random erasure. The enhancement between pictures fuses two or more images to enhance a single image, including Mixup and Mosaic. The teacher model in the invention is used for dynamically generating the pseudo labels for the unlabeled images and filtering the pseudo labels. The student model uses the pseudo tags for semi-supervised training. The student model updates its own weights by learning, and the teacher updates weights from the student model by EMA. Thus, a positive feedback system is formed, as training is carried out, the teacher model can generate a pseudo tag with higher quality, and the student model can learn further from the pseudo tag with higher quality. In the initial stage, pre-training is carried out through a small amount of true value data, parameters are copied to a teacher model and a student model, and then semi-supervised training is carried out.

Pseudo tag generation and filtering system: the strong and weak double-branch structures of the teacher model are identical, the input feature images are identical, the strong branch reasoning of the teacher model generates candidate pseudo tag frames and confidence scores, and the weak branch reasoning of the teacher model generates the candidate pseudo tag frames and the confidence scores. Both pseudo tag boxes IOU are calculated. In the invention, a pseudo tag frame obtained by strong branch reasoning and confidence score are selected as pseudo tags of semi-supervised training of a student model in the next stage. The confidence score of the strong branch is used as a quality basis for judging and classifying the pseudo tag. The IOU of the two branches is used as the quality basis for judging the pseudo tag regression frame, so that the problem of positioning quality measurement of the pseudo tag regression frame which is not solved in other methods is solved, and the IOU is not solved in the single-branch model structure. Further, the method for filtering the pseudo tag comprises the following steps: filtering the candidate pseudo tag frames for one time by adopting non-maximum suppression, and dividing the pseudo tag into a high-quality classification tag and a low-quality classification tag according to a confidence threshold tau; dividing the pseudo tag into a high-quality regression tag and a low-quality regression tag according to the IOU threshold epsilon; a set of pseudo tags is referred to as a high quality pseudo tag when it is both a high quality classification tag and a high quality regression tag; when a set of pseudo tags is both a low quality classification tag and a low quality regression tag, it is not eligible to participate in the next semi-supervised training; otherwise, it is called a low quality pseudo tag; thereby completing the quality division of the pseudo tag.

In the pseudo tag training system: the reasonable training problem of the pseudo tag and the truth tag is solved by utilizing the double-branch structure of the student model. Specifically, strong and weak branches share the rest of the network except the ROI head: the pseudo tag and the truth data are sampled at a 1:1 ratio for each input, which is essentially an oversampling strategy for the truth data. This approach can solve the problem of false tag and true data sample imbalance to some extent. The strong branches of the student model only train clean data, namely true value tag data and high quality pseudo tags, and the weak branches train clean data and dirty data simultaneously, namely input true value tags and all filtered pseudo tag data (high quality and low quality pseudo tags). Compared with a single-branch structure, the ROI strong and weak double-branch network of the student model performs independent loss calculation on clean data and dirty data, and prevents low-quality pseudo tag noise from directly interfering with the weight of the strong-branch network. Compared with two completely independent model structures, the shared part of the student network can extract common effective information from clean data and dirty data so as to well eliminate supervision inconsistency of the two independent model structures.

Furthermore, to further enhance the feature extraction capability of the shared network portion, we introduce CBAM before the last, ROIHead, of the shared network.

The beneficial effects of the invention are as follows:

the invention introduces a strong and weak double-branch structure to measure the quality of the pseudo tag, and trains the true value tag and the noisy pseudo tag more reasonably; for the teacher model, strong branch prediction results of the teacher model are used as pseudo tags, and weak branch prediction results assist in measuring the quality of the pseudo tags and filtering the pseudo tags. For student models, strong and weak branches can be better trained by using pseudo tag data and true tag data.

Drawings

FIG. 1 is a schematic diagram of the overall structure of the present invention;

in the figure: 1 teacher model, 2 student model, 3 teacher model strong branch, 4 teacher model weak branch, 5 student model strong branch, 6 student model weak branch, 7CBAM,8EMA,9 strong data enhancement, 10 weak data enhancement.

Detailed Description

The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present invention.

From the whole framework, our teacher student model includes two training phases, namely a pre-training phase and a semi-supervised training phase. During the pre-training phase, we train the student model with a small amount of truth data. In the beginning of semi-supervised training, we copy the initialization parameters of the student model to the teacher model. Training a teacher model in the STAC to generate pseudo tag data at one time and not update the generated pseudo tags results in the student model training being based on poor quality and non-updated pseudo tags, limiting its performance. In our method, the teacher model dynamically generates pseudo tags for the student models, and the student models update parameters for the teacher model through EMA. Along with the training, the parameters of the teacher model are continuously updated, and then a pseudo tag with higher quality is generated, so that the student model can obtain better supervision data.

Pre-training stage: a small amount of true value data is trained in a full supervision mode, random initialization is adopted for strong and weak branches, and all supervision data can pass through the strong and weak branches. To this end, we first use the available supervision data to optimize our model. The supervised penalty of the pre-training phase includes six penalties: RPN classification loss, RPN regression loss, ROI strong branch classification loss, ROI strong branch regression loss, ROI weak branch classification loss, and ROI weak branch regression loss.

Semi-supervised training phase: after the pre-training phase, semi-supervised training begins.

The first stage of semi-supervised training is that the teacher model generates pseudo labels for unlabeled picture reasoning after weak data enhancement and filters the pseudo labels. Specifically, first, strong branch reasoning of the teacher model generates candidate pseudo tag boxes and confidence scores, and weak branch reasoning thereof generates pseudo tag boxes and confidence scores. And calculating the intersection ratio of the strong branch pseudo tag frame and the weak branch pseudo tag frame. In the face of the reasoning results of the strong branch and the weak branch, a pseudo tag frame obtained by strong branch reasoning and confidence score are selected as pseudo tags of semi-supervised training of a student model in the next stage. And the quality basis of the classified pseudo tag is used for judging. And the quality basis of the pseudo tag regression frame is used for judging.

We filter the pseudo tags using the following method: to avoid duplication of frames, filtering candidate pseudo tag frames once by adopting non-maximum value suppression (NMS, non-maximum suppression), and dividing the pseudo tags into high-quality classification tags and low-quality classification tags according to a confidence threshold tau; the pseudo tags are classified into high quality regression tags and low quality regression tags according to the IOU threshold ε. We refer to a high quality pseudo tag when a set of pseudo tags are both high quality classification tags and high quality regression tags. When a set of pseudo tags is both a low quality classification tag and a low quality regression tag, it is not eligible to participate in the next semi-supervised training. In the rest of the cases we refer to as low quality pseudo tags. Thus, the generation and quality division of the pseudo tag are completed.

The second stage of semi-supervised training is that the student model is trained with clean and dirty data. And (3) carrying out strong data enhancement on the true label data and the false label data after carrying out balanced sampling, and then sending the data enhancement to a student model for training. For the network sharing part: the RPN network of the student model generates a suggestion box for the image and then extracts ROI features from the suggestion box using ROIAlign. These ROI features are further characterized by CBAM. For the dual-branch structure of the student model, the features obtained in the last step are sent to the strong branch and weak branch ROI network, clean data (true value data and high quality pseudo tag) are sent only to the strong branch calculation loss, and clean data and dirty data (low quality pseudo tag) are sent to the weak branch calculation loss. The total ROI loss in the semi-supervised training process comprises classification and regression loss of the ROI strong branch, classification loss of the ROI weak branch and regression loss of the ROI weak branch.

The above is a complete training process, and in the reasoning process we only use the strong branch 3 of the teacher model.

The present invention follows the STAC using Faster RCNN with a feature pyramid network, employing an ImageNet pre-training initialized ResNet-50 as the backbone network. We use the confidence threshold τ=0.7 and the IOU threshold ε=0.6 as the basis for pseudo tag filtering and reasoning on the strong branches of the teacher model. The batch size was 32, with 16 marked images and 16 unmarked images, and the learning rate was set to 0.02.EMA coefficient is set to α=0.0005.

Table 1 is a table of the present invention (outer) and other prior art methods to evaluate IoU =0.5 on MS-COCO2017val at different data rates: mAP results of 0.95. The different ratios represent marked data and the remainder represent unmarked data, with COCO-full representing the total MS-COCO train2017 as marked data and MS-COCO additional as unmarked data. The errors in the table are from multiple sets of experimental results for different random seeds. Experiments at MS-COCO settings are shown in Table 1. It can be seen from the table that the method of the present invention increased about 10 mAPs after semi-supervised training compared to the supervised baseline at a 1%/2%/5%/10% label data ratio. We tested the best results with tag data at a 2%/10% ratio and COCO-full. In summary, experimental results on MS-COCO verify that our method is effective and that the superiority is significant when the tag data is relatively small.

TABLE 1

The foregoing describes specific embodiments of the present invention. It is to be understood that the invention is not limited to the particular embodiments described above, and that various changes or modifications may be made by those skilled in the art within the scope of the appended claims without affecting the spirit of the invention. The embodiments of the present application and features in the embodiments may be combined with each other arbitrarily without conflict.

Claims

1. A semi-supervised target detection method based on a teacher student model and strong and weak branches is characterized by comprising the following steps:

2. The semi-supervised target detection method based on a teacher student model and strong and weak branches according to claim 1, wherein strong branch reasoning of the teacher model generates candidate pseudo tag frames and confidence scores, weak branch reasoning thereof generates pseudo tag frames and confidence scores, and the pseudo tag frames IOU of the two are calculated; selecting a pseudo tag frame obtained by strong branch reasoning and a confidence score as a pseudo tag of semi-supervised training of a student model in the next stage; and taking the confidence scores of the strong branches as the quality basis of the judgment classification pseudo tag, and taking the IOUs of the two branches as the quality basis of the judgment pseudo tag regression frame.

3. The semi-supervised target detection method based on teacher student models and strong and weak branches as claimed in claim 2, wherein the method for filtering pseudo tags is as follows: filtering the candidate pseudo tag frames for one time by adopting non-maximum suppression, and dividing the pseudo tag into a high-quality classification tag and a low-quality classification tag according to a confidence threshold tau; dividing the pseudo tag into a high-quality regression tag and a low-quality regression tag according to the IOU threshold epsilon; a set of pseudo tags is referred to as a high quality pseudo tag when it is both a high quality classification tag and a high quality regression tag; when a set of pseudo tags is both a low quality classification tag and a low quality regression tag, it is not eligible to participate in the next semi-supervised training; otherwise, it is called a low quality pseudo tag; thereby completing the quality division of the pseudo tag.

4. The method for detecting the semi-supervised target based on the teacher student model and the strong and weak branches according to claim 1, wherein when the student model is trained, the truth label data and the quality-divided high-quality pseudo labels are sent to the strong branch calculation loss of the student model, and the truth label data and all the filtered pseudo labels are sent to the weak branch calculation loss of the student model, so that the influence of noise in dirty data on the strong branch weight of the teacher model is avoided.

5. The method for semi-supervised object detection based on teacher student models and strong and weak branches according to claim 1, wherein in the teacher student models, a convolution attention module CBAM is introduced before the last ROI head of the shared network to enhance the feature extraction capability of the shared network part.

6. The method for detecting a semi-supervised target based on a teacher student model and strong and weak branches according to claim 1, wherein the weak data is enhanced to random horizontal flip; the strong data enhancement comprises the enhancement of a single picture and the enhancement among a plurality of pictures, the single picture data enhancement is only processed in one image, the single picture data enhancement comprises random horizontal overturn, photometric distortion, random Gaussian blur and random erasure, and the enhancement among the pictures fuses two or more images to enhance the single image, and the single image comprises Mixup and Mosaic.

7. A semi-supervised target detection system based on a teacher student model and strong and weak branches, which is characterized in that the system is used for realizing the method of any one of claims 1-6, and comprises a pseudo tag generation and filtering system, a pseudo tag training system and a data enhancement system; the method comprises the steps of carrying out a first treatment on the surface of the

The pseudo tag generation and filtering system consists of strong branches and weak branches of a teacher model; the method comprises the steps of dynamically generating and filtering pseudo tags for a student model;

the pseudo tag training system consists of strong branches and weak branches of a student model; the system is used for learning from the pseudo tag and the truth tag to update the parameters and updating the parameters for the teacher model through the EMA;

the data enhancement system is used for enhancing data, carrying out weak data enhancement on data input into the teacher model and carrying out strong data enhancement on data input into the student model.