CN115641316A - Weak supervision change detection method and device based on background mixed data expansion technology - Google Patents
Weak supervision change detection method and device based on background mixed data expansion technology Download PDFInfo
- Publication number
- CN115641316A CN115641316A CN202211332063.8A CN202211332063A CN115641316A CN 115641316 A CN115641316 A CN 115641316A CN 202211332063 A CN202211332063 A CN 202211332063A CN 115641316 A CN115641316 A CN 115641316A
- Authority
- CN
- China
- Prior art keywords
- background
- image pair
- change detection
- enhancement
- pair
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a method and a device for detecting weak supervision change based on a background mixed data expansion technology, wherein the method comprises the following steps: constructing a weak supervision training set, a background guide set and an augmentation data set; enhancing an input image pair with a mask, background, through a background-aware enhancement operation to obtain a new image pair
Obtaining the final output after expansion through an enhancement strategy; continuously updating the change detection model parameters by using a learning algorithm to finally obtain an optimal model; and finally, testing by using the trained model. The device comprises: a processor and a memory. The present invention uses a background guide set to enrich potential background variations in training examples; through back grAn out-aware enhancement operation to help the change detection model see different background changes; generalization capability is enhanced by a consistency loss function.
Description
Technical Field
The invention relates to the field of data expansion, in particular to a method and a device for detecting weak supervision change based on a background mixed data expansion technology.
Background
Change detection (Change detection) is a technique capable of detecting a Change area in a pair of two-time images. Early change detection techniques were based on algebra, such as: gradient (gradient), change Vector Analysis (CVA), etc. These conventional change detection models cannot deal with noise of background change, and some complex models are gradually proposed. For example: models such as CNN (convolutional neural network), GAN (generative countermeasure network), and the like are also increasingly applied to change detection. In recent studies, transformations have also been used for change detection. The change detection method mentioned in document [1] combines a hierarchically structured transform encoder and a multilayer perceptual (MLP) decoder in a twin network structure, effectively presenting the multi-scale remote details required for accurate CDs. These methods have achieved good results, but since the change detection data set does not contain all the environmental transformations, a data extension suitable for change detection is necessary.
Data Augmentation (Data Augmentation) is a method that can effectively improve the generalization capability of depth models. At present, the data expansion technology is mainly applied to classification and detection tasks. The CutOut data expansion method described in document [2] cuts out a rectangular area from an input image, and then fills pixels of the area with 0; the random erasure algorithm introduced in the document [3] randomly selects a rectangular frame for an input image, and then randomly replaces the pixel value of the rectangular frame, thereby achieving the effect of data expansion; the CutMix method introduced in the document [4] cuts off a part of the region of the image, and then fills the cut-off region with the pixel values of other data in a training set so as to improve the robustness of the model; the Mixup image mixing enhancement method introduced in the document [5] adds two sample data according to a certain proportion to obtain new sample data. By correctly adopting some data expansion methods, the problem of insufficient data quantity can be solved, and the generalization capability and robustness of the model can be rapidly improved.
The data expansion methods described above are only suitable for classification or detection tasks, but not for change detection. It is therefore necessary to provide a method for change detection.
Reference to the literature
[1]Bandara,W.G.C.；and Patel,V.M.2022.A Transformer-Based Siamese Network for Change Detection.arXivpreprint arXiv:2201.01293.
[2]DeVries,T.；and Taylor,G.W.2017.Improved regularization of convolutional neural networks with cutout.arXiv preprint arXiv:1708.04552.
[3]Zhong Z,Zheng L,Kang G,et al.Random Erasing Data Augmentation[C]//AAAI.2020:13001-13008.
[4]Yun,S.；Han,D.；Oh,S.J.；Chun,S.；Choe,J.；and Yoo,Y.2019.Cutmix:Regularization strategy to train strong classifiers with localizable features.In Proceedings of the IEEE/CVF international conference on computer vision,6023–6032.
[5]Zhang H,Cisse M,Dauphin Y N,et al.mixup:Beyond Empirical RiskMinimization[C]//International Conference on Learning Representations.2018.
Disclosure of Invention
The invention provides a method and a device for detecting weak supervision change based on background mixed Data expansion technology (BGMix), provides a new Consistency Loss function (Augmented and Real Data Consistency Loss), and constructs a background bootstrap set; training a depth model for change detection in a weakly supervised manner using background mixing techniques; enriching potential context changes in training examples using a context bootstrap set; the background-aware enhancement operation helps the change detection model to see different background changes; the generalization ability is enhanced by a consistency loss function, described in detail below:
a method for detecting weakly supervised change based on background mixed data extension technology, the method comprising:
constructing a weak supervision training set, a background guide set and an augmentation data set;
enhancing an input image pair with a mask, background, through a background-aware enhancement operation to obtain a new image pair
Obtaining the final output after expansion through an enhancement strategy;
continuously updating the change detection model parameters by using a learning algorithm to finally obtain an optimal model; and finally, testing by using the trained model.
Wherein the enhancing the input image pair by the background perception enhancing operation uses a mask, a background to obtain a new image pair
The method specifically comprises the following steps:
randomly sampling a background pair < B from the background boot set 1 ,B 2 Using augmentation operations in augmented datasets to pair input images < I 1 ,I 2 The treatment is carried out.
Further, the obtaining of the expanded final output through the enhancement strategy specifically includes:
for each augmentation path, sampling three operations from the augmentation dataset and stacking to construct a new operation; randomly sampling an operation from the newly constructed operations, the sampling operation being for the mask C, background B and image pair I 1 ,I 2 Mixing the three to obtain a new image pair;
and repeating the steps, acquiring K new image pairs after K times of enhancement, and mixing the K new image pairs with the randomly sampled weights.
The method comprises the following steps of continuously updating change detection model parameters by using a learning algorithm, and finally obtaining an optimal model specifically:
the final loss function is:
wherein the content of the first and second substances,
wherein [ ·]Is a join operation, psi (-) refers to a pre-trained VGG16 network for perceptual feature extraction, cos (-) is a trigonometric function,
to an enhanced image pair;
a weakly supervised change detection apparatus based on background mixed data extension techniques, the apparatus comprising: a processor and a memory, the memory having stored therein program instructions, the processor calling upon the program instructions stored in the memory to cause the apparatus to perform any of the method steps described.
The technical scheme provided by the invention has the beneficial effects that:
1. the invention provides a data expansion method, which is used for improving the performance of a change detection model, and a background guide set B is constructed and used for enriching the potential background of an input image;
2. according to the method, background-aware enhanced operation is used to help a change detection model to see different background changes; the invention enables an original image pair to be paired by constructing an augmented set of operations<I 1 ,I 2 >Conversion to a new image pair
3. The invention uses Augmented&Real Data Consistency Loss evaluation of enhanced image pairs with different backgrounds
And original image pair<I 1 ,I 2 >The similarity between the two images is used for reducing errors caused by image background change and enhancing the generalization capability of the change detection model.
Drawings
FIG. 1 is a flow chart of a method for detecting weakly supervised change based on background mixed data expansion technique;
FIG. 2 is a flow chart of a BGMix function proposed by the present invention;
fig. 3 is a schematic structural diagram of BGMix according to the present invention;
FIG. 4 is a diagram illustrating the generation result of the proposed method.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in further detail below.
The embodiment of the invention provides a background mixing enhancement technology specially designed for change detection aiming at the problem of insufficient performance of the change detection of a remote sensing image, and can mix an original input image pair so that a change detection model can well detect a change area.
A background mixing enhancement technique specifically designed for change detection, the technique comprising the steps of:
1. constructing a weak supervision training set T, a background guide set B and an augmentation operation set O
The method comprises the following steps: images with image-level labels are prepared to construct a weakly supervised training set T, for example: AICD (change detection reference in aerial imagery) and BCD (building change detection) datasets; collecting N image pairs to construct a background guide set B, each image pair including only background changes and no target changes; an augmented operation set O is set that includes M enhanced operations, for example: posterize, rotate, equalize, and the like.
2. Performing BGMix enhancement operations
The BGMix enhancement operation is subdivided into two operations: background-aware Augmentation operation and Augmentation strategy.
Background-aware augmentation operation: first, a background pair is randomly sampled from a background boot set<B 1 ,B 2 > (ii). The input image pair is then subjected to an augmentation operation using the augmented dataset<I 1 ,I 2 >And (4) processing.
The operation function is specifically as follows:
wherein o is j For transforming image pairs<I 1 ,I 2 >Operation of α j To change the background operation of the original image pair, C is a change mask,
is passing through o j And (5) obtaining an image pair after operation.
This formula can be further described as:
wherein, the corresponding elements of the two matrixes are multiplied, and Rep is a function for replacing the original image background image.
The operation steps of the Augmentation strand are as follows: k augmented paths are set, for each augmented path, three operations are first sampled from the augmented operation set O and stacked to build a new operation. Then, an operation is randomly sampled from the newly constructed operations, the sampling operation being used to pair the mask C, the background B and the image pair I 1 ,I 2 The three are mixed to obtain a new image pair and achieve the purpose of enhancement. And repeating the steps, acquiring K new image pairs after K times of enhancement, and mixing the K new image pairs with the randomly sampled weights. Finally, the mixed image pair is further mixed with the original input image to obtain the final outputAnd (6) outputting the image.
3. Performing a learning algorithm
From a training data set T train Sampling an image pair and passing phi θ (. The) performs change detection. Then, an enhanced image pair is obtained by the BGMix function. Thereafter, the defined loss function is calculated and the model parameters are updated.
Wherein the loss function comprises 5 parts, the first part being to
And
the perceptual similarity between as a first loss function, which can be described as:
wherein, [ ·]Is a join operation, psi (-) refers to a pre-trained VGG16 network for perceptual feature extraction, cos (-) is a trigonometric function,
as an enhanced image pair.
The second part randomly samples a background pair from the background guide set B<B 1 ,B 2 >. Then use this background to replace < I 1 ,I 2 Background > and a new image pair is obtained, the specific function being:
I′ 1 ＝Rep(I 1 ，B 1 ，C),I′ 2 ＝Rep(I 2 ，B 2 ，C)， (5)
wherein, I' 1 ,I′ 2 For replacing the original image pair < I by Rep function 1 ,I 2 Image pair obtained background.
In addition, a new background pair can be synthesized<I 1 ,I 2 >Background area sticking in (1)Is adhered to<B 1 ,B 2 >To the corresponding position in (a). Then a new background pair can be obtained, the specific function being:
B′ 1 ＝Rep(B 1 ，I 1 ，C),B′ 2 ＝Rep(B 2 ，I 2 ，C)， (6)
wherein, B' 1 ,B′ 2 Is a background pair obtained by a Rep function.
A new loss function is then defined to evaluate<I 1 ,I 2 >And<I′ 1 ,I′ 2 >、<B 1 ,B 2 >and<B′ 1 ,B′ 2 >similarity. The loss function can be described as:
the third part embeds more background context from the background pair, which will facilitate the identification of small area areas. The loss function is described as:
where SSIM (·) is a loss of structural similarity.
The fourth section uses two discriminators as the penalty for the antagonism to make the convolutional neural network phi θ (. Cndot.) produced accurate results of the changes. The loss function is described as the following function:
wherein D is 1 And D 2 Two discriminators.
Finally, to control potential prediction errors due to background variations, a penalty function is set such that φ is used as input when the background pair is taken as input θ (. Cndot.) predict all zero mask. The loss function is described as the following function：
Where the number 0 is a tensor of all 0's. Through all the above loss functions, a final loss function can be obtained, and the specific functions are as follows:
wherein the content of the first and second substances,
is defined as the function:
wherein, { lambda ] i For each of the balanced equations.
4. Testing of networks
After the third step, the trained models can be obtained, and then the images of the test set are read to test the trained change detection models (FCD, AFA, and WCD).
In summary, the embodiment of the present invention uses the BGMix data extension method to enhance the input image pair I by Background-aware augmentation operation using the Background boot set B and the change mask C 1 ,I 2 To obtain a new image pair
Obtaining the final output after expansion through the Augmentation strand; then, continuously updating the parameters of the change detection model by using a learning algorithm to finally obtain an optimal model; and finally, testing by using the trained model.
A weakly supervised change detection apparatus based on background mixed data extension technology, the apparatus comprising: a processor and a memory, the memory having stored therein program instructions, the processor calling the program instructions stored in the memory to cause the apparatus to perform any of the method steps of:
constructing a weak supervision training set, a background guide set and an augmentation data set;
enhancing an input image pair with a mask, background, through a background-aware enhancement operation to obtain a new image pair
Obtaining the final output after expansion through an enhancement strategy;
continuously updating the change detection model parameters by using a learning algorithm to finally obtain an optimal model; and finally, testing by using the trained model.
Wherein an input image pair is enhanced with a mask, background, by a background-aware enhancement operation to obtain a new image pair
The method specifically comprises the following steps:
randomly sampling a background pair from a background boot set<B 1 ,B 2 >Using augmented operation in augmented dataset to pair input images<I 1 ,I 2 >And (6) processing.
Further, obtaining the expanded final output through the enhancement strategy specifically includes:
for each augmentation path, sampling three operations from the augmentation dataset and stacking to construct a new operation; randomly sampling an operation from the newly constructed operations, the sampling operation being for the mask C, background B and image pair I 1 ,I 2 Mixing the three to obtain a new image pair;
and repeating the steps, acquiring K new image pairs after K times of enhancement, and mixing the K new image pairs with the randomly sampled weights.
The method comprises the following steps of continuously updating change detection model parameters by using a learning algorithm, and finally obtaining an optimal model:
the final loss function is：
Wherein, the first and the second end of the pipe are connected with each other,
wherein [ ·]Is a join operation, psi (-) refers to a pre-trained VGG16 network for perceptual feature extraction, cos (-) is a trigonometric function,
to an enhanced image pair;
in the embodiment of the present invention, except for the specific description of the model of each device, the model of other devices is not limited as long as the device can perform the above functions.
Those skilled in the art will appreciate that the drawings are only schematic illustrations of preferred embodiments, and the above-described embodiments of the present invention are merely provided for description and do not represent the merits of the embodiments.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (5)
1. A method for detecting weak supervision change based on background mixed data expansion technology is characterized by comprising the following steps:
constructing a weak supervision training set, a background guide set and an augmentation data set;
enhancing output using mask, background through background awareness enhancement operationsEntering the image pair to obtain a new image pair
Obtaining the final output after expansion through an enhancement strategy;
continuously updating the change detection model parameters by using a learning algorithm to finally obtain an optimal model; and finally, testing by using the trained model.
2. The method of claim 1, wherein the enhancement of the input image pair by the background-aware enhancement operation using the mask and the background to obtain a new image pair
The method specifically comprises the following steps:
randomly sampling a background pair from a background boot set<B 1 ,B 2 >Using augmented operation in augmented dataset to pair input images<I 1 ,I 2 >And (6) processing.
3. The method for detecting weakly supervised change based on background mixed data extension technology as claimed in claim 1, wherein the obtaining of the extended final output through the enhancement strategy is specifically:
for each augmentation path, sampling three operations from the augmentation dataset and stacking to construct a new operation; randomly sampling an operation from the newly constructed operations, the sampling operation being for the mask C, background B and image pair I 1 ,I 2 Mixing the three to obtain a new image pair;
and repeating the steps, acquiring K new image pairs after K times of enhancement, and mixing the K new image pairs with the randomly sampled weights.
4. The method according to claim 1, wherein the parameters of the change detection model are continuously updated by using a learning algorithm, and the finally obtained optimal model specifically comprises:
the final loss function is:
wherein, the first and the second end of the pipe are connected with each other,
wherein [ ·]Is a connection operation, psi (-) refers to a pre-trained VGG16 network for perceptual feature extraction, cos (-) is a trigonometric function,
to an enhanced image pair;
5. a weakly supervised change detection apparatus based on background mixed data extension technology, the apparatus comprising: a processor and a memory, the memory having stored therein program instructions, the processor calling upon the program instructions stored in the memory to cause the apparatus to perform any of the method steps described.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211332063.8A CN115641316A (en) | 2022-10-28 | 2022-10-28 | Weak supervision change detection method and device based on background mixed data expansion technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211332063.8A CN115641316A (en) | 2022-10-28 | 2022-10-28 | Weak supervision change detection method and device based on background mixed data expansion technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115641316A true CN115641316A (en) | 2023-01-24 |
Family
ID=84946890
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211332063.8A Pending CN115641316A (en) | 2022-10-28 | 2022-10-28 | Weak supervision change detection method and device based on background mixed data expansion technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115641316A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117095136A (en) * | 2023-10-19 | 2023-11-21 | 中国科学技术大学 | Multi-object and multi-attribute image reconstruction and editing method based on 3D GAN |
-
2022
- 2022-10-28 CN CN202211332063.8A patent/CN115641316A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117095136A (en) * | 2023-10-19 | 2023-11-21 | 中国科学技术大学 | Multi-object and multi-attribute image reconstruction and editing method based on 3D GAN |
| CN117095136B (en) * | 2023-10-19 | 2024-03-29 | 中国科学技术大学 | Multi-object and multi-attribute image reconstruction and editing method based on 3D GAN |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | Teinet: Towards an efficient architecture for video recognition | |
| Kim et al. | Deep convolutional neural models for picture-quality prediction: Challenges and solutions to data-driven image quality assessment | |
| CN114202672A (en) | Small target detection method based on attention mechanism | |
| CN112183360A (en) | Lightweight semantic segmentation method for high-resolution remote sensing image | |
| CN108171701B (en) | Significance detection method based on U network and counterstudy | |
| CN111311563A (en) | Image tampering detection method based on multi-domain feature fusion | |
| CN113591968A (en) | Infrared weak and small target detection method based on asymmetric attention feature fusion | |
| CN111325165A (en) | Urban remote sensing image scene classification method considering spatial relationship information | |
| CN115641316A (en) | Weak supervision change detection method and device based on background mixed data expansion technology | |
| CN111008570B (en) | Video understanding method based on compression-excitation pseudo-three-dimensional network | |
| CN113706544A (en) | Medical image segmentation method based on complete attention convolution neural network | |
| Zhou et al. | Image quality assessment using kernel sparse coding | |
| CN116205962A (en) | Monocular depth estimation method and system based on complete context information | |
| Li et al. | Image reflection removal using end‐to‐end convolutional neural network | |
| CN109377498A (en) | Interactive mode based on Recognition with Recurrent Neural Network scratches drawing method | |
| Yu et al. | Intelligent corner synthesis via cycle-consistent generative adversarial networks for efficient validation of autonomous driving systems | |
| TWI803243B (en) | Method for expanding images, computer device and storage medium | |
| CN115862119A (en) | Human face age estimation method and device based on attention mechanism | |
| CN115760670A (en) | Unsupervised hyperspectral fusion method and unsupervised hyperspectral fusion device based on network implicit prior | |
| Huang et al. | A Multi-Stage Vision Transformer for Fine-grained Image Classification | |
| CN114841870A (en) | Image processing method, related device and system | |
| CN114549500A (en) | High-dimensional image target defect detection model based on axial self-attention | |
| Zhou et al. | Learning visual representation of underwater acoustic imagery using transformer-based style transfer method | |
| Quan et al. | Unsupervised deep learning for phase retrieval via teacher-student distillation | |
| Wang et al. | Image intelligent automatic processing analysis based on artificial intelligence vision generator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |