CN116152666A - An Adaptive Learning Method for Cross-Domain Remote Sensing Images Considering the Heterogeneity of Surface Object and Phenology - Google Patents

Abstract

The invention discloses a cross-domain remote sensing image self-adaptive learning method considering the heterogeneity of ground feature weathers, which comprises the following steps: given a label Y with split _A Source phase dataset X of (1) _A And a target phase data set X without a tag _B Training M _st Generating a style migration sample considering the heterogeneity of the ground feature weathers; training M using style migration samples _seg Reducing the category characteristic distribution difference of the source domain and the target domain and simultaneously constructing M _st And M _seg Is a bidirectional optimized learning mechanism for improving cross-time-phase remote sensingImage domain self-adaptive learning capability to complete X _B Semantic segmentation tasks of (a). The method has the advantages of reducing the category characteristic distribution difference between different time domains and improving the domain self-adaptive learning effect of the model; the information interaction of the style migration process and the semantic segmentation process is enhanced.

Description

An adaptive learning method for cross-domain remote sensing images considering the heterogeneity of ground objects and phenology

Technical Field

The present invention belongs to the technical field of domain adaptive learning, and in particular relates to a cross-domain remote sensing image adaptive learning method that takes into account the heterogeneity of ground objects and phenology.

Background Art

Remote sensing technology has become an important means of studying land cover classification due to its wide coverage, high spatiotemporal resolution, and large amount of information. In the context of global economic integration, the scale of land cover classification is no longer limited to the country, but has expanded to the regional or global scale. Traditional manual interpretation and classification methods are labor-intensive and slow to update, and are far from meeting the needs of modern map making. In recent years, advances in artificial intelligence technologies such as deep learning have promoted the development of intelligent remote sensing image interpretation, and a large number of data-driven remote sensing image interpretation methods based on deep learning have emerged.

However, the good performance of various existing data-driven remote sensing image deep learning models requires that the test data and training data meet the independent and identically distributed assumption. In the real application scenarios of multi-temporal remote sensing image classification, the labeled training data (source domain) and the unlabeled test data (target domain) often come from different data distributions and have significant visual style differences, resulting in poor performance of the source domain model in the target domain. In order to solve this problem, in recent years, many research works have been devoted to using deep neural networks to learn the mapping of different image domains. The motivation is to transfer the style of remote sensing images in one domain to another specified domain, so that the generated transferred samples are closer to the specified domain samples in visual style, thereby supporting cross-temporal remote sensing image domain adaptive learning. Yang et al. proposed in the non-patent document "FG-GAN: AFine-Grained Generative Adversarial Network for Unsupervised SAR-to-Optical Image Translation, IEEE Trans. Geosci. Remote Sensing, vol. 60, pp. 1–11, 2022, doi: 10.1109/TGRS.2022.3165371" to integrate densely connected modules and residual modules in the generative network and use a multi-scale discriminant network to enhance the style transfer model's ability to characterize the radiation characteristics of remote sensing images. Tasar et al. in the non-patent literature “DAugNet: Unsupervised, Multisource, Multitarget, and Life-Long Domain Adaptationfor Semantic Segmentation of Satellite Images,” IEEE Trans. Geosci. Remote Sensing, vol. 59, no. 2, pp. 1067–1081, Feb. 2021, doi: 10.1109/TGRS.2020.3006161” use the statistics of each channel of the image feature to describe the style characteristics of the image, and simply adjust the mean and variance of each channel of the input image feature through adaptive instance normalization to match the style characteristics of the target remote sensing image. Zhang et al. in the non-patent literature “Remote Sensing Image Translation via Style-Based Recalibration Module and Improved Style Discriminator,” IEEE Geosci. Remote Sensing Lett., vol. 19, pp. 1–5, 2022, doi: 10.1109/LGRS.2021.3068558” introduces a style-based feature calibration module, which assigns learning weights to each channel of image features according to the importance of the statistics of each channel of image features to the style transfer of remote sensing images, so that the style transfer network can more quickly obtain the style information that needs to be focused on in the remote sensing images.

However, the current technical methods for adaptive learning across temporal remote sensing image domains using style transfer samples implicitly assume that the visual style changes of all objects in the remote sensing scene are unidirectional. They mainly focus on reducing the image radiation differences generated during the external imaging process of the remote sensing scene, and ignore the mining of factors affecting the phenological differences of objects within the scene. The impact of phenological differences on the visual style changes of remote sensing images is mainly reflected in two aspects. On the one hand, compared with objects such as artificial surfaces that are not phenologically sensitive, phenologically sensitive objects are particularly susceptible to seasonal cycles and undergo morphological changes, such as plant germination, leaf expansion, leaf discoloration, and leaf shedding. On the other hand, the phenological laws of different objects are heterogeneous ^[3] . For example, the growing season of forests is relatively long, and generally only one phenological periodic change occurs in a year, while the growing season of farmland crops is short, and farmland with two or even three crops a year may undergo multiple phenological periodic changes. Therefore, when these technical methods are used to identify target remote sensing images of geographical areas with severe spectral mixing and relatively fragmented ecological landscapes, the identification results are difficult to distinguish the boundaries of phenologically sensitive objects, and are particularly prone to confusing phenologically sensitive objects (such as farmland) and phenologically insensitive objects (such as artificial surfaces).

According to the current technical research background, the following problems need to be solved for the technical method of using style transfer samples for cross-temporal remote sensing image domain adaptive learning: (1) The visual style changes of remote sensing scenes are affected by both the image radiation differences generated by the external imaging process and the phenological differences of the objects inside the scene. However, the style transfer samples generated by the existing technology cannot simulate the heterogeneity of the phenology of the objects, which hinders the classification effect of domain adaptation. (2) The style transfer network fails to interact with the semantic segmentation network during the feature learning process, resulting in insufficient information transmitted by the style transfer samples to the domain adaptive learning process, which limits the domain adaptive learning ability of the model.

Summary of the invention

In view of this, the framework proposed in the present invention includes a style transfer network M _st and a semantic segmentation network M _seg . Given a source temporal dataset X _{A with segmentation labels Y A and} a target temporal dataset X _B without labels, the goal is to train M _st to generate style transfer samples that take into account the heterogeneity of ground objects and phenology, and use these style transfer samples to train M _seg to reduce the difference in the distribution of category features between the source domain and the target domain. At the same time, a bidirectional optimization learning mechanism of M _st and M _seg is constructed to improve the adaptive learning ability of cross-temporal remote sensing image domains and complete the semantic segmentation task of X _B.

The present invention discloses a cross-domain remote sensing image adaptive learning method that takes into account the heterogeneity of ground objects and phenology. The method is applied to a style transfer network M _st and a semantic segmentation network M _seg , and the method comprises the following steps:

Given a source temporal dataset _XA with segmentation labels _YA and a target temporal dataset _XB without labels, train _Mst to generate style transfer samples that take into account the heterogeneity of ground objects and phenology;

The style transfer samples are used to train _Mseg to reduce the difference in category feature distribution between the source domain and the target domain. At the same time, a bidirectional optimization learning mechanism of _Mst and _Mseg is constructed to improve the adaptive learning ability of cross-temporal remote sensing image domain and complete the semantic segmentation task of _XB .

和对应尺度的地物分割图

对于地物类别k，得到类别特征图

用类别特征通道维的均值和方差表示该图像的地物类别风格

其中，Furthermore, given an image feature map

And the corresponding scale of the ground feature segmentation map

For the ground feature category k, we get the category feature map

The mean and variance of the channel dimension of the category feature are used to represent the category style of the image.

in,

和

分别表示不同图像域的地物类别特征风格参数集合，其中，μ^k表示均值，σ^k表示方差，β^k，γ^k表示μ^k，σ^k的数学期望；假定对于地物类别k，图像域共包含N^k个采样像素，则use

and

They represent the feature style parameter sets of different object categories in different image domains, where μ ^k represents the mean, σ ^k represents the variance, β ^k and γ ^k represent the mathematical expectations of μ ^k and σ ^k . Assuming that for object category k, the image domain contains N ^k sample pixels in total, then

First, randomly select a part of samples and use formula (2) to initialize β ^k , γ ^k . During the model training process, use moving average to gradually update μ ^k , γ ^k :

β ^k ←λβ ^k +(1-λ)μ ^k (F)

γ ^k ←λγ ^k +(1-λ)σ ^k (F)

Where λ is the momentum coefficient, which is set to 0.9999;

The embedding features are regularized by class by adaptive segmentation sample normalization, which is defined as follows:

表示地物类别特征风格参数集合，

表示物候敏感因子，如果类别k是物候敏感地物，w^k＝1，否则w^k＝0，即对物候敏感地物类别特征进行风格正则化。in

Represents a set of feature style parameters for feature categories.

represents the phenological sensitivity factor. If category k is a phenologically sensitive feature, w ^k = 1, otherwise w ^k = 0, that is, style regularization is performed on the category characteristics of phenologically sensitive features.

Furthermore, the process of style transfer taking into account the heterogeneity of ground objects and phenology includes:

组合，F_B和其伪分割图

与

进行组合；Different domain spatial samples X _A and X _B are passed through the domain encoder to obtain embedded features F _A and F _B , F _A and its segmentation map Y _A and the style parameters of the object category

Combination, F _B and its pseudo segmentation map

and

Make a combination;

Performing style regularization on the phenological sensitive land feature category features through the category-by-category regularization constraints to obtain F _AB and F _BA , and then passing through a domain decoder to obtain style transfer samples X _AB and X _BA ;

Use adversarial learning to train the style transfer network: define _GA and _GB as generators of image domains _XA and _XB , D _A and _DB as discriminators of image domains _XA and _XB , and the discriminator adopts a semantic segmentation network structure, which can not only distinguish real samples and style transfer samples, but also correctly classify the objects in the real samples.

Furthermore, the adversarial loss function of the discriminator is defined as:

表示由M_seg(初始为源域模型)对目标域样本x_B预测得到的伪分割图；Where _yA represents the true segmentation map of the source domain sample _xA ,

represents the pseudo segmentation map predicted by _Mseg (initially the source domain model) for the target domain sample _xB ;

Correspondingly, the adversarial loss function of the generator is defined as:

以及X_BA和

生成交叉风格迁移样本X_ABA和X_BAB，因此，最小化交叉重建一致损失：By combining X _AB and

and _XBA and

Generate cross-style transfer samples _XABA and _XBAB , thus minimizing the cross-reconstruction consistency loss:

以及X_B和

生成的重建样本X_AA和X_BB与原始样本也应该保持一致，并最小化自重建一致损失：At the same time, by combining X _A and

and X _B and

The generated reconstructed samples X _AA and X _BB should also be consistent with the original samples and minimize the self-reconstruction consistency loss:

The objective loss function of the final generator is defined as:

表示第m次模型双向优化，

表示初始模型为源域模型，

为

生成的目标域伪分割图；Furthermore, the bidirectional optimization of the model in the bidirectional optimization learning mechanism includes two directions: (M _seg →M _st and (M _st →M _seg );

represents the mth model bidirectional optimization,

Indicates that the initial model is the source domain model,

for

Generated pseudo segmentation map of the target domain;

The optimization direction (M _seg →M _st ) indicates that the target domain pseudo-labels predicted by the semantic segmentation network are used to train a style transfer network that takes into account the heterogeneity of ground objects and phenology;

Given the prediction result p _B =M _seg (x _B ) of _Mseg in the target domain, a confidence threshold d will be set to screen p _B , and the prediction results with high confidence will be selected as pseudo labels for the training of _Mst ;

其伪标签

表示为：For the target pixel

Its pseudo label

It is expressed as:

The optimization direction (M _st →M _seg ) indicates optimizing the semantic segmentation network using style transfer samples. First, use the source domain data and its true segmentation map to train M _seg :

得到p_B＝M_seg(x_B)以及p_BA＝M_seg(x_BA)；Then, given the trained M _st, the style transfer result of the target domain sample is

We obtain p _B =M _seg (x _B ) and p _BA =M _seg (x _BA );

The predictions of M _seg for the target domain samples and their style transfer results should be consistent, therefore, minimizing the prediction consistency loss function:

At the same time, for the high confidence region max(p _B ＞d) of p _B , the mutual learning loss function of the migration samples is minimized:

Therefore, the objective loss function for semantic segmentation domain adaptation is defined as:

The beneficial effects of the present invention are as follows:

(1) Starting from the heterogeneity of landforms and phenology, the method of the present invention designs an adaptive segmentation sample normalization module (AdaSIN) to perform regularization constraints on the embedded features on a category-by-category basis, so that the style transfer network can generate style transfer samples that take into account the heterogeneity of landforms and phenology. Compared with the style transfer samples generated by traditional methods, the style transfer samples that take into account the heterogeneity of landforms and phenology have more advantages in reducing the differences in the distribution of category features between different temporal domains, which is conducive to improving the domain adaptive learning effect of the model, and can be extended to other task scenarios with multi-temporal feature data, such as cross-domain scene classification, cross-domain semantic segmentation, and change detection.

(2) Due to the design of the bidirectional optimization learning mechanism of the style transfer network and the semantic segmentation network, the method of the present invention strengthens the information interaction between the style transfer process and the semantic segmentation process, and further improves the domain adaptive learning ability of the model.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of the style transfer process that takes into account the heterogeneity of landform phenology, where rectangular and triangular shapes represent phenologically sensitive landform categories, and circular shapes represent phenologically insensitive landform categories;

Figure 2 is the pseudo code of the bidirectional optimization algorithm of the style transfer network and the semantic segmentation network;

Figure 3 is a semantic segmentation diagram of different domain adaptation learning methods;

FIG4 is a sensitivity analysis diagram of the confidence threshold parameter.

DETAILED DESCRIPTION

The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

The framework proposed in the present invention includes a style transfer network M _st and a semantic segmentation network M _seg . Given a source temporal data set X _A with segmentation labels Y _A and a target temporal data set X _B without labels, the goal of the present invention is to train M _st to generate style transfer samples that take into account the heterogeneity of ground objects and phenology, and use these style transfer samples to train M _seg to reduce the distribution difference of category features between the source domain and the target domain, and at the same time construct a bidirectional optimization learning mechanism of M _st and M _seg to improve the adaptive learning ability of cross-temporal remote sensing image domains and complete the semantic segmentation task of X _B. Next, the present invention first explains how to generate style transfer samples that take into account the heterogeneity of ground objects and phenology, and then introduces the bidirectional optimization learning mechanism of M _st and M _seg in detail.

(1) Style transfer that takes into account the heterogeneity of ground features and phenology

和对应尺度的地物分割图

从地物物候的异质性规律出发，图像场景中不同地物类别应当具有不同的风格特征。因此，对于地物类别k，可得类别特征图

本发明用类别特征通道维的均值和方差表示该图像的地物类别风格

其中，Given an image feature map

And the corresponding scale of the ground feature segmentation map

Based on the heterogeneity of ground objects and phenology, different ground object categories in the image scene should have different style characteristics. Therefore, for ground object category k, the category feature map can be obtained:

The present invention uses the mean and variance of the category feature channel dimension to represent the category style of the image.

in,

和

分别表示不同图像域的地物类别特征风格参数集合，其中β^k，γ^k表示μ^k，σ^k的数学期望。假定对于地物类别k，图像域共包含N^k个采样像素，则The present invention is used

and

They represent the feature style parameter sets of different object categories in different image domains, where β ^k and γ ^k represent the mathematical expectations of μ ^k and σ ^k . Assuming that for object category k, the image domain contains N ^k sample pixels, then

This calculation method takes up too much computing resources and is not conducive to model training. Therefore, the present invention first randomly selects a part of samples and initializes β ^k , γ ^k using formula (2), and gradually updates β ^k , γ ^k using moving average during model training:

β ^k ←λβ ^k +(1-λ)μ ^k (F)

γ ^k ←λγ ^k +(1-λ)σ ^k (F) (3)

Where λ is the momentum coefficient, which is set to 0.9999. Therefore, the present invention can perform regularization constraints on the embedded features by category through Adaptive Segmented Instance Normalization (AdaSIN), which is defined as follows:

表示地物类别特征风格参数集合，

表示物候敏感因子，如果类别k是物候敏感地物，w^k＝1，否则w^k＝0，即对物候敏感地物类别特征进行风格正则化。in

Represents a set of feature style parameters for feature categories.

represents the phenological sensitivity factor. If category k is a phenologically sensitive feature, w ^k = 1, otherwise w ^k = 0, that is, style regularization is performed on the category characteristics of phenologically sensitive features.

组合，F_B和其伪分割图

与

进行组合，并通过公式(4)对物候敏感地物类别特征进行风格正则化得到F_AB和F_BA，随后经过域解码器得到风格迁移样本X_AB和X_BA。为了保证X_A与风格迁移样本X_BA以及X_B与风格迁移样本X_AB的地物类别特征分布尽可能接近，本发明使用对抗学习训练风格迁移网络。定义G_A和G_B为图像域X_A和X_B的生成器，D_A和D_B为图像域X_A和X_B的鉴别器。鉴别器采用语义分割网络结构，其既需要判别真实样本和风格迁移样本，也需要对真实样本中的地物进行正确分类。因此，鉴别器的对抗损失函数定义为：The process of style transfer taking into account the heterogeneity of ground objects is described as follows: As shown in Figure 1, different domain spatial samples X _A and X _B are passed through the domain encoder to obtain embedded features _FA and _FB , _FA and its segmentation map Y _A and ground object category style parameters

Combination, F _B and its pseudo segmentation map

and

The phenologically sensitive ground object category features are combined, and the style regularization is performed on the phenologically sensitive ground object category features through formula (4) to obtain F _AB and F _BA , and then the style transfer samples X _AB and X _BA are obtained through the domain decoder. In order to ensure that the ground object category feature distribution of X _A and the style transfer sample X _BA and X _B and the style transfer sample X _AB are as close as possible, the present invention uses adversarial learning to train the style transfer network. _GA and G _B are defined as generators of the image domain X _A and X _B , and D _A and D _B are discriminators of the image domain X _A and X _B. The discriminator adopts a semantic segmentation network structure, which needs to distinguish between real samples and style transfer samples, and also needs to correctly classify the ground objects in the real samples. Therefore, the adversarial loss function of the discriminator is defined as:

表示由M_seg(初始为源域模型)对目标域样本x_B预测得到的伪分割图。对应地，生成器的对抗损失函数定义为：Where _yA represents the true segmentation map of the source domain sample _xA ,

represents the pseudo segmentation map predicted by _Mseg (initially the source domain model) for the target domain sample _xB . Correspondingly, the adversarial loss function of the generator is defined as:

以及X_BA和

生成交叉风格迁移样本X_ABA和X_BAB。因此，本发明最小化交叉重建一致损失：In order to ensure that _XA and _XAB , _XB and _XBA maintain semantic consistency after style transfer, the present invention first combines _XAB and

and _XBA and

Generate cross-style transfer samples _XABA and _XBAB . Therefore, the present invention minimizes the cross-reconstruction consistency loss:

以及X_B和

生成的重建样本X_AA和X_BB与原始样本也应该保持一致，并最小化自重建一致损失：At the same time, the present invention combines X _A and

and X _B and

The generated reconstructed samples X _AA and X _BB should also be consistent with the original samples and minimize the self-reconstruction consistency loss:

Therefore, the objective loss function of the final generator is defined as:

(2) Bidirectional optimization learning mechanism

表示第m次模型双向优化，详细优化过程见图2，

表示初始模型为源域模型，

为

生成的目标域伪分割图。The bidirectional optimization of the model includes two directions: (M _seg →M _st ) and (M _st →M _seg ).

It represents the mth bidirectional optimization of the model. The detailed optimization process is shown in Figure 2.

Indicates that the initial model is the source domain model,

for

Generated pseudo segmentation map of the target domain.

其伪标签

表示为：The optimization direction (M _seg →M _st ) indicates that the target domain pseudo-label predicted by the semantic segmentation network is used to train the style transfer network that takes into account the heterogeneity of the ground objects and phenology. Given the prediction result p _B =M _seg (x _B ) of M _seg in the target domain, the present invention sets a confidence threshold d to screen p _B , and the prediction results with high confidence will be selected as pseudo-labels for the training of M _st . For the target pixel

Its pseudo label

It is expressed as:

The optimization direction (M _st →M _seg ) indicates optimizing the semantic segmentation network using style transfer samples. The present invention first trains M _seg using source domain data and its true segmentation map:

本发明可以得到p_B＝M_seg(x_B)以及p_BA＝M_seg(x_BA)。M_seg对目标域样本及其风格迁移结果的预测应该保持一致，因此，本发明最小化预测一致性损失函数：Then, given the trained M _st, the style transfer result of the target domain sample is

The present invention can obtain p _B =M _seg (x _B ) and p _BA =M _seg (x _BA ). The prediction of M _seg for the target domain sample and its style transfer result should be consistent. Therefore, the present invention minimizes the prediction consistency loss function:

At the same time, for the high confidence region max(p _B >d) of p _B , the present invention minimizes the mutual learning loss function of the migration samples:

Therefore, the objective loss function for semantic segmentation domain adaptation is defined as:

In order to verify the effectiveness of the method of the present invention, a comparative experiment on cross-temporal semantic segmentation was conducted on the same dataset between the present invention and two typical domain adaptive learning technologies, namely, a domain adaptive learning method based on self-training (CBST) and a domain adaptive learning method using style transfer samples (DAugnet). The present invention uses overall accuracy (OA), Kappa coefficient (Kappa) and weighted intersection over union (FWIoU) as overall classification evaluation indicators, and uses intersection over union (IoU) as the classification evaluation indicator for individual classes. From the quantitative indicators (see Table 1), compared with the baseline, CBST has the worst improvement effect on cross-temporal semantic segmentation, and the method of the present invention has the best improvement effect. Compared with the domain adaptive learning method DAugnet that uses traditional style transfer samples, the method of the present invention uses style transfer samples that take into account the heterogeneity of ground objects and phenology for domain adaptive learning, and has 3.58%, 1.2% and 2.3% improvements on the overall indicators OA, Kappa and FWIoU, respectively.

From the qualitative results (see Figure 3), the traditional domain adaptive learning method is more likely to confuse cultivated land and water bodies, while the method of the present invention uses style transfer samples that take into account the phenological heterogeneity of land objects to reduce the distribution differences of category features between different temporal domains, and constructs a two-way optimization learning mechanism of style transfer network and semantic segmentation network, which significantly reduces the classification confusion of cultivated land and water bodies. This shows that the method of the present invention has significant advantages in distinguishing phenologically sensitive land objects (cultivated land) from phenologically insensitive land objects (water bodies).

Table 1 Comparative experimental results of semantic segmentation of different domain adaptation learning methods (%)

In order to verify the feasibility of the method of the present invention, a set of multi-temporal remote sensing image data in Xiangtan City, Hunan Province, China was selected for experiments. The data was sampled from the GF-2 sensor with a resolution of 2m. The source domain data set was sampled from 2018 and the target domain data set was sampled from 2019. The source domain and the target domain each contain 4232 remote sensing images (both with a size of 512×512 pixels), which contain 6 types of ground feature labels, namely null, cultivated land, forest land, grassland, water body and artificial surface. The source domain images and labels and the target domain images are used for domain adaptive model training, and the target domain labels are used for domain adaptive model testing. The method of the present invention was compared with two typical domain adaptive learning methods. The experimental results are shown in Figure 3 and Table 1. The method of the present invention shows more outstanding domain adaptive learning ability. In addition, this paper discusses three issues: (1) the contribution of style transfer samples that take into account the heterogeneity of ground objects and phenology to the semantic segmentation of remote sensing images across temporal domains; (2) the role of the bidirectional optimization learning mechanism of style transfer networks and semantic segmentation networks; and (3) the impact of the confidence threshold on model optimization during the target pseudo-label generation process.

Table 2 Ablation experimental results of each component of the method of the present invention (%), where ST represents pseudo-label self-learning, AdaIN represents adaptive sample normalization, AdaSIN represents the adaptive segmentation sample normalization proposed by the present invention, and m represents the number of bidirectional optimization learning

First, it can be seen from the ablation experiment results (see Table 2) that compared with the pseudo-label self-learning method (ST), the effect of introducing style transfer samples in the domain adaptive learning process is more obvious. However, the style transfer samples generated by the traditional style transfer method using adaptive sample normalization (Adaptive Instance Normalization, AdaIN) only consider reducing the image radiation difference but ignore the heterogeneity of the phenology of the ground objects, which limits the domain adaptive learning ability of the model. Starting from the law of the heterogeneity of the ground objects and phenology, the method of the present invention designs adaptive segmented sample normalization (Adaptive Segmented Instance Normalization, AdaSIN) to perform regular constraints on the embedded features by category, so that the style transfer network can generate style transfer samples that take into account the heterogeneity of the ground objects and phenology. Compared with AdaIN, the AdaSIN proposed by the method of the present invention improves the domain adaptive learning ability of the model, so that the various indicators of the experimental results are significantly improved.

Secondly, the method of the present invention uses the output of the semantic segmentation network as pseudo-label information to input the style transfer network, and constructs a bidirectional optimization learning mechanism for the style transfer network and the semantic segmentation network. As can be seen from Table 2, when the number of bidirectional optimization learning times m>1, the experimental results are further improved after each bidirectional optimization, which shows that the bidirectional optimization learning mechanism strengthens the information interaction between the style transfer process and the domain adaptation learning process, and further improves the domain adaptation learning ability of the model.

Finally, in order to study the influence of the confidence threshold d on model optimization during pseudo-label generation, the present invention sets d in the range of [0.4, 0.8] and conducts a series of model optimization experiments. As shown in Figure 4, when d is set to 0.7, the model achieves the best performance. When d is set too small (for example, less than 0.5), the pseudo-label has more erroneous information and the model performance is weaker; when d is between 0.6 and 0.7, the impact on model optimization is not obvious; when d is set too large (for example, greater than 0.7), the model performance decreases slightly. The possible reason for the performance degradation caused by using a larger d is that the number of pseudo-labels it generates is reduced, which limits the retraining of the model.

The beneficial effects of the present invention are as follows:

(1) Starting from the heterogeneity of landforms and phenology, the method of the present invention designs an adaptive segmentation sample normalization module (AdaSIN) to perform regularization constraints on the embedded features on a category-by-category basis, so that the style transfer network can generate style transfer samples that take into account the heterogeneity of landforms and phenology. Compared with the style transfer samples generated by traditional methods, the style transfer samples that take into account the heterogeneity of landforms and phenology have more advantages in reducing the differences in the distribution of category features between different temporal domains, which is conducive to improving the domain adaptive learning effect of the model, and can be extended to other task scenarios with multi-temporal feature data, such as cross-domain scene classification, cross-domain semantic segmentation, and change detection.

(2) Due to the design of the bidirectional optimization learning mechanism of the style transfer network and the semantic segmentation network, the method of the present invention strengthens the information interaction between the style transfer process and the semantic segmentation process, and further improves the domain adaptive learning ability of the model.

As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment, and any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims (5)

1. A cross-domain remote sensing image adaptive learning method taking into account the heterogeneity of ground objects and phenology, characterized in that the method is applied to a style transfer network M _st and a semantic segmentation network M _seg , and the method comprises the following steps: Given a source temporal dataset _XA with segmentation labels _YA and a target temporal dataset _XB without labels, train _Mst to generate style transfer samples that take into account the heterogeneity of ground objects and phenology; The style transfer samples are used to train _Mseg to reduce the difference in the distribution of category features between the source domain and the target domain, and a bidirectional optimization learning mechanism of _Mst and _Mseg is constructed to improve the adaptive learning ability of cross-temporal remote sensing image domain and complete the semantic segmentation task of _XB . The bidirectional optimization of the model in the bidirectional optimization learning mechanism includes two directions: (M _seg →M _st ) and (M _st →M _seg );

represents the mth model bidirectional optimization,

Indicates that the initial model is the source domain model,

for

The generated target domain pseudo segmentation map; the (M _seg →M _st ) optimization direction indicates that the target domain pseudo labels predicted by the semantic segmentation network are used to train the style transfer network that takes into account the heterogeneity of ground objects and phenology; the (M _st →M _seg ) optimization direction indicates that the semantic segmentation network is optimized using style transfer samples. 2. The cross-domain remote sensing image adaptive learning method considering the heterogeneity of ground objects and phenology according to claim 1 is characterized in that, given an image feature map

And the corresponding scale of the ground feature segmentation map

For the ground feature category k, we get the category feature map

The mean and variance of the channel dimension of the category feature are used to represent the category style of the image.

in,

use

and

They represent the feature style parameter sets of different object categories in different image domains, where μ ^k represents the mean, σ ^k represents the variance, β ^k and γ ^k represent the mathematical expectations of μ ^k and σ ^k . Assuming that for object category k, the image domain contains N ^k sample pixels in total, then

First, randomly select a part of samples and use the above formula to initialize β ^k , γ ^k . During the model training process, use moving average to gradually update β ^k , γ ^k : β ^k ←λβ ^k +(1-λ)μ ^k (F) γ ^k ←λγ ^k +(1-λ)σ ^k (F) Where λ is the momentum coefficient, which is set to 0.9999; The embedding features are regularized by class by adaptive segmentation sample normalization, which is defined as follows:

in

Represents a set of feature style parameters for feature categories.

represents the phenological sensitivity factor. If category k is a phenologically sensitive feature, w ^k = 1, otherwise w ^k = 0, that is, style regularization is performed on the category characteristics of phenologically sensitive features. 3. The cross-domain remote sensing image adaptive learning method considering the heterogeneity of ground object phenology according to claim 2, characterized in that the process of style transfer considering the heterogeneity of ground object phenology comprises: Different domain spatial samples X _A and X _B are passed through the domain encoder to obtain embedded features F _A and F _B , F _A and its segmentation map Y _A and the style parameters of the object category

Combination, F _B and its pseudo segmentation map

and

Make a combination; Performing style regularization on the phenological sensitive land feature category features through the category-by-category regularization constraints to obtain F _AB and F _BA , and then passing through a domain decoder to obtain style transfer samples X _AB and X _BA ; Use adversarial learning to train the style transfer network: define _GA and _GB as generators of image domains _XA and _XB , D _A and _DB as discriminators of image domains _XA and _XB , and the discriminator adopts a semantic segmentation network structure, which can not only distinguish real samples and style transfer samples, but also correctly classify the objects in the real samples. 4. The cross-domain remote sensing image adaptive learning method taking into account the heterogeneity of ground objects and phenology according to claim 3, characterized in that the adversarial loss function of the discriminator is defined as:

Where _yA represents the true segmentation map of the source domain sample _xA ,

represents the pseudo segmentation map predicted by _Mseg for the target domain sample _xB ; Correspondingly, the adversarial loss function of the generator is defined as:

By combining X _AB and

and _XBA and

Generate cross-style transfer samples _XABA and _XBAB , thus minimizing the cross-reconstruction consistency loss:

At the same time, by combining X _A and

and X _B and

The generated reconstructed samples X _AA and X _BB should also be consistent with the original samples and minimize the self-reconstruction consistency loss:

The objective loss function of the final generator is defined as:

5. The cross-domain remote sensing image adaptive learning method taking into account the heterogeneity of ground objects and phenology according to claim 4, characterized in that for the optimization direction (M _seg →M _st ): Given the prediction result p _B =M _seg (x _B ) of _Mseg in the target domain, a confidence threshold d is set to screen p _B , and the prediction results with high confidence are selected as pseudo labels for the training of _Mst ; For the target pixel

Its pseudo label

It is expressed as:

For the optimization direction (M _st →M _seg ): First, use the source domain data and its true segmentation map to train _Mseg :

Given the trained M _st, the style transfer result of the target domain sample

We obtain p _B =M _seg (x _B ) and p _BA =M _seg (x _BA ); The predictions of M _seg for the target domain samples and their style transfer results should be consistent, therefore, minimizing the prediction consistency loss function:

At the same time, for the high confidence region max(p _B >d) of p _B , minimize the mutual learning loss function of the migration sample:

Therefore, the objective loss function for semantic segmentation domain adaptation is defined as:

Images