TWI840637B - Training method and training system of generative adversarial network for image cross domain conversion - Google Patents

Abstract

A training method and a training system of generative adversarial network for image cross domain conversion are provided. The generative confrontation network is used for performing image cross domain conversion with preserved structure. The training method of generative confrontation network for image multimodal conversion includes the following steps. A first real image is inputted to a first generator to obtain a first generated image. The first real image has a first mode, and the first generated image has a second mode. The first generated image is inputted to a second generator to obtain a first reconstructed image. The parameters of the first generator and the second generator are updated according to a plurality of loss values, so as to achieve image multimodal conversion with cyclically consistent or similar structure.

Description

Training method and training system for generative adversarial network for performing multimodal image conversion

This case is about a training method and training system for a generative adversarial network that performs multimodal image conversion.

In recent years, the rise of deep learning technology has greatly improved the recognition rate of various image recognition technologies. Deep learning models require a large amount of labeled data for training. However, labeled data is the most important and never public asset for relevant R&D units/companies in product development. Even if many small and medium-sized enterprises have the ability to develop deep learning models, they are limited by manpower or material resources, making the amount of self-labeled data extremely limited, making it impossible for them to compete with international manufacturers in productization, especially in applications such as vehicle object detection/semantic segmentation/instance segmentation. In these applications, especially semantic segmentation and instance segmentation, it is necessary to label each pixel in the image, which consumes a huge amount of manpower costs.

After the introduction of the Generative Adversarial Network (GAN) for performing multimodal image conversion, deep learning models began to have the ability to generate images. Many subsequent studies have further evolved to the ability to convert images from style A (modality) to style B (modality), etc. Using such powerful image conversion capabilities, image segmentation or object detection may achieve the so-called multimodal application (cross domain adaptation), that is, using GAN to convert labeled training data from the original modality (source domain) to the target modality (target domain). This increases the versatility and robustness of image segmentation or object detection models in the target modality (target domain). However, the main reason why existing models still cannot achieve such a function is that although existing GANs can achieve a significant change in image style, the objects or pixels of a specific category marked in the image often change their positions after the transformation, which will require the image to be re-labeled, making GAN unable to be used in image style transformation of labeled data.

Therefore, researchers are developing a generative adversarial network that performs multimodal image conversion, hoping that it can convert an original modality image into target modality images of varying degrees, and the structure of each converted target modality image can remain consistent or similar to that before conversion.

This case is about a training method and training system for a generative adversarial network that performs multimodal image conversion.

According to an embodiment of the present case, a training method for a generative adversarial network that performs image multimodal conversion is proposed. The generative adversarial network is used to perform structure-preserving image multimodal conversion. The training method for a generative adversarial network that performs image multimodal conversion includes the following steps. Input a first real image to a first generator to obtain a first generated image. The first real image has a first modality, and the first generated image has a second modality. Input the first generated image to a second generator to obtain a first structural reconstruction image. The first structural reconstruction image has a first modality. Input the first generated image to a first discriminator to calculate a first loss value. Input the first generated image to a first image segmentation network to calculate a second loss value. Input the first structural reconstruction image to a second discriminator to calculate a third loss value. Input the first structured reconstructed image to a second image segmentation network to calculate a fourth loss value. Update the parameters of the first generator and the second generator to train a generative adversarial network that performs image multimodal conversion. The parameters of the first generator and the second generator are updated based on at least the first loss value, the second loss value, the third loss value, and the fourth loss value.

According to another embodiment of the present invention, a training system for a generative adversarial network for performing image multimodal conversion is provided. The generative adversarial network is used for performing structure-preserving image multimodal conversion. The training system for a generative adversarial network for performing image multimodal conversion includes a first generator, a second generator, a first discriminator, a second discriminator, a first image segmentation network, and a second image segmentation network. The first generator is used for receiving a first real image to obtain a first generated image. The first real image has a first modality. The first generated image has a second modality. The second generator is used for receiving the first generated image to obtain a first structural reconstruction image. The first structural reconstruction image has a first modality. The first discriminator is used for receiving the first generated image to calculate a first loss value. The second discriminator is used to receive the first structural reconstruction image to calculate a second loss value. The first image segmentation network is used to receive the first generated image to calculate a third loss value. The second image segmentation network is used to receive the first structural reconstruction image to calculate a fourth loss value. The parameters of the first generator and the second generator are updated at least according to the first loss value, the second loss value, the third loss value and the fourth loss value to train the generative adversarial network for performing image multimodal conversion.

In order to have a better understanding of the above and other aspects of this case, the following is a specific example, and the attached drawings are explained in detail as follows:

100,200: Training system for generative adversarial networks for image multimodal transformation

D1: First discriminator

D2: Second discriminator

D _x ,D _y : discriminator

,

:編碼器 E _x ,E _y ,

,

:Encoder

FK1: First generated image

FK2: Second generation image

G1: First generator

G2: Second generator

GT1: The first real image

GT2: Second Reality Image

G _x ,G _y : generator

LS1: First loss value

LS2: Second loss value

LS3: Third loss value

LS4: Fourth loss value

LS5: Fifth loss value

LS6: Sixth loss value

LS7: Seventh loss value

LS8: Eighth loss value

l _mce : comparison function

N(z): Random noise

P _x ,P _y : Splitter

RC1: First structural reconstruction image

RC2: Second structural reconstruction image

S1: The first image segmentation network

S2: Second image segmentation network

S110,S120,S130,S170,S150,S160,S170,S210,S220,S230,S240,S250,S260,S270: Steps

SG1: The image segmentation truth of the first real image

SG2: Image segmentation truth of the second ground truth image

x:First real image

:第一真實影像之影像分割真值

:The first real image segmentation truth

:第二生成影像

: Second generation image

:第二生成影像的影像分割

: Image segmentation of the second generated image

x _rec : First structural reconstruction image

:第一結構重建影像的影像分割

: Image segmentation of the first structural reconstruction image

y: Second real image

:第二真實影像之影像分割真值

:Second ground truth image segmentation

:第一生成影像

:First generation image

:第一生成影像的影像分割

: Image segmentation of the first generated image

y _rec : Second structural reconstruction image

:第二結構重建影像的影像分割

: Image segmentation of the second structure reconstruction image

FIG. 1 is a block diagram of a training system for a generative adversarial network for performing image multimodal transformation according to an embodiment.

FIG. 2 shows a flow chart of a method for training a generative adversarial network for performing image multimodal conversion according to an embodiment.

FIG. 3 shows a block diagram of a training system for a generative adversarial network for performing image multimodal conversion according to another embodiment.

FIG. 4 shows a flow chart of a method for training a generative adversarial network for performing image multimodal conversion according to another embodiment.

Figure 5 shows the network architecture of the generative adversarial network that performs image multimodal transformation.

Figure 6 illustrates an example of the original first real image.

Figure 7 shows an example of the first generated image of the night after adding random noise.

Figure 8 shows an example of the first structure reconstruction image.

Figure 9 illustrates the image segmentation truth value of the first real image.

Figure 10 illustrates the image segmentation of the first generated image.

Figure 11 illustrates the image segmentation results of the first structural reconstruction image.

This case proposes a method for training a generative adversarial network that performs image multimodal transformation. The trained generative adversarial network can perform structure-preserving image multimodal transformation. The so-called "structure" refers to the objects and their positions in the image. "Structure preservation" means that the generated image can preserve the objects and their positions in the original image. Taking driving images as an example, daytime driving images can be expanded into nighttime driving images as training samples for deep learning through the generative adversarial network trained in this case. The object positions of the expanded nighttime driving images are consistent or similar to those of the daytime driving images, and the structure is preserved without the need for re-labeling. It not only solves the problem of decreased recognition rate of driving object detection due to insufficient labeled data, but also avoids the need for a large amount of labeled data. Among them, the generative adversarial network in this case is applied to the expansion of training samples. The training samples are, for example, images, and the application models of the training samples are various image object detection models. The input data of the image object detection model is an image, and the output data is the location of the object.

The training method of the generative adversarial network for performing multimodal image conversion proposed in this case includes a forward cycle architecture (forward cycle) and a backward cycle architecture (backward cycle). The forward cycle architecture is for training the conversion from the first mode to the second mode, and the backward cycle architecture is for training the conversion from the second mode to the first mode. The combination of the forward cycle architecture and the backward cycle architecture can make the conversion between the first mode and the second mode quite accurate.

In one embodiment, the training method of the generative adversarial network for performing image multimodal conversion may only have a forward-looking loop architecture. The forward-looking loop architecture is first described below.

Please refer to Figure 1, which shows a block diagram of a training system 100 for a generative adversarial network that performs image multimodal conversion according to an embodiment. The training system 100 for a generative adversarial network that performs image multimodal conversion in Figure 1 only includes a forward loop architecture. The training system 100 for a generative adversarial network that performs image multimodal conversion includes a first generator G1, a second generator G2, a first discriminator D1, a second discriminator D2, a first image segmentation network S1, and a second image segmentation network S2. The functions of each component are summarized as follows. The first generator G1 and the second generator G2 are used to perform mode and style conversion. The first discriminator D1 and the second discriminator D2 are used to verify the generated data. The first image segmentation network S1 and the second image segmentation network S2 are used for object segmentation and labeling, such as semantic segmentation networks. The first generator G1, the second generator G2, the first discriminator D1, the second discriminator D2, the first image segmentation network S1 and the second image segmentation network S2 are, for example, a circuit, a chip, a circuit board or a storage device for storing program codes.

The main purpose of the first generator G1 is to convert the first real image GT1 of the first mode (for example, daytime) into the first generated image FK1 of the second mode (for example, nighttime). After the first generator G1 is injected with random noise N(z), the first generated image FK1 is verified by the first discriminator D1 through learning and training. In addition, in order to maintain the structural consistency or similarity between the first generated image FK1 and the first real image GT1, the first image segmentation network S1 can assist the first generated image FK1 in structural confirmation to ensure that the structure before and after the image conversion is preserved or similar. The above-mentioned first mode and second mode are explained by taking daytime and night as examples. In another embodiment, the first mode and the second mode can also be in a situation where the style difference is not much, such as sunny and cloudy.

Furthermore, in order to prevent the mode and style of the image conversion from being too strong, the second generator G2 converts the first generated image FK1 of the second mode back to the first structural reconstruction image RC1 of the first mode, and passes the verification of the second discriminator D2. In addition, the first structural reconstruction image RC1 and the first real image GT1 also need to maintain structural consistency or similarity. The second image segmentation network S2 can assist the first structural reconstruction image RC1 in structural confirmation to ensure that the structure before and after the image conversion is consistent or similar. The operation of the above-mentioned components is described in detail below through a flowchart. The so-called "structural reconstruction" means that the objects and their positions of the first structural reconstruction image RC1 are reconstructed so that the objects of the first structural reconstruction image RC1 are similar to the objects and their positions of the first real image GT1.

Please refer to Figures 1 and 2, Figure 2 shows a flow chart of a method for training a generative adversarial network for performing image multimodal conversion according to an embodiment. The method for training a generative adversarial network for performing image multimodal conversion in Figure 2 only includes a forward loop architecture. In step S110, as shown in Figure 1, a first real image GT1 is input to a first generator G1 to obtain a first generated image FK1. The first real image GT1 has a first mode, and the first generated image FK1 has a second mode. The first real image GT1 is captured by an image sensor. The image sensor is, for example, an infrared image capture device, a photocoupler, a complementary metal oxide semiconductor optical sensing element, or any combination thereof. Taking a camera for capturing driving images as an example, it can be installed in the front, back, left, right, or any position that can capture images around the vehicle. The first mode is, for example, daytime, and the second mode is, for example, night, rainy day, sunset, snowy day, etc.

As shown in Figure 1, the first generator G1 is injected with random noise N(z) to obtain the first generated image FK1. Please refer to Figures 6 to 11. Figure 6 illustrates the original first real image GT1, which is a road condition taken during the day. Figure 7 illustrates the first generated image FK1 simulated at night after adding random noise N(z). Adding different levels of random noise N(z) can obtain different levels of first generated images FK1, such as evening or late at night.

Next, in step S120, as shown in FIG. 1, the first generated image FK1 is input to the second generator G2 to obtain the first structural reconstruction image RC1. The first structural reconstruction image RC1 has a first modality. That is, the first real image GT1 of the original first modality is converted into the first generated image FK1 of the second modality, and then converted back to the first structural reconstruction image RC1 of the first modality. Please refer to FIG. 8, which illustrates the first structural reconstruction image RC1.

Then, in step S130, as shown in FIG. 1, the first generated image FK1 is input to the first discriminator D1 to calculate a first loss value LS1. The first loss value LS1 represents whether the converted first generated image FK1 of the second modality is similar to the second real image GT2 of the second modality (indicated in FIG. 3). The second real image GT2 does not need to completely or almost overlap with the first real image GT1, as long as they are generally similar.

Next, in step S140, as shown in FIG. 1, the first generated image FK1 is input to the first image segmentation network S1 to calculate a second loss value LS2. In this step, the second loss value LS2 is calculated based on the image segmentation ground truth SG1 of the first real image GT1 and the image segmentation of the first generated image FK1. Please refer to FIG. 9, which illustrates the image segmentation ground truth SG1 of the first real image GT1. Please refer to FIG. 10, which illustrates the image segmentation of the first generated image FK1. The second loss value LS2 represents whether the structure of the first generated image FK1 is consistent or similar to the structure of the first real image GT1. Considering the second loss value LS2 during the training process of the generative adversarial network can ensure that the object position of the first generated image FK1 generated by the first generator G1 will not change or be similar.

Then, in step S150, as shown in FIG. 1, the first structured reconstructed image RC1 is input to the second discriminator D2 to calculate a third loss value LS3. The third loss value LS3 represents whether the first structured reconstructed image RC1 of the first modality after conversion is similar to the first real image GT1 of the first modality. The third loss value LS3 is considered in the training process of the generative adversarial network to ensure that the first generated image FK1 of the second modality generated by the first generator G1 can also be correctly restored to the first modality.

Next, in step S160, as shown in FIG. 1, the first structural reconstructed image RC1 is input to the second image segmentation network S2 to calculate a fourth loss value LS4. In this step, the fourth loss value LS4 is calculated based on the image segmentation truth value SG1 of the first real image GT1 and the image segmentation of the first structural reconstructed image RC1. Please refer to FIG. 11, which illustrates an example of the image segmentation result of the first structural reconstructed image RC1. The fourth loss value LS4 represents whether the structure of the first structural reconstructed image RC1 is consistent or similar to the structure of the first real image GT1. Considering the fourth loss value LS4 during the training process of the generative adversarial network can ensure that the object position of the first structural reconstructed image RC1 generated by the second generator G2 will not change or be similar. Steps S130 to S160 are not limited to the order shown in Figure 2, and can be executed simultaneously or changed to any order, as long as they are executed before step S170.

Then, in step S170, the parameters of the first generator G1 and the second generator G2 are updated to train the generative adversarial network for performing image multimodal conversion. In the implementation of the forward cycle architecture, the parameters of the first generator G1 and the second generator G2 are updated according to the first loss value LS1, the second loss value LS2, the third loss value LS3 and the fourth loss value LS4.

The above content is a forward loop architecture. In the process of training the generative adversarial network, the first loss value LS1 can ensure that the first generated image FK1 of the second modality after conversion is similar to the second real image GT2 of the second modality (marked in Figure 3). The second loss value LS2 can ensure that the object position of the first generated image FK1 generated by the first generator G1 will not change. The third loss value LS3 can ensure that the first generated image FK1 of the second modality generated by the first generator G1 can also be restored to the first modality. The fourth loss value LS4 can ensure that the object position of the first structural reconstruction image RC1 generated by the second generator G2 will not change.

As mentioned above, the combination of the forward loop architecture and the backward loop architecture can make the conversion between the first mode and the second mode quite accurate. The following will continue to explain the combination of the forward loop architecture and the backward loop architecture.

Please refer to FIG. 3, which shows a training system 200 for a generative adversarial network for performing image multimodal conversion according to another embodiment. The training system 200 for a generative adversarial network for performing image multimodal conversion in FIG. 3 includes both a forward loop architecture and a backward loop architecture. The training system 200 for a generative adversarial network for performing image multimodal conversion includes a first generator G1, a second generator G2, a first discriminator D1, a second discriminator D2, a first image segmentation network S1, and a second image segmentation network S2. The upper half of FIG. 3 is a forward loop architecture, and the lower half of FIG. 3 is a backward loop architecture. The forward loop architecture is the same as described above and will not be repeated here. The operation of each component is described in detail below through a flow chart.

Please refer to Figures 3-4. Figure 4 shows a flow chart of a training method for a generative adversarial network for performing image multimodal conversion according to another embodiment. The training method for a generative adversarial network for performing image multimodal conversion in Figure 4 includes both a forward loop architecture and a backward loop architecture. Steps S110-S160 are the forward loop architecture, and steps S210-S270 are the backward loop architecture. The forward loop architecture of steps S110-S160 is the same as described above, and will not be repeated here.

In step S210, the second real image GT2 is input to the second generator G2 to obtain a second generated image FK2. The second real image GT2 has a second mode, and the second generated image FK2 has a first mode. The second real image GT2 is captured by an image sensor. The image sensor is, for example, an infrared image capture device, a photocoupler, a complementary metal oxide semiconductor optical sensing element, or any combination thereof. The first mode is, for example, daytime, and the second mode is, for example, night, rainy day, sunset, snowy day, etc.

Next, in step S220, as shown in FIG. 3, the second generated image FK2 is input to the first generator G1 to obtain a second structural reconstruction image RC2. The second structural reconstruction image RC2 has a second modality. As shown in FIG. 3, the first generator G1 is injected with random noise N(z) to obtain the second structural reconstruction image RC2. That is, the second real image GT2 of the original second modality is converted into the second generated image FK2 of the first modality, and then converted back to the second structural reconstruction image RC2 of the second modality.

Then, in step S230, as shown in FIG. 3, the second generated image FK2 is input to the second discriminator D2 to calculate a fifth loss value LS5. The fifth loss value LS5 represents whether the second generated image FK2 of the first modality after conversion is similar to the first real image GT1 of the first modality.

Next, in step S240, as shown in FIG. 3, the second generated image FK2 is input to the second image segmentation network S2 to calculate a sixth loss value LS6. In this step, the sixth loss value LS6 is calculated based on the image segmentation ground truth SG2 of the second real image GT2 and the image segmentation of the second generated image FK2. The sixth loss value LS6 represents whether the structure of the second generated image FK2 is consistent or similar to the structure of the second real image GT2. The sixth loss value LS6 is considered in the training process of the generative adversarial network for performing image multimodal conversion to ensure that the object position of the second generated image FK2 generated by the second generator G2 will not change.

Then, in step S250, as shown in FIG. 3, the second structure reconstructed image RC2 is input to the first discriminator D1 to calculate a seventh loss value LS7. The seventh loss value LS7 represents whether the second structure reconstructed image RC2 of the second modality after conversion is similar to the second real image GT2 of the second modality. Considering the seventh loss value LS7 during the training process of the generative adversarial network can ensure that the second generated image FK2 of the first modality generated by the second generator G2 can also be correctly restored to the second modality.

Next, in step S260, as shown in FIG. 3, the second structured reconstructed image RC2 is input to the first image segmentation network S1 to calculate an eighth loss value LS8. In this step, the eighth loss value LS8 is calculated based on the image segmentation truth value SG2 of the second real image GT2 and the image segmentation of the second structured reconstructed image RC2. The eighth loss value LS8 represents whether the structure of the second structured reconstructed image RC2 is consistent or similar to the structure of the second real image GT2. Considering the eighth loss value LS8 during the training process of the generative adversarial network can ensure that the object position of the second structured reconstructed image RC2 generated by the first generator G1 will not change.

Then, in step S270, the parameters of the first generator G1 and the second generator G2 are updated to train the generative adversarial network for performing image multimodal conversion. In the embodiment of the forward loop architecture combined with the backward loop architecture, the parameters of the first generator G1 and the second generator G2 are updated according to the first loss value LS1, the second loss value LS2, the third loss value LS3, the fourth loss value LS4, the fifth loss value LS5, the sixth loss value LS6, the seventh loss value LS7 and the eighth loss value LS8.

The above content is a combination of a forward loop architecture and a backward loop architecture. In the process of training a generative adversarial network that performs image multimodal conversion, the first loss value LS1 can be used to ensure that the first generated image FK1 of the second modality after conversion is similar to the real image of the second modality. The second loss value LS2 can be used to ensure that the object position of the first generated image FK1 generated by the first generator G1 does not change. The third loss value LS3 can be used to ensure that the first generated image FK1 of the second modality generated by the first generator G1 can also be restored to the first modality. The fourth loss value LS4 can be used to ensure that the object position of the first structural reconstruction image RC1 generated by the second generator G2 does not change. The fifth loss value LS5 can be used to ensure that the second generated image FK2 of the first modality after conversion is similar to the first real image GT1 of the first modality. The sixth loss value LS6 can ensure that the structure of the second generated image FK2 is consistent or similar to the structure of the second real image GT2. The seventh loss value LS7 can ensure that the second structure reconstructed image RC2 of the second modality after conversion is similar to the second real image GT2 of the second modality. The eighth loss value LS8 can ensure that the structure of the second structure reconstructed image RC2 is consistent or similar to the structure of the second real image GT2.

The following further explains the contents and calculation methods of the first loss value LS1, the second loss value LS2, the third loss value LS3, the fourth loss value LS4, the fifth loss value LS5, the sixth loss value LS6, the seventh loss value LS7 and the eighth loss value LS8 in detail through the network architecture diagram.

。生成器G_x的功能在於將第一模態轉換為第二模態。鑑別器D_x負責判斷轉換後的第一生成影像

是否與真實的第二真實影像y相似。 Please refer to Figure 5, which shows the network architecture of the generative adversarial network for performing image multimodal transformation. The first real image x of the first modality is input to the encoder _Ex and the generator _Gx , and after adding random noise N(z), the first generated image of the second modality is output.

The function of the generator _Gx is to convert the first mode into the second mode. The discriminator _Dx is responsible for judging the first generated image after the conversion.

Is it similar to the real second real image y?

In the forward cycle, the first loss value LS1 is calculated as follows:

In equation (1), the goal of the generator _Gx in network learning is to reduce the loss, and the goal of the discriminator _Dx is to maximize the loss. For example, if the discriminator _Dx works perfectly, _Dx (y) in equation (1) will output 1, that is, log _Dx (y)=0. For the data _Gx ( _Ex (x,z),z) generated by the noise-injected generator _Gx , _Dx ( _Gx ( _Ex (x,z),z))=0, so log(1- _Dx ( _Gx ( _Ex (x,z),z))=0.

。 The calculation method of formula (1) can also be used in the same way to calculate the seventh loss value LS7 using the same generator G _x and discriminator D _x in the backward loop architecture. The equation is:

.

輸入至編碼器E_x及生成器G_x，並加入隨機雜訊N(z)後，輸出第二模態之第二結構重建影像y_rec。使用相同生成器G_x與鑑別器D_x所產生的第一損失值LS1與第七損失值LS7主要的差別在於，前者是判斷轉換後之第二模態的第一生成影像

是否夠真實，後者是判斷退後循環架構中所重建之第二模態的第二結構重建影像y_rec是否夠真實。 Among them, the second generated image of the first mode

After inputting to the encoder _Ex and the generator _Gx and adding random noise N(z), the second structure reconstructed image y _rec of the second mode is output. The main difference between the first loss value LS1 and the seventh loss value LS7 generated by the same generator _Gx and discriminator _Dx is that the former is the first generated image of the second mode after the judgment conversion.

Whether it is realistic enough, the latter is to judge whether the second structure reconstructed image y _rec of the second mode reconstructed in the backward loop framework is realistic enough.

Similarly, in the backward loop architecture, the fifth loss value LS5 generated by the generator G _y and the discriminator D _y is calculated as shown in the following formula (2).

。生成器G_y的功能在於將第二模態轉換為第一模態。鑑別器D_y負責判斷轉換後的第二生成影像

是否與真實的第一真實影像x相似。值得一提的是，在第二模態轉換至第一模態時，並不需要加入隨機雜訊N(z)。 The second real image y of the second modality is input to the encoder E _y and the generator G _y , and then the second generated image of the first modality is output.

The function of the generator G _y is to convert the second mode into the first mode. The discriminator D _y is responsible for judging the second generated image after the conversion.

Is it similar to the real first real image x? It is worth mentioning that when the second mode is converted to the first mode, there is no need to add random noise N(z).

。 The calculation method of formula (2) can also be used in the forward loop architecture to calculate the third loss value LS3 using the same generator G _y and discriminator D _y . The equation is:

.

In addition, the calculation method of the second loss value LS2 of the forward loop structure is as follows.

輸入編碼器

及分割器P_y後，獲得影像分割

，第一真實影像x之影像分割真值(Ground-Truth)

與影像分割

利用比對函數l_mce進行比對。假始生成器G_x轉換的第一生成影像

真實且結構與第一真實影像x一致或近似，則第二損失值LS2則會相當的小。 As shown in Figure 5, the first generated image

Input Encoder

After the segmentation device P _y , the image segmentation is obtained.

, the ground-truth value of the first ground-truth image x

Image segmentation

The comparison function l _mce is used for comparison. The first generated image transformed by the initial generator G _x

If the image x is real and its structure is consistent or similar to the first real image x, the second loss value LS2 will be quite small.

Similarly, the calculation method of the sixth loss value LS6 of the backward loop structure is as follows (4).

輸入編碼器

及分割器P_x後，獲得影像分割

，第二真實影像y之影像分割真值(Ground-Truth)

與影像分割

進行比對。假始生成器G_y轉換的第二生成影像

真實且結構與第二真實影像y一致，則第六損失值LS6則會相當的小。 As shown in Figure 5, the second generated image

Input Encoder

After the segmentation device P _x , the image segmentation is obtained.

, the ground-truth value of the second ground-truth image y

Image segmentation

The second generated image transformed by the initial generator G _y

If the image y is real and its structure is consistent with the second real image y, the sixth loss value LS6 will be quite small.

In addition, the calculation method of the fourth loss value LS4 in the forward loop structure is as follows (5).

及分割器P_x後，獲得影像分割

，第一真實影像x之影像分割真值(Ground-Truth)

與影像分割

進行比對，假始結構一致或近似，則第四損失值LS4則會相當的小。 As shown in FIG. 5 , the first structurally reconstructed image x _rec is input to the encoder.

After the segmentation device P _x , the image segmentation is obtained.

, the ground-truth value of the first ground-truth image x

Image segmentation

If the initial structures are consistent or similar, the fourth loss value LS4 will be quite small.

Similarly, the calculation method of the eighth loss value LS8 of the backward loop structure is as follows (6).

Finally, the entire network objective function is

,

,D_x與D_y。 The goal of the entire network optimization is to find the best G _x , G _y , P _x , P _y , _Ex , E _y ,

,

,D _x and D _y .

After the entire network is trained, the same first-modality image and different random noise N(z) are input into the generator G _x to generate several second-modality images with the same structure as the first-modality image, where the overall brightness of the second-modality images and the brightness of the car lights in the images are different.

The technology disclosed herein can be used for offline training of vehicle object detection/semantic segmentation/instance segmentation models, and these models can be implemented using machine learning and deep learning technologies.

According to the above description, this case has at least the following advantages: The generative adversarial network in this case can transform various modes of varying degrees (such as driving weather), and the position of the markers in the image will not change or will be similar.

The generated images in this case will not change or approximate the marking positions, so the markings of the original real images can be used continuously, thus greatly reducing the cost of manual marking.

In addition, this case can convert a real image and its labeling into several generated images of other modalities. These generated images of other modalities can be used to train or improve the recognition rate of object detection/semantic segmentation/instance segmentation models in other modalities.

In summary, although the present invention has been disclosed as an embodiment, it is not intended to limit the present invention. Those with common knowledge in the technical field to which the present invention belongs can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of the patent application attached hereto.

100: Training system for generative adversarial networks for image multimodal transformation

D1: First discriminator

D2: Second discriminator

FK1: First generated image

G1: First generator

G2: Second generator

GT1: The first real image

LS1: First loss value

LS2: Second loss value

LS3: Third loss value

LS4: Fourth loss value

N(z): Random noise

RC1: First structural reconstruction image

S1: The first image segmentation network

S2: Second image segmentation network

SG1: The image segmentation truth of the first real image

Claims (26)

A training method for a generative adversarial network for performing image multimodal conversion, wherein the generative adversarial network is used for performing structure-preserving image multimodal conversion, and the training method for the generative adversarial network for performing image multimodal conversion includes: inputting a first real image to a first generator to obtain a first generated image, wherein the first real image has a first mode, and the first generated image has a second mode; the first generator inputs the first generated image to a second generator to obtain a first structural reconstruction image, wherein the first structural reconstruction image has the first mode; the first generator inputs the first generated image to a first discriminator to calculate a first loss value; the first generator inputs the first generated image to a first discriminator to calculate a first loss value; The first generator inputs the first structure reconstructed image to a second discriminator to calculate a third loss value; the second generator inputs the first structure reconstructed image to a second image segmentation network to calculate a fourth loss value; and the first generator and the second generator update the parameters of the first generator and the second generator to train the generative adversarial network for performing image multimodal conversion, the parameters of the first generator and the second generator are updated at least according to the first loss value, the second loss value, the third loss value and the fourth loss value, and the updated generative adversarial network is used for performing structure-preserving image multimodal conversion so that the expanded image does not need to be re-labeled. A method for training a generative adversarial network for performing image multimodal conversion as described in claim 1, wherein the first loss value is calculated based on a second real image. A method for training a generative adversarial network for performing multimodal image conversion as described in claim 1, wherein the second loss value is calculated based on a true image segmentation value of the first true image. A method for training a generative adversarial network for performing image multimodal transformation as described in claim 1, wherein the third loss value is calculated based on the first true image. A method for training a generative adversarial network for performing multimodal image conversion as described in claim 1, wherein the fourth loss value is calculated based on a true image segmentation value of the first true image. The training method of the generative adversarial network for performing image multimodal conversion as described in claim 1 further includes: inputting a second real image to the second generator to obtain a second generated image, the second real image has the second modality, and the second generated image has the first modality; inputting the second generated image to the first generator to obtain a second structural reconstruction image, the second structural reconstruction image has the second modality; inputting the second generated image to the second discriminator to calculate a fifth loss value; inputting the The second generated image is input to the second image segmentation network to calculate a sixth loss value; the second structure reconstructed image is input to the first discriminator to calculate a seventh loss value; and the second structure reconstructed image is input to the first image segmentation network to calculate an eighth loss value; wherein the parameters of the first generator and the second generator are updated according to the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, the sixth loss value, the seventh loss value and the eighth loss value. A method for training a generative adversarial network for performing image multimodal conversion as described in claim 6, wherein the fifth loss value is calculated based on the first true image. A method for training a generative adversarial network for performing image multimodal conversion as described in claim 6, wherein the sixth loss value is calculated based on an image segmentation true value of the second real image. A method for training a generative adversarial network for performing image multimodal conversion as described in claim 6, wherein the seventh loss value is calculated based on the second real image. A method for training a generative adversarial network for performing image multimodal conversion as described in claim 6, wherein the eighth loss value is calculated based on an image segmentation truth value of the second real image. A method for training a generative adversarial network for performing multimodal image conversion as described in claim 1, wherein the first mode and the second mode include daytime, nighttime, rainy days, and snowy days. A method for training a generative adversarial network for performing image multimodal conversion as described in claim 1, wherein the first image segmentation network and the second image segmentation network are semantic segmentation networks. A method for training a generative adversarial network for performing image multimodal conversion as described in claim 1, wherein the first generator further obtains the first generated image based on a random noise. A training system for a generative adversarial network for performing image multimodal conversion, wherein the generative adversarial network performs structure-preserving image multimodal conversion, and the training system for the generative adversarial network for performing image multimodal conversion comprises: a first generator, receiving a first real image to obtain a first generated image, wherein the first real image has a first modality, and the first generated image has a second modality; a second generator, receiving the first generated image to obtain a first structural reconstruction image, wherein the first structural reconstruction image has the first modality; a first discriminator, receiving the first generated image to calculate a first loss value; a second A second discriminator receives the first structural reconstruction image to calculate a second loss value; a first image segmentation network receives the first generated image to calculate a third loss value; and a second image segmentation network receives the first structural reconstruction image to calculate a fourth loss value; wherein the parameters of the first generator and the second generator are updated at least according to the first loss value, the second loss value, the third loss value and the fourth loss value to train the generative adversarial network for performing image multimodal conversion, and the updated generative adversarial network is used for performing structure-preserving image multimodal conversion so that the augmented image does not need to be re-labeled. A training system for a generative adversarial network that performs image multimodal transformation as described in claim 14, wherein the first loss value is calculated based on a second real image. A training system for a generative adversarial network that performs image multimodal transformation as described in claim 14, wherein the second loss value is calculated based on the first real image. A training system for a generative adversarial network that performs image multimodal transformation as described in claim 14, wherein the third loss value is calculated based on a true image segmentation value of the first true image. A training system for a generative adversarial network that performs multimodal image conversion as described in claim 14, wherein the fourth loss value is calculated based on a true image segmentation value of the first real image. A training system for a generative adversarial network that performs image multimodal conversion as described in claim 14, wherein the second generator further receives a second real image to obtain a second generated image, the second real image has the second modality, and the second generated image has the first modality; the first generator further receives the second generated image to obtain a second structural reconstruction image, the second structural reconstruction image has the second modality; the second discriminator further receives the second generated image to calculate a fifth loss value; the first discriminator further receives the second structural reconstruction image The second image segmentation network further receives the second generated image to calculate a seventh loss value; and the first image segmentation network further receives the second structural reconstruction image to calculate an eighth loss value; wherein the parameters of the first generator and the second generator are updated according to the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, the sixth loss value, the seventh loss value and the eighth loss value to train the generative adversarial network that performs image multimodal conversion. A training system for a generative adversarial network that performs image multimodal transformation as described in claim 19, wherein the fifth loss value is calculated based on the first real image. A training system for a generative adversarial network that performs image multimodal transformation as described in claim 19, wherein the sixth loss value is calculated based on the second real image. A training system for a generative adversarial network that performs image multimodal transformation as described in claim 19, wherein the seventh loss value is calculated based on a true image segmentation value of the second real image. A training system for a generative adversarial network that performs image multimodal transformation as described in claim 19, wherein the eighth loss value is calculated based on a true image segmentation value of the second real image. A training system for a generative adversarial network that performs multimodal image conversion as described in claim 14, wherein the first mode and the second mode include daytime, nighttime, sunny days, rainy days, and snowy days. A training system for a generative adversarial network for performing image multimodal transformation as described in claim 14, wherein the first image segmentation network and the second image segmentation network are semantic segmentation networks. A training system for a generative adversarial network that performs image multimodal transformation as described in claim 14, wherein the first generator further obtains the first generated image based on a random noise.