CN116030365A - Model training method, apparatus, computer device, storage medium, and program product - Google Patents

Abstract

The present application relates to a model training method, device, computer equipment, storage medium and program product. The method includes: acquiring a sample low-resolution image, a sample visible light image, and a sample high-resolution image; inputting the sample low-resolution image to The first adversarial generator of the first adversarial network in the image conversion model generates a pseudo visible light image; the pseudo visible light image is input to the first auxiliary generator to obtain a first pseudo low resolution image; using the pseudo visible light image, the The first pseudo low-resolution image, the sample low-resolution image and the sample visible light image are used to train the first confrontation network; the sample low-resolution image, the pseudo visible light image and the sample high-resolution image, and train the second confrontation network in the image conversion model. The image conversion model trained by this method can convert low-resolution images into clear high-resolution images.

Description

Model training method, device, computer equipment, storage medium and program product

technical field

The present application relates to the technical field of artificial intelligence, in particular to a model training method, device, computer equipment, storage medium and program product.

Background technique

Power systems play a vital role in social production and daily life. Multi-angle and multi-period inspections of the power system help us monitor the operating status of power equipment, help us accurately understand the operation and maintenance of each link of the power system, and facilitate timely maintenance by operation and maintenance personnel. To inspect the power system, in addition to using the image acquisition equipment on the drone to take pictures of the power system to obtain images during the day, it is also necessary to use infrared imagers to obtain images of the power system through infrared imaging technology at night.

The images obtained by drone photography or infrared imaging technology are low-resolution images with low resolution and need to be processed. Using traditional models to process low-resolution images, the resulting high-resolution images are not clear enough, so improvements are urgently needed.

Contents of the invention

Based on this, it is necessary to provide a model training method, device, computer equipment, storage medium and program product for the above technical problems.

In a first aspect, the present application provides a model training method. The methods include:

Obtain sample low-resolution images, sample visible light images and sample high-resolution images;

The sample low-resolution image is input to the first confrontation generator of the first confrontation network in the image conversion model to generate a pseudo visible light image;

inputting the pseudo visible light image to a first auxiliary generator to obtain a first pseudo low resolution image;

training the first adversarial network by using the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image and the sample visible light image;

The second confrontation network in the image conversion model is trained by using the sample low-resolution image, the pseudo-visible light image, and the sample high-resolution image.

In one of the embodiments, the training of the first adversarial network by using the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image and the sample visible light image includes:

training a first adversarial discriminator of the first adversarial network by using the pseudo visible light image and the sample visible light image;

The first adversarial generator is trained by using the first pseudo low-resolution image and the sample low-resolution image.

In one of the embodiments, the training of the first adversarial generator by using the first pseudo low-resolution image and the sample low-resolution image includes:

Training the first adversarial generator according to the sample low-resolution image based on a loss function;

determining a norm loss based on the first pseudo low-resolution image and the sample low-resolution image;

Using the norm loss, the trained first adversarial generator is optimized.

In one of the embodiments, the training of the second confrontation network in the image conversion model by using the sample low-resolution image, the pseudo-visible light image and the sample high-resolution image includes:

Fusing the low-resolution image of the sample with the pseudo-visible light image to obtain a fused image;

The fusion image is input to the second confrontation generator of the second confrontation network in the image conversion model to generate a pseudo high-resolution image;

The second adversarial network is trained by using the pseudo high-resolution image and the sample high-resolution image.

In one of the embodiments, the method also includes:

Inputting the pseudo high-resolution image to a second auxiliary generator to obtain a second pseudo low-resolution image;

Verifying the rationality of the trained second adversarial generator of the second adversarial network according to the second pseudo low-resolution image and the sample low-resolution image.

In one of the embodiments, the first adversarial generator of the first adversarial network is a variational autoencoder; the second adversarial generator of the second adversarial network is a residual network.

In a second aspect, the present application also provides a device for optimizing a target detection model. The devices include:

An image acquisition module, configured to acquire a sample low-resolution image, a sample visible light image and a sample high-resolution image;

The first generation module is used to input the sample low-resolution image to the first confrontation generator of the first confrontation network in the image conversion model to generate a pseudo visible light image;

The second generating module is configured to input the pseudo-visible light image to the first auxiliary generator to obtain a first pseudo-low-resolution image;

A first training module, configured to train the first adversarial network by using the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image and the sample visible light image;

The second training module is configured to use the sample low-resolution image, the pseudo-visible light image, and the sample high-resolution image to train the second confrontation network in the image conversion model.

In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Obtain sample low-resolution images, sample visible light images and sample high-resolution images;

The sample low-resolution image is input to the first confrontation generator of the first confrontation network in the image conversion model to generate a pseudo visible light image;

inputting the pseudo visible light image to a first auxiliary generator to obtain a first pseudo low resolution image;

training the first adversarial network by using the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image and the sample visible light image;

The second confrontation network in the image conversion model is trained by using the sample low-resolution image, the pseudo-visible light image, and the sample high-resolution image.

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

Obtain sample low-resolution images, sample visible light images and sample high-resolution images;

The sample low-resolution image is input to the first confrontation generator of the first confrontation network in the image conversion model to generate a pseudo visible light image;

inputting the pseudo visible light image to a first auxiliary generator to obtain a first pseudo low resolution image;

training the first adversarial network by using the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image and the sample visible light image;

The second confrontation network in the image conversion model is trained by using the sample low-resolution image, the pseudo-visible light image, and the sample high-resolution image.

In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

Obtain sample low-resolution images, sample visible light images and sample high-resolution images;

The sample low-resolution image is input to the first confrontation generator of the first confrontation network in the image conversion model to generate a pseudo visible light image;

inputting the pseudo visible light image to a first auxiliary generator to obtain a first pseudo low resolution image;

training the first adversarial network by using the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image and the sample visible light image;

The second confrontation network in the image conversion model is trained by using the sample low-resolution image, the pseudo-visible light image, and the sample high-resolution image.

In the above model training method, device, computer equipment, storage medium and program product, a pseudo-visible light image is generated by inputting the sample low-resolution image into the first confrontation generator of the first confrontation network, and the pseudo-visible light image is input into the first auxiliary generator , to obtain the first pseudo-resolved image. Use the pseudo visible light image, the first pseudo resolution image, the sample low resolution image and the sample visible light image to train the first confrontation network; use the sample low resolution image, pseudo visible light image and sample high resolution image to train the second in the image conversion model The adversarial network is trained. The image conversion model trained through this training can convert low-resolution images into clear high-resolution images.

Description of drawings

Fig. 1 is the flowchart of model training method in an embodiment;

Fig. 2 is a flowchart of training the first confrontation network in one embodiment;

Fig. 3 is a flowchart of training the second confrontation network in one embodiment;

Fig. 4 is the flowchart of model training method in another embodiment

Fig. 5 is a structural block diagram of a model training device in an embodiment;

Fig. 6 is a structural block diagram of a model training device in another embodiment;

Fig. 7 is a structural block diagram of a model training device in yet another embodiment;

Figure 8 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

In one embodiment, as shown in FIG. 1 , a model training method is provided, which can be applied to a terminal or a server. Wherein, the terminal may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and the like. The server can be implemented by an independent server or a server cluster composed of multiple servers. Taking this method applied to the server as an example to illustrate, it specifically includes the following steps:

S101. Acquire a sample low-resolution image, a sample visible light image, and a sample high-resolution image.

Specifically, the sample image set can be obtained by collecting the image of the sample through the image acquisition device on the UAV or the tower base camera. Each sample image set can contain multiple sample groups, and each sample group includes a sample low-resolution image, a sample visible light image, and a sample high-resolution image; the three sample images included in each sample group correspond to The scene is the same.

S102. Input the sample low-resolution image to the first adversarial generator of the first adversarial network in the image conversion model to generate a pseudo visible light image.

In this embodiment, the so-called image conversion model is a model for converting a low-resolution image into a high-resolution model; optionally, the image conversion model includes two confrontation networks; wherein, the first confrontation network is used to convert Low resolution images are converted to visible light images.

Specifically, the sample low-resolution image is input to the first adversarial generator of the first adversarial network, and the first adversarial generator generates a pseudo visible light image based on the pixel probability distribution of the sample low-resolution image. Optionally, the sample low-resolution images in each sample group correspond to a pseudo visible light image.

S103. Input the pseudo visible light image to the first auxiliary generator to obtain a first pseudo low-resolution image.

In this embodiment, the first auxiliary generator is used to assist in training the first adversarial network, and further, is used to assist in training the first adversarial generator in the first adversarial network.

Specifically, the first adversarial discriminator of the first adversarial network can be used to discriminate the pseudo visible light image corresponding to each sample group from the sample visible light discrimination image in each sample group, and obtain the visible light image of the sample identified by the first adversarial discriminator. Pseudo visible light images with large differences; the acquired pseudo visible light images are input to the first auxiliary generator, and the first auxiliary generator generates a first pseudo resolution image based on the pixel probability distribution of the pseudo visible light images.

S104. Train the first adversarial network by using the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image, and the sample visible light image.

Specifically, the training loss can be determined based on the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image and the sample visible light image according to a preset loss function, and the training loss is used to train the first adversarial network.

S105, using the sample low-resolution image, the pseudo-visible light image, and the sample high-resolution image to train the second adversarial network in the image conversion model.

Optionally, the second confrontation network in the image conversion model is used to convert the visible light image into a high-resolution image. The second confrontation network includes a second confrontation generator and a second confrontation discriminator; further, the first confrontation generator is different from the second confrontation generator, and optionally, the first confrontation generator of the first confrontation network is a variation Autoencoder; Second Adversarial Network The second adversarial generator is a residual network. Variational autoencoders are generative network structures based on variational Bayesian inference. Unlike the traditional autoencoder, which describes the latent space numerically, it describes the observations of the latent space probabilistically. The residual network is easy to optimize, and can increase the accuracy by increasing the depth. The internal residual block uses skip connections, which alleviates the gradient disappearance problem caused by increasing the depth in the deep neural network.

Specifically, according to a preset loss function, the training loss can be determined based on the sample low-resolution image, the pseudo-visible light image and the sample high-resolution image, and the second adversarial network in the image conversion model can be trained.

It can be understood that after the first adversarial network and the second adversarial network are trained, a trained image conversion model can be obtained. Furthermore, the target low-resolution image collected in real time can be input into the trained image conversion model to obtain a cleaned high-resolution image.

In the above model training method, the sample low-resolution image is input into the first adversarial generator of the first adversarial network to generate a pseudo-visible light image, and the pseudo-visible light image is input into the first auxiliary generator to obtain a first pseudo-resolution image. Use the pseudo-visible light image, the first pseudo-resolution image, the sample low-resolution image and the sample visible light image to train the first confrontation network; use the sample low-resolution image, pseudo-visible light image and sample high-resolution image to train the second in the image conversion model The adversarial network is trained. The image conversion model trained through this training can convert low-resolution images into clear high-resolution images.

In one of the embodiments, step S104 is further refined, which may specifically include the following steps:

S201. Train the first adversarial discriminator of the first adversarial network by using the pseudo visible light image and the sample visible light image.

Specifically, based on a preset loss function, the training loss is determined according to the pseudo visible light image and the sample visible light image; the training loss is used to train the first adversarial discriminator.

Among them, the loss function can be:

中，

代表伪可见光图像，真值接近1，由于1-1＝0，

无限逼近负数，即第一对抗判别器往loss的方向优化。Among them, log1 is close to 0, x is the sample visible light image, and logD ₁ (x) is close to 0; in

middle,

Represents a pseudo-visible light image, the true value is close to 1, because 1-1=0,

Infinitely approaching negative numbers, that is, the first confrontation discriminator is optimized in the direction of loss.

S202. Train the first adversarial generator by using the first pseudo low-resolution image and the sample low-resolution image.

Optionally, based on the loss function, the first adversarial generator can be trained according to the sample low-resolution image; the norm loss is determined according to the first pseudo low-resolution image and the sample low-resolution image; The trained first adversarial generator is optimized. Among them, the loss function is defined as follows:

Among them, KL represents the divergence, x represents the sample low-resolution image, z represents sampling, pn is the prior distribution of the image, q _v is a parameterized distribution used to approximate the real situation, generally replaced by a normal distribution, and L represents sampling points.

Specifically, the sample low-resolution image can be substituted into the loss function, and the first adversarial generator can be optimized for the purpose of minimizing the loss function; then, based on the norm loss function (such as the L1 norm loss function), according to the first pseudo-low Distinguish the image and the sample low-resolution image, determine the norm loss; use the norm loss to adjust the network parameters of the trained first adversarial generator.

It can be understood that, in this embodiment, the first auxiliary generator is introduced to optimize the first adversarial generator, so that the image output by the trained first adversarial generator is closer to the input.

In this embodiment, the first adversarial discriminator of the first adversarial network is trained by using the pseudo visible light image and the sample visible light image, and the first adversarial generator is trained by using the first pseudo low resolution image and the sample low resolution image For training, a specific method for training the first adversarial network is given.

In one of the embodiments, step S105 is further refined, as shown in FIG. 3 , which may specifically include the following steps:

S301. Fusion the sample low-resolution image and the pseudo-visible light image to obtain a fusion image.

Specifically, when fusing the sample low-resolution image and the pseudo visible light image, the texture of the fused image can be adjusted based on the sample low-resolution image, so that the fused image contains more texture information. The resulting fused image has the same brightness as the sample low-resolution image and has the same gradient as the pseudo-visible image.

Among them, the fused image is expressed as:

代表基础层图像；in,

Represents the base layer image;

代表细节层图像。

Represents a detail layer image.

The texture information of the fused image is as follows:

Specifically, _IV represents the pseudo-visible light image, b represents the label of the sample low-resolution image (the real image is 1, and the unlabeled one is 0), and D(I _v ) represents the recognition result of _IV .

Among them, the content loss of the fused image is as follows:

Among them, where H and W represent the height and width of the input image, ||·||F represents the matrix norm, If f is the infrared image, ^I _i is the visible light image, Δ is the derivation symbol, which means to calculate the gradient of the image.

S302. Input the fused image to the second adversarial generator of the second adversarial network in the image conversion model to generate a pseudo high-resolution image.

Optionally, the second adversarial generator is mainly used to reconstruct the resolution of the fused image, and then input the given low-resolution fused image into the second adversarial generator, and the second adversarial generator restores the into corresponding pseudo high-resolution images.

S303, using the pseudo high-resolution image and the sample high-resolution image to train the second adversarial network.

Specifically, the second adversarial discriminator in the second adversarial network can be trained by using the pseudo high-resolution image and the sample high-resolution image, combined with the loss function shown in the following formula (6).

Among them, V and C represent the channel size and quantity of the characteristic spectrum respectively, and ||·||||||F represents the Frobenius norm. I ^HR represents the sample high-resolution image, and I ^SR represents the pseudo-high-resolution image.

Further, the second adversarial generator in the second adversarial network of this embodiment may be a pre-trained residual network, which can be used directly.

Optionally, the pseudo-high-resolution image can be input to the second auxiliary generator to obtain the second pseudo-low-resolution image; according to the second pseudo-low-resolution image and the sample low-resolution image, verify the second trained adversarial network. Rationality against generators. Among them, the second auxiliary generator and the second adversarial generator have a network inversion relationship, and the network parameters are the same, but the gradient update parameter time will be different.

Specifically, the rationality of the second adversarial generator can be judged by comparing the similarity between the second pseudo low-resolution image and the sample low-resolution image. If the similarity between the second pseudo low-resolution image and the sample low-resolution image is greater than a threshold, then It shows that the second adversarial generator is unreasonable. If the similarity between the second pseudo low-resolution image and the sample low-resolution image is less than the threshold, it means that the second adversarial generator is reasonable. Furthermore, when the second adversarial generator is unreasonable, the second adversarial generator can be re-optimized.

In this embodiment, the fused image is obtained by fusing the sample low-resolution image and the pseudo visible light image, and the fused image is input to the second adversarial generator of the second adversarial network in the image conversion model to generate a pseudo high-resolution image, Then, the pseudo high-resolution image and the sample high-resolution image are used to train the second adversarial network, and the specific method of training the second adversarial network is given.

In one embodiment, as shown in Figure 4, a preferred example of a model training method is provided, and the specific implementation process may include:

S401. Acquire a sample low-resolution image, a sample visible light image, and a sample high-resolution image.

S402. Input the sample low-resolution image to the first adversarial generator of the first adversarial network in the image conversion model to generate a pseudo visible light image.

S403. Input the pseudo visible light image to the first auxiliary generator to obtain a first pseudo low resolution image.

S404. Train the first adversarial discriminator of the first adversarial network by using the pseudo visible light image and the sample visible light image.

S405. Train the first adversarial generator by using the first pseudo low-resolution image and the sample low-resolution image.

S406. Fusion the sample low-resolution image and the pseudo-visible light image to obtain a fusion image.

S407. Input the fused image to the second adversarial generator of the second adversarial network in the image conversion model to generate a pseudo high-resolution image.

S408, using the pseudo high-resolution image and the sample high-resolution image to train the second confrontation network.

It should be understood that although the steps in the flow charts involved in the above embodiments are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flow charts involved in the above embodiments may include multiple steps or stages, and these steps or stages are not necessarily executed at the same time, but may be executed at different times, The execution order of these steps or stages is not necessarily performed sequentially, but may be performed in turn or alternately with other steps or at least a part of steps or stages in other steps.

Based on the same inventive concept, an embodiment of the present application further provides an object detection model optimization device for implementing the above-mentioned method for optimizing an object detection model. The solution to the problem provided by the device is similar to the implementation described in the above method, so the specific limitations in the embodiment of the optimization device for one or more target detection models provided below can be referred to above for the target detection model The limitation of the optimization method will not be repeated here.

In one embodiment, as shown in the figure, a model training device 1 is provided, as shown in Figure 5, comprising: an image acquisition module 10, a first generation module 20, a second generation module 30, a first training module 40, Two training modules 50, wherein:

An image acquisition module 10, configured to acquire a sample low-resolution image, a sample visible light image and a sample high-resolution image;

The first generation module 20 is used to input the sample low-resolution image to the first confrontation generator of the first confrontation network in the image conversion model to generate a pseudo visible light image;

The second generating module 30 is configured to input the pseudo-visible light image to the first auxiliary generator to obtain a first pseudo-low-resolution image;

The first training module 40 is used to train the first confrontation network by using the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image and the sample visible light image;

The second training module 50 is used to train the second adversarial network in the image conversion model by using the sample low-resolution image, the pseudo-visible light image and the sample high-resolution image.

In one of the embodiments, as shown in FIG. 6, the above-mentioned first training module 40 also includes the following units:

A discriminator training unit 41, configured to train the first adversarial discriminator of the first adversarial network by using the pseudo visible light image and the sample visible light image;

The generator training unit 42 is configured to train the first adversarial generator by using the first pseudo low-resolution image and the sample low-resolution image.

In one of the embodiments, the above-mentioned generator training unit 42 is also used for:

Based on the loss function, the first confrontation generator is trained according to the sample low-resolution image; according to the first pseudo low-resolution image and the sample low-resolution image, the norm loss is determined;

In one of the embodiments, the second training module 50 is refined on the basis of FIG. 5 or FIG. 6 . For example, on the basis of Fig. 5, the second training module 50 is refined, as shown in Fig. 7, the above-mentioned second training module 50 also includes the following units:

An image fusion unit 51, configured to fuse the sample low-resolution image and the pseudo-visible light image to obtain a fusion image;

The high-resolution generation unit 52 is used to input the fused image to the second confrontation generator of the second confrontation network in the image conversion model to generate a pseudo high-resolution image;

The second training unit 53 is configured to use the pseudo high-resolution image and the sample high-resolution image to train the second confrontation network.

In one of the embodiments, the above-mentioned model training device 1 also includes the following modules:

The third generation module is used to input the pseudo-high-resolution image to the second auxiliary generator to obtain the second pseudo-low-resolution image;

The verification module is used to verify the rationality of the second adversarial generator of the trained second adversarial network according to the second pseudo low-resolution image and the sample low-resolution image.

In one of the embodiments, the first adversarial generator of the first adversarial network is a variational autoencoder; the second adversarial generator of the second adversarial network is a residual network.

Each module in the above-mentioned model training device can be fully or partially realized by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure may be as shown in FIG. 8 . The computer device includes a processor, memory and a network interface connected by a system bus. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs and databases. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer equipment is used to store the initial height of the corrugated pipe to be tested and the height, temperature and performance change rate of the standard corrugated pipe and other related data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a model training method is realized.

Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a partial structure related to the solution of this application, and does not constitute a limitation on the computer equipment to which the solution of this application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, a computer program is stored in the memory, and the processor implements the following steps when executing the computer program:

Obtain sample low-resolution images, sample visible light images and sample high-resolution images;

Inputting the sample low-resolution image to the first confrontation generator of the first confrontation network in the image conversion model to generate a pseudo visible light image;

inputting the pseudo-visible light image to the first auxiliary generator to obtain a first pseudo-low-resolution image;

Using the pseudo-visible light image, the first pseudo-low-resolution image, the sample low-resolution image and the sample visible-light image to train the first confrontation network;

The second adversarial network in the image conversion model is trained by using sample low-resolution images, pseudo visible light images and sample high-resolution images.

In one embodiment, when the processor executes the logic of training the first confrontation network by using the pseudo-visible light image, the first pseudo-low-resolution image, the sample low-resolution image and the sample visible-light image in the computer program, the following steps are also implemented:

The first adversarial discriminator of the first adversarial network is trained by using the pseudo visible light image and the sample visible light image; the first adversarial generator is trained by using the first pseudo low resolution image and the sample low resolution image.

In one embodiment, when the processor executes the logic of training the first adversarial generator using the first pseudo low-resolution image and the sample low-resolution image in the computer program, the following steps are also implemented:

Based on the loss function, the first adversarial generator is trained according to the sample low-resolution image; the norm loss is determined according to the first pseudo low-resolution image and the sample low-resolution image; the norm loss is used to train the first adversarial The generator is optimized.

In one embodiment, when the computer program executes the logic of training the second confrontation network in the image conversion model by using the sample low-resolution image, the pseudo-visible light image and the sample high-resolution image, the following steps are also implemented:

The sample low-resolution image is fused with the pseudo-visible light image to obtain a fused image; the fused image is input to the second confrontation generator of the second confrontation network in the image conversion model to generate a pseudo-high-resolution image; the pseudo-high-resolution image and the sample High-resolution images for training the second adversarial network.

In one embodiment, the computer program also implements the following steps when executing the logic:

Input the pseudo-high-resolution image to the second auxiliary generator to obtain a second pseudo-low-resolution image; verify the rationality of the trained second confrontation network's second confrontation generator according to the second pseudo-low-resolution image and the sample low-resolution image sex.

In one embodiment, the first adversarial generator of the first adversarial network is a variational autoencoder; the second adversarial generator of the second adversarial network is a residual network.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Obtain sample low-resolution images, sample visible light images and sample high-resolution images;

Inputting the sample low-resolution image to the first confrontation generator of the first confrontation network in the image conversion model to generate a pseudo visible light image;

inputting the pseudo-visible light image to the first auxiliary generator to obtain a first pseudo-low-resolution image;

Using the pseudo-visible light image, the first pseudo-low-resolution image, the sample low-resolution image and the sample visible-light image to train the first confrontation network;

The second adversarial network in the image conversion model is trained by using sample low-resolution images, pseudo visible light images and sample high-resolution images.

In one embodiment, the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image and the sample visible light image are used in the computer program, and the logic for training the first confrontation network is executed by the processor to further implement the following steps:

The first adversarial discriminator of the first adversarial network is trained by using the pseudo visible light image and the sample visible light image; the first adversarial generator is trained by using the first pseudo low resolution image and the sample low resolution image.

In one embodiment, the first pseudo low-resolution image and the sample low-resolution image are used in the computer program, and when the logic for training the first confrontation generator is executed by the processor, the following steps are also implemented:

Based on the loss function, the first adversarial generator is trained according to the sample low-resolution image; the norm loss is determined according to the first pseudo low-resolution image and the sample low-resolution image; the norm loss is used to train the first adversarial The generator is optimized.

In one embodiment, when the logic of training the second confrontation network in the image conversion model is executed by the processor, the following steps are implemented by using the sample low-resolution image, the pseudo-visible light image and the sample high-resolution image in the computer program:

The sample low-resolution image is fused with the pseudo-visible light image to obtain a fused image; the fused image is input to the second confrontation generator of the second confrontation network in the image conversion model to generate a pseudo-high-resolution image; the pseudo-high-resolution image and the sample High-resolution images for training the second adversarial network.

In one embodiment, when the logic in the computer program is executed by the processor, the following steps are also implemented:

Input the pseudo-high-resolution image to the second auxiliary generator to obtain a second pseudo-low-resolution image; verify the rationality of the trained second confrontation network's second confrontation generator according to the second pseudo-low-resolution image and the sample low-resolution image sex.

In one embodiment, the first adversarial generator of the first adversarial network is a variational autoencoder; the second adversarial generator of the second adversarial network is a residual network.

In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

Obtain sample low-resolution images, sample visible light images and sample high-resolution images;

Inputting the sample low-resolution image to the first confrontation generator of the first confrontation network in the image conversion model to generate a pseudo visible light image;

inputting the pseudo-visible light image to the first auxiliary generator to obtain a first pseudo-low-resolution image;

Using the pseudo-visible light image, the first pseudo-low-resolution image, the sample low-resolution image and the sample visible-light image to train the first confrontation network;

The second adversarial network in the image conversion model is trained by using sample low-resolution images, pseudo visible light images and sample high-resolution images.

In one embodiment, the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image and the sample visible light image are used in the computer program, and the logic for training the first confrontation network is executed by the processor to further implement the following steps:

The first adversarial discriminator of the first adversarial network is trained by using the pseudo visible light image and the sample visible light image; the first adversarial generator is trained by using the first pseudo low resolution image and the sample low resolution image.

In one embodiment, the first pseudo low-resolution image and the sample low-resolution image are used in the computer program, and when the logic for training the first confrontation generator is executed by the processor, the following steps are also implemented:

Based on the loss function, the first adversarial generator is trained according to the sample low-resolution image; the norm loss is determined according to the first pseudo low-resolution image and the sample low-resolution image; the norm loss is used to train the first adversarial The generator is optimized.

In one embodiment, when the logic of training the second confrontation network in the image conversion model is executed by the processor, the following steps are implemented by using the sample low-resolution image, the pseudo-visible light image and the sample high-resolution image in the computer program:

The sample low-resolution image is fused with the pseudo-visible light image to obtain a fused image; the fused image is input to the second confrontation generator of the second confrontation network in the image conversion model to generate a pseudo-high-resolution image; the pseudo-high-resolution image and the sample High-resolution images for training the second adversarial network.

In one embodiment, when the logic in the computer program is executed by the processor, the following steps are also implemented:

Input the pseudo-high-resolution image to the second auxiliary generator to obtain a second pseudo-low-resolution image; verify the rationality of the trained second confrontation network's second confrontation generator according to the second pseudo-low-resolution image and the sample low-resolution image sex.

In one embodiment, the first adversarial generator of the first adversarial network is a variational autoencoder; the second adversarial generator of the second adversarial network is a residual network.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the computer programs can be stored in a non-volatile computer-readable storage medium. , when the computer program is executed, it may include the procedures of the embodiments of the above-mentioned methods. Wherein, any reference to storage, database or other media used in the various embodiments provided in the present application may include at least one of non-volatile and volatile storage. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive variable memory (ReRAM), magnetic variable memory (Magnetoresistive Random Access Memory, MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (Phase Change Memory, PCM), graphene memory, etc. The volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a blockchain-based distributed database, etc., but is not limited thereto. The processors involved in the various embodiments provided by this application can be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., and are not limited to this.

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above examples only express several implementation modes of the present application, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be determined by the appended claims.

Claims (10)

1. A method of model training, the method comprising:

acquiring a sample low-fraction image, a sample visible light image and a sample high-resolution image;

inputting the sample low-resolution image to a first countermeasure generator of a first countermeasure network in an image conversion model to generate a pseudo visible light image;

inputting the pseudo visible light image into a first auxiliary generator to obtain a first pseudo low-resolution image;

training the first countermeasure network with the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image, and the sample visible light image;

and training a second reactance network in the image conversion model by adopting the sample low-resolution image, the pseudo visible light image and the sample high-resolution image.

2. The method of claim 1, wherein training the first countermeasure network with the pseudo visible light image, the first pseudo low resolution image, the sample low resolution image, and the sample visible light image comprises:

training a first countermeasure discriminator of the first countermeasure network using the pseudo visible light image and the sample visible light image;

training the first countermeasure generator using the first pseudo low resolution image and the sample low resolution image.

3. The method of claim 2, wherein training the first countermeasure generator with the first pseudo low resolution image and the sample low resolution image comprises:

training the first countermeasure generator based on a loss function from the sample low fraction image;

determining a norm loss from the first pseudo low resolution image and the sample low resolution image;

the trained first countermeasure generator is optimized with the norm loss.

4. The method of claim 1, wherein training the second antagonism network in the image conversion model using the sample low resolution image, the pseudo visible image, and the sample high resolution image comprises:

fusing the sample low-resolution image and the pseudo visible light image to obtain a fused image;

inputting the fused image to a second countermeasure generator of a second countermeasure network in the image conversion model to generate a pseudo high-resolution image;

training the second countermeasure network with the pseudo high resolution image and the sample high resolution image.

5. The method according to claim 4, wherein the method further comprises:

inputting the pseudo high-resolution image into a second auxiliary generator to obtain a second pseudo low-resolution image;

verifying the rationality of a second countermeasure generator of the trained second countermeasure network from the second pseudo low resolution image and the sample low resolution image.

6. The method of any one of claims 1-5, wherein the first countermeasure generator of the first countermeasure network is a variational automatic encoder; the second countermeasure generator of the second countermeasure network is a residual network.

7. A model training apparatus, the apparatus comprising the following modules:

the image acquisition module is used for acquiring a sample low-fraction image, a sample visible light image and a sample high-resolution image;

the first generation module is used for inputting the sample low-resolution image into a first countermeasure generator of a first countermeasure network in the image conversion model to generate a pseudo visible light image;

the second generation module is used for inputting the pseudo visible light image into the first auxiliary generator to obtain a first pseudo low-resolution image;

the first training module is used for training the first countermeasure network by adopting the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image;

and the second training module is used for training a second reactance network in the image conversion model by adopting the sample low-resolution image, the pseudo visible light image and the sample high-resolution image.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.

9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.

Images