WO2021035979A1

WO2021035979A1 - Image filling method and apparatus based on edge learning, terminal, and readable storage medium

Info

Publication number: WO2021035979A1
Application number: PCT/CN2019/118150
Authority: WO
Inventors: 王健宗; 王义文
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-08-23
Filing date: 2019-11-13
Publication date: 2021-03-04
Also published as: CN111062877A

Abstract

Provided are an image filling method and apparatus based on edge learning, a terminal, and a readable storage medium, belonging to the field of image processing. The method comprises the following steps: determining a background grayscale image on the basis of a grayscale image corresponding to an original image and a mask image corresponding to the original image; then, determining a background edge image on the basis of an edge image corresponding to the original image and the mask image; generating a missing edge image on the basis of the background grayscale image, the background edge image, the mask image and a first preset dilated convolution model; and generating, on the basis of the background image corresponding to the original image, the background edge image, the missing edge image, the mask image and a second preset dilated convolution model, a filling image corresponding to the original image. Edge repairing of an image to be reconstructed is realized through a two-stage adversarial model, and color filling is then performed, so that a filled area better reproduces fine details of the image.

Description

Edge learning image filling method, device, terminal and readable storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on August 23, 2019, the application number is 201910784055.9, and the invention title is "Image filling method, device, terminal and readable storage medium for edge learning", all of which The content is incorporated in the application by reference.

Technical field

This application relates to the field of image processing, and in particular to an image filling method, device, terminal and readable storage medium for edge learning.

Background technique

Image restoration refers to the process of reconstructing missing or damaged parts of images and videos. For example, in museums, this work is often performed by experienced museum administrators or art restoration engineers; in the digital world, image restoration is also called image interpolation or video interpolation, which refers to the use of complex algorithms to replace lost or damaged items. Image data mainly replaces some small areas and defects.

At present, by filling an existing image patch segment or using diffusion filling based on the background, when the image is repaired, the repaired area is blurred due to excessive smoothing, and the ideal detail effect is not reconstructed.

Summary of the invention

The main purpose of this application is to provide an edge learning image filling method, device, terminal, and readable storage medium, aiming to solve the technical problem that the existing image restoration cannot reconstruct ideal details.

To achieve the above objective, the present application provides an image filling method for edge learning, and the image filling method for edge learning includes the following steps:

Determine the background grayscale image based on the grayscale image corresponding to the original image and the mask image corresponding to the original image;

Determining a background edge image based on the edge image corresponding to the original image and the mask image;

Generating a missing edge image based on the background grayscale image, the background edge image, the mask image, and the first preset cavity convolution model;

Based on the background image corresponding to the original image, the background edge image, the missing edge image, the mask image, and the second preset cavity convolution model, a filled image corresponding to the original image is generated.

In addition, in order to achieve the above object, the present application also provides a terminal, the terminal including a processor, a memory, and computer-readable instructions stored on the memory and executable by the processor, wherein the computer can When the read instruction is executed by the processor, the steps of the above-mentioned edge learning image filling method are realized.

In addition, in order to achieve the above objective, the present application also provides a readable storage medium having computer readable instructions stored on the readable storage medium, and the computer readable instructions implement any of the above The steps of the image filling method of edge learning.

This application provides an image filling method for edge learning, which determines the background grayscale image based on the grayscale image corresponding to the original image and the mask image corresponding to the original image, and then determines the background grayscale image based on the edge image corresponding to the original image and the mask image A background edge image, and then based on the background grayscale image, the background edge image, the mask image, and the first preset cavity convolution model, a missing edge image is generated, and then based on the background image corresponding to the original image , The background edge image, the missing edge image, the mask image, and the second preset cavity convolution model to generate a filled image corresponding to the original image. The two-stage confrontation model realizes the edge restoration of the image to be reconstructed, and then performs color filling, so that the filled area can better reproduce the fine details of the image.

Description of the drawings

FIG. 1 is a schematic diagram of the hardware structure of an image filling terminal for edge learning involved in a solution of an embodiment of the application;

2 is a schematic flowchart of a first embodiment of an image filling method for edge learning according to this application;

3 is a schematic flowchart of a second embodiment of an image filling method for edge learning according to this application;

Fig. 4 is a schematic diagram of functional modules of an image filling device for edge learning in this application.

detailed description

It should be understood that the specific embodiments described here are only used to explain the application, and not used to limit the application.

The edge learning image filling method involved in the embodiments of the present application is mainly applied to a terminal, and the edge terminal may be a device with display and processing functions such as a PC, a portable computer, and a mobile terminal.

Referring to FIG. 1, FIG. 1 is a schematic diagram of the hardware structure of the edge learning image filling terminal involved in the solution of the embodiment of the application. In the embodiment of the present application, the edge learning image filling terminal may include a processor 1001 (for example, a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 may include a display (Display), an input unit such as a keyboard (Keyboard); the network interface 1004 may optionally include a standard wired interface, a wireless interface (Such as WI-FI interface); the memory 1005 can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as a disk memory. The memory 1005 can optionally be a storage device independent of the aforementioned processor 1001 .

Those skilled in the art can understand that the hardware structure shown in FIG. 1 does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or a combination of certain components, or different component arrangements.

Continuing to refer to FIG. 1, the memory 1005 as a readable storage medium in FIG. 1 may include an operating system, a network communication module, and computer readable instructions.

In FIG. 1, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 can call the computer-readable instructions stored in the memory 1005, and execute the edge learning image filling method provided by the embodiment of the present application .

The embodiment of the present application provides an image filling method of edge learning.

Referring to FIG. 2, FIG. 2 is a schematic flowchart of a first embodiment of an image filling method for edge learning according to this application.

In this embodiment, the image filling method for edge learning includes the following steps:

Step S10: Determine a background grayscale image based on the grayscale image corresponding to the original image and the mask image corresponding to the original image;

In this embodiment, the original image is the image to be repaired, and there are pixel deletions in the image; grayscale images are images with only one sampled color per pixel, and this type of image is usually displayed from the darkest black to the brightest white The mask image is an image derived from the original image, that is, the pixel value of the pixel in the missing area in the original image is defined as 1, the area outside the missing area is defined as the background, and the pixel value of the pixel in the background is defined as 0 , That is to say, the value of the pixel in the mask image is either 1 or 0.

Specifically, for convenience of description, the definition of the original image represented by I _gt, gray scale image is represented by I _gray, represented by the mask image M, where, M _i = 0 represents background, M _i = 1 represents a defective area, the background gray Image use

Indicates that the background gray image is determined according to the gray image I _gray corresponding to the original image and the mask image M corresponding to the original image

The formula is as follows, where □ represents the Hadamard product.

Hadamard product (Hadamard product) is a kind of operation of matrix. If A=(aij) and B=(bij) are two matrices of the same order, if cij=aij×bij, then matrix C=(cij) is called A The Hadamard product of B and B, or the basic product.

Step S20: Determine a background edge image based on the edge image corresponding to the original image and the mask image;

In the present embodiment, for convenience of description, the definition of an edge image of the original image corresponding to C _gt, the mask image is represented by M, where, M _i = 0 represents background, M _i = 1 represents a deleted region, corresponding to the original image Edge image C _gt and mask image M, determine the background edge image

The formula is as follows, where □ represents the Hadamard product.

Step S30, training a first preset cavity convolution model based on the background grayscale image, the background edge image, and the mask image;

In this embodiment, the original image is an image with missing pixels, and the area of the missing image also includes the edge of the image. In this application, in order to repair the image, the missing edge of the image is first repaired. The first preset hollow convolution model is a convolutional neural network. The convolutional neural network mimics the visual perception mechanism of biology. It can perform supervised learning and unsupervised learning. Convolution kernel parameter sharing and layer in its hidden layer The sparsity of the connections enables the convolutional neural network to learn grid features, such as pixels and audio, with a small amount of calculation, with stable effects and no additional feature engineering requirements for the data.

Specifically, the background grayscale image, the background edge image, and the mask image are used as samples to train the first preset hole convolution model until the first preset hole convolution model converges, wherein the first preset hole volume The output of the product model is the missing edge image. Whether the missing edge image output in the training process of the first preset hole convolution model is accurate, it needs to be further discriminated by the first discriminator. The first discriminator adopts the PatchGAN architecture and only judges the structure within the patch range . For the convenience of description, the first preset hole convolution model is represented by G ₁ , and the missing edge image is represented by C _pred . The calculation formula is as follows:

among them,

Is the background grayscale image, M is the mask image, the background edge image

Step S40: Determine whether the first preset hole convolution model converges based on a first discriminator, where the terminal includes the first discriminator;

In this embodiment, whether the missing edge image output during the training process of the first preset hole convolution model is accurate needs to be further identified by the first discriminator. When the first preset hole convolution model converges, the output missing edge image is relatively accurate.

Specifically, step S40 includes:

Step a, based on the first input data and the first discriminator, determine a first countermeasure loss function corresponding to the first preset hole convolution model, wherein the first input data includes the grayscale image, The edge image and the missing edge image;

In this embodiment, to determine whether the first preset hole convolution model converges, it is necessary to calculate the loss function of the first preset hole convolution model, where the loss function of the first preset hole convolution model includes the counter loss function And feature matching function.

Specifically, for the convenience of description, the first discriminator is _{denoted by D 1} , then according to the gray image I _gray , the edge image C _gt , the missing edge image C _pred and the first discriminator D ₁ , the first adversarial loss function L _{adv is calculated , 1} , the formula is as follows:

Step b: Determine the feature matching loss function corresponding to the first preset hole convolution model based on the second input data and the first discriminator, wherein the second input data includes the edge image and the Missing edge image;

In this embodiment, for the convenience of description, the feature matching loss function is represented by L _FM . According to the edge image C _gt , the missing edge image C _pred and the first discriminator D ₁ , the feature matching loss function L _{FM is} calculated. The formula is as follows Show:

Step c: Determine the loss function of the first preset hole convolution model based on the first counter loss function and the feature matching loss function;

_{In this embodiment, the loss function L G1} of the first preset hole convolution model is further calculated according to the first confrontation loss function L _adv,1 and the feature matching loss function L _FM , and the formula is as follows:

Among them, λ _adv,1 and λ _FM are regularization parameters. Optionally, λ _adv,1 =1 and λ _FM =10.

Step d: Determine whether the first preset hole convolution model converges based on the loss function of the first preset hole convolution model.

In this embodiment, according to the first confrontation loss function L _adv,1 and the feature matching loss function L _FM _{, the loss function L G1} of the first preset hole convolution model is calculated, and according to the preset convergence judgment rule, according to the loss The function L _G1 determines whether the first preset hole convolution model converges. The preset convergence judgment rule is not limited in this application. Optionally, the preset convergence judgment rule is training on the first preset hole convolution model In the process, when the difference between the loss function of this training and the loss function of the last training is less than the preset value, it is determined that the first preset cavity convolution model has converged. If the loss function of this training is compared with the loss of the last training When the difference of the function is greater than the preset value, it is determined that the first preset cavity convolution model does not converge, the parameters of the first preset cavity convolution model need to be updated, and the updated first preset cavity convolution model continues to be performed training.

Step S50, when it is determined that the first preset cavity convolution model is converged, determine that the first preset cavity convolution model is the first target cavity convolution model;

In this embodiment, during the model training process, when the model converges, it means that the model training is finished, and the model convergence is judged according to the loss function of the model. Specifically, in the training process of the first preset hole convolution model, if the difference between the loss function of this training and the loss function of the last training is less than the preset value, it is determined that the first preset hole convolution model converges Therefore, the current first preset cavity convolution model is used as the first target cavity convolution model.

Step S60: Generate a missing edge image based on the background grayscale image, the background edge image, the mask image, and the first target cavity convolution model.

In this embodiment, the first target hole convolution model is a trained model, and the model is convergent. For the convenience of description, the first target hole convolution model uses G _F1 , then according to the background gray image

Background edge image

The mask image M and the first target hole convolution model G _F1 generate the missing edge image C _pred , the formula is as follows:

Step S70: Generate a filled image corresponding to the original image based on the background image corresponding to the original image, the background edge image, the missing edge image, the mask image, and the second preset cavity convolution model.

In this embodiment, after the missing edge image is generated according to the background grayscale image, the background edge image, the mask image, and the first target hole convolution model, the image is further filled. First, determine the overall edge according to the missing edge image and the background edge image, and then fill in the images inside and outside the edge. Specifically, according to the background image, background edge image, missing edge image, and mask image corresponding to the original image And a second preset cavity convolution model to generate a filled image corresponding to the original image.

Further, in an embodiment, after step S40, the method further includes:

Step S80, when it is determined that the first preset hole convolution model does not converge, update the first learning rate of the first preset hole convolution model according to the first preset rule;

In this embodiment, in the training process of the first preset hole convolution model, when the difference between the loss function of this training and the loss function of the last training is greater than the preset value, it is determined that the first preset hole convolution The model does not converge. At this time, it is necessary to update the parameters of the first preset hole convolution model, and continue to train the updated first preset hole convolution model. The learning rate determines the model parameters, so the learning rate of the first preset hole convolution model is updated according to the preset rule, and then the parameters of the first preset hole convolution model are adjusted.

In step S90, the first preset hole convolution model is updated based on the updated first learning rate, the updated first preset hole convolution model is used as the first preset hole convolution model, and the execution based on the The step of training a first preset cavity convolution model on the background grayscale image, the background edge image, and the mask image.

In this embodiment, the learning rate determines the model parameters, so after updating the learning rate of the first preset hole convolution model according to the preset rule, the first preset hole convolution model is further updated according to the updated first learning rate, That is, the parameters of the first preset hole convolution model are updated. Next, continue to train the first preset hole convolution model, that is, continue to perform the step of training the first preset hole convolution model based on the background grayscale image, the background edge image, and the mask image .

The edge learning image filling method proposed in this embodiment implements the edge repair of the image to be reconstructed through a two-stage confrontation model, and then performs color filling, so that the filled area can better reproduce the fine details of the image.

Based on the first embodiment, a second embodiment of the image filling method for edge learning of the present application is proposed. Referring to FIG. 3, in this embodiment, step S70 includes:

Step S71: Determine a composite edge image based on the missing edge image, the background edge image, and the mask image;

In this embodiment, after the missing edge image is generated according to the background grayscale image, the background edge image, the mask image, and the first target hole convolution model, the overall edge is continuously determined based on the missing edge image and the background edge image. Specifically, for the convenience of description, the composite edge image is represented by C _copm , and the formula is as follows, where C _gt represents the edge image corresponding to the original image, M represents the mask image, C _pred represents the missing edge image, and □ represents the Hadamard product.

C _copm ＝C _gt □(1-M)+C _pred □M

Step S72, training a second preset cavity convolution model based on the background image corresponding to the original image and the composite edge image;

In this embodiment, the second preset hole convolution model is also a kind of convolutional neural network, and its training samples are background images and composite edge images corresponding to the original image. When the second preset hole convolution model converges, the model The training stops, where the output of the second preset hole convolution model is the filled image corresponding to the original image. Whether the filled image output during the training process of the second preset hole convolution model is accurate needs to be further identified by the second discriminator. For the convenience of description, the second preset hole convolution model is G ₂ , and the filled image is represented by I _pred . The color image I _gt corresponding to the original image, I _gt and (1-M) are subjected to the Hadamard product operation to obtain the original image corresponding Background image

The calculation formula of _{I pred} for filling the image is as follows:

Step S73: Determine whether the second preset hole convolution model converges based on the second discriminator and the preset convolutional neural network;

In this embodiment, whether the filled image output during the training process of the second preset hole convolution model is accurate or not needs to be further discriminated by the second discriminator. The second discriminator adopts the PatchGAN architecture, which is only in the patch range Judge the structure within. When the second preset cavity convolution model converges, the output filled image is relatively accurate.

Specifically, step S73 includes:

Step e, based on the third input data and the second discriminator, determine a second confrontation loss function corresponding to the second preset hole convolution model;

In this embodiment, to determine whether the second preset hole convolution model converges, it is necessary to calculate the loss function of the second preset hole convolution model, where the loss function of the second preset hole convolution model includes the L1 loss function , The second confrontation loss function, perceptual loss function and style loss function. Among them, the L1 loss function is a preset value, which is determined by the actual situation. The purpose of the L1 loss function is to ensure correct scaling. The L1 loss function is based on the size of the mask image M Standardize.

Further, calculating a second function against loss, for convenience of description, the second authentication Used D ₂ represents, according to the third input data and a second discriminator determining a second predetermined cavity corresponding to a second convolutional model loss function against , Where the third input data includes the color image I _gt corresponding to the original image, the filled image I _pred and the composite edge image C _copm , and the second counter loss function L _{adv is} calculated according to the third input data and the second discriminator D ₂ _{, 2} , the formula is as follows:

Step f: Determine the perceptual loss function corresponding to the second preset hole convolution model based on the fourth input data and the preset convolutional neural network;

In this embodiment, the perceptual loss function determines the result that is not similar to the label by defining a distance metric between the activation maps of the pre-trained network. For the convenience of description, φ _{i is used to} represent the activation map of the i-th layer of the preset convolutional neural network, N _i is the number of elements in the i-th activation layer, and the fourth input data includes the color image I _gt corresponding to the original image and padding For the image I _pred _{, the perceptual loss function L perc} corresponding to the second preset hole convolution model is calculated, and the formula is as follows:

Step g: Determine a style loss function corresponding to the second preset hole convolution model based on the fifth input data and a preset convolutional neural network;

In this embodiment, the filled image I _pred and (1-M) are subjected to the Hadamard product operation to obtain

Wherein, M is the mask image, M _i = 0 represents background, M _i = 1 indicates the deleted region; the color image corresponding to the original image I _gt Hadamard product operation carried out with (1-M), to give

The fifth input parameter includes

with

_{Then calculate the style loss function L style} corresponding to the second preset hole convolution model, where,

Is the matrix constructed in the active layer φ _i , the formula is as follows:

Step h, determining the loss function of the second preset hole convolution model based on the second counter loss function, the perceptual loss function, and the style loss function;

_{In this embodiment, the loss function L G2 of the} second preset hole convolution model is further determined according to the L1 loss function, the second confrontation loss function L _adv,2 , the perceptual loss function L _perc and the style loss function L _style , where λ _L1 , λ _{adv, 2} , λ _p and λ _s are regularization parameters, the formula is as follows:

Step i: Determine whether the second preset hole convolution model converges based on the loss function of the second preset hole convolution model.

_{In this embodiment, the loss function L G2} of the second preset hole convolution model is calculated according to the L1 loss function, the second counter loss function L _adv,2 , the perceptual loss function L _perc and the style loss function L _style , according to The preset convergence judgment rule determines whether the second preset cavity convolution model converges according to the loss function L _G2 . The preset convergence judgment rule is not limited in this application. Optionally, the preset convergence judgment rule is During the training process of the second preset cavity convolution model, when the difference between the loss function of this training and the loss function of the last training is less than the preset value, it is determined that the second preset cavity convolution model has converged. When the difference between the loss function and the last training loss function is greater than the preset value, it is determined that the second preset hole convolution model does not converge, and the parameters of the second preset hole convolution model need to be updated, and the updated The second preset cavity convolution model is trained.

Step S74, when it is determined that the second preset cavity convolution model is converged, determine that the second preset cavity convolution model is a second target cavity convolution model;

In this embodiment, during the model training process, when the model converges, it means that the model training is finished, and the model convergence is judged according to the loss function of the model. Specifically, in the training process of the second preset hole convolution model, if the difference between the loss function of this training and the loss function of the last training is less than the preset value, it is determined that the second preset hole convolution model converges Therefore, the current second preset cavity convolution model is used as the second target cavity convolution model.

Step S75: Generate a filled image corresponding to the original image based on the background image corresponding to the original image, the composite edge image, and the second target cavity convolution model.

In this embodiment, the second target hole convolution model is a trained model, and the model is convergent. For the convenience of description, the second target hole convolution model is represented by G _F2 , and the color image corresponding to the original image is represented by I _gt Indicates that I _gt and (1-M) perform the Hadamard product operation to obtain the background image corresponding to the original image

According to the background image corresponding to the original image

The composite edge image C _copm and the second target hole convolution model G _{F2 are used} to generate the filling image I _pred , and the calculation formula is as follows:

Step S76: When it is determined that the second preset hole convolution model does not converge, update the second learning rate of the second preset hole convolution model according to a second preset rule;

In this embodiment, during the training process of the second preset hole convolution model, when the difference between the loss function of this training and the loss function of the last training is greater than the preset value, it is determined that the second preset hole convolution The model does not converge. At this time, it is necessary to update the parameters of the second preset hole convolution model, and continue to train the updated second preset hole convolution model. The learning rate determines the model parameters, so the learning rate of the second preset hole convolution model is updated according to the preset rule, and then the parameters of the second preset hole convolution model are adjusted.

In step S77, the second preset hole convolution model is updated based on the updated second learning rate, the updated second preset hole convolution model is used as the second preset hole convolution model, and the execution based on the second preset hole convolution model is continued. The step of training a second preset cavity convolution model on the background image corresponding to the original image and the composite edge image.

In this embodiment, the learning rate determines the model parameters, so after updating the learning rate of the second preset hole convolution model according to the preset rule, further, the second preset hole convolution model is updated according to the updated second learning rate , That is, update the parameters of the second preset hole convolution model. Next, continue to train the second preset hole convolution model, that is, continue to perform the step of determining a composite edge image based on the missing edge image, the background edge image, and the mask image.

The edge learning image filling method proposed in this embodiment realizes the color filling after the edge repair of the image to be reconstructed through the confrontation model, so that the filled area can better reproduce the fine details of the image.

In addition, the embodiment of the present application also provides an image filling device for edge learning.

Referring to FIG. 4, FIG. 4 is a schematic diagram of functional modules of an image filling device for edge learning in this application.

In this embodiment, the image filling device for edge learning includes:

The first determining module 10 is configured to determine the background grayscale image based on the grayscale image corresponding to the original image and the mask image corresponding to the original image;

The second determining module 20 is configured to determine a background edge image based on the edge image corresponding to the original image and the mask image;

The training module 30 is configured to train a first preset cavity convolution model based on the background grayscale image, the background edge image, and the mask image;

The judging module 40 is configured to determine whether the first preset hole convolution model converges based on the first discriminator;

The third determining module 50, when determining that the first preset cavity convolution model converges, determines that the first preset cavity convolution model is the first target cavity convolution model;

The first generating module 60 generates a missing edge image based on the background grayscale image, the background edge image, the mask image, and the first target cavity convolution model;

The second generating module 70 generates a filled image corresponding to the original image based on the background image corresponding to the original image, the background edge image, the missing edge image, the mask image, and the second preset cavity convolution model .

In addition, an embodiment of the present application also provides a readable storage medium, and the computer-readable storage medium may be a non-volatile readable storage medium.

The readable storage medium of the present application stores computer readable instructions, and when the computer readable instructions are executed by a processor, the steps of the above-mentioned edge learning image filling method are realized.

For the method implemented when the computer-readable instruction is executed, refer to the various embodiments of the image filling method for edge learning in this application, which will not be repeated here.

It should be noted that in this article, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, It also includes other elements that are not explicitly listed, or elements inherent to the process, method, article, or system. Without more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or system that includes the element.

The serial numbers of the foregoing embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

Through the description of the above implementation manners, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , Magnetic disks, optical disks), including several instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the method described in each embodiment of the present application.

The above are only the preferred embodiments of the application, and do not limit the scope of the patent for this application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of the application, or directly or indirectly applied to other related technical fields , The same reason is included in the scope of patent protection of this application.

Claims

An edge learning image filling method applied to a terminal, characterized in that the edge learning image filling method includes the following steps:

Determine the background grayscale image based on the grayscale image corresponding to the original image and the mask image corresponding to the original image;

Determining a background edge image based on the edge image corresponding to the original image and the mask image;

Training a first preset cavity convolution model based on the background grayscale image, the background edge image, and the mask image;

Determining whether the first preset hole convolution model converges based on a first discriminator, where the terminal includes the first discriminator;

When it is determined that the first preset cavity convolution model converges, determining that the first preset cavity convolution model is a first target cavity convolution model;

Generating a missing edge image based on the background grayscale image, the background edge image, the mask image, and the first target hole convolution model;

Based on the background image corresponding to the original image, the background edge image, the missing edge image, the mask image, and the second preset cavity convolution model, a filled image corresponding to the original image is generated.
5. The edge learning image filling method according to claim 1, wherein after the step of determining whether the first preset hole convolution model is converged based on the first discriminator, the method further comprises:

When it is determined that the first preset hole convolution model does not converge, update the first learning rate of the first preset hole convolution model according to a first preset rule;

The first preset hole convolution model is updated based on the updated first learning rate, the updated first preset hole convolution model is used as the first preset hole convolution model, and the execution based on the background gray level is continued. The step of training a first preset cavity convolution model on the image, the background edge image, and the mask image.
The method for image filling of edge learning according to claim 1, wherein the step of determining whether the first preset hole convolution model has converged based on the first discriminator comprises:

Based on the first input data and the first discriminator, a first countermeasure loss function corresponding to the first preset hole convolution model is determined, wherein the first input data includes the gray image, the edge An image and the missing edge image;

Determine the feature matching loss function corresponding to the first preset hole convolution model based on the second input data and the first discriminator, wherein the second input data includes the edge image and the missing edge image ；

Determining a loss function of the first preset cavity convolution model based on the first counter loss function and the feature matching loss function;

Determine whether the first preset hole convolution model converges based on the loss function of the first preset hole convolution model.
The image filling method for edge learning according to claim 1, wherein the terminal includes a second discriminator, and the background image corresponding to the original image, the background edge image, and the missing edge image are based on , The mask image and the second preset cavity convolution model, and the step of generating a filled image corresponding to the original image includes:

Determining a composite edge image based on the missing edge image, the background edge image, and the mask image;

Training a second preset cavity convolution model based on the background image corresponding to the original image and the composite edge image;

Determining whether the second preset cavity convolution model converges based on the second discriminator and the preset convolutional neural network;

When it is determined that the second preset cavity convolution model converges, determining that the second preset cavity convolution model is a second target cavity convolution model;

Based on the background image corresponding to the original image, the composite edge image, and the second target cavity convolution model, a filled image corresponding to the original image is generated.
The image filling method for edge learning according to claim 4, wherein after the step of determining whether the second preset hole convolution model has converged based on the second discriminator and the preset convolutional neural network, include:

When it is determined that the second preset hole convolution model does not converge, update the second learning rate of the second preset hole convolution model according to a second preset rule;

The second preset hole convolution model is updated based on the updated second learning rate, the updated second preset hole convolution model is used as the second preset hole convolution model, and the corresponding operation based on the original image is continued. The step of training a second preset cavity convolution model on the background image and the composite edge image.
The image filling method for edge learning according to claim 4, wherein the step of determining whether the second preset hole convolution model is converged based on a second discriminator and a preset convolutional neural network comprises:

Based on the third input data and the second discriminator, determine a second counter loss function corresponding to the second preset hole convolution model;

Determine the perceptual loss function corresponding to the second preset hole convolution model based on the fourth input data and the preset convolutional neural network;

Determining a style loss function corresponding to the second preset hole convolution model based on the fifth input data and a preset convolutional neural network;

Determining a loss function of the second preset hole convolution model based on the second counter loss function, the perceptual loss function, and the style loss function;

Determine whether the second preset hole convolution model converges based on the loss function of the second preset hole convolution model.
An image filling device for edge learning, characterized in that the image filling device for edge learning comprises:

The first determining module is configured to determine the background grayscale image based on the grayscale image corresponding to the original image and the mask image corresponding to the original image;

The second determining module is configured to determine the background edge image based on the edge image corresponding to the original image and the mask image;

A training module, configured to train a first preset cavity convolution model based on the background grayscale image, the background edge image, and the mask image;

The first judgment module is configured to determine whether the first preset hole convolution model converges based on the first discriminator;

A third determining module, when determining that the first preset cavity convolution model converges, determining that the first preset cavity convolution model is the first target cavity convolution model;

A first generating module, which generates a missing edge image based on the background grayscale image, the background edge image, the mask image, and the first target cavity convolution model;

The second generating module generates a filled image corresponding to the original image based on the background image corresponding to the original image, the background edge image, the missing edge image, the mask image, and the second preset cavity convolution model.
8. The image filling device for edge learning according to claim 7, wherein the image filling device for edge learning further comprises:

The first update module is configured to update the first learning rate of the first preset hole convolution model according to the first preset rule when it is determined that the first preset hole convolution model does not converge;

The first loop module is configured to update the first preset hole convolution model based on the updated first learning rate, use the updated first preset hole convolution model as the first preset hole convolution model, and continue A step of training a first preset cavity convolution model based on the background grayscale image, the background edge image, and the mask image is performed.
8. The edge learning image filling device according to claim 7, wherein the judging module comprises:

The first determining unit is configured to determine a first countermeasure loss function corresponding to the first preset hole convolution model based on the first input data and the first discriminator, wherein the first input data includes the A grayscale image, the edge image, and the missing edge image;

The second determining unit is configured to determine the feature matching loss function corresponding to the first preset hole convolution model based on the second input data and the first discriminator, wherein the second input data includes the edge An image and the missing edge image;

A third determining unit, configured to determine the loss function of the first preset hole convolution model based on the first counter loss function and the feature matching loss function;

The first determining unit is configured to determine whether the first preset hole convolution model converges based on the loss function of the first preset hole convolution model.
8. The image filling device for edge learning according to claim 7, wherein the second generating module comprises:

A fourth determining unit, configured to determine a composite edge image based on the missing edge image, the background edge image, and the mask image;

A training unit, configured to train a second preset cavity convolution model based on the background image corresponding to the original image and the composite edge image;

A second judgment unit, configured to determine whether the second preset hole convolution model converges based on the second discriminator and the preset convolutional neural network, wherein the terminal includes a second discriminator;

A fifth determining unit, configured to determine that the second preset cavity convolution model is a second target cavity convolution model when it is determined that the second preset cavity convolution model converges;

The generating unit is configured to generate a filled image corresponding to the original image based on the background image corresponding to the original image, the composite edge image, and the second target cavity convolution model.
The image filling device for edge learning according to claim 10, wherein the image filling device for edge learning further comprises:

The second update module is configured to update the second learning rate of the second preset hole convolution model according to a second preset rule when it is determined that the second preset hole convolution model does not converge;

The second loop module is configured to update the second preset hole convolution model based on the updated second learning rate, use the updated second preset hole convolution model as the second preset hole convolution model, and continue A step of training a second preset hole convolution model based on the background image corresponding to the original image and the composite edge image is performed.
9. The image filling device for edge learning according to claim 10, wherein the image filling device for edge learning further comprises:

A first calculation module, configured to determine a second countermeasure loss function corresponding to the second preset hole convolution model based on the third input data and the second discriminator;

The second calculation module is configured to determine the perceptual loss function corresponding to the second preset hole convolution model based on the fourth input data and a preset convolutional neural network;

The third calculation module is configured to determine the style loss function corresponding to the second preset hole convolution model based on the fifth input data and a preset convolutional neural network;

A fourth calculation module, configured to determine the loss function of the second preset hole convolution model based on the second confrontation loss function, the perceptual loss function, and the style loss function;

The second judgment module is configured to determine whether the second preset hole convolution model converges based on the loss function of the second preset hole convolution model.
A terminal, characterized in that the terminal includes a processor, a memory, and computer readable instructions stored on the memory and executable by the processor, wherein the computer readable instructions are When the processor executes, the following steps are implemented:

Determine the background grayscale image based on the grayscale image corresponding to the original image and the mask image corresponding to the original image;

Determining a background edge image based on the edge image corresponding to the original image and the mask image;

Training a first preset cavity convolution model based on the background grayscale image, the background edge image, and the mask image;

Determining whether the first preset hole convolution model converges based on a first discriminator, where the terminal includes the first discriminator;

When it is determined that the first preset cavity convolution model converges, determining that the first preset cavity convolution model is a first target cavity convolution model;

Generating a missing edge image based on the background grayscale image, the background edge image, the mask image, and the first target hole convolution model;

Based on the background image corresponding to the original image, the background edge image, the missing edge image, the mask image, and the second preset cavity convolution model, a filled image corresponding to the original image is generated.
The terminal according to claim 13, wherein after the step of determining whether the first preset hole convolution model is converged based on the first discriminator, the method further comprises:

When it is determined that the first preset hole convolution model does not converge, update the first learning rate of the first preset hole convolution model according to a first preset rule;

The first preset hole convolution model is updated based on the updated first learning rate, the updated first preset hole convolution model is used as the first preset hole convolution model, and the execution based on the background gray level is continued. The step of training a first preset cavity convolution model on the image, the background edge image, and the mask image.
The terminal according to claim 13, wherein the step of determining whether the first preset hole convolution model has converged based on the first discriminator comprises:

Based on the first input data and the first discriminator, a first countermeasure loss function corresponding to the first preset hole convolution model is determined, wherein the first input data includes the gray image, the edge An image and the missing edge image;

Determine the feature matching loss function corresponding to the first preset hole convolution model based on the second input data and the first discriminator, wherein the second input data includes the edge image and the missing edge image ；

Determining a loss function of the first preset cavity convolution model based on the first counter loss function and the feature matching loss function;

Determine whether the first preset hole convolution model converges based on the loss function of the first preset hole convolution model.
The terminal according to claim 13, wherein the terminal comprises a second discriminator, the background image corresponding to the original image, the background edge image, the missing edge image, and the mask The image and the second preset cavity convolution model, and the steps of generating a filled image corresponding to the original image include:

Determining a composite edge image based on the missing edge image, the background edge image, and the mask image;

Training a second preset cavity convolution model based on the background image corresponding to the original image and the composite edge image;

Determining whether the second preset cavity convolution model converges based on the second discriminator and the preset convolutional neural network;

When it is determined that the second preset cavity convolution model converges, determining that the second preset cavity convolution model is a second target cavity convolution model;

Based on the background image corresponding to the original image, the composite edge image, and the second target cavity convolution model, a filled image corresponding to the original image is generated.
The terminal according to claim 16, wherein after the step of determining whether the second preset hole convolution model has converged based on the second discriminator and the preset convolutional neural network, the method further comprises:

When it is determined that the second preset hole convolution model does not converge, update the second learning rate of the second preset hole convolution model according to a second preset rule;

The second preset hole convolution model is updated based on the updated second learning rate, the updated second preset hole convolution model is used as the second preset hole convolution model, and the corresponding operation based on the original image is continued. The step of training a second preset cavity convolution model on the background image and the composite edge image.
A readable storage medium, characterized in that computer readable instructions are stored on the readable storage medium, and when the computer readable instructions are executed by a processor, the following steps are implemented:

Determine the background grayscale image based on the grayscale image corresponding to the original image and the mask image corresponding to the original image;

Determining a background edge image based on the edge image corresponding to the original image and the mask image;

Training a first preset cavity convolution model based on the background grayscale image, the background edge image, and the mask image;

Determining whether the first preset hole convolution model converges based on a first discriminator, where the terminal includes the first discriminator;

When it is determined that the first preset cavity convolution model converges, determining that the first preset cavity convolution model is a first target cavity convolution model;

Generating a missing edge image based on the background grayscale image, the background edge image, the mask image, and the first target hole convolution model;

Based on the background image corresponding to the original image, the background edge image, the missing edge image, the mask image, and the second preset cavity convolution model, a filled image corresponding to the original image is generated.
18. The computer-readable storage medium of claim 18, wherein the step of obtaining contract information corresponding to the issuance request when the issuance request is received comprises: the determining of the issuance based on the first discriminator After the step of whether the first preset hole convolution model converges, it further includes:

When it is determined that the first preset hole convolution model does not converge, update the first learning rate of the first preset hole convolution model according to a first preset rule;

The first preset hole convolution model is updated based on the updated first learning rate, the updated first preset hole convolution model is used as the first preset hole convolution model, and the execution based on the background gray level is continued. The step of training a first preset cavity convolution model on the image, the background edge image, and the mask image.
18. The computer-readable storage medium according to claim 18, wherein after the step of determining whether the first preset hole convolution model is converged based on the first discriminator, the method further comprises:

When it is determined that the first preset hole convolution model does not converge, update the first learning rate of the first preset hole convolution model according to a first preset rule;

The first preset hole convolution model is updated based on the updated first learning rate, the updated first preset hole convolution model is used as the first preset hole convolution model, and the execution based on the background gray level is continued. The step of training a first preset cavity convolution model on the image, the background edge image, and the mask image.