JP2023543964A - Image processing method, image processing device, electronic device, storage medium and computer program - Google Patents

Abstract

The present disclosure provides an image processing method, an image processing device, an electronic device, and a storage medium, and relates to the field of artificial intelligence, especially the field of computer vision and deep learning, and can be applied to scenes such as facial image processing and face identification. . The specific technical proposals are as follows. That is, an image to be processed is generated based on the first target image and the second target image, and the identity information of the object in the image to be processed is matched with the identity information of the object in the first target image. , generate a decoupling image set based on the second target image and the image to be processed, where the decoupling image set includes a head decoupling image corresponding to a head region of the object in the image to be processed and the image to be processed; generating a fused image based on the decoupling image set, including an inpainting decoupling image corresponding to to-be-inpainted information associated with the object in the fused image, where identity information and texture of the object in the fused image; The information is matched with the identity information and texture information of the object in the image to be processed, respectively, and the information to be repaired associated with the object in the fused image is repaired.

Description

cross reference

This application claims priority to the Chinese patent application filed on August 25, 2021 and with application number 202110985605.0, the entire contents of which are incorporated by reference into this application.

The present disclosure relates to the field of artificial intelligence technology, especially the field of computer vision and deep learning, and can be applied to scenes such as facial image processing and face identification. Specifically, the present invention relates to an image processing method, an image processing device, an electronic device, and a storage medium.

With the development of the Internet and the development of artificial intelligence technology centered on deep learning, computer vision technology has been widely applied in various fields.

Research on facial images of objects is one of the important research topics in the field of computer vision because objects can reflect their inner feelings and convey interaction information through rich facial expressions. . Along with this, research on image replacement technology that combines and transforms facial images of objects into images is also emerging. Image replacement is applied to various scenes such as movie editing or virtual characters.

The present disclosure provides an image processing method, an image processing device, an electronic device, and a storage medium.

According to one aspect of the present disclosure, an image processing method is provided. The image processing method generates an image to be processed based on a first target image and a second target image, wherein identity information of an object in the image to be processed is the identity of an object in the first target image. matching the texture information of the object in the image to be processed with the texture information of the object in the second target image, and based on the second target image and the image to be processed, a decoupling image. a head decoupling image corresponding to a head region of an object in the image to be processed and information to be repaired associated with the object in the image to be processed; and generating a fused image based on the decoupled image set, wherein identity information and texture information of an object in the fused image each correspond to an identity of an object in the image to be processed. information and texture information and that the information to be repaired that is related to the object in the fused image is repaired.

According to another aspect of the present disclosure, an image processing device is provided. The image processing device generates an image to be processed based on a first target image and a second target image, wherein identity information of an object in the image to be processed is the identity of an object in the first target image. a first generation module that matches the texture information of the object in the image to be processed with the texture information of the object in the second target image, based on the second target image and the image to be processed; generating a decoupling image set, wherein the decoupling image set includes a head decoupling image corresponding to a head region of an object in the to-be-processed image and to-be-repaired information associated with the object in the to-be-processed image; a second generation module comprising a corresponding repaired decoupled image and a fused image based on the decoupled image set, wherein the identity information and texture information of an object in the fused image are respectively associated with the image to be processed; and a third generation module in which the information to be repaired related to the object in the fused image is repaired.

According to another aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, and the instructions executing the at least one processor. By being executed by one processor, the at least one processor can perform the method.

According to another aspect of the disclosure, a non-transitory computer-readable storage medium having computer instructions stored thereon causes the computer to perform the method described above.

According to another aspect of the present disclosure, a computer program product is provided that includes a computer program that, when executed by a processor, implements the method described above.

It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the disclosure or to limit the scope of the disclosure. Other features of the disclosure will be readily understood from the following description.

The drawings are used to better understand the technical solution and are not intended to limit the disclosure.
FIG. 1 schematically depicts an exemplary system architecture in which an image processing method and apparatus according to embodiments of the present disclosure can be applied. FIG. 2 schematically shows a flowchart of an image processing method according to an embodiment of the present disclosure. FIG. 3 schematically shows a schematic diagram of a process of generating an image to be processed according to an embodiment of the present disclosure. FIG. 4 schematically shows a schematic diagram of an image processing process according to an embodiment of the present disclosure. FIG. 5 schematically shows a block diagram of an image processing device according to an embodiment of the present disclosure. FIG. 6 schematically shows a block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of the present disclosure.

The following describes exemplary embodiments of the present disclosure with reference to the drawings and herein includes various details of embodiments of the present disclosure for ease of understanding and which are to be considered as exemplary. It is. Accordingly, those skilled in the art will appreciate that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for the sake of clarity and brevity, well-known functions and structures will not be described in the following description.

In the process of realizing the concept of the present disclosure, image replacement is realized by face replacement, that is, replacing the five face senses, ignoring other information other than the face area, such as head information and skin color information, and head information is , hair and head type, etc. As a result, the identity similarity of the image after replacement is low, which further affects the replacement effect of image replacement.

The following example can explain a case where the identity similarity of the images after replacement is easily caused to be low. For example, it is necessary to replace the head area of object a in image A with the head area of object b in image B. The skin color of object b is black, and the skin color of object a is yellow. If skin color information is ignored when replacing the five facial senses, a situation will arise in which the five facial senses of the object in the replaced image are yellow and the skin color of the face is black, and the identity similarity of the replaced image will be low. .

To this end, embodiments of the present disclosure provide a technical solution for generating a fusion result with high similarity of identity information through multi-stage head replacement fusion. That is, an image to be processed is generated based on the first target image and the second target image, a decoupling image set is generated based on the second target image and the image to be processed, and the decoupling image set is Based on this, the object's identity information and texture information are matched with the object's identity information and texture information in the image to be processed, respectively, and a fused image in which the information to be repaired is repaired is generated. Since the information to be repaired related to the object in the fused image has been repaired, identity similarity in the fused image is improved, and the replacement effect of image replacement is further improved.

FIG. 1 schematically depicts an exemplary system architecture in which an image processing method and apparatus according to embodiments of the present disclosure can be applied.

It should be noted that FIG. 1 is only an example to which the system architecture of the embodiments of the present disclosure may be applied, and is helpful for those skilled in the art to understand the technical content of the present disclosure; The examples are not meant to imply that they cannot be used in other devices, systems, environments, or scenes. For example, in another embodiment, an example system architecture in which the image processing method and apparatus may be applied may include a terminal device, but the terminal device does not need to interact with a server, and embodiments of the present disclosure It is possible to realize the image processing method and apparatus provided by.

As shown in FIG. 1, a system architecture 100 according to the embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is used as a medium to provide a communication link between terminal equipment 101 , 102 , 103 and server 105 . Network 104 may include various connection types, such as wired and/or wireless communication links.

A user can receive or send messages, etc. by interacting with a server 105 over a network 104 using terminal equipment 101 , 102 , 103 . The terminal devices 101, 102, 103 are equipped with various communication client applications, such as knowledge browsing applications, web page browser applications, search applications, real-time communication tools, mailbox clients, and/or social platform software (to name a few). May be installed.

The terminal devices 101, 102, 103 may be various electronic devices that have a display and support viewing web pages, including, but not limited to, smartphones, tablet computers, laptop computers, desktop computers, and the like. Not limited.

The server 105 may be a server that provides various services, such as providing a background management server (by way of example only) that supports content viewed by users on the terminal devices 101, 102, 103. The background management server performs processing such as analysis on data such as received user requests, and sends processing results (for example, web pages, information, or data acquired or generated in response to user requests) to the terminal device. Can give feedback.

The server 105 may be a cloud server, called a cloud computing server or a cloud host, and is one of the host products of the cloud computing service system, which is a traditional physical host and a VPS service (Virtual Private Server, VPS). ), which is highly difficult to manage and has low service expandability. The server 105 may be a distributed system server or a blockchain coupled server.

It should be noted that the image processing method provided by the embodiments of the present disclosure can generally be performed by the terminal device 101, 102, or 103. Accordingly, the image processing device provided by the embodiment of the present disclosure may be installed in the terminal device 101, 102, or 103.

Alternatively, the image processing methods provided by embodiments of the present disclosure may be generally performed by the server 105. Accordingly, image processing devices provided by embodiments of the present disclosure may generally be installed on the server 105. Image processing methods provided by embodiments of the present disclosure may be performed by a server or server cluster that is different from server 105 and that is capable of communicating with terminal devices 101 , 102 , 103 and/or server 105 . Accordingly, the image processing device provided by the embodiments of the present disclosure may be installed in a server or server cluster that is different from the server 105 and that is capable of communicating with the terminal devices 101 , 102 , 103 and/or the server 105 .

For example, the server 105 generates an image to be processed based on the first target image and the second target image, and the identity information of the object in the image to be processed matches the identity information of the object in the first target image. , the texture information of the object in the image to be processed is matched with the texture information of the object in the second target image, and a decoupling image set is generated based on the second target image and the image to be processed; includes a head decoupling image corresponding to the head region of the object in the image to be processed and a repaired decoupling image corresponding to information to be repaired related to the object in the image to be processed, and based on the decoupled image set, A fused image is generated, the object's identity information and texture information in the fused image are matched with the object's identity information and texture information in the image to be processed, respectively, and the information to be repaired related to the object in the fused image is repaired. . Alternatively, a server or server cluster capable of communicating with the terminal devices 101, 102, 103 and/or the server 105 generates an image to be processed based on the first target image and the second target image, and generates the image to be processed based on the second target image and the second target image. A decoupled image set is generated based on the images to be processed, and a fused image is generated based on the decoupled image set.

It should be understood that the number of terminal equipment, networks and servers in FIG. 1 is merely exemplary. It can have any number of terminal devices, networks and servers as required.

FIG. 2 schematically shows a flowchart of an image processing method according to an embodiment of the present disclosure.

As shown in FIG. 2, the method 200 includes operations S210-S230.

In operation S210, an image to be processed is generated based on the first target image and the second target image, where the identity information of the object in the image to be processed is matched with the identity information of the object in the first target image. , the texture information of the object in the image to be processed is matched with the texture information of the object in the second target image.

In operation S220, a decoupling image set is generated based on the second target image and the image to be processed, where the decoupling image set is a head decoupling image corresponding to a head region of the object in the image to be processed. A repair decoupling image corresponding to the image and the information to be repaired related to the object in the image to be processed is included.

In operation S230, a fused image is generated based on the decoupled image set, where the identity information and texture information of the object in the fused image are matched with the identity information and texture information of the object in the image to be processed, respectively, and The information to be repaired related to the object in the fused image has been repaired.

According to embodiments of the present disclosure, the first target image can be understood to be an image that provides identity information of the first object, and the second target image can be understood as an image that provides texture information of the second object. It can be understood that The texture information may include facial texture information, and the facial texture information may include at least one of facial posture information and facial expression information. In the first target image, the object can be understood as a first object, and in the second target image, the object can be understood as a second object. If it is necessary to replace the texture information of the object in the first target image with the texture information of the object in the second target image, the first target image can be called a driven image and the second target image can be called a driving image.

According to embodiments of the present disclosure, the number of first target images may include one or more. The first target image may be a video frame in a video or a still image. The second target image may be a video frame in a video or a still image. For example, the number of first target images can include a plurality, and the identity information of the object in the plurality of first target images is the same.

According to embodiments of the present disclosure, the image to be processed is such that the object identity information matches the object identity information in the first target image, and the object texture information matches the object texture information in the second target image. That is, the object in the image to be processed is the first object, and the texture information of the object is the texture information of the second object.

According to embodiments of the present disclosure, the decoupling image set may include a head decoupling image and a repair decoupling image. A head decoupled image can be understood as an image corresponding to the head region of an object in the image to be processed, i.e. obtained by extracting relevant features of the head region of the object from the image to be processed. It is an image. A repair decoupling image can be understood as an image containing information to be repaired about objects in the image to be processed. The information to be repaired may include at least one of skin color information and missing information. The skin color information may include the skin color of the face.

According to embodiments of the present disclosure, a fused image can be understood as an image obtained after completing an inpainting operation on the information to be inpainted, and objects in the fused image are the same as objects in the image to be processed. That is, the identity information of the object in the fused image matches the identity information of the object in the image to be processed, and the texture information of the object matches the texture information of the object in the image to be processed.

According to embodiments of the present disclosure, a first target image and a second target image are obtained, the first target image and the second target image are processed, an image to be processed is obtained, a second target image and the second target image are obtained, and the second target image and the second target image are processed. Processing the exponent images to obtain a decoupled image set; processing the decoupled image set to obtain a fused image. Processing the first target image and the second target image to obtain the image to be processed includes extracting the object's identity information from the first target image, extracting the object's texture information from the second target image, and extracting the object's identity information from the second target image. and obtaining an image to be processed based on the texture information.

According to an embodiment of the present disclosure, a fused image is generated based on a decoupled image set, and since information to be repaired related to an object in the fused image is repaired, identity similarity in the fused image is improved; Furthermore, the replacement effect of image replacement is improved.

According to embodiments of the present disclosure, the repaired decoupled image includes a first decoupled image and a second decoupled image. The identity information of the object in the first decoupled image is matched with the identity information of the object in the image to be processed, and the skin color information of the object in the first decoupled image is matched with the skin color information of the object in the second target image. The second decoupling image is a difference image between the head area of the object in the image to be processed and the head area of the object in the second target image. The information to be repaired related to the object in the fused image is repaired because the skin color information of the object in the fused image matches the skin color information of the object in the second target image, and the pixel value of the pixel in the difference image is set in advance. indicates that the specified conditions are met.

According to embodiments of the present disclosure, in order to improve the replacement effect of image replacement, the skin color information of the object in the image to be processed is matched with the skin color information of the object in the driving image (i.e., the second target image); The missing region between the head region of the object in the image to be processed and the head region of the object in the second target image needs to be repaired.

According to embodiments of the present disclosure, the first decoupling image matches the skin color information of the object in the image to be processed and the skin color information of the object in the second target image. can play a role. The first decoupling image may be a mask image of the five senses of the face having complexion.

According to embodiments of the present disclosure, the second decoupling image serves to repair the missing region between the head region of the object in the image to be processed and the head region of the object in the second target image. I can do it. The second decoupling image can be understood as a difference image, which may be a difference image between the head area of the object in the image to be processed and the head area of the object in the second target image. . The difference image may be a mask image.

According to an embodiment of the present disclosure, the difference image includes a plurality of pixels, each pixel has a corresponding pixel value, and the fact that the pixel value of a pixel point in the difference image meets a preset condition means that The histogram distribution of the multiple pixel values matches the preset histogram distribution, the average variance of the multiple pixel values is less than or equal to the preset variance threshold, and the sum of the multiple pixel values is less than or equal to the preset threshold. can include one of the following.

According to an embodiment of the present disclosure, the head decoupling images include a third decoupling image, a fourth decoupling image, and a fifth decoupling image. The third decoupled image includes a tone image of the head region of the object in the image to be processed. The fourth decoupled image includes a binarized image of the head region of the object in the image to be processed. The fifth decoupled image includes an image obtained from the second target image and the fourth decoupled image.

According to an embodiment of the present disclosure, the fourth decoupling image is a binary image of the head area of the object in the image to be processed, that is, a binary image of the background and foreground of the head area of the object in the image to be processed. A value mask image may be included. The fifth decoupling image may be a difference image between the second target image and the fourth decoupling image. The fifth decoupling image can be understood to be an image obtained by subtracting the head area of the object in the second target image and then placing the head area of the object in the fourth decoupling image in the subtraction area. .

According to embodiments of the present disclosure, generating a decoupled image set based on a second target image and an image to be processed includes generating a decoupled image set based on a second target image and an image to be processed. This may include obtaining. A second decoupled image is obtained based on the second target image and the image to be processed. A third decoupled image is obtained based on the image to be processed. A fourth decoupled image is obtained based on the image to be processed. A fifth decoupled image is obtained based on the second target image and the fourth decoupled image.

According to embodiments of the present disclosure, generating a fused image based on a decoupled image set may include the following operations.

A fusion model is utilized to process the decoupled image set to obtain a fusion image, where the fusion model includes a generator in a first generative adversarial network model.

According to the embodiment of the present disclosure, the fusion model restores information to be restored, thereby more naturally merging the fused image obtained using the fusion model with the background of the virtual person. The fusion model decouples the skin color information of the object in the second target image, the head area of the object in the image to be processed, and the background information in the second target image to realize skin color matching and repair of the missing area in the image. Skin color matching is used to change the skin color information of the object in the image to be processed to the skin color information of the object in the second target image, and repairing the image of the missing area is to change the skin color information of the object in the image to be processed to the skin color information of the object in the image to be processed. By setting the pixel value of a pixel in the difference image between the area and the head area of the object in the second target image, the pixel value matches a preset condition.

According to an embodiment of the present disclosure, the fusion model may be a model obtained by training using deep learning. The fusion model may include a generator in the first generative adversarial network model, ie, the generator in the first generative adversarial network model is utilized to process the decoupled image set to obtain the fusion model.

According to embodiments of the present disclosure, the generative adversarial network model may include a deep convolutional generative adversarial network model, a bulldozer distance-based generative adversarial network model, a conditional generative adversarial network model, or the like. The generative adversarial network model may include a generator and a discriminator. The generator and discriminator can include neural network models. Neural network models can include Unet models. The Unet model can include two symmetric parts, namely, the front part model is the same as the general convolutional network model, which includes a convolution layer and a downsampling layer, and contains the context information in the image (i.e. between relationships) can be extracted. The back part model is basically symmetrical with the front part and includes a convolution layer and an upsampling layer, thereby achieving the purpose of segmenting the output image. In addition, the Unet model further utilizes feature fusion, that is, fuses the features of the downsampling part of the front part and the features of the upsampling part of the rear part to obtain more accurate context information and achieve a better segmentation effect. achieve.

According to embodiments of the present disclosure, the generator of the first generative adversarial network model may include a Unet model.

According to an embodiment of the present disclosure, the fusion model is obtained by training in the following manner: obtaining a first sample image set, the first sample image set including a plurality of first sample images; . Each first sample image is processed to obtain a sample decoupled image set. A first generative adversarial network model is trained using the plurality of sample decoupled image sets to obtain a trained first generative adversarial network model. A generator in the trained first generative adversarial network model is determined as a fusion model. The sample decoupling image set may include a head decoupling image corresponding to a head region of the object in the first sample image and a repair decoupling image corresponding to information to be repaired related to the object in the first sample image. .

According to embodiments of the present disclosure, training a first adversarial generative network model using a plurality of sample decoupled image sets and obtaining a trained first adversarial generative network model comprises: processing each sample decoupled image set of the plurality of sample decoupled image sets using a generator in the generative network model to obtain a sample fused image corresponding to each sample decoupled image set. . A generator and a discriminator in the first generative adversarial network model are alternately trained based on the plurality of sample fused images and the first set of sample images to obtain a trained first generative adversarial network model.

According to embodiments of the present disclosure, the head decoupling image corresponding to the head region of the object in the first sample image may include a first sample decoupling image and a second sample decoupling image. The identity information of the object in the first sample decoupled image corresponds to the identity information of the object in the first sample image, and the skin color information of the object in the first sample decoupled image corresponds to preset skin color information. The second sample decoupled image is a difference image between the head area of the object in the first sample image and a preset head area.

According to an embodiment of the present disclosure, the repaired decoupled image corresponding to the information to be repaired about the object in the first sample image includes the third sample decoupled image, the fourth sample decoupled image, and the fifth sample decoupled image. can be included. The third sample decoupled image may include a tone image of the head region of the object in the first sample image. The fourth sample decoupled image may include a binarized image of the head region of the object in the first sample image. The fifth sample decoupled image may include an image obtained from the fourth sample decoupled image.

According to embodiments of the present disclosure, the fusion model is trained using a first identity information loss function, a first image feature matching loss function, a first discriminant feature matching loss function, and a first discriminator loss function. It is.

According to embodiments of the present disclosure, an identity information loss function is used to achieve identity information matching. An image feature matching loss function can be used to achieve matching of texture information. The discriminant feature matching loss function can be used to match texture information in the discriminator space as much as possible. The classifier loss function is used to ensure that the generated images have high resolution as much as possible.

According to embodiments of the present disclosure, the identity information loss function can be determined based on equation (1) below.

The image feature matching loss function can be determined based on equation (2) below.

The discriminant feature matching loss function can be determined based on the following equation (3).

The discriminator loss function can be determined based on the following equation (4).

According to embodiments of the present disclosure, a first identity information loss function is used to achieve matching of the object's identity information in the first sample image and the object's identity information in the sample fused image. The first image feature matching loss function can be used to achieve matching between the texture information of the object in the first sample image and the texture information of the object in the sample fused image. The first discriminant feature matching loss function can be used to achieve matching between the texture information of the object in the first sample image in the classifier space and the texture information of the object in the sample fused image. The first discriminator loss function is used to ensure as much as possible that the sample fused images have high resolution.

According to embodiments of the present disclosure, generating an image to be processed based on the first target image and the second target image may include the following operations.

Processing the first target image using an identity extraction module in the driving model to obtain identity information of the object in the first target image. A texture extraction module in the driving model is used to process the second target image to obtain texture information of the object in the second target image. A stitching module in the driving model is used to process identity information and texture information to obtain stitching information. The stitching information is processed by a generator in the driving model to obtain an image to be processed.

According to embodiments of the present disclosure, the driving model decouples the identity information of the object in the first target image and the texture information of the object in the second target image, and decouples the object's identity information in the first target image and the object's texture information in the second target image. used to complete face replacement.

According to embodiments of the present disclosure, the driving model may include an identity extraction module, a texture extraction module, a stitching module, and a generator. The driving model generator may be a second generative adversarial network model generator. The identity extraction module is used to extract identity information of an object. A texture extraction module may be used to extract texture information of an object. The stitching module is used to stitch identity information and texture information. A driving model generator is used to generate a fused image based on the stitching information.

According to embodiments of the present disclosure, the identity extraction module may be a first encoder, the texture extraction module may be a second encoder, and the stitching module is an MLP (Multilayer Perceptron). There may be. The first encoder and the second encoder may include a Visual Geometry Group (VGG) model.

According to embodiments of the present disclosure, the stitching information includes a plurality, and the generator of the driving model includes N cascaded deep units, where N is an integer greater than 1.

Processing the stitching information by a generator in the driving model and obtaining an image to be processed may include the following operations.

For the i-th deep unit among the N deep units, the i-th level jump information corresponding to the i-th deep unit is processed using the i-th deep unit, and the i-th level feature information is processed. , where the i-th level jump information includes the (i-1)-th level feature information and the i-th level stitching information, where i is greater than 1 and less than or equal to N. An image to be processed is generated based on the Nth level feature information.

According to embodiments of the present disclosure, the driving model generator may include N cascaded deep units. Deep units at each level have corresponding stitching information. Different level deep units are used to extract different deep features of the image. The input of a deep unit at each level may include two parts: feature information corresponding to a deep unit above the deep unit at the level and stitching information corresponding to the deep unit at the level.

According to an embodiment of the present disclosure, a driving model is trained and obtained in the following manner. That is, a second sample image set and a third sample image set are obtained, the second sample image set including a plurality of second sample images, and the third sample image set including a plurality of third sample images. Processing the second sample image using an identity extraction module to obtain identity information of objects in the second sample image. Processing the third sample image using a texture extraction module to obtain texture information of the object in the third sample image. Processing the identity information of the object in the second sample image and the texture information of the object in the third sample image using a stitching module to obtain sample stitching information, and using a generator to generate the sample stitching information. Process and obtain simulation images. A trained driving model is obtained by training an identity extraction module, a texture extraction module, a splicing module, and a second generative adversarial network model using the second sample image set and the simulation image set.

According to embodiments of the present disclosure, the driving model utilizes a second identity information loss function, a second target image feature matching loss function, a second discriminant feature matching loss function, a second discriminator loss function, and a cyclic matching loss function. This is what they were trained to do.

According to embodiments of the present disclosure, a second identity information loss function is used to achieve matching of the object's identity information in the second sample image and the object's identity information in the simulation image. The second image feature matching loss function can be used to achieve matching between the texture information of the object in the second sample image and the texture information of the object in the simulation image. The second discriminant feature matching loss function can be used to achieve matching between the texture information of the object in the second sample image in the discriminator space and the texture information of the object in the simulation image. The second classifier loss function is used to ensure that the simulated images have high resolution as much as possible. A circular matching loss function can be used to improve the driving model's ability to retain texture information of objects in the third sample image.

According to embodiments of the present disclosure, the cyclic matching loss function is determined based on the actual result and the predicted result generated by the driving model, where the actual result includes actual identity information and actual texture information of the object in the actual image. , the prediction result includes predicted identity information and predicted texture information of the object in the simulation image.

According to an embodiment of the present disclosure, the actual identity information of the object in the actual image can be understood as the identity information of the object in the second sample image mentioned above. The actual texture information of the object in the actual image can be understood as the texture information of the object in the third sample image described above.

According to embodiments of the present disclosure, the cyclic matching loss function can be determined based on equations (5) to (7) below.

According to embodiments of the present disclosure, the image processing method may further include the following operations.

Enhancement processing is performed on the fused image to obtain an enhanced image.

According to an embodiment of the present disclosure, in order to improve the resolution of the fused image, resolution enhancement processing is performed on the fused image to obtain the enhanced image, so that the resolution of the enhanced image is greater than the resolution of the fused image.

According to embodiments of the present disclosure, performing an enhancement process on a fused image and obtaining an enhanced image may include the following operations.

The fused image is processed using an enhanced model to obtain an enhanced image, where the enhanced model includes a generator in a third generative adversarial network model.

According to embodiments of the present disclosure, an enhanced model is used to improve the resolution of an image. The reinforcement model may include a generator in a third generative adversarial network model. The third generative adversarial network model may include a PSFR (Progressive Semantic-Aware Style)-GAN.

The image processing method according to the embodiment of the present disclosure will be further described below with reference to FIGS. 3 to 4 and specific embodiments.

FIG. 3 schematically shows a schematic diagram of a process of generating an image to be processed according to an embodiment of the present disclosure.

As shown in FIG. 3, in the process 300, the first target image set 301 includes a first target image 3010, a first target image 3011, a first target image 3012, and a first target image 3013. The driving model includes an identity extraction module 303, a texture extraction module 305, a stitching module 307 and a generator 309.

The identity extraction module 303 is used to process the first target image set 301 and extract identity information 3040 of the object in the first target image 3010, identity information 3041 of the object in the first target image 3011, and object identity information 3041 of the object in the first target image 3012. The object identity information 3042 and the object identity information 3043 in the first target image 3013 are acquired. Based on the identity information 3040, the identity information 3041, the identity information 3042, and the identity information 3043, the average identity information 304 is obtained, and the average identity information 304 is determined as the identity information 304 of the first target image.

The second target image 302 is processed using the texture extraction module 305 to obtain texture information 306 of the object in the first target image 302 .

The stitching module 307 is used to process the identity information 304 and texture information 306 to obtain a stitching information set 308, which includes stitching information 3080, stitching information 3081, and stitching information 3082. include.

The stitching information set 308 is processed using the generator 309 to obtain an image 310 to be processed. The identity information of the object in the image to be processed 310 is matched with the identity information of the object in the first target image. The texture information of the object in the image to be processed 310 is matched with the texture information of the object in the second target image 302.

FIG. 4 schematically shows a schematic diagram of an image processing process according to an embodiment of the present disclosure.

As shown in FIG. 4, in the process 400, a driving model 403 is used to process a first target image 401 and a second target image 402 to obtain an image 404 to be processed.

A first decoupled image 4050 in the decoupled image set 405 is obtained based on the second target image 402 and the image to be processed 404 . A second decoupled image 4051 in the decoupled image set 405 is obtained based on the second target image 402 and the image to be processed 404 . Based on the image 404 to be processed, a third decoupled image 4052 in the decoupled image set 405 is obtained. A fourth decoupled image 4053 in the decoupled image set 405 is obtained based on the image 404 to be processed. A fifth decoupled image 4054 in the decoupled image set 405 is obtained based on the second target image 402 and the fourth decoupled image 4053.

The fusion model 406 is utilized to understand the decoupled image set 405 and obtain the fused image 407.

In the technical proposal of this disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of related user personal information shall all comply with the provisions of relevant laws and regulations and adopt necessary security measures. , and does not violate public order and morals.

In the technical solution of the present disclosure, the user's permission or consent is obtained before obtaining or collecting the user's personal information.

FIG. 5 schematically shows a block diagram of an image processing device according to an embodiment of the present disclosure.

As shown in FIG. 5, the image processing apparatus 500 may include a first generation module 510, a second generation module 520, and a third generation module 530.

The first generation module 510 generates an image to be processed based on the first target image and the second target image. Here, the identity information of the object in the image to be processed is matched with the identity information of the object in the first target image, and the texture information of the object in the image to be processed is matched with the texture information of the object in the second target image. .

A second generation module 520 generates a decoupled image set based on the second target image and the image to be processed. Here, the decoupling image set includes a head decoupling image corresponding to the head region of the object in the image to be processed and a repair decoupling image corresponding to information to be repaired related to the object in the image to be processed. include.

A third generation module 530 generates a fused image based on the decoupled image set. Here, the identity information and texture information of the object in the fused image are matched with the identity information and texture information of the object in the image to be processed, respectively, and the information to be repaired related to the object in the fused image is repaired.

According to embodiments of the present disclosure, the repaired decoupled image includes a first decoupled image and a second decoupled image. The identity information of the object in the first decoupled image is matched with the identity information of the object in the image to be processed, and the skin color information of the object in the first decoupled image is matched with the skin color information of the object in the second target image. The second decoupling image is a difference image between the head area of the object in the image to be processed and the head area of the object in the second target image. Here, the information to be repaired related to the object in the fused image is repaired because the skin color information of the object in the fused image matches the skin color information of the object in the second target image, and the pixel of the pixel in the difference image is Indicates that the value matches a preset condition.

According to an embodiment of the present disclosure, the head decoupling images include a third decoupling image, a fourth decoupling image, and a fifth decoupling image. The third decoupled image includes a tone image of the head region of the object in the image to be processed. The fourth decoupled image includes a binarized image of the head region of the object in the image to be processed. The fifth decoupled image includes an image obtained from the second target image and the fourth decoupled image.

According to embodiments of the present disclosure, the third generation module 530 may include a first processing unit.

The first processing unit processes the decoupled image set using the fusion model to obtain a fused image. Here, the fusion model includes the generator in the first generative adversarial network model.

According to embodiments of the present disclosure, the fusion model is trained using a first identity information loss function, a first image feature matching loss function, a first discriminant feature matching loss function, and a first discriminator loss function. It is.

According to embodiments of the present disclosure, the first generation module 510 may include a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit.

The second processing unit processes the first target image using the identity extraction module in the driving model to obtain identity information of the object in the first target image.

The third processing unit processes the second target image using a texture extraction module in the driving model to obtain texture information of the object in the second target image.

A fourth processing unit processes identity information and texture information using a stitching module in the driving model to obtain stitching information.

The fifth processing unit processes the stitching information using a generator in the driving model to obtain an image to be processed.

According to embodiments of the present disclosure, the stitching information includes a plurality, and the generator of the driving model includes N cascaded deep units, where N is an integer greater than 1.

The fifth processing unit may include a processing subunit and a generation subunit.

The processing subunit processes jump information of the i-th level corresponding to the i-th deep unit using the i-th deep unit for the i-th deep unit among the N deep units; Obtain i-level feature information. Here, the i-th level jump information includes (i-1)-th level feature information and i-th level stitching information. Here, i is greater than 1 and less than or equal to N.

The generation subunit generates an image to be processed based on the Nth level feature information.

According to embodiments of the present disclosure, the driving model utilizes a second identity information loss function, a second image feature matching loss function, a second discriminant feature matching loss function, a second discriminator loss function, and a cyclic matching loss function. This training has been carried out.

According to embodiments of the present disclosure, the cyclic matching loss function is determined based on the actual result and the predicted result generated by the driving model, where the actual result includes actual identity information and actual texture information of the object in the actual image. , the prediction result includes predicted identity information and predicted texture information of the object in the simulation image.

According to embodiments of the present disclosure, the image processing device 500 may further include a processing module.

The processing module performs an enhancement process on the fused image and obtains an enhanced image.

According to embodiments of the disclosure, the disclosure further provides electronic devices, readable storage media, and computer program products.

According to embodiments of the present disclosure, an electronic device includes at least one processor and a memory communicatively coupled to the at least one processor, wherein the memory includes instructions executable by the at least one processor. is stored and the instructions are executed by at least one processor, thereby allowing at least one processor to execute the image processing method as described above.

According to embodiments of the present disclosure, a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions causing a computer to perform the image processing method described above.

According to embodiments of the present disclosure, a computer program product includes a computer program product that, when executed by a processor, implements an image processing method as described above.

According to embodiments of the disclosure, the disclosure further provides electronic devices, readable storage media, and computer program products.

According to embodiments of the present disclosure, an electronic device includes at least one processor and a memory communicatively coupled to the at least one processor, wherein the memory includes instructions executable by the at least one processor. are stored and the instructions are executed by the at least one processor, thereby enabling the at least one processor to perform the method.

According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions causing a computer to perform the method described above.

According to embodiments of the present disclosure, a computer program product includes a computer program product that, when executed by a processor, implements a method as described above.

FIG. 6 schematically shows a block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of the present disclosure. Electronic equipment is intended to refer to various types of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, large format computers, and other suitable computers. Electronic devices may also represent various types of mobile devices, such as personal digital assistants, mobile phones, smart phones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions depicted herein are illustrative only and are not intended to limit implementation of the disclosure as described and/or required herein.

As shown in FIG. 6, the electronic device 600 includes a computing unit 601, which is a computer program stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage unit 608. Various appropriate actions and processing may be performed based on the . The RAM 603 can further store various programs and data necessary for operating the electronic device 600. Computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. An input/output (I/O) interface 605 is also connected to the bus 604 .

A plurality of components in the electronic device 600 are connected to an I/O interface 605, and include an input unit 606 such as a keyboard and a mouse, an output unit 607 such as various types of displays and speakers, and an output unit 607 such as a magnetic disk, an optical disk, etc. a storage unit 608, and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information/data with other devices via a computer network such as the Internet and/or various telecommunication networks.

Computing unit 601 may be a variety of general purpose and/or special purpose processing modules with processing and computing capabilities. Some examples of the calculation unit 601 include a central processing unit (CPU), a GPU (Graphics Processing Unit), various dedicated artificial intelligence (AI) calculation chips, calculation units for various machine learning model algorithms, and a DSP (Digital Signal Processor). ), as well as any suitable processor, controller, microcontroller, etc. The calculation unit 601 performs the methods and processes described above, such as image processing methods, for example. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed on electronic device 600 via ROM 1002 and/or communication unit 609. When a computer program is loaded into RAM 1003 and executed by calculation unit 601, one or more steps of the image processing method described above may be performed. Alternatively, in another embodiment, the computing unit 601 may be configured to perform the image processing method in any other suitable manner (eg, via firmware).

Various embodiments of the systems and techniques described herein include digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and application specific standard products (ASSPs). ), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments are implemented in one or more computer programs that are executed and/or interpreted on a programmable system that includes at least one programmable processor. The programmable processor may be a special purpose or general purpose programmable processor and receives data and instructions from a storage system, at least one input device, and at least one output device, and receives data and instructions from a storage system, at least one input device, and at least one output device. The method may include being able to transmit instructions to the storage system, the at least one input device, and the at least one output device.

Program code for implementing the methods of this disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program codes may be implemented in a flowchart and/or block format. The functions/operations specified in the diagram are performed. The program code may be executed entirely on the device, partially on the device, partially on the device as a separate software package, and partially on a remote device, or It may be performed entirely on a remote device or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium and includes a program for use in or in combination with an instruction-execution system, device, or electronic device. Or it may be memorized. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or electronic device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connection through one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

A computer may implement the systems and techniques described herein to provide interaction with a user, and the computer may include a display device (e.g., a CRT (cathode ray tube) or a liquid crystal display (LCD) monitor), a keyboard and a pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other types of devices may further provide interaction with the user, for example, the feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or haptic feedback). and may receive input from the user in any form, including audio input, speech input, or tactile input.

The systems and techniques described herein may be used in a computing system that includes background components (e.g., a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components. a system (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein); The present invention may be implemented in a computing system that includes any combination of background components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks illustratively include local area networks (LANs), wide area networks (WANs), and the Internet.

A computer system may include a client and a server. Clients and servers are generally remote and typically interact via a communications network. The relationship between client and server is created by a computer program running on the relevant computer and having a client-server relationship. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

It should be understood that various types of flows illustrated above may be used and operations may be re-sorted, added, or removed. For example, each operation described in the present invention may be performed in parallel, sequentially, or in a different order, provided that the desired result of the technical solution of the present disclosure can be achieved. The specification is not limited here.

The specific embodiments do not limit the protection scope of the present disclosure. Those skilled in the art should appreciate that various modifications, combinations, subcombinations, and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

The present disclosure relates to the field of artificial intelligence technology, especially the field of computer vision and deep learning, and can be applied to scenes such as facial image processing and face identification. Specifically, the present invention relates to an image processing method, an image processing device, an electronic device , a storage medium , and a computer program .

According to another aspect of the disclosure, a computer program product is provided that, when executed by a processor, implements the method described above.

According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program .

According to embodiments of the present disclosure, there is provided a computer program that , when executed by a processor, implements the image processing method as described above .

According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program .

According to embodiments of the present disclosure, a computer program is provided that, when executed by a processor, implements such a method .

Claims (20)

An image processing method, comprising:
generating an image to be processed based on a first target image and a second target image, wherein identity information of an object in the image to be processed is matched with identity information of an object in the first target image; The texture information of the object in the image to be processed matches the texture information of the object in the second target image;
generating a decoupling image set based on the second target image and the to-be-processed image, wherein the decoupling image set corresponds to a head region of an object in the to-be-processed image; and a repair decoupling image corresponding to information to be repaired related to an object in the image to be processed;
generating a fused image based on the decoupled image set, wherein the identity information and texture information of an object in the fused image are matched with the identity information and texture information of an object in the image to be processed, respectively; Information to be repaired related to an object in the fused image is repaired. the repaired decoupled image includes a first decoupled image and a second decoupled image;
The identity information of the object in the first decoupled image is matched with the identity information of the object in the image to be processed, and the skin color information of the object in the first decoupled image is matched with the skin color information of the object in the second target image. death,
The second decoupling image is a difference image between the head area of the object in the image to be processed and the head area of the object in the second target image,
Here, the fact that the information to be repaired related to the object in the fused image is repaired means that the skin color information of the object in the fused image matches the skin color information of the object in the second target image, and 2. The method of claim 1, further comprising indicating that a pixel value of a pixel meets a preset condition. The head decoupling image includes a third decoupling image, a fourth decoupling image, and a fifth decoupling image,
The third decoupling image includes a gradation image of a head area of the object in the image to be processed,
The fourth decoupling image includes a binarized image of the head area of the object in the image to be processed,
The method according to claim 1 or 2, wherein the fifth decoupled image includes an image obtained based on the second target image and the fourth decoupled image. Generating the fused image based on the decoupled image set comprises:
A method according to any one of claims 1 to 3, comprising processing the decoupled image set using a fusion model comprising a generator in a first generative adversarial network model to obtain the fused image. . The fusion model is obtained by training using a first identity information loss function, a first image feature matching loss function, a first discriminant feature matching loss function, and a first discriminator loss function. Method described. Generating an image to be processed based on the first target image and the second target image includes:
processing the first target image using an identity extraction module in a driving model to obtain identity information of an object in the first target image;
processing the second target image using a texture extraction module in the driving model to obtain texture information of an object in the second target image;
processing the identity information and the texture information using a stitching module in the driving model to obtain stitching information;
The method according to any one of claims 1 to 5, comprising processing the stitching information using a generator in the driving model and obtaining the image to be processed. The stitching information includes a plurality of stitching information, the generator in the driving model includes N cascaded deep units, and N is an integer greater than 1;
Processing the stitching information using a generator in the driving model to obtain the image to be processed,
For the i-th deep unit among the N deep units, jump information of the i-th level corresponding to the i-th deep unit is processed using the i-th deep unit, and the jump information of the i-th level is processed. , where the i-th level jump information includes the (i-1)-th level feature information and the i-th level stitching information, where i is greater than 1 and less than or equal to N. and
7. The method of claim 6, comprising: generating the image to be processed based on Nth level feature information. The driving model is obtained by training using a second identity information loss function, a second image feature matching loss function, a second discriminant feature matching loss function, a second discriminator loss function, and a cyclic matching loss function. The method according to claim 6 or 7. The cyclic matching loss function is determined based on the actual result and the predicted result generated by the driving model, the actual result includes actual identity information and actual texture information of the object in the actual image, and the predicted result includes: 9. The method of claim 8, comprising predicted identity information and predicted texture information of the object in a simulated image. The method according to any one of claims 1 to 8, further comprising performing an enhancement process on the fused image to obtain an enhanced image. An image processing device,
generating an image to be processed based on a first target image and a second target image, wherein identity information of an object in the image to be processed is matched with identity information of an object in the first target image; a first generation module that matches texture information of an object in the image to be processed with texture information of an object in the second target image;
A decoupling image set is generated based on the second target image and the to-be-processed image, wherein the decoupling image set includes a head-decoupling image corresponding to a head region of an object in the to-be-processed image; a second generation module including a repaired decoupled image corresponding to information to be repaired related to an object in the image to be processed;
generating a fused image based on the decoupled image set, wherein the identity information and texture information of an object in the fused image are matched with the identity information and texture information of an object in the image to be processed, respectively; An image processing device including: a third generation module in which information to be repaired related to an object in the fused image is repaired. the repaired decoupled image includes a first decoupled image and a second decoupled image;
The identity information of the object in the first decoupled image is matched with the identity information of the object in the image to be processed, and the skin color information of the object in the first decoupled image is matched with the skin color information of the object in the second target image. death,
The second decoupling image is a difference image between the head area of the object in the image to be processed and the head area of the object in the second target image,
Here, the fact that the information to be repaired related to the object in the fused image is repaired means that the skin color information of the object in the fused image matches the skin color information of the object in the second target image, and The device according to claim 11, wherein the pixel value of the pixel indicates that a preset condition is met. The head decoupling image includes a third decoupling image, a fourth decoupling image, and a fifth decoupling image,
The third decoupling image includes a tone image of a head area of the object in the image to be processed,
The fourth decoupling image includes a binarized image of the head area of the object in the image to be processed,
The apparatus according to claim 11 or 12, wherein the fifth decoupled image includes an image obtained based on the second target image and the fourth decoupled image. The third generation module is
14. A first processing unit for processing the decoupled image set using a fusion model comprising a generator in a first generative adversarial network model to obtain the fused image. The device described. The fusion model is obtained by training using a first identity information loss function, a first image feature matching loss function, a first discriminant feature matching loss function, and a first discriminator loss function. The device described. The first generation module is
a second processing unit that processes the first target image using an identity extraction module in a driving model to obtain identity information of an object in the first target image;
a third processing unit that processes the second target image using a texture extraction module in the driving model to obtain texture information of an object in the second target image;
a fourth processing unit that processes the identity information and the texture information using a stitching module in the driving model to obtain stitching information;
The apparatus according to any one of claims 11 to 15, further comprising a fifth processing unit that processes the stitching information using a generator in the driving model and obtains the image to be processed. The stitching information includes a plurality of stitching information, the driving model includes N cascaded deep units, and N is an integer greater than 1;
The fifth processing unit generates jump information of the i-th level corresponding to the i-th deep unit using the i-th generator for the i-th deep unit among the N deep units. processing to obtain i-th level feature information, where the i-th level jump information includes (i-1)-th level feature information and i-th level stitching information, where i is a processing subunit greater than 1 and less than or equal to N;
17. The apparatus according to claim 16, further comprising: a generation subunit that generates the image to be processed based on Nth level feature information. An electronic device,
at least one processor;
a memory communicatively connected to the at least one processor;
The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, thereby causing the at least one processor to operate according to any one of claims 1 to 10. The method described can be carried out. A non-transitory computer-readable storage medium having computer instructions stored thereon;
The computer instructions cause the computer to perform a method according to any one of claims 1 to 10. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images