KR20220123091A - Image processing methods, devices, devices and storage media - Google Patents

Abstract

The present invention discloses an image processing method, apparatus, apparatus and storage medium, the method comprising: acquiring a first facial feature of a first facial image obtained based on a facial feature of a source image and a target image; obtaining a second facial feature by mapping the first facial feature, wherein a distribution of at least some of the second facial features matches a feature distribution of the target image; and acquiring a target facial image according to the first facial image and the second facial feature.

Description

Image processing methods, devices, devices and storage media

[Cross Citation of Related Applications]

This patent application claims the priority of the Chinese patent application filed on October 21, 2020 with the application number "202011135767.7" and the Chinese patent application filed on November 4, 2020 with the application number "202011218375.7". , the entire contents of which are incorporated herein by reference.

The present invention relates to computer vision technology, and more particularly to an image processing method, apparatus, apparatus and storage medium.

Image processing such as face exchange has a wide application value in digital entertainment and film industry, for example, the effect of an actor's band can be achieved through face exchange.

Currently, face exchange is implemented by migrating the facial features of the source image to the face region in the target image and performing face pose alignment. have.

The present invention provides an image processing method.

According to the first aspect, the method further comprises: obtaining a first facial feature of a first facial image, wherein the first facial image is obtained based on the facial feature of a source image and a target image; obtaining a second facial feature by mapping the first facial feature, wherein a distribution of at least some of the second facial features matches a feature distribution of the target image; and obtaining a target face image according to the first face image and the second face feature.

In any one embodiment of the present invention, the acquiring the first facial feature of the first facial image includes: acquiring at least one of facial coordinate information and facial normal vector information of the first facial image; performing feature extraction processing on the first face image through a feature encoding network to obtain encoding feature information; and obtaining a first facial feature of the first face image according to at least one of face coordinate information and face normal vector information of the first face image and the encoding feature information.

In any one embodiment of the present invention, the step of obtaining a second facial feature by mapping the first facial feature comprises performing mapping on the first feature of a pixel in the first facial image through a feature mapping network. to obtain the second characteristic of the pixel, wherein a distance between the probability distribution of the second characteristic of the pixel and the probability distribution of the third characteristic of the corresponding pixel in the target image satisfies a setting condition; Here, a first feature of a pixel in the first facial image belongs to the first facial feature, and a second feature of the pixel belongs to the second facial feature.

In any one embodiment of the present invention, the step of obtaining a target facial image according to the first facial image and the second facial feature comprises performing decoding on the second facial feature through a feature decoding network to perform a facial acquiring an image of the area; and obtaining the target face image according to an image other than the face region in the first face image and an image of the face region obtained by decoding.

In any one embodiment of the present invention, the encoding characteristic information includes nth order characteristic information; The step of obtaining the first facial feature of the first face image according to at least one of face coordinate information and face normal vector information of the first face image and the encoding feature information may include: all M-order features of the n-th order feature information connecting the information with at least one of face coordinate information and face normal vector information of the first face image, respectively, to obtain M-order linked feature information; and obtaining the first facial feature according to the M-order connection feature information and the subsequent feature information, wherein the subsequent feature information includes feature information other than the previous M-order feature information among the encoding feature information and n and M are natural numbers, and M<n.

In any one embodiment of the present invention, the step of obtaining the second facial feature by mapping the first facial feature comprises performing mapping on the M-order connected feature information to match the feature distribution of the target image. obtaining M-order mapping feature information; The step of obtaining an image of a face region by performing decoding on the second facial feature through a feature decoding network includes: performing decoding on the M-order mapping feature information and the subsequent feature information to obtain an image of the face region including the steps of

In any one embodiment of the present invention, the method further comprises performing end-to-end training on the feature encoding network, the feature mapping network, and the feature decoding network, wherein in each generation training the feature The training of the encoding network and the feature decoding network is performed sequentially with the training of the feature mapping network.

In any one embodiment of the present invention, the feature mapping network is obtained using training of an optimized migration network, the optimized migration network comprising the feature mapping network and a distance estimation network, the network of training The loss is a mapping loss used to indicate the difference between a probability distribution of a second feature of a pixel in the first facial image determined by the distance estimation network and a probability distribution of a third feature of that pixel in the target image. include

In any one embodiment of the present invention, the feature encoding network and the feature decoding network are obtained using appearance migration network training, the appearance migration network comprising the feature encoding network and the feature decoding network, The network loss of training includes: a first loss for indicating a difference between a second feature and a first feature of a pixel in the first facial image; and a second loss to indicate a difference between a second characteristic of a pixel in the first facial image and a third characteristic of a corresponding pixel in the target image.

In any one embodiment of the present invention, the appearance migration network further comprises a face reconstruction network, wherein the face reconstruction network is used to obtain a reconstructed face image according to the facial feature reconstruction of the target image, The network loss of training further includes a third loss for indicating a difference between a second feature of a pixel in the first facial image and a fourth feature of that pixel in the facial reconstruction image of the facial reconstruction network output.

In any one embodiment of the present invention, the appearance migration network further comprises a discrimination network, and the trained network loss is determined by the discrimination network according to the pixel classification result of the mixed image sample and the labeling information of the mixed image sample. a fourth loss used to indicate a difference between, wherein the mixed image sample comprises an image obtained by mixing pixels in the target facial image and pixels in the target image or the facial reconstruction image; The labeling information indicates a generated image pixel or an actual image pixel.

According to the second aspect, the apparatus provides an image processing apparatus, wherein the apparatus is used to obtain a first facial feature of a first facial image obtained based on a facial feature of a source image and a target image a first acquisition unit to be; a mapping unit used to map the first facial feature in which a distribution of at least some of the second facial features matches a feature distribution of the target image to obtain a second facial feature; and a second acquiring unit used to acquire a target facial image according to the first facial image and the second facial feature.

According to a third aspect, there is provided an electronic device, wherein the electronic device includes a memory for storing computer instructions executable in a process, and executing the image processing method according to any embodiment of the present disclosure when executing the computer instructions includes a process for

According to a fourth aspect, there is provided a storage medium in which a computer program is stored. When the program is executed by a processor, the image processing method according to any embodiment of the present invention is implemented.

In the image processing method, apparatus, apparatus and storage medium according to one or more embodiments of the present invention, a first facial feature of a first facial image obtained based on a facial feature of a source image and a facial feature of a target image to obtain a second facial feature by mapping the first facial feature, and the distribution of at least some of the second facial features matches the feature distribution of the target image, so that the accuracy of appearance migration from the source image to the target image Since the target face image is acquired according to the first facial image and the second facial feature, the continuity and consistency between the facial region and other regions in the target facial image can be improved. It can improve the quality of the face image.

It is to be understood that the foregoing general description and the following detailed description are illustrative and interpretative only and are not intended to limit the invention.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of this specification, show embodiments consistent with this specification, and together with the specification, serve to explain the principles of the invention.
1 shows a flowchart of an image processing method according to at least one embodiment of the present invention.
2 is a schematic diagram of a method for acquiring facial features in an image processing method according to at least one embodiment of the present invention.
3 is a schematic diagram illustrating a facial feature mapping method among image processing methods according to at least one embodiment of the present invention.
4A is a schematic diagram of a method for training a feature encoding network and a feature decoding network in an image processing method according to at least one embodiment of the present invention.
4B is a schematic diagram of another method for training a feature encoding network and a feature decoding network in an image processing method according to at least one embodiment of the present invention.
4C is a schematic diagram of another method for training a feature encoding network and a feature decoding network in an image processing method according to at least one embodiment of the present invention.
5 is a structural schematic diagram of an image processing apparatus according to at least one embodiment of the present invention.
6 is a structural diagram of an electronic device according to at least one embodiment of the present invention.

Examples will be described in detail herein, examples of which are shown in the drawings. When the following description includes drawings, like numbers in different drawings refer to the same or similar elements unless otherwise indicated. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. Rather, these are merely examples of apparatuses and methods consistent with some aspects of the present disclosure and as detailed in the appended claims.

In the present invention, the term "and/or" is merely an association relationship that describes an associated object, and indicates that three relationships may exist, for example, A and/or B is only A, A and B are simultaneously It can represent three cases in which only B exists and B. In addition, the term "at least one" used in the present invention denotes any one of several or any combination of at least two of several, e.g. For example, including at least one of A, B, and C may indicate that it includes any one or a plurality of elements selected from the set consisting of A, B, and C.

Embodiments of the present invention may be applied to computer systems/servers along with many other general purpose or dedicated computing systems, environments, and/or configurations. Examples of such computing systems, environments and/or configurations are personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor based systems, set top boxes, programmable consumer electronics, networked Distributed cloud computing technology environments including, but not limited to, personal computers, small computer systems, large computer systems and any of the above.

1 shows a flowchart of an image processing method according to at least one embodiment of the present invention. As shown in FIG. 1 , the method includes steps 101 to 103 .

In step 101, a first facial feature of a first facial image is acquired.

In an embodiment of the present invention, the first facial feature of the obtained first facial image may include various aspects such as a structure and a pattern, for example, a shape of a face included in the first facial image; It may include features such as size and orientation.

Here, the first face image is obtained based on the facial features of the source image and the target image.

In an embodiment of the present invention, both the source image and the target image include a face region, and the first face image may be generated according to a facial feature of the face region in the source image and the target image. For example, the first facial image may be obtained by migrating facial features of the facial region in the source image to the facial region in the target image, and aligning the facial pose in the source image with the facial pose in the target image. Here, the facial features of the source image may be obtained by performing feature extraction on a facial region in the source image. Exemplarily, the face regions in the source image and the target image may be a human face region, an animal face region, or a designated region of another virtual object.

When the first face image is obtained based on the facial features of the source image and the target image, when the skin color or lighting difference between the source image and the target image is large, other than the face region and the face region in the generated first face image Other areas of , may have large differences in appearance, i.e., may have a visual discontinuity.

In step 102, a second facial feature is obtained by mapping the first facial feature, and a distribution of at least some of the second facial features is matched with a feature distribution of the target image.

When the appearance difference between the source image and the target image is large, the feature distribution of the two images does not generally match, and if the mapping is performed without guidance of positioning information, the appearance may be vaguely migrated. Here, the positioning information is information indicating the location of the mapped feature.

In an embodiment of the present invention, by mapping the first facial feature, the distribution of at least some of the obtained second facial features is matched with the feature distribution of the target image, so that the appearance migration from the facial feature of the source image to the target image is obtained. accuracy can be improved.

In step 103, a target facial image is acquired according to the first facial image and the second facial feature.

The target face image may be understood as an image formed after migrating facial features of the face region in the source image to the face region of the target image and further optimizing the features of the region where the migration occurs through feature mapping. Visually, the target face image can reflect not only the features of the face region in the source image, but also features of regions other than the face region in the target image, so that a natural face replacement effect can be obtained.

Since the second facial feature closer to the feature distribution of the target image is obtained through feature mapping, the image of the facial region obtained according to the second facial feature is closer to the feature distribution of the target image in appearance, such as color space distribution. Therefore, in the target face image obtained according to the second facial feature and the first face image, the problem that the appearance of the face region is inconsistent and discontinuous with other regions is eliminated.

In an embodiment of the present invention, the second facial feature is obtained by mapping the first facial feature through the first facial feature of the first facial image obtained based on the facial feature of the source image and the target image, and the second facial feature is obtained. 2 the distribution of at least some of the facial features matches the feature distribution of the target image to improve the accuracy of migration of the appearance from the facial features of the facial region in the source image to the target image; By acquiring a target face image according to the first facial image and the second facial feature, it is possible to improve the continuity and consistency of the facial region and other regions in the target facial image, thereby improving the quality of the target facial image. have.

In some embodiments, the first facial feature of the first facial image may be acquired in the following manner.

First, at least one of face coordinate information and face normal vector information of the first face image is acquired.

Here, the face coordinate information is used to characterize geometrical information of a face included in an image, such as Projected Normalized Coordinate Code (PNCC) information of a face point cloud; The face normal vector information is used to characterize face illumination direction information.

In one example, such as obtaining a face point cloud through 3D Dense Face Alignment (3DDFA), a face point cloud can be obtained from an image including a face using a 3D face fitting model, Face coordinate information and/or face normal vector information may be obtained by rendering and projecting a face point cloud on a two-dimensional image.

In another aspect, the encoding feature information may be obtained by performing feature extraction processing on the face region in the first face image through the feature encoding network, that is, encoding the first face image through the feature coding network. Thus, high-order features can be obtained.

The first facial feature of the first face image may be obtained by combining face coordinate information and/or face normal vector information with the encoding feature information.

In an embodiment of the present invention, combining face coordinate information and/or face normal vector information of a first face image with encoding feature information obtained by a feature encoding network to obtain a first facial feature of the first face image; By mapping the first facial feature to a second facial feature matching the feature distribution of the target image, the distribution of facial geometric information and/or lighting direction information of the generated target facial image is consistent with the target image, When there is a large difference in skin color or illumination between the image and the target image, it is possible to remove the problem that the appearance of the face region in the first face image is inconsistent and discontinuous with other regions.

In some embodiments, the encoding feature information output from the feature encoding network includes nth-order feature information, that is, n different size feature maps are output. Among the nth-order characteristic information, M-th order connection characteristic information is obtained by linking with at least one of face coordinate information and face normal vector information of the first face image, respectively, through M-order characteristic information, and also M-order connection characteristic information and The first facial feature is acquired according to the subsequent feature information. Here, the second-order feature information includes feature information other than the previous M-order feature information among the n-th feature information. In the above embodiment, n and M are natural numbers, and M<n.

또는 타겟 이미지

)에 대해, 한 방면으로는, 특징 인코딩 네트워크(201)를 통해 상기 입력 이미지의 인코딩 특징 정보를 획득하고, 상기 인코딩 특징 정보는 4차 특징 정보를 포함하며; 다른 한 방면으로는, 3차원 피팅 모델(202)을 통해 상기 입력 이미지에 포함된 얼굴의 얼굴 포인트 클라우드를 획득하고, 상기 얼굴 포인트 클라우드를 렌더링하여 PNCC 정보 및/또는 법선 벡터 정보를 획득하고, 또한 상기 PNCC 정보 및/또는 법선 벡터 이미지를 투영하여 PNCC 이미지 및/또는 법선 벡터 이미지를 획득한다. 이어서, 상기 PNCC 이미지 및/또는 법선 벡터 이미지를 각각 전 3차 특징 정보와 매칭하는 사이즈로 스케일링한 후, 스케일링된 상기 PNCC 이미지 및/또는 법선 벡터 이미지가 각각 전 3차 특징 정보와 연결되어 연결 특징 정보

를 획득한다. 연결 특징 정보

및 4차 특징 정보

를 연결하여 상기 입력 이미지의 제1 얼굴 특징을 형성한다. 도 2에 도시된 제1 얼굴 특징을 획득하기 위한 구조는 감지 인코더라고 칭한다. 2 is a schematic diagram illustrating a method of acquiring a first facial feature in an image processing method according to at least one embodiment of the present invention. As shown in Figure 2, the input image X (eg, the first face image

or target image

), in one aspect, obtain encoding characteristic information of the input image through a characteristic encoding network 201, wherein the encoding characteristic information includes quaternary characteristic information; In another aspect, a face point cloud of a face included in the input image is obtained through a three-dimensional fitting model 202, and PNCC information and/or normal vector information is obtained by rendering the face point cloud, and also A PNCC image and/or a normal vector image is obtained by projecting the PNCC information and/or a normal vector image. Then, the PNCC image and/or the normal vector image are each scaled to a size matching the full tertiary feature information, and then the scaled PNCC image and/or the normal vector image are respectively connected to the full tertiary feature information to connect features Information

to acquire Connection feature information

and quaternary characteristic information

to form a first facial feature of the input image. The structure for acquiring the first facial feature shown in FIG. 2 is called a sensory encoder.

In some embodiments, a first facial feature to a second facial feature may be mapped via a feature mapping network.

In one example, mapping is performed on the first facial feature of the first facial image pixel through a feature mapping network to obtain a second feature corresponding to the pixel, and a probability distribution of the second feature corresponding to the pixel and a distance between the probability distribution of the third feature of the corresponding pixel in the target image satisfies a setting condition; Here, a first feature corresponding to a pixel in the first facial image belongs to a first facial feature, and a second feature corresponding to the pixel belongs to the second facial feature.

If the distance between the probability distribution of the second feature of the pixel and the probability distribution of the third feature of the corresponding pixel in the target image satisfies a preset condition, the distance is less than a preset threshold value or the distance reaches a minimum include that Here, the set threshold may be set according to application demand, and the embodiment of the present invention is not limited thereto.

In some embodiments, some features included in the first facial feature may be mapped to a second facial feature that matches a feature distribution of a target image, and another feature of the first facial feature is directly transmitted to obtain an equivalent feature do.

For example, when the first facial feature information includes M-order connected feature information and subsequent feature information, mapping is performed on the M-order connected feature information to obtain M-order mapping feature information that matches the feature distribution of the target image. can be obtained; In addition, the second facial feature obtained by directly transmitting the subsequent feature information includes M-order mapping feature information and subsequent feature information.

3 is a schematic diagram illustrating a first facial feature mapping method among image processing methods according to at least one embodiment of the present invention.

으로 표시하고, 여기서

는 인코딩된 특징 정보를 나타내는 바, 예를 들어, 콘볼루션 신경망을 이용하여 픽셀 공간을 은닉층 공간으로 인코딩하여 얻은 k차원 특징 정보이고,

는 3차원 PNCC 특징과 같은 얼굴 좌표 정보를 나타내고,

은 3차원 법선 벡터 정보와 같은 법선 벡터 정보를 나타낸다. 예를 들어, 제1 얼굴 이미지

내의 픽셀의 제1 특징은 제1 특징 벡터

로 표시될 수 있고, 상기 픽셀의 제2 특징은 제2 특징 벡터

로 표시될 수 있으며, 상기 타겟 이미지

내의 픽셀의 제3 특징은 제3 특징 벡터

로 표시될 수 있다.In this example, the feature vector of the pixel

marked as, where

represents the encoded feature information, for example, k-dimensional feature information obtained by encoding the pixel space into the hidden layer space using a convolutional neural network,

represents face coordinate information such as three-dimensional PNCC features,

denotes normal vector information such as 3D normal vector information. For example, the first face image

The first feature of the pixel in the first feature vector

may be represented as , wherein the second feature of the pixel is a second feature vector

may be displayed as, and the target image

the third feature of the pixel in the third feature vector

can be displayed as

를 통해 상기 제1 얼굴 이미지 내의 픽셀의 제1 특징 벡터

에 대해 매핑을 수행하여 픽셀 {m,n}의 제2 특징 벡터

를 획득한다. 여기서 i는 특징의 차수를 나타내고, {m,n}은 픽셀 위치를 나타내며, 여기서 m은 행 번호, n은 열 번호,

는 제1 얼굴 이미지의 제i차 특징 정보를 나타내며,

는 제1 얼굴 이미지의 제i 차 특징 정보에 대응하는 매핑 특징 정보를 가리킨다.As shown in Figure 3, first the mapping function

a first feature vector of a pixel in the first facial image via

The second feature vector of the pixel {m,n} by performing mapping on

to acquire where i denotes the order of the feature, {m,n} denotes the pixel position, where m is the row number, n is the column number,

represents the ith characteristic information of the first face image,

denotes mapping feature information corresponding to the i-th feature information of the first face image.

를 도입하여 이미지 내의 픽셀의 특징 벡터의 확률 분포를 결정할 수 있으며, 따라서 제2 특징 벡터

와 상기 타겟의 이미지 내의 픽셀에 대응하는 제3 특징 벡터

사이의 와서스테인(Wasserstein) 거리를 결정할 수 있는 바, 즉 상기 제1 얼굴 이미지 내의 픽셀의 제2 특징 벡터의 확률 분포와 상기 타겟 이미지 내의 상응한 픽셀의 제3 특징 벡터의 확률 분포 사이의 거리를 결정할 수 있다. 매핑 함수

를 조정하여 제2 특징 벡터의 확률분포와 제3 특징 벡터의 확률분포 사이의 거리가 설정된 조건을 만족시키도록 하는 바, 예를 들어 설정된 임계값보다 작거나 또는 상기 거리를 최소치로 조정하여 제2 특징 벡터의 분포가 타겟 이미지의 제3 특징 벡터의 분포에 컨버징되도록 함으로써, 상기 제1 얼굴 이미지 내의 픽셀의 제1 특징 벡터의 최적화 마이그레이션을 구현한다.In one example, the evaluation function

can determine the probability distribution of the feature vectors of pixels in the image by introducing

and a third feature vector corresponding to a pixel in the image of the target.

It is possible to determine a Wasserstein distance between can decide mapping function

is adjusted so that the distance between the probability distribution of the second feature vector and the probability distribution of the third feature vector satisfies a set condition, for example, it is smaller than a set threshold or adjusts the distance to a minimum value By allowing the distribution of the feature vector to be converged with the distribution of the third feature vector of the target image, an optimized migration of the first feature vector of the pixel in the first face image is realized.

The above process can also be expressed as a minimax problem via Equation (1):

는 기대값을 나타내고,

는 1차 립쉬츠(1-Lipschitz)의 제약을 받으며, 이미지 내의 픽셀의 특징 벡터의 확률 분포를 획득하기 위함이고,

는 제1 얼굴 이미지

이 타겟 이미지

로 변환한 기초 상에, 픽셀의 제1 특징 벡터에 대해 매핑을 수행하는 것을 가리킨다.here,

represents the expected value,

is constrained by the first order Lipschitz (1-Lipschitz), to obtain the probability distribution of the feature vector of the pixel in the image,

is the first face image

this target image

It refers to performing mapping on the first feature vector of the pixel on the basis of the transformation.

의 기능은 특징 매핑 네트워크를 통해 구현될 수 있고, 상기 평가 함수

의 기능은 평가 네트워크를 통해 구현될 수 있다.In an embodiment of the present invention, the mapping function

The function of may be implemented through a feature mapping network, and the evaluation function

The function of can be implemented through an evaluation network.

after performing mapping on the first facial feature of the first facial image, performing decoding on the second facial feature through a feature decoding network to obtain an image of a facial region; In addition, the target face image is obtained according to an image other than the face region in the first face image and an image of the face region obtained by decoding.

For example, when the second facial feature obtained through mapping includes M-order mapping feature information and subsequent feature information, decoding is performed on the M-order mapping feature information and the subsequent feature information to obtain an image of the face region can be obtained.

와 타겟 이미지

에 따라 제1 얼굴 이미지

을 획득하고, 상기 제1 얼굴 이미지

을 감지 인코더(401)에 입력하여 제1 얼굴 특징

를 획득한다. 4A is a schematic diagram of an image processing method according to at least one embodiment of the present invention. As shown in Figure 4a, the source image

and target image

1st face image according to

to obtain the first face image

to the sensory encoder 401 to input the first facial feature

to acquire

를 사용하여 제1 얼굴 특징 중의

에 대해 매핑을 수행하여 매핑 특징

을 획득하고, 또한, 얼굴 특징

에 대해 직접 전송하며(동등 특징으로 매핑); 얼굴 디코더(402)(특징 디코딩 네트워크)는 매핑 특징

및 얼굴 특징

에 대해 디코딩을 수행하여 얼굴 영역의 이미지를 획득하고, 상기 얼굴 영역의 이미지 및 상기 제1 얼굴 이미지의 상기 얼굴 영역 이외의 이미지에 따라 타겟 얼굴 이미지

을 획득한다.feature mapping network

of the first facial feature using

Mapping features by performing a mapping on

to obtain, and also facial features

send directly to (map to equivalence features); Face decoder 402 (feature decoding network) is a feature mapping feature

and facial features

performing decoding to obtain an image of a face region, and a target face image according to the image of the face region and an image other than the face region of the first face image

to acquire

The following describes the training process of a feature encoding network, a feature mapping network, and a feature decoding network applied in an embodiment of the present invention.

In an embodiment of the present invention, end-to-end training is performed on the feature encoding network, the feature mapping network, and the feature decoding network. Here, in each generation training, training of the feature encoding network and the feature decoding network and training of the feature mapping network are sequentially performed. That is, first, in one-generation training, the feature encoding network and the feature decoding network are jointly trained, and then the feature mapping network is trained, and in this way, each generation training is alternately performed.

First, the training process of the feature mapping network will be described.

The feature mapping network is obtained using training of an optimized migration network, wherein the optimized migration network includes the feature mapping network and a distance estimation network, wherein the distance estimation network comprises pixels in the first face image. is used to determine the distance between the probability distribution of the second feature of , and the probability distribution of the third feature of the corresponding pixel in the target image.

The network loss performing the feature mapping network training is the difference between the probability distribution of the second feature of the pixel in the first face image determined by the distance estimation network and the probability distribution of the third feature of the corresponding pixel in the target image. It contains the mapping loss used to point to .

In one example, parameters of the feature mapping network may be adjusted by minimizing the mapping loss.

In one example, it is possible to alternately train the feature mapping network and the distance estimation network. For example, in one-generation training, the network parameters of the distance estimation network are adjusted by maximizing the mapping loss, and then the network parameters of the feature mapping network are adjusted by minimizing the mapping loss, in this way. Take turns training each generation.

The mapping loss can be expressed by Equation (2).

는 기대값,

는 1차 립쉬츠(1-Lipschitz)의 제약을 받으며, 이미지 내의 픽셀의 특징 벡터의 확률 분포를 획득하기 위함이고,

는 제1 얼굴 이미지

이 타겟 이미지

로 변환한 기초 상에, 제1 얼굴 이미지의 픽셀｛m,n｝의 제i 차 특징 벡터에 대해 매핑을 수행하는 것을 가리킨다.where i is the order of the feature, m is the row number of the pixel, n is the column number of the pixel, M is the total number of rows, N is the total number of columns,

is the expected value,

is constrained by the first order Lipschitz (1-Lipschitz), to obtain the probability distribution of the feature vector of the pixel in the image,

is the first face image

this target image

It indicates that mapping is performed on the i-th feature vector of pixels m,n , of the first face image, on the basis of the transformation.

The following describes the training process of the feature encoding network and the feature decoding network. The training of the feature encoding network and the feature decoding network can be divided into three types. In the first case, training is performed using the feature encoding network and the feature decoding network itself without the help of other networks. In this case, the feature-in coding network and the feature-decoding network are jointly called an appearance migration network, and the network loss for training is:

a first loss for indicating a difference between a second feature and a first feature of a pixel in the first facial image; and

and a second loss to indicate a difference between a second characteristic of a pixel in the first facial image and a third characteristic of a corresponding pixel in the target image.

In one example, parameters of the feature encoding network and the feature decoding network may be adjusted by minimizing the first loss and the second loss.

The loss of training can be expressed as Equation (3):

는 vgg 네트워크를 사용하여 얻은 이미지의 특징을 나타내며, 여기서 제 1 손실은 타겟 얼굴 이미지와 제1 얼굴 이미지 사이의 내용 상의 차이를 반영할 수 있고, 제2 손실은 타겟 얼굴 이미지와 타겟 이미지 사이의 외관 상의 차이를 반영할 수 있다. here

denotes the features of the image obtained using the vgg network, where the first loss may reflect a difference in content between the target face image and the first face image, and the second loss may reflect the appearance between the target face image and the target image. differences may be reflected.

The second case is a case in which training can be performed on the feature encoding network and the feature decoding network using the face reconstruction network. In this case, the feature encoding network, the feature decoding network, and the face reconstruction network may be jointly referred to as an appearance migration network. Here, the face reconstruction network is used to obtain a reconstructed face image obtained by reconstructing according to the facial features of the target face image.

로부터 획득된 얼굴 특징

을 직접 전송하고, 또한 얼굴 디코더(404)를 통해 얼굴 특징

를 디코딩함으로써 재구성 이미지

를 획득한다. 4B is a schematic diagram of a method for training a feature encoding network and a feature decoding network in an image processing method according to at least one embodiment of the present invention. As shown in FIG. 4B , the face reconstruction network includes a sensory encoder 403 and a face decoder 404 that share weights together with a sensory encoder 401 and a face decoder 402 . Target image via sensing encoder 403

facial features obtained from

directly transmits the facial features, and also through the facial decoder 404

Reconstructed image by decoding

to acquire

In this case, the network loss of the training further includes a third loss in addition to the first loss and the second loss, wherein the third loss is output by the face reconstruction network and the second feature of the pixel in the first face image. used to indicate the difference between the fourth feature of the corresponding pixel in the reconstructed face image.

The trained loss can be expressed using Equation (4).

Here, the first loss and the second loss are the same as in Equation (3), and the third loss may reflect a difference in appearance between the reconstructed face image and the target image.

A third case is a case in which training can be performed on the feature encoding network and the feature decoding network using the mixed network and the discrimination network. In this case, the feature encoding network, the feature decoding network, the mixed network and the discrimination network, or the face reconstruction network may be jointly referred to as an appearance migration network. wherein the blending network is used to mix pixels in the target face image and pixels in the target image/the reconstructed image to obtain a blended image; The discriminative network is used to determine the classification result of pixels in the blended image, i.e., the pixel is a pixel of a target face image (generated image pixel) or a pixel of a target image or a reconstructed face image (real image). pixel) or not.

를 사용하여 혼합 이미지

를 생성하는 바, 예를 들어 수학식 (5)에 의해 혼합 이미지

을 생성한다.4C is a schematic diagram of another method for training a feature encoding network and a feature decoding network in an image processing method according to at least one embodiment of the present invention. As shown in Figure 4c, the mixing network 405 is a randomly generated mask.

blend image using

Create a bar, e.g. a mixed image by Equation (5)

create

는 마스크,

는 타겟 얼굴 이미지,

는 타겟 이미지를 나타낸다.here

is a mask,

is the target face image,

represents the target image.

에 대해 혼합 이미지의 픽셀이 실제 이미지 픽셀인지 생성된 이미지 픽셀인지 여부를 예측한다.The discriminative network 406 is a mixed image

Predict whether a pixel in the blended image is an actual image pixel or a generated image pixel.

In this case, the training loss can be expressed as Equation (6):

는 혼합 이미지,

는 마스크,

는 혼합 이미지에서 픽셀 기반 와서스테인(Wasserstein) 거리를 예측하는 함수이고,

은 MSD 함수가 1차 립쉬츠(Lipschitz) 제한을 받는다는 것을 나타낸다.where E is the expected value,

is a mixed image,

is a mask,

is a function to predict the pixel-based Wasserstein distance in the mixed image,

indicates that the MSD function is subject to a first-order Lipschitz constraint.

In the case of this training method, the parameters of the feature encoding network and the feature decoding network are maximized by maximizing the loss for the part judged to be false (loss discrimination) and by minimizing the loss for the part judged to be true (loss generation). can be adjusted for

In one example, the network loss for training the feature encoding network and the feature decoding network can be expressed as Equation (7):

는 가중 계수로서 각 부분의 손실의 중요도에 따라 결정될 수 있다.here,

is a weighting factor and may be determined according to the importance of the loss of each part.

The above has described the training process of various neural networks used in the image processing method of the embodiment of the present invention in conjunction with FIGS. 4A to 4C. The facial features shown in FIGS. 4A to 4C are only examples and are not limiting. In a specific embodiment, if necessary, the facial features may include local features such as, for example, features of eyes, nose, mouth, etc., and may also include global features of the entire face, or local features and It can include both global features and other facial features.

5 is a structural schematic diagram of an image processing apparatus according to an embodiment of the present invention. As shown in FIG. 5 , the apparatus includes a first facial image obtained based on facial features of a source image and a target image. a first acquiring unit 501 used to acquire facial features; a mapping unit (502), used for mapping the first facial feature in which distribution of at least some of the second facial features matches the feature distribution of the target image to obtain a second facial feature; and a second acquiring unit, used to acquire a target facial image according to the first facial image and the second facial feature.

In some embodiments, the first acquiring unit is specifically configured to: acquire at least one of face coordinate information and face normal vector information of the first face image; performing feature extraction processing on the first face image through a feature encoding network to obtain encoding feature information; used to obtain a first facial feature of the first face image according to at least one of face coordinate information and face normal vector information of the first face image and the encoding feature information.

In some embodiments, the mapping unit is specifically used to map a first feature of a pixel in the first facial image through a feature mapping network to obtain a second feature of the pixel, wherein the second feature of the pixel is a distance between a probability distribution of a feature and a probability distribution of a third feature of a corresponding pixel in the target image satisfies a setting condition; Here, a first feature of a pixel in the first facial image belongs to the first facial feature, and a second feature of the pixel belongs to the second facial feature.

In some embodiments, the second obtaining unit is specifically configured to: perform decoding on the second facial feature through a feature decoding network to obtain an image of a facial region; used to obtain the target face image according to an image other than the face region in the first face image and an image of the face region obtained by decoding.

In some embodiments, the encoding characteristic information includes nth-order characteristic information; Acquiring the first facial feature of the first face image according to at least one of face coordinate information and face normal vector information of the first face image and the encoding feature information includes, specifically, all M orders of the nth order feature information. connecting the feature information with at least one of face coordinate information and face normal vector information of the first face image, respectively, to obtain M-order linked feature information; used to obtain the first facial feature according to the M-th order connection feature information and the subsequent feature information, wherein the subsequent feature information includes feature information other than the previous M-order feature information among the encoding feature information, n and M is a natural number, and M<n.

In some embodiments, when the mapping unit is used to map the first facial feature to obtain a second facial feature, specifically, the mapping is performed on the M-th order connected feature information to characterize the target image used to obtain M-order mapping feature information matching the distribution; When the second obtaining unit is used to obtain an image of a face region by performing decoding on the second facial feature through a feature decoding network, specifically, in the M-th order mapping feature information and the subsequent feature information It is used to obtain an image of the face region by performing decoding on the face region.

In some embodiments, the apparatus further comprises a training unit for performing end-to-end training on the feature encoding network, the feature mapping network, and the feature decoding network, wherein in each generation training, the feature encoding network and the The training of the feature decoding network and the training of the feature mapping network are performed sequentially.

In some embodiments, the feature mapping network is obtained using training of an optimized migration network, the optimized migration network includes the feature mapping network and a distance estimation network, and the network loss of the training is: a mapping loss used to indicate a difference between a probability distribution of a second feature of a pixel in the first facial image determined by the network and a probability distribution of a third feature of that pixel in the target image.

In some embodiments, the feature encoding network and the feature decoding network are obtained using appearance migration network training, wherein the appearance migration network includes the feature encoding network, the feature decoding network, and the network loss of the training is: a first loss for indicating a difference between a second feature and a first feature of a pixel in the first facial image; and a second loss to indicate a difference between a second characteristic of a pixel in the first facial image and a third characteristic of a corresponding pixel in the target image.

In some embodiments, the appearance migration network further comprises a face reconstruction network, the face reconstruction network is used to obtain a reconstructed face image according to the facial feature reconstruction of the target image, and the network loss of the training is , a third loss for indicating a difference between a second feature of a pixel in the first facial image and a fourth feature of that pixel in the facial reconstruction image of the facial reconstruction network output.

In some embodiments, the appearance migration network further comprises a discrimination network, wherein the trained network loss indicates a difference between the pixel classification result of the mixed image sample determined by the discrimination network and the labeling information of the mixed image sample. and a fourth loss used for: wherein the mixed image sample includes an image obtained by mixing pixels in the target face image and pixels in the target image or the face reconstruction image, and wherein the labeling information is generated Points to an image pixel or an actual image pixel.

6 illustrates an electronic device provided by at least one embodiment of the present invention, the device comprising a processor 602 and a memory 601 used to store computer instructions executable by the processor. , the processor is used to execute the image processing method according to any embodiment of the present specification when executing the computer instruction.

At least one embodiment of the present specification further provides a storage medium in which a computer program is stored, and when the program is executed by a processor, the imaging method according to any embodiment of the present specification is implemented. .

In an embodiment of the present invention, the computer-readable storage medium may have various forms. For example, in another example, the machine-readable storage medium may include random access memory (RAM), volatile memory, and non-volatile memory. It may be a volatile memory, flash memory, a memory driver (eg, a hard disk driver), a solid state hard disk, any type of memory disk (eg, CD, dvd, etc.), or a similar storage medium, or a combination thereof. In particular, the computer readable medium may also be paper or other suitable medium capable of printing a program. Using such a medium, such a program may be obtained electrically (eg, optically scanned), compiled in an appropriate manner, interpreted, processed, and then stored on a computer medium.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and all modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (20)

An image processing method comprising:
acquiring a first facial feature of a first facial image, wherein the first facial image is acquired based on facial features of a source image and a target image;
obtaining a second facial feature by mapping the first facial feature, wherein a distribution of at least some of the second facial features matches a feature distribution of the target image; and
acquiring a target facial image according to the first facial image and the second facial feature
An image processing method, characterized in that. According to claim 1,
Acquiring a first facial feature of the first facial image includes:
obtaining at least one of face coordinate information and face normal vector information of the first face image;
performing feature extraction processing on the first face image through a feature encoding network to obtain encoding feature information; and
acquiring a first facial feature of the first face image according to at least one of face coordinate information and face normal vector information of the first face image and the encoding feature information
An image processing method, characterized in that. 3. The method of claim 1 or 2,
obtaining a second facial feature by mapping the first facial feature,
performing mapping on a first feature of a pixel in the first facial image through a feature mapping network to obtain a second feature of the pixel, wherein a probability distribution of the second feature of the pixel and a probability distribution of the second feature in the target image a distance between the probability distributions of the third feature of the corresponding pixel satisfies the setting condition;
a first feature of a pixel in the first facial image belongs to the first facial feature, and a second feature of the pixel belongs to the second facial feature
An image processing method, characterized in that. 4. The method according to any one of claims 1 to 3,
Acquiring a target facial image according to the first facial image and the second facial feature comprises:
performing decoding on the second facial feature through a feature decoding network to obtain an image of a facial region; and
acquiring the target face image according to an image other than the face region in the first face image and an image of the face region obtained by decoding
An image processing method, characterized in that. 5. The method according to any one of claims 2 to 4,
the encoding characteristic information includes nth-order characteristic information;
Acquiring the first facial feature of the first face image according to at least one of face coordinate information and face normal vector information of the first face image and the encoding feature information comprises:
obtaining M-order linked feature information by linking all M-order feature information of the n-th feature information to at least one of face coordinate information and face normal vector information of the first face image, respectively; and
obtaining the first facial feature according to the M-order connection feature information and the subsequent feature information, wherein the subsequent feature information includes feature information other than the previous M-order feature information among the encoding feature information, n and M is a natural number, and M<n
An image processing method, characterized in that. 6. The method of claim 5,
obtaining a second facial feature by mapping the first facial feature,
performing mapping on the M-order connection feature information to obtain M-order mapping feature information matching the feature distribution of the target image;
Acquiring an image of a facial region by performing decoding on the second facial feature through a feature decoding network includes:
and obtaining an image of the face region by performing decoding on the M-order mapping feature information and the subsequent feature information. 7. The method according to any one of claims 2 to 6,
Further comprising the step of performing end-to-end training on the feature encoding network, the feature mapping network, and the feature decoding network, wherein in each generation training, the training of the feature encoding network and the feature decoding network is of the feature mapping network. training and sequential
An image processing method, characterized in that. 8. The method of claim 7,
The feature mapping network is obtained using training of an optimized migration network, the optimized migration network includes the feature mapping network and the distance estimation network, and the network loss of the training is:
a mapping loss used to indicate a difference between a probability distribution of a second feature of a pixel in the first facial image determined by the distance estimation network and a probability distribution of a third feature of that pixel in the target image.
An image processing method, characterized in that. 8. The method of claim 7,
The feature encoding network and the feature decoding network are obtained using appearance migration network training, wherein the appearance migration network includes the feature encoding network and the feature decoding network, and the network loss of the training is:
a first loss for indicating a difference between a second feature and a first feature of a pixel in the first facial image;
a second loss to indicate a difference between a second characteristic of a pixel in the first facial image and a third characteristic of a corresponding pixel in the target image;
An image processing method, characterized in that. 10. The method of claim 9,
The appearance migration network further includes a face reconstruction network, wherein the face reconstruction network is used to obtain a reconstructed face image according to the facial feature reconstruction of the target image, and the network loss of the training is:
a third loss to indicate a difference between a second feature of a pixel in the first facial image and a fourth feature of that pixel in the facial reconstruction image of the facial reconstruction network output
An image processing method, characterized in that. 11. The method of claim 9 or 10,
The appearance migration network further comprises a discriminative network, and the trained network loss is
a fourth loss used to indicate a difference between the pixel classification result of the mixed image sample determined by the discrimination network and the labeling information of the mixed image sample,
The mixed image sample includes an image obtained by mixing pixels in the target face image and pixels in the target image or the face reconstruction image, and the labeling information points to a generated image pixel or points to an actual image pixel.
An image processing method, characterized in that. An image processing device comprising:
a first acquiring unit used to acquire a first facial feature of a first facial image acquired based on the facial feature of the source image and the target image;
a mapping unit used to map the first facial feature in which a distribution of at least some of the second facial features matches a feature distribution of the target image to obtain a second facial feature; and
a second acquiring unit used to acquire a target facial image according to the first facial image and the second facial feature;
Image processing device, characterized in that. 13. The method of claim 12,
The first acquiring unit,
acquiring at least one of face coordinate information and face normal vector information of the first face image;
performing feature extraction processing on the first face image through a feature encoding network to obtain encoding feature information;
used to obtain a first facial feature of the first face image according to at least one of face coordinate information and face normal vector information of the first face image and the encoding feature information
Image processing device, characterized in that. 14. The method of claim 12 or 13,
The mapping unit is
is used to map a first feature of a pixel in the first facial image through a feature mapping network to obtain a second feature of the pixel, wherein the probability distribution of the second feature of the pixel and the corresponding pixel in the target image the distance between the probability distributions of the third feature satisfies the setting condition;
a first feature of a pixel in the first facial image belongs to the first facial feature, and a second feature of the pixel belongs to the second facial feature
Image processing device, characterized in that. 15. The method according to any one of claims 12 to 14,
The second acquisition unit,
perform decoding on the second facial feature through a feature decoding network to obtain an image of a facial region;
used to obtain the target face image according to an image other than the face region in the first face image and an image of the face region obtained by decoding
Image processing device, characterized in that. 16. The method according to any one of claims 12 to 15,
the encoding characteristic information includes nth-order characteristic information;
when the first obtaining unit is used to obtain a first facial feature of the first face image according to the encoding feature information and at least one of face coordinate information and face normal vector information of the first face image;
connecting all M-order feature information of the n-th feature information with at least one of face coordinate information and face normal vector information of the first face image, respectively, to obtain M-order linked feature information;
used to obtain the first facial feature according to the M-order connection feature information and the subsequent feature information, wherein the subsequent feature information includes feature information other than the pre-M-order feature information among the encoding feature information, n and M is a natural number, where M < n
An image processing method, characterized in that. 17. The method of claim 16,
when the mapping unit is used to map the first facial feature to obtain a second facial feature,
is used to perform mapping on the M-order connection feature information to obtain M-order mapping feature information matching the feature distribution of the target image;
when the second acquiring unit is used to perform decoding on the second facial feature through a feature decoding network to acquire an image of a facial region,
used to obtain an image of the face region by performing decoding on the M-order mapping feature information and the subsequent feature information
Image processing device, characterized in that. 18. The method according to any one of claims 13 to 17,
A training unit for performing end-to-end training on the feature encoding network, the feature mapping network, and the feature decoding network, wherein training of the feature encoding network and the feature decoding network and the feature mapping network in each generation training training is performed in sequence
Image processing device, characterized in that. As an electronic device,
12. A method comprising: a memory and a process, wherein the memory is used to store computer instructions executable in the process, the processor executing the image processing method according to any one of claims 1 to 11 when executing the computer instructions used to
Electronic device, characterized in that. A storage medium in which a computer program is stored, comprising:
When the program is executed by the processor, the image processing method according to any one of claims 1 to 11 is implemented
A storage medium, characterized in that.

Images