KR20240070392A - Electronic device for processing image and method for operating the same - Google Patents

Abstract

An electronic device that processes images and a method of operating the same are disclosed. The operating method of the electronic device includes acquiring an initial image feature matrix of the face image based on a multi-level feature map of the face image, and initial facial a priori features of the face image based on the last level feature map of the multi-level feature map. Obtaining a matrix and based on the initial image feature matrix and the initial facial a priori feature matrix, using one or more cascaded encoders, obtain a super-resolution image of the face image and/or key point coordinates of the face image Includes actions.

Description

Electronic device for processing images and method of operating the same {ELECTRONIC DEVICE FOR PROCESSING IMAGE AND METHOD FOR OPERATING THE SAME}

The disclosure below relates to electronic devices that process images and methods of operating the same.

Recently, with the development of deep neural network technology, face super-resolution (FSR) technology has advanced significantly. FSR can mainly be performed based on convolutional neural network (CNN), generative adversarial network (GAN), ensemble learning, or reinforcement learning. To improve the performance of FSR, complex network structure design may be required. However, as the network structure becomes more complex, it can lead to increases in memory size, amount of computation, and parameters, which can increase the training time and computation cost of the network. Additionally, FSR performance can be improved by utilizing face prior information, but labeling of additional face prior information may be required in the FSR method utilizing face prior information.

A method of operating an electronic device according to an embodiment includes acquiring an initial image feature matrix of the face image based on a multi-level feature map of the face image, and obtaining the face image based on a last-level feature map of the multi-level feature map. Obtaining an initial facial a priori feature matrix and based on the initial image feature matrix and the initial facial a priori feature matrix, using one or more cascaded encoders, a super-resolution image of the facial image and/or a super-resolution image of the facial image Includes an operation to obtain key point coordinates.

The operation of acquiring a super-resolution image of the face image involves obtaining a fused image feature matrix based on the initial image feature matrix and the initial facial a priori feature matrix using a cross attention module included in the one or more encoders. An operation of obtaining an updated image feature matrix of the face image based on the fused image feature matrix using a first deformable attention module included in the one or more encoders, and the updated image feature matrix And it may include an operation of acquiring a super-resolution image of the face image based on the face image.

The operation of acquiring the key point coordinates of the face image involves obtaining a fused image feature matrix based on the initial image feature matrix and the initial facial a priori feature matrix using a cross attention module included in the one or more encoders. The operation of obtaining updated facial a priori features of the facial image based on the fused image feature matrix and the initial facial a priori feature matrix using a second deformable attention module included in the one or more encoders. An operation of predicting key point coordinates of the face image based on the updated facial a priori features and initial key point coordinates of the facial image, wherein the initial key point coordinates of the facial image are based on the initial facial a priori feature matrix. Obtained by - may include.

Each of the one or more encoders includes a first network, a second network, and a third network, the first network including a cross attention module, the second network including a first deformable attention module, and the third network includes a second deformable attention module, and the operation of obtaining a super-resolution image of the face image and/or key point coordinates of the face image includes, for each encoder, an image feature matrix corresponding to the current encoder and a face a priori. Obtaining, based on the feature matrix, a fused image feature matrix of the current encoder using a first network, based on the fused image feature matrix of the current encoder and a facial a priori feature matrix corresponding to the current encoder, a second operation; Obtaining, using a network, an updated facial a priori feature matrix of the current encoder, based on the fused image feature matrix of the current encoder, using a third network, obtaining an updated image feature matrix of the current encoder Obtaining and/or obtaining a super-resolution image of the face image based on the face image and the updated image feature matrix of the last encoder of the one or more encoders and/or the updated face a priori feature matrix of the last encoder and An operation of predicting key point coordinates of the face image based on the initial key point coordinates of the face image, and when the current encoder is a first encoder among one or more encoders, the image feature matrix corresponding to the current encoder is is the initial image feature matrix, and the facial a priori feature matrix corresponding to the current encoder is the initial facial a priori feature matrix, and if the current encoder is not the first encoder, the image feature matrix corresponding to the current encoder is the current encoder is the updated image feature matrix of the previous encoder, and the facial a priori feature matrix corresponding to the current encoder may be the updated facial a priori feature matrix of the previous encoder of the current encoder.

The operation of acquiring the super-resolution image of the face image and/or the key point coordinates of the face image is based on the updated image feature matrix corresponding to the last encoder cascaded, using an upsampling amplification network to generate a first offset. Obtaining the super-resolution image based on the first offset and the facial image and/or based on the updated facial a priori feature matrix corresponding to the cascaded last encoder, using a key point prediction network It may include obtaining an offset and obtaining key point coordinates of the face image predicted based on the second offset and the initial key point coordinates of the face image.

The first network of each encoder further includes a layer normalization layer and a feedforward network layer, and the operation of obtaining the fused image feature matrix of the current encoder includes an image feature matrix corresponding to the current encoder with location information embedded, and location information. Input the facial a priori feature matrix corresponding to the current encoder embedded and the facial a priori feature matrix corresponding to the current encoder as the query vector, key vector, and value vector, respectively, into the cross attention module, and the cascaded cross attention module, layer normalization layer, and It may include an operation of acquiring the fused image feature matrix of the current encoder through a feedforward network layer.

The operation of obtaining an updated image feature matrix of the current encoder using a third network based on the fused image feature matrix of the current encoder is an operation of determining the normalization position of each feature among the fused image feature matrix of the current encoder. - The normalization position indicates the normalization position of a feature in the feature map corresponding to each feature in the corresponding feature map - Near the normalization position of each feature according to a preset rule in each feature map of the multi-level feature map In the operation of determining K normalization positions, weighted summation is performed on L*K features corresponding to the K normalization positions of each feature map of the multi-level feature map in the fused image feature matrix of the current encoder. It may include an operation of acquiring features corresponding to each feature in the fused image feature matrix as features in the updated image feature matrix of the current encoder, where L is the number of feature maps of the multi-level feature map.

The second network of each encoder further includes a self-attention module, and the operation of acquiring the updated facial a priori feature matrix of the current encoder is based on the facial a priori feature matrix corresponding to the current encoder, using the self-attention module. Obtaining a self-attention facial a priori feature matrix corresponding to the current encoder and, based on the self-attention facial a priori feature matrix corresponding to the current encoder and the fused image feature matrix of the current encoder, using a first deformable attention module. The operation may include obtaining an updated facial a priori feature matrix of the current encoder.

The self-attention module includes a cascaded self-attention layer, a layer normalization layer, and a feedforward network layer, and the operation of acquiring the self-attention face a priori feature matrix of the current encoder is a face a priori feature matrix corresponding to the current encoder with embedded location information. The feature matrix, the facial a priori feature matrix corresponding to the current encoder with embedded position information, and the facial a priori feature matrix corresponding to the current encoder are input into the self-attention layer as the query matrix, key matrix, and value matrix, respectively, and a cascaded self-attention layer is formed. , Obtaining a self-attention facial a priori feature matrix of the current encoder through a layer normalization layer and a feedforward network layer, and obtaining an updated facial a priori feature matrix of the current encoder using the second deformable attention module. The operation is to determine the normalization position of each feature in the self-attention face a priori feature matrix of the current encoder in the last level feature map - the normalization position is the feature in the last level feature map corresponding to each feature in the last level feature map. represents the normalization positions of -, an operation of determining K normalization positions near the normalization positions in a final level feature map according to a preset rule, and K corresponding to the K normalization positions in the updated image feature matrix of the current encoder. It may include determining K features, adding weights to the K features, and obtaining features corresponding to each feature in the self-attention facial a priori feature matrix as features of the updated facial a priori feature matrix of the current encoder. .

An electronic device according to an embodiment includes a processor and a memory storing instructions, and when the instructions are executed by the processor, cause the electronic device to execute the method of any one of claims 1 to 9. .

The electronic device according to one embodiment includes a first acquisition unit configured to acquire an initial image feature matrix of the face image based on a multi-level feature map of the face image, and a first acquisition unit configured to acquire an initial image feature matrix of the face image based on a last-level feature map of the multi-level feature map. a super-resolution image of the facial image using a second acquisition unit configured to acquire an initial facial a priori feature matrix of the image and one or more encoders cascaded based on the initial image feature matrix and the initial facial a priori feature matrix and/or a third acquisition unit configured to acquire key point coordinates of the face image.

1 is a diagram illustrating a method of operating an electronic device according to an embodiment.
FIG. 2 is a diagram illustrating an operation of acquiring an initial image feature matrix and an initial facial a priori feature matrix according to an embodiment.
FIG. 3 is a diagram illustrating the structure of a first network layer among one or more encoders according to an embodiment.
FIG. 4 is a diagram illustrating an operation of updating an image feature matrix based on a deformable attention mechanism according to an embodiment.
Figure 5 is a diagram for explaining the structure of a second network layer according to an embodiment.
FIG. 6 is a diagram for explaining an operation of acquiring a super-resolution image of a face image and/or key point coordinates of a face image, according to an embodiment.
Figure 7 is a diagram showing an electronic device according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

In this document, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, “A, Phrases such as “at least one of B, or C,” and “one or a combination of two or more of A, B, and C” each refer to any one of the items listed together with that phrase, or any possible combination thereof. may include. Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 is a diagram illustrating a method of operating an electronic device according to an embodiment.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. Operations 101 to 103 may be performed by at least one component (eg, host processor, accelerator, memory, etc.) of the electronic device.

Electronic devices are devices that process images, for example, various computing devices such as mobile phones, smart phones, tablets, e-book devices, laptops, personal computers, desktops, workstations or servers, smart watches, smart glasses, HMDs ( Head-Mounted Display), or various wearable devices such as smart clothing, various home appliances such as smart speakers, smart TVs, or smart refrigerators, smart cars, smart kiosks, IoT (Internet of Things) devices, and WAD (Walking Assist Device) , drones, or robots, but is not limited to the examples described above. The image may include a face, but is not limited to the above-described example, and may include various objects depending on the embodiment. For convenience of explanation herein, the electronic device may also be referred to as an image processing device or a facial image processing device.

In operation 101, the electronic device obtains an initial image feature matrix of the face image based on the multi-level feature map of the face image. For example, the electronic device may obtain the initial image feature matrix of the face image by flattening and cascading the multi-level feature map of the face image.

In operation 102, the electronic device obtains an initial facial a priori feature matrix of the facial image based on the last level feature map of the multi-level feature map. For example, the electronic device may acquire the initial facial a priori feature matrix of the face image using a fully connected network based on the last level feature map of the multi-level feature map. Each level feature map within the multi-level feature map may have the same number of channels.

Hereinafter, the operation of the electronic device will be described in detail with reference to FIG. 2.

FIG. 2 is a diagram illustrating an operation of acquiring an initial image feature matrix and an initial facial a priori feature matrix according to an embodiment.

Referring to FIG. 2, the electronic device can extract four-level pyramid feature maps F1, F2, F3, and F4 of the input face image through a trained convolutional neural network (e.g., ResNet18). The electronic device can obtain features F1', F2', F3', and F4' by projecting the features of each feature map to have the same number of channels through a 1*1 convolutional network. The electronic device flattens and cascades the acquired four-level feature maps F1', F2', F3', and F4' to create an initial image feature matrix of the face image. can be obtained. among them, may represent the features of the ith row and mth column of the initial image feature matrix, M may represent the number of rows of the initial image feature matrix, and C may represent the number of columns of the initial image feature matrix. For example, C may represent the same number of channels (e.g., C=256). For convenience of explanation, may represent the initial image feature matrix.

In this specification, for convenience of explanation, a face image may also be referred to as an LR image (low resolution image), and a map may also be referred to as a matrix. Additionally, for convenience of explanation, the following description is based on an example in which four levels of feature maps are extracted from an LR image.

For example, a facial a priori feature matrix can be obtained through linear projection on the spatial dimension in F4', and N represents the number of features in the facial a priori feature matrix.

For example, the initial facial a priori feature matrix of the LR image can be obtained according to Equation 1 below.

In equation 1 above, may represent a fully connected operation.

In operation 103, the electronic device generates a super-resolution image of the face image and/or key point coordinates of the face image using one or more cascaded encoders, based on the initial image feature matrix and the initial facial a priori feature matrix. coordinate).

For example, the operation of acquiring a super-resolution image of a face image is fused based on the initial image feature matrix and the initial facial a priori feature matrix using a cross-attention module included in one or more encoders. An operation of acquiring a fused image feature matrix, an operation of obtaining an updated image feature matrix of a face image based on the fused image feature matrix using a first deformable attention module included in one or more encoders. , It may include an operation of acquiring a super-resolution image of the face image based on the updated image feature matrix and the face image.

In this specification, for convenience of explanation, a module may also be referred to as a layer. For example, a cross attention module may also be referred to as a cross attention layer, and a layer normalization layer may also be referred to as a normalization module.

For example, the operation of acquiring the key point coordinates of a face image involves obtaining a fused image feature matrix based on the initial image feature matrix and the initial facial a priori feature matrix using a cross attention module included in one or more encoders. An operation, using a second deformable attention module included in the one or more encoders, to obtain updated facial a priori features of the face image based on the fused image feature matrix and the initial facial a priori feature matrix, the updated facial a priori features. and predicting key point coordinates of the face image based on the initial key point coordinates of the face image. At this time, the initial key point coordinates of the face image can be obtained based on the initial facial a priori feature matrix. For example, the electronic device performs full concatenation on the initial facial a priori feature matrix to obtain the initial key point coordinates of the face image. can do.

For example, each encoder of the one or more encoders includes a first network, a second network, and a third network, the first network includes a cross attention module, and the second network includes a first deformable attention module. And the third network includes a second deformable attention module, wherein the operation of acquiring the super-resolution image of the face image and/or the key point coordinates of the face image is, for each encoder, An operation of obtaining a fused image feature matrix of the current encoder using a first network, based on the image feature matrix and the face a priori feature matrix, the fused image feature matrix of the current encoder and the face a priori feature matrix corresponding to the current encoder. Based on this, an operation of obtaining an updated facial a priori feature matrix of the current encoder using a second network, and based on the fused image feature matrix of the current encoder, an updated image feature matrix of the current encoder using a third network. Obtaining, based on the face image and the updated image feature matrix of the last encoder of one or more encoders, a super-resolution image of the face image, and/or of the updated face a priori feature matrix of the last encoder and the face image. Based on the initial key point coordinates, the operation of predicting the key point coordinates of the face image may be included. If the current encoder is the first encoder among one or more encoders, the image feature matrix corresponding to the current encoder may be the initial image feature matrix, and the face a priori feature matrix corresponding to the current encoder may be the initial face a priori feature matrix. If the current encoder is not the first encoder, the image feature matrix corresponding to the current encoder is the updated image feature matrix of the previous encoder of the current encoder, and the face a priori feature matrix corresponding to the current encoder is the updated image feature matrix of the previous encoder of the current encoder. It may be a facial a priori feature matrix.

For example, the one or more encoders may include a first encoder and a second encoder, but are not limited to the examples described above.

For example, the electronic device may obtain the fused image feature matrix M12 of the first encoder using the first network of the first encoder, based on the initial image feature matrix M11 and the initial facial a priori feature matrix F11. The electronic device may obtain an updated facial a priori feature matrix F12 of the first encoder based on the initial facial a priori feature matrix F11 and the fused image feature matrix M12 of the first encoder, using the second network of the first encoder. there is. Based on the fused image feature matrix M12 of the first encoder, the electronic device may obtain the updated image feature matrix M13 of the first encoder using the third network of the first encoder. Based on the updated facial a priori feature matrix F12 of the first encoder and the updated image feature matrix M13 of the first encoder, the electronic device uses the first network of the second encoder to generate a fused image feature matrix M22 of the second encoder. can be obtained. Based on the fused image feature matrix M22 of the second encoder and the updated facial a priori feature matrix F12 of the first encoder, the electronic device uses the second network of the second encoder to update the facial a priori feature matrix of the second encoder. You can obtain F22. Based on the fused image feature matrix M22 of the second encoder, the electronic device may obtain the updated image feature matrix M23 of the second encoder using the third network of the second encoder.

For example, the electronic device may acquire a super-resolution image of the face image based on the updated image feature matrix M23 of the second encoder and the face image.

For example, the electronic device may predict the key point coordinates of the face image based on the updated facial a priori feature matrix F22 of the second encoder and the initial key point coordinates of the face image.

For example, the operation of obtaining a super-resolution image of a face image and/or key point coordinates of a face image may be performed using an up sampling amplification network, based on the updated image feature matrix corresponding to the last encoder cascaded. ), and obtaining a super-resolution image based on the first offset and the face image and/or based on the updated face a priori feature matrix corresponding to the last encoder cascaded, a key point prediction network. Obtaining a second offset using , and obtaining key point coordinates of the predicted face image based on the second offset and the initial key point coordinates of the face image, wherein the key point coordinates of the face image are the initial key point coordinates of the face image. It can be obtained by performing full concatenation on the a priori feature matrix.

For example, the electronic device may obtain a first offset using an upsampling amplification network based on the updated image feature matrix M23 of the second encoder, and obtain a super-resolution image based on the first offset and the face image. there is. Based on the updated facial a priori feature matrix F22 of the second encoder, the electronic device obtains a second offset using a key point prediction network, and predicts the face image based on the second offset and the initial key point coordinates of the face image. You can obtain key point coordinates. At this time, the initial key point coordinates of the face image can be obtained by performing full concatenation on the initial facial a priori feature matrix.

For example, the first network of each encoder further includes a layer normalization (LN) layer and a feedforward network (FFN) layer, where the fused image feature matrix of the current encoder The operation of acquiring is an image feature matrix corresponding to the current encoder with embedded position information, a facial a priori feature matrix corresponding to the current encoder with embedded position information, and a facial a priori feature matrix corresponding to the current encoder, respectively, as a query vector and key. It includes the operation of inputting a vector and a value vector to the cross-attention module, and obtaining the fused image feature matrix of the current encoder through the cascaded cross-attention module, layer normalization layer, and feedforward network layer.

FIG. 3 is a diagram illustrating the structure of a first network layer among one or more encoders according to an embodiment.

Referring to FIG. 3, the first network may include a cross attention module, a layer normalization layer, and a feedforward network layer.

For example, the image feature matrix Q corresponding to the current encoder with embedded location information can be obtained through Equation 2 below.

In Equation 2 above, of the corresponding original feature map in the original feature map. Indicates the location of the corresponding feature, may represent an image feature matrix corresponding to each encoder without embedded location information.

for example, Features of this first level feature map If you correspond to Is It can be obtained based on is in the first level feature map can indicate the location of

For example, the key matrix K of the input cross attention module can be obtained through Equation 3 below.

In Equation 3 above, Is Indicates the position in the last level feature map (that is, the last level feature) of the feature corresponding to the feature map, where: may represent a facial a priori feature matrix corresponding to each encoder that does not include location information. for example, Some features of The corresponding features in the last level feature map are ego, is in the last level feature map. can indicate the location of

Additionally, the value matrix of the input cross attention module is It can be. The output MHCA(Q, K, V) of the cross attention module can be expressed as Equation 4 below.

In equation 4 above, may be the dimension of the row vector of the key matrix.

Based on the output of the cross-attention module, the electronic device uses a layer normalization layer and a feedforward network layer to generate the fused image feature matrix of each encoder. can be obtained. According to an embodiment of the present disclosure, the electronic device obtains a fused image feature matrix by inputting the corresponding image feature matrix and face a priori feature matrix of each encoder into a multi-head attention layer, and based on this cross attention mechanism. Since the obtained fused image feature matrix integrates facial a priori feature information, it can better reflect the correlation between facial image features.

For convenience of explanation, the feature positions of the image feature matrix of the original feature map corresponding to the features of the original feature map may be expressed as feature positions of the image feature matrix of the original feature map.

For example, the operation of acquiring the updated image feature matrix of the current encoder using a third network is the operation of determining the normalization position of each feature among the fused image feature matrix of the current encoder - at this time, the normalization position is the corresponding Indicates the normalization position of the feature in the feature map corresponding to each feature in the feature map - An operation of determining K normalization positions near the normalization position of each feature according to a preset rule in each feature map of the multi-level feature map; In the fused image feature matrix of the current encoder, weighted sum is performed on L*K features corresponding to the K normalized positions of each feature map of the multi-level feature map to correspond to each feature in the fused image feature matrix of the current encoder. It includes an operation of acquiring the feature as a feature in the updated image feature matrix of the current encoder, where L is the number of feature maps of the multi-level feature map, and may be, for example, L=4.

For example, the third network may include a deformable-attention layer, a residual summation and layer normalization layer (Add&Norm), and a feedforward network layer (FFN).

Below, an example of obtaining features from the encoder's updated image feature matrix using a third network will be described.

For example, in the fused image feature matrix output by the first network, layer information and position information corresponding to each feature can be inserted and added to each feature by Equation 5 below.

In Equation 5 above, represents the ith feature in the fused image feature matrix, represents the original feature map (that is, some level feature map in a multi-level feature map) corresponding to the ith feature, represents the position in the original feature map of the feature corresponding to the ith feature in the original feature map corresponding to the ith feature, may represent the i-th feature to which layer information and location information are added.

For example, the spatial location of each feature in its original feature map using its normalized coordinates. can represent, may represent the normalized spatial location of the ith feature in the fused image feature matrix in its corresponding original feature map. For example, (0,0) and (1,1) may represent the normalized spatial positions corresponding to the upper left and lower right features of the original feature map, respectively. These normalized coordinates can be used as reference points for sampling relevant features.

For example, in the fused image feature matrix In the case of. The normalized coordinates of the corresponding original feature map are And the electronic device is By sampling multiple features from the surroundings, cast It can be updated with . Normalized coordinates corresponding to a plurality of sampling features can be expressed as Equation 6 below.

In Equation 6 above, are the normalized coordinates corresponding to the features sampled from the original feature map, , k=1, ??, K, and K may be a preset value.

After determining the normalization coordinates corresponding to the multi-sampling features, the electronic device generates a fused image matrix based on the determined normalization coordinates. Features corresponding to can be determined, and using equation 7 below, can be decided.

In Equation 7 above, class is the learnable weight matrix, Can be obtained from Equation 8 or Equation 9 below.

For example, the electronic device may include a feature of a second level feature map of the pyramid feature map. Some features of the fused image feature matrix, corresponding to For, in the second level feature map Normalized position of Determine, and based on Equation 6 above, K coordinates nearby can be determined. At this time, the K coordinates may correspond to K coordinates in the second level feature map, and the K coordinates may correspond to K features in the fused image matrix. In other words, for the K features corresponding to the K coordinates in the fused image feature matrix, the electronic device uses Equation 7 above to Updated features corresponding to can be calculated.

As mentioned earlier, electronic devices In calculating, By sampling K features from the fused image feature matrix based only on the original feature map (e.g., second-level feature map) corresponding to can be obtained.

In another embodiment, an electronic device may be configured to integrate multi-level image features. For , K features are sampled from each level feature map, and through Equation 10 below: can be obtained.

In Equation 10 above, L is the number of multi-level feature maps (for example, if the extracted pyramid feature map is a 4-level feature map, L = 4), may represent the kth feature sampled from the fused image matrix based on the jth level feature map. The position coordinates corresponding to are And at this time Through Equation 11 below, It can be obtained through a linear projection of .

For example, the electronic device may use some features of the fused image feature matrix. For , K coordinates of each level feature map can be determined through Equation 11 above, and K features corresponding to the K coordinates of each level feature map among the fused image feature matrices can be determined. In this case, the electronic device can determine K*L features from the fused image feature matrix, and the updated can be decided.

FIG. 4 is a diagram illustrating an operation of updating an image feature matrix based on a deformable attention mechanism according to an embodiment.

In Figure 4, the fused image feature matrix Image feature matrix updated using a deformable attention mechanism based on An example diagram of obtaining is shown.

Referring to Figure 4, the electronic device By using the features corresponding to the multi-level feature map of Update the features corresponding to each level feature map, and through this, the feature matrix Features corresponding to each feature map can be obtained. Figure 4 shows an example to explain the structure of the third network layer.

For example, electronic devices About some features of Obtain K features from the features corresponding to the first level feature map, Obtain K features from the features corresponding to the second level feature map, Obtain K features from the features corresponding to the third level feature map, K features are acquired from the features corresponding to the first level feature map, and updated features of some features are obtained based on the obtained 4K features. can be determined as a feature corresponding to some feature. thus, For example, the feature corresponding to the first level feature map may include information of each level feature map.

Each feature of By integrating the information of the features corresponding to the multi-level feature maps, this allows better consideration of the information corresponding to the lower-level feature maps, thereby better reflecting the correlation between local features of the face image. .

The method of sampling K features based on each feature map described in this specification is only an example, and collecting K or L*K features in another method to update the features of the second image matrix is also possible without limitation. It can be applied.

For example, the fused image feature matrix can be updated into an updated image feature matrix by sampling only the features of some feature maps.

For example, the second network of each encoder further includes a self-attention module, wherein the operation of obtaining the updated facial a priori feature matrix of the current encoder is based on the facial a priori feature matrix corresponding to the current encoder, An operation of using an attention module to obtain a self-attention face a priori feature matrix corresponding to the current encoder, based on the self-attention face a priori feature matrix corresponding to the current encoder and the fused image feature matrix of the current encoder, a first deformable attention It may include an operation of obtaining an updated facial a priori feature matrix of the current encoder using a module.

For example, the self-attention module includes a cascaded self-attention layer, a layer normalization layer, and a feedforward network layer, where the operation of acquiring the self-attention face a priori feature matrix of the current encoder is the current encoder with embedded location information. Input the corresponding facial a priori feature matrix, the facial a priori feature matrix corresponding to the current encoder with embedded location information, and the facial a priori feature matrix corresponding to the current encoder as the query matrix, key matrix, and value matrix, respectively, into the self-attention layer, It may include an operation of acquiring the self-attention facial a priori feature matrix of the current encoder through a cascaded self-attention layer, a layer normalization layer, and a feedforward network layer. At this time, the operation of acquiring the updated facial a priori feature matrix of the current encoder using the second deformable attention module is the operation of determining the normalization position of each feature among the self-attention facial a priori feature matrix of the current encoder in the last level feature map. - At this time, the normalization position represents the normalization position of the feature in the last level feature map corresponding to each feature in the last level feature map. -, Determine K normalization positions near the normalization position in the final level feature map according to preset rules. The operation is to determine the K features corresponding to the K normalization positions in the updated image feature matrix of the current encoder, add the weights to the K features, and select the features corresponding to each feature in the self-attention face a priori feature matrix to the current encoder. It may include an operation of acquiring features of the updated facial a priori feature matrix.

Figure 5 is a diagram for explaining the structure of a second network layer according to an embodiment.

Figure 5 is a diagram for explaining the structure of a second network layer according to an embodiment.

Referring to Figure 5, first, the electronic device generates a self-attention facial a priori feature matrix based on the self-attention module. can be obtained. At this time, the self-attention module may include a self-attention layer, a normalization layer (Add&Norm), and a feedforward network layer (FFN). The query matrix, key matrix, and value matrix of the input self-attention layer are, respectively, a facial a priori feature matrix corresponding to each encoder with embedded location information, a facial a priori feature matrix corresponding to each encoder with embedded location information, and a facial a priori feature matrix corresponding to each encoder. It may be a facial a priori feature matrix. Electronic devices can learn dependencies between facial a priori features by updating the facial a priori feature matrix based on a self-attention mechanism. Therefore, the self-attention facial a priori features obtained based on the self-attention mechanism can reflect the structural information of the facial a priori features and better represent the input image.

Self-attention layer output of self-attention module Can be expressed as Equation 12 below.

In Equation 12 above, Q, K, and V may represent a query matrix, key matrix, and value matrix respectively input to the self-attention layer.

electronic devices Based on this, using the layer normalization layer (Add&Norm) and feedforward network layer (FFN) can be obtained.

Then, the electronic device Facial a priori feature matrix updated using a deformable attention mechanism based on can be obtained.

Referring to FIG. 2, the deformable attention module may include a deformable attention layer, a layer normalization layer (LN), and a feedforward network layer (FFN).

Below, the updated facial a priori feature matrix. An example of the calculation process for will be described.

For example, an electronic device may use a self-attention facial a priori feature matrix For , we can first determine the normalized position of the features of the original feature map in the corresponding original feature map (i.e., the last level feature map). For example, (0,0) and (1,1) are the spatial positions of features in the last level feature map, top left and bottom right, respectively. can represent.

Then, the electronic device follows equation 13 below: Determine K locations in the surrounding area.

In Equation 13 above, Is ego, Is The facial a priori features of the embedded location information corresponding to can be expressed.

The electronic device can determine features corresponding to the K positions in the fused image feature matrix. For example, in the fused image feature matrix, there are features corresponding to each normalization position of the final level feature map, and K features of the fused image feature matrix can be determined based on the K normalization positions.

after that, is one of the updated facial a priori feature matrices based on the K features based on Equation 14 below. As a feature corresponding to, can be updated with

In Equation 14 above, W is the learnable weight matrix, and, or It can be.

FIG. 6 is a diagram illustrating an operation of acquiring a super-resolution image of a face image and/or key point coordinates of a face image according to an embodiment.

Referring to FIG. 6, the T cascaded encoders may obtain the updated image feature matrix and/or the updated facial a priori feature matrix of the last encoder. The electronic device may obtain a super-resolution image of the face image by obtaining a first offset through an upsampling network, and/or obtain key point coordinates of the predicted face image by obtaining a second offset through a key point prediction network. You can.

For example, the electronic device cuts out features corresponding to the first-level feature map from the updated image feature matrix of the last encoder, and outputs the first-level feature map through a convolution layer and a pixel shuffle layer. By upsampling and amplifying the corresponding features, the super-resolution image and LR images (i.e., offset between input images) can be obtained. The electronic device can acquire a super-resolution image based on Equation 15 below.

For example, the electronic device determines the offset between the key point coordinates of the face image predicted through MLP (Multi-layer Perceptron) and the key point coordinates of the initial face image based on Equation 16 below. can be obtained.

In equation 16 above, may represent the facial a priori feature matrix of the last encoder.

For example, electronic devices use a three-layer fully connected network with a Rectified Linear Unit (ReLU) activation function to can be obtained. At this time, the first two layers are composed of linear complete connection followed by a ReLU activation function, and the last layer is composed of full connection through full connection. can be printed directly.

After is acquired, the predicted key point coordinates of the face image can be obtained by Equation 17 below.

In equation 17 above, Is represents a function, may represent the initial key point coordinates of the face image.

For example, the entire model used to obtain key point coordinates of a super-resolution image and/or a predicted face image may be trained using a Loss function for FSR and a Loss function for face marking.

For example, the loss in FSR may include Pixel loss, Adversarial loss, and Perceptual loss, while the loss in face marking may include Consistency loss and Separation constraint. ) may be included.

FSR can restore a low-resolution face image to a high-resolution face image, and the FSR network can be trained to obtain a corresponding high-resolution correct image by inputting a low-resolution learning image into the FSR network. Through this, FSR performance can be effectively improved while reducing the complexity of the FSR network without additional facial a priori information.

A facial image processing method in an electronic device according to an embodiment of the present disclosure has been described with reference to FIGS. 1 to 6 . Hereinafter, an electronic device for processing a face image according to an embodiment of the present disclosure will be described with reference to FIG. 7.

Figure 7 is a diagram showing an electronic device according to an embodiment.

Referring to FIG. 7 , the electronic device 700 that processes a facial image may include a first acquisition unit 701, a second acquisition unit 702, and a third acquisition unit 703. However, the electronic device 700 is not limited to the examples described above, and may additionally include other components, and one or more components of the electronic device 700 may be divided or combined.

For example, the first acquisition unit 701 may acquire the initial image feature matrix of the face image based on the multi-level feature map of the face image.

For example, the second acquisition unit 702 may acquire the initial facial a priori feature matrix of the face image based on the last level feature map of the multi-level feature map.

For example, the third acquisition unit 703 acquires a super-resolution image of the face image and/or key point coordinates of the face image using one or more encoders cascaded based on the initial image feature matrix and the initial facial a priori feature matrix. can do.

For example, the third acquisition unit 703 acquires a fused image feature matrix based on the initial image feature matrix and the initial facial a priori feature matrix using a cross attention module included in one or more encoders, and one Using the first deformable attention module included in the above encoders, an updated image feature matrix of the face image is obtained based on the fused image feature matrix, and an updated image feature matrix of the face image is obtained based on the updated image feature matrix and the face image. Super-resolution images can be obtained.

For example, the third acquisition unit 703 acquires a fused image feature matrix based on the initial image feature matrix and the initial facial a priori feature matrix using a cross attention module included in one or more encoders, and one Using the second deformable attention module included in the above encoders, updated facial a priori features of the face image are obtained based on the fused image feature matrix and the initial facial a priori feature matrix, and the updated facial a priori features and the face image are obtained. Based on the initial key point coordinates of , the key point coordinates of the face image can be predicted. At this time, the initial key point coordinates of the face image can be obtained based on the initial facial a priori feature matrix.

For example, each of the one or more encoders includes a first network, a second network, and a third network, the first network including a cross attention module, the second network including a first deformable attention module, and The third network may include a second deformable attention module.

For example, the third acquisition unit 703, for each encoder, uses the first network to determine the fused image feature matrix of the current encoder, based on the image feature matrix and the face a priori feature matrix corresponding to the current encoder. Obtain, based on the fused image feature matrix of the current encoder and the facial a priori feature matrix corresponding to the current encoder, use a second network to obtain an updated facial a priori feature matrix of the current encoder, and obtain the fused image feature matrix of the current encoder. Based on the image feature matrix, an updated image feature matrix of the current encoder can be obtained using a third network. The third acquisition unit 703 acquires a super-resolution image of the face image based on the face image and the updated image feature matrix of the last encoder of the one or more encoders, and/or updated face a priori features of the last encoder. Based on the matrix and the initial key point coordinates of the face image, the key point coordinates of the face image can be predicted. At this time, if the current encoder is the first encoder among one or more encoders, the image feature matrix corresponding to the current encoder may be an initial image feature matrix, and the facial a priori feature matrix corresponding to the current encoder may be an initial facial a priori feature matrix. If the current encoder is not the first encoder, the image feature matrix corresponding to the current encoder is the updated image feature matrix of the previous encoder of the current encoder, and the face a priori feature matrix corresponding to the current encoder is the updated image feature matrix of the previous encoder of the current encoder. It may be a facial a priori feature matrix.

For example, the third acquisition unit 703 obtains the first offset using an upsampling amplification network, based on the updated image feature matrix corresponding to the last encoder cascaded, and based on the first offset and the face image. acquire a super-resolution image, and/or obtain a second offset using a key point prediction network based on the updated facial a priori feature matrix corresponding to the last encoder cascaded, and obtain the second offset and the initial key of the face image. Based on the point coordinates, the key point coordinates of the predicted face image can be obtained. At this time, the initial key point coordinates of the face image can be obtained by completely concatenating the initial facial a priori feature matrix.

For example, the first network of each encoder may further include a level normalization layer and a feedforward network layer.

For example, the third acquisition unit 703 may include an image feature matrix corresponding to the current encoder with embedded position information, a facial a priori feature matrix corresponding to the current encoder with embedded position information, and a facial a priori feature matrix corresponding to the current encoder with embedded position information. By inputting the matrices as the query vector, key vector, and value vector into the cross-attention module, respectively, the fused image feature matrix of the current encoder can be obtained through the cascaded cross-attention module, layer normalization layer, and feedforward network layer.

For example, the third acquisition unit 703 determines the normalization position of each feature in the fused image feature matrix of the current encoder - where the normalization position is in the feature map corresponding to each feature in the corresponding feature map. Indicates the normalization position of the feature -, Determine K normalization positions near the normalization position of each feature according to preset rules in each feature map of the multi-level feature map, and determine the multi-level feature map in the fused image feature matrix of the current encoder. Weighted summation is performed on the L*K features corresponding to the K normalized positions of each feature map, and the features corresponding to each feature in the fused image feature matrix of the current encoder are combined with the features in the updated image feature matrix of the current encoder. It can be obtained with At this time, L may be the number of feature maps of the multi-level feature map.

For example, the second network of each encoder further includes a self-attention module, wherein the third acquisition unit determines the corresponding to the current encoder using the self-attention module, based on the facial a priori feature matrix corresponding to the current encoder. Obtain a self-attention face a priori feature matrix, and based on the self-attention face a priori feature matrix corresponding to the current encoder and the fused image feature matrix of the current encoder, use the first deformable attention module to generate an updated face of the current encoder. An a priori feature matrix can be obtained.

For example, the self-attention module may include a cascaded self-attention layer, a layer normalization layer, and a feedforward network layer. For example, the third acquisition unit 703 may include a facial a priori feature matrix corresponding to the current encoder with embedded position information, a facial a priori feature matrix corresponding to the current encoder with embedded position information, and a facial a priori feature matrix corresponding to the current encoder with embedded position information. Input the feature matrix into the self-attention layer as the query matrix, key matrix, and value matrix, respectively, to obtain the self-attention face a priori feature matrix of the current encoder through the cascaded self-attention layer, layer normalization layer, and feedforward network layer, and finally In the level feature map, determine the normalization position of each feature among the self-attention face a priori feature matrix of the current encoder - at this time, the normalization position represents the normalization position of the feature in the last level feature map corresponding to each feature in the last level feature map. -, Determine K normalization positions near the normalization positions in the final level feature map according to preset rules, determine K features corresponding to the K normalization positions in the updated image feature matrix of the current encoder, and K features By adding the weights, the features corresponding to each feature in the self-attention facial a priori feature matrix can be obtained as features of the updated facial a priori feature matrix of the current encoder.

According to an embodiment of the present disclosure, an electronic device is provided, the electronic device includes a processor and a memory storing a computer program, and the computer program, when executed by the processor, can implement a facial image processing method.

According to one embodiment, the electronic device 700 may include a memory (not shown) and a processor (not shown).

Memory may contain instructions that can be read by the computer. The processor can perform the operations described above as instructions stored in memory are executed on the processor. The memory may be volatile memory or non-volatile memory.

A processor is a device that executes instructions or programs or controls the electronic device 700 and may include, for example, a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), or a Neural Processing Unit (NPU). However, it is not limited to the examples described above.

In addition, the electronic device 700 can process the operations described above.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be saved in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may store program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (12)

In a method of operating an electronic device,
Obtaining an initial image feature matrix of the face image based on a multi-level feature map of the face image;
Obtaining an initial facial a priori feature matrix of the face image based on the last level feature map of the multi-level feature map; and
An operation of obtaining a super-resolution image of the face image and/or key point coordinates of the face image using one or more cascaded encoders, based on the initial image feature matrix and the initial facial a priori feature matrix.
Including,
How electronic devices work.
According to paragraph 1,
The operation of acquiring a super-resolution image of the face image is
Obtaining a fused image feature matrix based on the initial image feature matrix and the initial facial a priori feature matrix using a cross attention module included in the one or more encoders;
Obtaining an updated image feature matrix of the face image based on the fused image feature matrix using a first deformable attention module included in the one or more encoders; and
Obtaining a super-resolution image of the face image based on the updated image feature matrix and the face image
Including,
How electronic devices work.
According to paragraph 2,
The operation of acquiring the key point coordinates of the face image is
Obtaining a fused image feature matrix based on the initial image feature matrix and the initial facial a priori feature matrix using a cross attention module included in the one or more encoders;
Obtaining updated facial a priori features of the face image based on the fused image feature matrix and the initial facial a priori feature matrix using a second deformable attention module included in the one or more encoders;
An operation of predicting key point coordinates of the face image based on the updated facial a priori features and initial key point coordinates of the face image - initial key point coordinates of the face image are obtained based on the initial facial a priori feature matrix became -;
Including,
How electronic devices work.
According to paragraph 3,
Each of the one or more encoders includes a first network, a second network, and a third network, the first network including a cross attention module, the second network including a first deformable attention module, and the third network includes a second deformable attention module,
The operation of acquiring a super-resolution image of the face image and/or key point coordinates of the face image
For each encoder, based on the image feature matrix and the facial a priori feature matrix corresponding to the current encoder, obtaining a fused image feature matrix of the current encoder using a first network;
Obtaining, using a second network, an updated facial a priori feature matrix of the current encoder, based on the fused image feature matrix of the current encoder and a facial a priori feature matrix corresponding to the current encoder;
Obtaining an updated image feature matrix of the current encoder using a third network, based on the fused image feature matrix of the current encoder; and
Obtaining a super-resolution image of the face image based on the face image and the updated image feature matrix of the last encoder of the one or more encoders and/or the updated face a priori feature matrix of the last encoder and the initial image of the face image An operation of predicting key point coordinates of the face image based on key point coordinates
Including,
When the current encoder is the first encoder among one or more encoders, the image feature matrix corresponding to the current encoder is the initial image feature matrix, and the face a priori feature matrix corresponding to the current encoder is the initial face a priori feature matrix,
When the current encoder is not the first encoder, the image feature matrix corresponding to the current encoder is the updated image feature matrix of the previous encoder of the current encoder, and the facial a priori feature matrix corresponding to the current encoder is the updated image feature matrix of the previous encoder of the current encoder. The updated facial a priori feature matrix of the previous encoder,
How electronic devices work.
According to paragraph 4,
The operation of acquiring a super-resolution image of the face image and/or key point coordinates of the face image,
Obtaining a first offset using an upsampling amplification network based on an updated image feature matrix corresponding to the last cascaded encoder, and obtaining the super-resolution image based on the first offset and the face image; and/or
Based on the updated facial a priori feature matrix corresponding to the last cascaded encoder, a second offset is obtained using a key point prediction network, and the face predicted based on the second offset and the initial key point coordinates of the face image An operation to obtain key point coordinates of an image
Including,
How electronic devices work.
According to clause 4,
The first network of each encoder further includes a layer normalization layer and a feedforward network layer,
The operation of acquiring the fused image feature matrix of the current encoder is
Cross the image feature matrix corresponding to the current encoder with embedded position information, the face a priori feature matrix corresponding to the current encoder with embedded position information, and the face a priori feature matrix corresponding to the current encoder as the query vector, key vector, and value vector, respectively. An operation to obtain the fused image feature matrix of the current encoder through the cascaded cross attention module, layer normalization layer, and feedforward network layer by inputting it to the attention module.
Including,
How electronic devices work.
According to clause 4,
The operation of obtaining an updated image feature matrix of the current encoder using a third network based on the fused image feature matrix of the current encoder is
Determining a normalized position of each feature in the fused image feature matrix of the current encoder, wherein the normalized position indicates a normalized position of a feature in the feature map corresponding to each feature in the corresponding feature map;
determining K normalization positions near the normalization positions of each feature according to a preset rule in each feature map of the multi-level feature map;
Weighted sum is performed on L*K features corresponding to K normalized positions of each feature map of the multi-level feature map in the fused image feature matrix of the current encoder, and each feature in the fused image feature matrix of the current encoder is Obtaining the corresponding feature as a feature in the updated image feature matrix of the current encoder, where L is the number of feature maps of the multi-level feature map;
Including,
How electronic devices work.
According to clause 4,
The second network of each encoder further includes a self-attention module,
The operation of obtaining the updated facial a priori feature matrix of the current encoder is
Based on the facial a priori feature matrix corresponding to the current encoder, obtaining a self-attention facial a priori feature matrix corresponding to the current encoder using a self-attention module; and
Based on the self-attention facial a priori feature matrix corresponding to the current encoder and the fused image feature matrix of the current encoder, obtaining an updated facial a priori feature matrix of the current encoder using a first deformable attention module.
Including,
How electronic devices work.
According to clause 8,
The self-attention module includes a cascaded self-attention layer, a layer normalization layer, and a feedforward network layer,
The operation of acquiring the self-attention face a priori feature matrix of the current encoder is
The facial a priori feature matrix corresponding to the current encoder with embedded position information, the facial a priori feature matrix corresponding to the current encoder with embedded position information, and the facial a priori feature matrix corresponding to the current encoder as the query matrix, key matrix, and value matrix, respectively. An operation to obtain the self-attention face a priori feature matrix of the current encoder by inputting it into the self-attention layer and through the cascaded self-attention layer, layer normalization layer, and feedforward network layer.
Including,
The operation of acquiring the updated facial a priori feature matrix of the current encoder using the second deformable attention module is
An operation of determining the normalization position of each feature among the self-attention face a priori feature matrix of the current encoder in the last level feature map - the normalization position is the normalization position of the feature in the last level feature map corresponding to each feature in the last level feature map. Indicates location -;
Determining K normalization positions near the normalization positions in the final level feature map according to a preset rule; and
Determine K features corresponding to the K normalized positions in the updated image feature matrix of the current encoder, add weights to the K features, and select the current features corresponding to each feature in the self-attention facial a priori feature matrix. Obtaining motion as a feature of the encoder's updated facial a priori feature matrix.
Including,
How electronic devices work.
A computer-readable recording medium storing a computer program that executes the method of any one of claims 1 to 9.
In electronic devices,
processor; and
Memory that stores instructions
Including,
When the instructions are executed by the processor, causing the electronic device to execute the method of any one of claims 1 to 9,
Electronic devices.
In electronic devices,
a first acquisition unit configured to acquire an initial image feature matrix of the face image based on a multi-level feature map of the face image;
a second acquisition unit configured to acquire an initial facial a priori feature matrix of the face image based on the last level feature map of the multi-level feature map; and
A third acquisition unit configured to acquire a super-resolution image of the face image and/or key point coordinates of the face image using the initial image feature matrix and one or more encoders cascaded based on the initial facial a priori feature matrix.
Including,
Electronic devices.