KR20230071720A - Method of predicting landmark coordinates of facial image and Apparatus thereof - Google Patents

Abstract

A method and apparatus for predicting landmark coordinates of a face image are provided. The method may include acquiring a multi-stage feature map of a face image through a convolutional neural network layer; Obtaining an initial query matrix by fully connecting the last stage feature maps of the multi-stage feature maps through a fully connected layer, wherein the initial query matrix represents initial features of landmarks of a face image, wherein the initial query matrix The number of elements in the matrix is equal to the number of landmarks in the face image -; obtaining a memory feature matrix by flattening and concatenating the multi-stage feature maps; and inputting the memory feature matrix and the initial query matrix to at least one decoder layer of the cascade to determine landmark coordinates of the face image.

Description

Method of predicting landmark coordinates of facial image and Apparatus thereof}

The present disclosure relates to the field of face landmark detection technology, and more specifically, to a method and apparatus for predicting landmark coordinates of a face image.

Facial landmark detection, also referred to as face alignment, aims to automatically find reference landmarks in face images. It is an important part of face analysis tasks such as face recognition, expression analysis, face frontalization and 3D face reconstruction.

Recent rapid development of deep neural network technology has greatly improved facial landmark detection. Existing facial landmark detection methods are largely divided into a coordinates regression-based method and a heat map regression-based method.

Coordinate regression-based methods directly map input images to landmark coordinates. In models using deep learning frameworks, input images are fed into a CNN model to obtain image features. Then, map image features directly to coordinates via a fully connected prediction layer. To increase the detection accuracy, the coordinate regression module is usually configured or integrated step-wise with the heatmap regression module.

The heatmap regression-based method first generates a heatmap based on given landmark coordinates used as a proxy for ground-truth (GT) values. Here, each heat map represents a probability of a landmark location. Models based on heatmap regression usually predict the heatmap through a full convolution network, and then obtain landmarks according to the probabilities of peak values in the heatmap. Heatmap-based models generally outperform coordinate regression-based models because they can maintain the spatial structure of image features.

Although the heatmap regression-based method has relatively high accuracy, 1) a post-processing step is required to map the heatmap probability output to coordinates, but the post-processing step is non-differentiable, so the end-to-end end) Disable training. 2) Considering the complexity of calculation, since the resolution of the heat map is generally lower than the resolution of the input image, quantization errors inevitably occur and further improvement is limited. 3) Generation of a hand-designed GT value heatmap introduces heuristic hyperparameters to tune.

Coordinate regression-based methods can bypass the limitations described above and implement end-to-end model training. In this case, if the fully connected prediction layer is used, the spatial structure of image features may be destroyed. The mapping from global features to landmark coordinates is like a black box where features and predictions are not aligned, which can degrade positioning performance.

There is a need for a method and device capable of more accurately predicting landmark coordinates of a face image, and the method and device can perform end-to-end training well without the need for heuristic post-processing compared to heatmap-based methods and devices. In addition, most relevant image features for coordinate prediction around landmarks can be extracted.

According to an embodiment of the present disclosure, a method of predicting landmark coordinates of a face image is provided, and the method includes acquiring a multi-stage feature map of the face image through a convolutional neural network layer. ; Obtaining an initial query matrix by fully connecting last stage feature maps of the multi-stage feature maps through a fully connected layer, the initial query matrix representing initial features of landmarks of the face image, The number of elements in the initial query matrix is equal to the number of landmarks of the face image -; obtaining a memory feature matrix by flattening and concatenating the multi-stage feature maps; and determining the landmark coordinates of the face image by inputting the memory feature matrix and the initial query matrix to at least one cascaded decoder layer.

Optionally, the decoder layer includes a cascaded self-attention module layer, a deformable attention module layer, and a landmark coordinate prediction layer, and the step of determining the landmark coordinates of the face image includes location information inputting the embedded initial query matrix and the initial query matrix to the self-attention module layer of the first decoder layer as a query matrix, a key matrix, and a value matrix; Transformable Attention obtaining an output matrix of a module layer, wherein the output matrix and the memory feature matrix of the self-attention module layer are the query matrix and the value matrix of the transformable attention module layer, and the transformation of the first decoder layer The landmark coordinates predicted in the previous decoder layer input to the possible attention module layer are initial landmark coordinates obtained based on the initial query matrix -; The output matrix of the transformable attention module layer of the current decoder layer and the output matrix of the transformable attention module layer in which the position information is embedded are cascaded into a value matrix of a self-attention module layer of the next decoder layer. , as a query matrix and a key matrix, inputting to the self-attention module layer of the next decoder layer; and obtaining the landmark coordinates of the face image predicted by the current decoder layer by inputting the output matrix of the deformable attention module layer of the current decoder layer to a landmark coordinate prediction layer of the current decoder layer. Step - landmark coordinates of the face image predicted by the last decoder layer are landmark coordinates of the final face image.

Optionally, the output matrix of the self-attention module layer of the decoder layer,

는 출력 행렬의 i번째 행 벡터를 나타내고,

는 셀프 어텐션 모듈 레이어에 입력된 쿼리 행렬의 i번째 행 벡터와 셀프 어텐션 모듈 레이어에 입력된 키 행렬의 j번째 행 벡터의 내적(dot product)을 정규화(normalization)하여 획득한 어텐션 가중치를 나타내고,

는 초기 쿼리 행렬 또는 이전 디코더 레이어의 변형 어텐션 레이어의 출력 행렬의 j번째 행 벡터를 나타내고, N은 얼굴 이미지의 랜드마크 개수를 나타낼 수 있다.is obtained as, where,

denotes the i-th row vector of the output matrix,

Represents the attention weight obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module layer and the j-th row vector of the key matrix input to the self-attention module layer,

represents a j-th row vector of an initial query matrix or an output matrix of a transformation attention layer of a previous decoder layer, and N may represent the number of landmarks in a face image.

Optionally, the output matrix of the transformable attention module layer of the decoder layer,

는 출력 행렬의 i번째 랜드마크의 업데이트된 특징을 나타내고,

는 변형 가능 어텐션 모듈 레이어에 입력된 쿼리 행렬에 대해 완전 연결 연산(fully connected operation) 및 softmax 연산을 수행하여 획득한 어텐션 가중치를 나타내고,

는 메모리 특징 행렬 중 k번째 기준점 좌표에 상응하는 특징을 나타내고, k번째 기준점 좌표와 이전 디코더 레이어에서 예측한 랜드마크 좌표의 i번째 랜드마크 좌표 사이의 위치 오프셋은 변형 가능 어텐션 모듈 레이어에 입력된 쿼리 행렬에 대해 완전 완결 연산을 수행하여 획득되고, K는 미리 설정된 값일 수 있다.is obtained as, where,

denotes the updated feature of the ith landmark of the output matrix,

represents an attention weight obtained by performing a fully connected operation and a softmax operation on the query matrix input to the transformable attention module layer,

denotes a feature corresponding to the k-th reference point coordinate in the memory feature matrix, and the location offset between the k-th reference point coordinate and the i-th landmark coordinate of the landmark coordinate predicted in the previous decoder layer is the query input to the transformable attention module layer. It is obtained by performing a full-complete operation on a matrix, and K may be a preset value.

Optionally, the landmark coordinates predicted by the decoder layer,

은 이전 디코더 레이어에서 예측한 랜드마크 좌표 또는 초기 랜드마크 좌표를 나타내고,

는

에 대한 y의 오프셋을 나타내는 랜드마크 좌표 예측 레이어의 출력을 나타낼 수 있다.Is obtained as, y represents the landmark coordinates predicted by the current decoder layer,

denotes the landmark coordinates predicted by the previous decoder layer or the initial landmark coordinates,

Is

It can represent the output of the landmark coordinate prediction layer representing the offset of y for .

Optionally, the convolutional network layer, the fully connected layer, and the at least one decoder layer are obtained through training using training image samples based on a regression loss function below,

는 회귀 손실을 나타내고,

는 각 디코더 레이어에서 예측한 트레이닝 이미지 샘플의 랜드마크 좌표를 나타내고,

는 트레이닝 이미지 샘플의 실제 랜드마크 좌표를 나타내고,

는 디코더 레이어의 수량이고,

은 디코더 레이어의 인덱스일 수 있다.From here,

represents the regression loss,

Represents the landmark coordinates of training image samples predicted by each decoder layer,

denotes the actual landmark coordinates of the training image sample,

is the quantity of decoder layers,

may be an index of a decoder layer.

According to another embodiment of the present disclosure, an apparatus for predicting landmark coordinates of a face image is provided, wherein the apparatus obtains a multi-stage feature map of the face image through a convolutional neural network layer, and a fully connected layer. Obtaining an initial query matrix by completely connecting the feature maps of the last stage of the multi-stage feature map through, wherein the initial query matrix represents initial features of landmarks of the face image, and the number of elements in the initial query matrix is the face equal to the number of landmarks in the image - an encoder configured to flatten and concatenate the multi-stage feature maps to obtain a memory feature matrix; and at least one cascaded decoder layer, wherein the at least one decoder layer determines the landmark coordinates of the face image based on the memory feature matrix and the initial query matrix received from the encoder. Configured to - may include.

According to another embodiment of the present disclosure, an apparatus for predicting landmark coordinates of a face image is provided, the apparatus comprising: a processor; and a memory storing one or more commands executed by the processor, wherein the processor executes the commands to obtain a multi-stage feature map of a face image through a convolutional neural network layer. ; Obtaining an initial query matrix by fully connecting last stage feature maps of the multi-stage feature maps through a fully connected layer, wherein the initial query matrix represents initial features of landmarks of the face image, and The number of elements in the initial query matrix is equal to the number of landmarks of the face image -; obtaining a memory feature matrix by flattening and concatenating the multi-stage feature maps; and determining the landmark coordinates of the face image by inputting the memory feature matrix and the initial query matrix to at least one cascaded decoder layer.

Optionally, in the device, the decoder layer includes a cascaded self-attention module layer, a transformable attention module layer, and a landmark coordinate prediction layer, and the self-attention module layer of the first decoder layer includes an initial query in which location information is embedded. and an initial query matrix to obtain an output matrix of a self-attention module layer of the first decoder layer based on the matrix and the initial query matrix, wherein the initial query matrix and the initial query matrix in which the location information is embedded are respectively the self-attention module layer of the first decoder layer. It is input as the query matrix, key matrix, and value matrix of the attention module layer, and the self-attention module layer of the decoder layer other than the first decoder layer is the deformable attention module layer of the previous decoder layer that is cascaded. Acquire an output matrix of a transformable attention module layer of a current decoder layer based on an output matrix and an output matrix of a transformable attention module layer of a previous decoder layer in which the location information is embedded, and the previous decoder layer The output matrix of the transformable attention module layer and the output matrix of the transformable attention module layer of the previous decoder layer in which the position information is embedded are a value matrix, a query matrix, and a key matrix of the self-attention module layer of the current decoder layer, , The deformable attention module layer of the decoder layer is based on the output matrix of the self-attention module layer of the current decoder layer, the memory feature matrix, and the landmark coordinates predicted by the cascaded previous decoder layer, and obtain the output matrix of the transformable attention module layer of the current decoder layer, wherein the output matrix and memory feature matrix of the self attention module layer of the current decoder layer are queried of the transformable attention module layer of the current decoder layer. matrix and a value matrix, the landmark coordinates predicted by the previous decoder layer of the first decoder layer are initial landmark coordinates obtained based on the initial query matrix, and the landmark coordinate prediction layer is the current decoder Acquiring the landmark coordinates of the face image predicted by the decoder layer based on the output matrix of the deformable attention module layer of the layer, and the landmark of the face image predicted by the last decoder layer. The coordinates may be landmark coordinates of the final face image.

Optionally, in the device, the self-attention module layer of the decoder layer,

는 출력 행렬의 i번째 행 벡터를 나타내고,

는 셀프 어텐션 모듈 레이어에 입력된 쿼리 행렬의 i번째 행 벡터와 셀프 어텐션 모듈 레이어에 입력된 키 행렬의 j번째 행 벡터의 내적을 정규화하여 획득한 어텐션 가중치를 나타내고,

는 초기 쿼리 행렬 또는 이전 디코더 레이어의 변형 어텐션 레이어의 출력 행렬의 j번째 행 벡터를 나타내고, N은 얼굴 이미지의 랜드마크 개수를 나타낼 수 있다.Obtain the output matrix with, where

denotes the i-th row vector of the output matrix,

denotes an attention weight obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module layer and the j-th row vector of the key matrix input to the self-attention module layer,

represents a j-th row vector of an initial query matrix or an output matrix of a transformation attention layer of a previous decoder layer, and N may represent the number of landmarks in a face image.

Optionally, in the device, the deformable attention module layer of the decoder layer,

는 출력 행렬의 i번째 랜드마크의 업데이트된 특징을 나타내고,

는 변형 가능 어텐션 모듈 레이어에 입력된 쿼리 행렬에 대해 완전 연결 연산 및 softmax 연산을 수행하여 획득한 어텐션 가중치를 나타내고,

는 메모리 특징 행렬 중 k번째 기준점 좌표에 상응하는 특징을 나타내고, k번째 기준점 좌표와 이전 디코더 레이어에서 예측한 랜드마크 좌표의 i번째 랜드마크 좌표 사이의 위치 오프셋은 변형 가능 어텐션 모듈 레이어에 입력된 쿼리 행렬에 대해 완전 완결 연산을 수행하여 획득되고, K는 미리 설정된 값일 수 있다.Obtain the output matrix with, where

denotes the updated feature of the ith landmark of the output matrix,

Represents the attention weight obtained by performing a fully connected operation and a softmax operation on the query matrix input to the transformable attention module layer,

denotes a feature corresponding to the k-th reference point coordinate in the memory feature matrix, and the location offset between the k-th reference point coordinate and the i-th landmark coordinate of the landmark coordinate predicted in the previous decoder layer is the query input to the transformable attention module layer. It is obtained by performing a full-complete operation on a matrix, and K may be a preset value.

Optionally, in the device, the landmark coordinate prediction layer of the decoder layer,

은 이전 디코더 레이어에서 예측한 랜드마크 좌표 또는 초기 랜드마크 좌표를 나타내고,

는

에 대한 y의 오프셋을 나타내는 랜드마크 좌표 예측 레이어의 출력을 나타낼 수 있다.Predict the landmark coordinates with , where y represents the landmark coordinates predicted by the current decoder layer,

denotes the landmark coordinates predicted by the previous decoder layer or the initial landmark coordinates,

Is

It can represent the output of the landmark coordinate prediction layer representing the offset of y for .

Optionally, the device is trained using training image samples based on the regression loss function below;

는 회귀 손실을 나타내고,

는 각 디코더 레이어에서 예측한 트레이닝 이미지 샘플의 랜드마크 좌표를 나타내고,

는 트레이닝 이미지 샘플의 실제 랜드마크 좌표를 나타내고,

는 디코더 레이어의 수량이고,

은 디코더 레이어의 인덱스일 수 있다.From here,

represents the regression loss,

Represents the landmark coordinates of training image samples predicted by each decoder layer,

denotes the actual landmark coordinates of the training image sample,

is the quantity of decoder layers,

may be an index of a decoder layer.

According to another embodiment of the present disclosure, a computer readable program storing a computer program implementing a method of predicting landmark coordinates of a face image according to any one of claims 1 to 6 when executed by a processor. A storage medium may be provided.

These and other objects and features of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings illustrating examples. From here,
1 is a flowchart illustrating a method for predicting facial landmark coordinates according to an embodiment of the present disclosure.
2 is a flowchart illustrating a method for predicting facial landmark coordinates according to an embodiment of the present disclosure.
3 is a diagram showing an example of a structure of a first decoder layer.
4 is a block diagram showing the structure of an apparatus for predicting landmark coordinates of a face image according to an embodiment of the present disclosure.
5 is a block diagram showing the structure of an apparatus for predicting landmark coordinates of a face image according to another embodiment of the present disclosure.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed and implemented in various forms. Therefore, the form actually implemented is not limited only to the specific disclosed embodiment, and the scope of the present disclosure includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as "first" or "second" may be used to describe various elements, but such terms should only be construed for the purpose of distinguishing one element from another. For example, a “first component” may be termed a “second component,” and similarly, a “second component” may also be termed a “first component.”

It should be understood that when a component is referred to as being “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but that one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

It is apparent to those skilled in the art that the singular forms "one", "one", "above" and "that" used herein may also include plural forms unless otherwise specified. The terms "comprising" and "containing" used in the embodiments of the present application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element and / or component, and the present description It does not exclude other features, information, data, steps, operations, elements, constituent elements and/or combinations thereof supported by the field. When an element is said to be “connected” or “coupled” to another element, that one element can be directly connected or coupled to another element, and a connection relationship between that one element and the other element will be established through an intermediate element. may be Also, “connection” or “combination” in this specification may include a wireless connection or a wireless combination. In this specification, the term "and/or" indicates at least one of the items defined by the term, and for example, "A and/or B" indicates implementation as "A" or implementation as "A and B".

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present disclosure, it should not be interpreted in an ideal or excessively formal meaning. don't

According to the method and apparatus for predicting landmark coordinates of a facial image of the present disclosure, landmark detection relates to N coordinate prediction, where N is the number of facial landmarks, for example, a total of six facial landmarks such as eye rims and mouth corners. Mark, and common 68-point and 98-point models. Using the self-attention module, we potentially learn structural dependencies between coordinates of landmarks, and output feature vectors represented by each landmark. Then, instead of retrieving the full image features (which slows down the convergence of the model), we focus on a small number of image features sampled around the reference point using a deformable attention module based on the image features. Here, information related to the predicted or initial predicted landmark coordinates of the previous layer (i.e., reference points) serves as a guide for extracting the most relevant image features around the landmark and refining the coordinates. used In addition, in the present disclosure, a randomly initialized query matrix is not used, and n features are extracted in a fully connected manner from the last layer of the backbone network and used as an initial query matrix. This design speeds up the convergence of the model and further improves the accuracy of landmark coordinate prediction in face images.

1 is a flowchart illustrating a method for predicting facial landmark coordinates according to an embodiment of the present disclosure.

Compared to the deformable DETR model, the method according to the present disclosure is more semantic based on image feature maps using a fully connected layer instead of using random numbers as the initial query embedding. Extract features that exist, and use the extracted features as initial query embeddings. Here, the number of queries is set to the number of landmarks required for prediction, and the initial query embedding based on the extracted features provides rough reference point locations related to images rather than standard landmark templates not related to images.

In addition, compared to the deformable DETR model, the method according to the present disclosure deletes encoder layers, keeps only convolutional layers and fully connected layers, and reduces the number of decoder layers (e.g., deformable DETR has 6 decoder layer, and may include only three or less decoder layers in the present disclosure). In addition, by using only the L1 norm loss (the absolute value representing the difference between the predicted value and the actual value) function for model training, it is possible to reduce the model size and save computational cost.

Referring to FIG. 1, an initial query matrix Qinit is obtained by fully concatenating the last stage feature maps, a memory feature matrix is obtained by flattening and concatenating multi-stage feature maps, and the memory feature matrix and the initial query matrix are used as inputs of the decoder layer. Then, the landmark coordinates of the face image are predicted through the decoder layer. Hereinafter, each step of the method will be described.

2 is a flowchart illustrating a method of predicting landmark coordinates of a face image according to an embodiment of the present disclosure.

Referring to FIG. 2 , in step 201, a multi-stage feature map (eg, the 4-stage feature map of FIG. 1) of a face image is acquired through a convolutional neural network layer. Those skilled in the art can understand that the extracted multi-stage feature map is a pyramid feature, the lower stage feature map represents the local feature of the image, and the upper stage feature map represents the global feature of the image. .

For example, a pyramidal feature of a face image may be acquired through a backbone network. This disclosure uses ResNet18, but is not limited to ResNet18. The size of the given input image is 256×256×3, and the sizes of all stage feature maps are 64×64×64, 32×32×128, 16×16×256, and 8×8×512.

을 획득한다. 여기에서, 초기 쿼리 행렬은 얼굴 이미지의 랜드마크의 초기 특징을 나타내고, 여기에서, 초기 쿼리 행렬의 엘리먼트 개수는 얼굴 이미지의 랜드마크 개수와 동일하다.

는 다음과 같이 얻을 수 있다.In step 202, the feature map of the last stage of the multi-stage feature map is fully connected through a fully connected layer to form an initial query matrix.

Acquire Here, the initial query matrix represents initial features of the landmarks of the face image, and the number of elements of the initial query matrix is equal to the number of landmarks of the face image.

can be obtained as:

Here, mark F is a feature map from the last layer of the backbone, and its size is expressed as (H×W)×C, where H and W are the spatial width and height of the feature map, and C is the feature of the feature map. dimension. FC denotes a fully connected layer and maps the spatial scale of each feature channel (H×W) to a vector of size N.

The query matrix Qinit has size N×C, where N is the number of landmarks in the face image and C is the feature dimension. This is essentially a learnable matrix, which extracts landmark-related features and transforms them into coordinates.

In step 203, a memory feature matrix is obtained by flattening and connecting the multi-stage feature maps.

For example, a typical encoder-decoder framework can be used. Specifically, a pyramid feature can be extracted from an input image using a backbone network as an encoder. Then, 1×1 convolution is applied to these feature maps to obtain feature maps with the same number of output channels. Then, these feature maps are flattened into a memory feature matrix M and connected (Flatten & Connect).

In step 204, the initial query matrix and the memory feature matrix are input to the cascaded at least one decoder layer to determine landmark coordinates of the face image.

According to an embodiment of the present disclosure, each decoder layer has the same structure except that an input matrix and an output matrix of each decoder layer are different.

For example, each decoder layer includes a cascaded self-attention module layer, a transformable attention module layer, and a landmark coordinate prediction layer. Those skilled in the art should understand that each decoder layer may additionally include other layers as needed.

For example, the step of determining landmark coordinates of a face image may include an initial query matrix in which location information is embedded, an initial query matrix in which location information is embedded, and an initial query matrix as a query matrix of a self-attention module layer of a first decoder layer. , and inputting the key matrix and the value matrix to the self-attention module layer of the first decoder layer.

By inputting the output matrix of the self-attention module layer of the current decoder layer, the memory feature matrix, and the landmark coordinates predicted by the previous decoder layer into the transformable attention module layer of the current decoder layer, the transformation of the transformable attention module layer Get the output matrix. Here, the output matrix and memory feature matrix of the self-attention module layer are the query matrix and the value matrix of the transformable attention module layer, where predictions from the previous decoder layer input to the transformable attention module layer of the first decoder layer The landmark coordinates are initial landmark coordinates obtained based on the initial query matrix.

As an exemplary embodiment, initial landmark coordinates may be obtained by performing a fully concatenated operation on a query matrix.

The output matrix of the transformable attention module layer of the current decoder layer and the output matrix of the transformable attention module layer in which the position information is embedded are converted to the value matrix, query matrix, and key matrix of the self-attention module layer of the next decoder layer that is cascaded. input to the self-attention module layer of the next decoder layer as ; The output matrix of the deformable attention module layer of the current decoder layer is input to the landmark coordinate prediction layer of the current decoder layer to obtain the landmark coordinates of the face image predicted by the current decoder layer. Here, the landmark coordinates of the face image predicted by the last decoder layer are the landmark coordinates of the final face image.

3 is a diagram showing an example of a structure of a first decoder layer, and other decoders may have the same structure.

Referring to FIG. 3 , each decoder layer may include a self-attention module layer and a transformable attention module layer. The self-attention module layer of each decoder layer may include a self-attention layer and a residual summation normalization layer.

, 위치 정보가 임베드된

및

이다. 다른 디코더 레이어의 셀프 어텐션 모듈 레이어의 쿼리 행렬, 키 행렬 및 값 행렬은, 각각 위치 정보가 임베드된 이전 레이어 디코더 레이어의 변형 가능 어텐션 모듈 레이어의 출력 행렬 및 이전 레이어 디코더 레이어의 변형 가능 어텐션 레이어의 출력 행렬이다.The query matrix, the key matrix, and the value matrix of the self-attention module layer of the first decoder layer are each embedded with position information.

, with location information embedded

and

am. The query matrix, key matrix, and value matrix of the self-attention module layer of another decoder layer are the output matrix of the transformable attention module layer of the decoder layer of the previous layer and the output of the transformable attention layer of the decoder layer of the previous layer, respectively. it is a matrix

The deformable attention module layer of each decoder layer may include a deformable-attention layer, a residual add & norm layer, and a feedforward neural network (FFN) layer. can

The output matrix of the self-attention module layer is used as a query matrix input for the transformable attention module layer (transformable attention module layer), and the memory feature matrix is a value matrix input to the transformable attention module layer.

임)만 입력으로 사용한다. 이는 랜드마크 간의 구조적 종속성을 학습한다. 해당 정보는 실제로 이미지와 무관하며, 랜드마크 위치에서 제스처, 표정 등을 캡처하고, 이는 랜드마크 포지셔닝에 있어 중요한 것으로 입증된다. 셀프 어텐션 모듈 레이어는 QP(제1 디코더 레이어가 위치 정보가 임베드된 초기 쿼리 행렬

이고, 다른 디코더 레이어가 이전 디코더 레이어인 위치 정보가 임베드된 변형 가능 어텐션 모듈 레이어에 대한 출력 행렬), QP(제1 디코더 레이어가 위치 정보가 임베드된 초기 쿼리 행렬

이고, 다른 디코더 레이어가 이전 디코더 레이어인 위치 정보가 임베드된 변형 가능 어텐션 모듈 레이어에 대한 출력 행렬) 및 Q(제1 디코더 레이어가

이고, 다른 디코더 레이어가 이전 디코더 레이어인 변형 가능 어텐션 모듈 레이어에 대한 출력 행렬)를 각각 쿼리 행렬, 키 행렬 및 값 행렬으로 취하고, 여기에서 QP=Q+P이고, P는 학습 가능한 위치 임베딩이다. 셀프 어텐션 모듈 레이어의 출력은 QE로 표현되며 다음과 같이 얻을 수 있다.The self-attention module layer is the query matrix Q (for the first decoder layer

i) is used as an input. It learns structural dependencies between landmarks. That information is actually image-independent and captures gestures, facial expressions, etc. at landmark locations, which proves to be important for landmark positioning. The self-attention module layer is a QP (initial query matrix in which the first decoder layer is embedded with positional information)

, where another decoder layer is the previous decoder layer, an output matrix for a transformable attention module layer in which position information is embedded), QP (an initial query matrix in which position information is embedded in the first decoder layer)

, where the other decoder layer is the previous decoder layer, the output matrix for the transformable attention module layer in which positional information is embedded) and Q (the first decoder layer is

, and another decoder layer takes the output matrix for the transformable attention module layer, which is the previous decoder layer) as the query matrix, key matrix, and value matrix, respectively, where QP = Q + P, where P is the learnable positional embedding. The output of the self-attention module layer is expressed as QE and can be obtained as follows.

(1)

(One)

는 출력 행렬의 i번째 행 벡터를 나타내고,

는 셀프 어텐션 모듈 레이어에 입력된 쿼리 행렬의 i번째 행 벡터와 셀프 어텐션 모듈 레이어에 입력된 키 행렬의 j번째 행 벡터의 내적(dot product)을 정규화(normalization)하여 획득한 결과를 나타내고,

는 초기 쿼리 행렬 또는 이전 디코더 레이어의 변형 가능 어텐션 모듈 레이어의 출력 행렬의 j번째 행 벡터를 나타내고, N은 랜드마크 개수를 나타낸다. 예를 들어, 입력 쿼리 행렬 Q, 키 행렬 K 및 값 행렬 V가 모두 N×C 크기 행렬인 경우,

이다.

는 L × L 행렬 α의 (i, j)번째 엘리먼트의 값을 나타내고,

는 키 행렬의 행 벡터의 차원(dimension)이다.From here,

denotes the i-th row vector of the output matrix,

Represents a result obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module layer and the j-th row vector of the key matrix input to the self-attention module layer,

represents the j-th row vector of the initial query matrix or the output matrix of the transformable attention module layer of the previous decoder layer, and N represents the number of landmarks. For example, if the input query matrix Q, key matrix K, and value matrix V are all matrices of size NxC,

am.

Represents the value of the (i, j)th element of the L × L matrix α,

is the dimension of the row vector of the key matrix.

Here, the sofmax operation is a well-known technique, and its meaning is as follows.

The Softmax operation normalizes all input values to (0,1) and makes the sum of all inputs equal to 1. The denominator of a formula represents the sum of all input indices, and the numerator represents the exponent of a particular value.

The transformable attention module layer of each decoder layer is based on the output matrix of the self-attention module layer, the memory feature matrix, and the landmark coordinate matrix predicted by the previous decoder layer (in case of the first decoder layer, this is the initial landmark coordinate matrix). We get the upgraded feature of the mark, i.e. the output matrix of the transformable attention module layer. Here, the initial landmark coordinate matrix is obtained by fully concatenating the initial query matrix. Those skilled in the art should understand that the landmark coordinates and the landmark coordinate matrix described herein have the same or similar meanings.

As an example, the output matrix of the transformable attention module layer can be obtained through:

는 i번째 랜드마크의 업데이트된 특징을 나타내고,

는 변형 가능 어텐션 모듈 레이어에 입력된 쿼리 행렬에 대해 전체 연결 연산과 소프트맥스 연산을 수행하여 얻은 어텐션 가중치를 나타내고,

는 메모리 특징 행렬에서 k번째 기준점에 해당하는 특징을 나타내고, 여기에서, k번째 기준점 좌표와 이전 디코더 레이어에서 예측한 랜드마크 좌표에서 i번째 랜드마크 좌표 사이의 위치 오프셋은 변형 가능 어텐션 모듈 레이어에 입력된 쿼리 행렬에 대해 완전 연결 연산을 수행하여 구한 것이다. 여기에서, K는 미리 설정된 값이다. 즉, k번째 기준점의 좌표는 i번째 랜드마크의 좌표에 상기 위치 오프셋을 더한 것으로, 상기 완전 연결 연산의 파라미터 행렬은 K와 관련이 있다. 구체적으로,

는 다음과 같이 구할 수 있다.From here,

represents the updated feature of the ith landmark,

represents the attention weight obtained by performing the full concatenation operation and the softmax operation on the query matrix input to the transformable attention module layer,

denotes a feature corresponding to the k-th reference point in the memory feature matrix, where the position offset between the k-th reference point coordinates and the i-th landmark coordinates from the landmark coordinates predicted in the previous decoder layer is input to the transformable attention module layer. It is obtained by performing a fully concatenated operation on the query matrix. Here, K is a preset value. That is, the coordinates of the k-th reference point are the coordinates of the i-th landmark plus the position offset, and the parameter matrix of the fully connected operation is related to K. Specifically,

can be obtained as:

는

의 k번째 엘리먼트이다. 여기에서,

는 변형 가능 어텐션 모듈 레이어의 입력 쿼리 행렬의 i번째 C차원의 행 벡터를 나타내고,

는 K×C의 행렬로,

에 대해 완전 연결 연산이 수행되는 완전 연결 연산 파라미터 행렬을 나타내며, 이는 학습 가능한 행렬이다. 위와 같이,

와

의 내적을 먼저 획득하고, 그런 다음 softmax 연산을 통해 내적을 정규화하여 어텐션 가중치

를 얻는다.

Is

is the kth element of From here,

Represents the i-th C-dimensional row vector of the input query matrix of the transformable attention module layer,

is a matrix of K × C,

Represents a fully connected operation parameter matrix in which a fully connected operation is performed on , which is a learnable matrix. as above,

and

first obtain the dot product of , and then normalize the dot product through softmax operation to

get

는 초기 랜드마크 좌표(제1 디코더 레이어에 대한) 또는 이전 디코더 레이어에서 예측한 랜드마크 좌표 행렬의 엘리먼트

(즉, 이전 디코더에서 예측한 i번째 랜드마크의 좌표)와 값 행렬 M의 좌표 인덱스로 위치 오프셋을 더하여 얻은 특징(예, 상기 얻은 좌표에 따라, M 중에서 해당 좌표에 해당하는 특징을 결정함)을 나타내고, k의 값이 1, …, K이므로, 각 랜드마크에 대해 M에서 K개의 특징을 인덱싱할 수 있다.

Is the initial landmark coordinate (for the first decoder layer) or an element of the landmark coordinate matrix predicted in the previous decoder layer

(i.e., the coordinates of the i-th landmark predicted by the previous decoder) and the feature obtained by adding the position offset to the coordinate index of the value matrix M (e.g., according to the obtained coordinates, the feature corresponding to the corresponding coordinate among M is determined) , and the value of k is 1, ... , K, so for each landmark, M to K features can be indexed.

은 입력된 쿼리 행렬을 통해 완전 연결로 얻은 K개의 기준점의 K번째 기준점(그 좌표는 i번째 기준점의 좌표에 위치 오프셋

을 더한 것임)의 위치와 i번째 랜드마크의 위치 사이의 상대 오프셋을 나타내고, 위치 오프셋은 다음 공식을 통해 얻을 수 있다.where position offset

is the K-th reference point of the K reference points obtained by full connection through the input query matrix (the coordinates are offset to the coordinates of the i-th reference point)

) represents the relative offset between the position of the i-th landmark and the position of the i-th landmark, and the position offset can be obtained through the following formula.

는

의 k번째 엘리먼트를 나타내고, k=1이고, …, K이고,

는 입력된 쿼리 행렬의 i번째 C 차원의 행 벡터를 나타내고,

는 2K×C 크기의 행렬을 나타내고,

에 대해 완전 연결 연산이 수행되는 완전 연결 연산 파라미터 행렬을 나타내며, 이는 학습 가능한 행렬이고, 2는 각 위치가 가로 좌표와 세로 좌표의 두 값으로 구성됨을 나타낸다.

Is

Represents the kth element of , k = 1, . . . , K,

Represents the i-th C-dimensional row vector of the input query matrix,

Represents a matrix of size 2K × C,

Represents a fully connected operation parameter matrix in which a fully connected operation is performed on , which is a learnable matrix, and 2 indicates that each position consists of two values of abscissa and ordinate.

K represents the number of reference points required for each landmark, and its size can be set in advance.

In order to more clearly understand the embodiments according to the present disclosure, the third element among the coordinates predicted by the above decoder layer is described as an example, and K=4 is set.

(현재 디코더 레이어가 제1 디코더 레이어인 경우,

은 초기 쿼리 행렬Qinit에 대해 완전 연결을 수행하여 얻은 3번째 랜드마크의 초기 좌표임)이다. 미리 설정된 K를 기반으로,

에 대해 완전 연결을 수행하여

를 얻을 수 있고,

는

,

,

, 및

의 4가지 엘리먼트를 포함한다.

+

하여

에 대응하는 첫 번째 기준점의 좌표를 얻고, 해당 좌표를 사용하여 메모리 특징 행렬의 엘리먼트

를 결정한다. 즉, 메모리 특징 행렬 중 좌표는

+

의 엘리먼트이다. 마찬가지로,

,

+

,

하여

,

,

와 대응하는 두 번째 기준점, 세 번째 기준점 및 네 번째 기준점을 얻고, 이를 통해 메모리 특징 중의 엘리먼트

,

및

를 결정한다. 최종적으로,

, 즉 세 번째 랜드마크의 업데이트된 특징을 얻을 수 있고, 다른 랜드마크의 업데이트된 특징 또한 동일한 방식으로 얻을 수 있다. 다시 말해, 변형 가능 어텐션 모듈 레이어의 출력 행렬은 얼굴 이미지의 랜드마크의 업데이트된 특징을 나타낸다.The third element is the coordinates of the third landmark predicted by the previous layer decoder.

(If the current decoder layer is the first decoder layer,

is the initial coordinate of the third landmark obtained by performing full linkage on the initial query matrix Qinit). Based on the preset K,

By performing full concatenation on

can be obtained,

Is

,

,

, and

contains four elements of

+

So

Get the coordinates of the first reference point corresponding to

decide That is, the coordinates of the memory feature matrix are

+

is an element of Likewise,

,

+

,

So

,

,

Obtain a second reference point, a third reference point, and a fourth reference point corresponding to

,

and

decide Finally,

, that is, the updated features of the third landmark can be obtained, and the updated features of other landmarks can also be obtained in the same way. In other words, the output matrix of the deformable attention module layer represents the updated feature of the landmark of the face image.

As described above, the transformable attention module layer takes QE as a query matrix, memory feature matrix M as a value matrix, and returns M from reference points (i.e., the initial landmark coordinate matrix or the landmark coordinate matrix predicted by the previous decoder layer). We focus only on a small group of features obtained by sampling (e.g., when calculating the features of the ith landmark, we use the features of the K points near the ith landmark).

을 얻을 수 있다. 다시 말해, QD를 감지 레이어의 입력으로 취하고, 출력은

이다.For example, after obtaining the output matrix of the transformable module layer, through the landmark coordinate prediction layer of each decoder layer, the offset of the landmark coordinates predicted by the current decoder layer relative to the landmark coordinates predicted by the previous decoder layer.

can be obtained. In other words, take a QD as an input to the sensing layer, and the output is

am.

For example, using a three-layer fully connected network with ReLU activation function, the first two layers consist of linear fully connected and ReLU activation function, the last layer directly outputs the coordinate deviation information through full connection, Do not add ReLU activation function later. For example, it is the same as the multilayer perceptron (MLP) of FIG. 3 . It takes QDs as input and outputs coordinate-related information. The formula of the ReLU activation function is as follows.

After obtaining yO, we can get the predicted landmark coordinates from the current decoder layer as follows:

은 초기 랜드마크 좌표 행렬(제1 디코더 레이어에 대한) 또는 이전 디코더 레이어에서 예측한 랜드마크 좌표를 나타내고,

는 랜드마크 좌표 예측 레이어의 지각 레이어의 출력을 나타내고,

에 대한 y의 상대적 오프셋을 나타낸다. 여기에서, 지각 레이어의 입력은 변형 가능 어텐션 모듈 레이어의 출력 행렬이다.where y denotes the predicted landmark coordinates in the current decoder layer,

denotes the landmark coordinates predicted by the initial landmark coordinate matrix (for the first decoder layer) or the previous decoder layer,

Represents the output of the perceptual layer of the landmark coordinate prediction layer,

Represents the offset of y relative to . Here, the input of the perceptual layer is the output matrix of the transformable attention module layer.

Here, the σ function is the prior art, and specifically, the σ function formula is as follows.

Finally, the landmark coordinates predicted by the last decoder layer are determined as the final predicted landmark coordinates.

For a clearer understanding of the present disclosure, a model with three decoder layers is used for explanation.

In the case of the first decoder layer, the query matrix, key matrix, and value matrix of the self-attention module layer are Qinit with embedded location information, Qinit with embedded location information, and Qinit, respectively.

In the case of the second decoder layer, the query matrix, key matrix, and value matrix of the self-attention module layer are the QE of the first decoder layer in which location information is embedded, the QE of the first decoder layer in which location information is embedded, and the first decoder layer, respectively. is the QE of

In the case of the third decoder layer, the query matrix, key matrix, and value matrix of the self-attention module layer are the QE of the second decoder layer in which location information is embedded, the QE of the second decoder layer in which location information is embedded, and the second decoder layer, respectively. is the QE of

The first decoder layer predicted landmark coordinates use an initial landmark coordinate matrix; The landmark coordinates predicted by the first decoder layer are used by the second decoder layer predicted landmark coordinates, and the landmark coordinates predicted by the second decoder layer are used by the third decoder layer predicted landmark coordinates. For those skilled in the art, the model having three decoder layers is just an example, the model may have one decoder layer or more decoder layers, and other decoder layers except for the first decoder layer are similar. It should be understood that it has input and output operations.

는 다음과 같다.For example, you can train a model using an L1 norm loss (absolute value representing the difference between the predicted and actual value) function between the coordinates of the landmarks in the predicted training image samples and the actual coordinates of the landmarks in the training image samples. and the regression loss function used for training

is as follows

은 각 디코더 레이어에서 예측한 트레이닝 이미지 샘플의 랜드마크 좌표를 나타내고,

는 트레이닝 이미지 샘플의 랜드마크의 실제 좌표를 나타내고,

는 디코더 레이어의 개수이고,

은 디코더 레이어의 인덱스이다.From here,

Represents the landmark coordinates of training image samples predicted by each decoder layer,

denotes the real coordinates of the landmarks of the training image sample,

is the number of decoder layers,

is the index of the decoder layer.

According to the facial landmark coordinate prediction method according to an embodiment of the present disclosure, end-to-end training of the entire prediction model is possible.

In the above, the method of predicting facial landmarks has been described with reference to FIGS. 1 to 3 . Hereinafter, an apparatus for predicting facial landmarks according to an embodiment of the present invention will be described with reference to FIGS. 4 and 5 .

Referring to FIG. 4 , an apparatus 400 for predicting facial landmarks according to another embodiment of the present disclosure may include an encoder 401 and a decoder 402 . Those skilled in the art should understand that the device 400 may additionally include other components, and the components included in the device 400 may be divided or combined.

For example, the encoder 401 acquires a multi-stage feature map of a face image through a convolutional neural network layer, and fully connects the last stage feature map of the multi-stage feature map through a fully connected layer to obtain an initial query matrix. Obtain, where the initial query matrix represents initial features of landmarks in the face image, where the number of elements in the initial query matrix is equal to the number of landmarks in the face image, and multi-stage feature maps are flattened and connected to memory It may be configured to obtain a feature matrix.

For example, the encoder may be implemented as a convolution module, a fully connected module, and a flatten and connect module, wherein the convolution module is configured to obtain a multi-stage feature map of a face image, and the fully connected module is configured to obtain the multi-stage feature map. A flattening and concatenating module is configured to obtain a memory feature matrix by flattening and concatenating the multi-stage feature maps.

For example, the decoder 402 may consist of at least one cascaded decoder layer, wherein the at least one decoder layer is a landmark of a facial image based on a memory feature matrix and an initial query matrix received from the encoder. It can be configured to determine coordinates.

For example, each of the at least one decoder layer includes a cascaded self-attention module layer, a transformable attention module layer, and a landmark coordinate prediction layer.

For example, the self-attention module layer of the first decoder layer may determine the self-attention module layer of the first decoder layer based on the initial query matrix in which the received location information is embedded, the initial query matrix in which the location information is embedded, and the initial query matrix. Is configured to obtain an output matrix of, wherein the initial query matrix in which the received location information is embedded, the initial query matrix in which the location information is embedded, and the initial query matrix are respectively a query matrix of the self-attention module layer of the first layer, a key It is entered as a matrix and a value matrix.

The self-attention module layer of each decoder layer other than the first decoder layer is based on the output matrix of the transformable attention module layer of the previous cascaded decoder layer and the output matrix of the transformable attention module layer of the previous decoder layer in which position information is embedded. , configured to obtain an output matrix of the transformable attention module layer of the current decoder layer, where the output matrix of the transformable attention module layer of the previous decoder layer and the transformable attention module layer of the previous decoder layer in which the positional information is embedded. The output matrix is the value matrix, query matrix, and key matrix of the self-attention module layer of the current decoder layer.

The transformable attention module layer of each decoder layer is the transformable attention module of the current decoder layer, based on the output matrix of the self-attention module layer of the current decoder layer, the memory feature matrix, and the landmark coordinates predicted by the cascaded previous decoder layer. configured to obtain an output matrix of a layer, wherein the output matrix and memory feature matrix of a self-attention module layer of a current decoder layer are a query matrix and a value matrix of a transformable attention module layer of a current decoder layer, wherein, The landmark coordinates predicted in the previous decoder layer of the first decoder layer are initial landmark coordinates obtained based on the initial query matrix.

Each landmark coordinate prediction layer is configured to obtain the landmark coordinates of the face image predicted by each decoder layer based on the output matrix of the deformable attention module layer of the current decoder layer, where the prediction by the last decoder layer The landmark coordinates of the face image are the landmark coordinates of the final face image.

For example, the device 400 may further include a fully connected layer that performs a fully connected operation on an initial query matrix to obtain initial coordinates of landmarks of a face image.

5 is a block diagram showing the structure of an apparatus 500 for predicting landmark coordinates of a face image according to another embodiment of the present disclosure.

In general, the electronic device 500 includes a processor 501 and a memory 502 .

Processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, or the like. The processor 1001 may be implemented using at least one hardware form among digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 501 may also include a main processor and an auxiliary processor, and the main processor is a processor that processes data in a wake-up state, also referred to as a CPU. The co-processor is a low-power processor for processing data in a standby state. In some embodiments, the processor 501 may be integrated with a graphics processing unit (GPU) used for rendering and drawing content to be displayed on a display screen. In some embodiments, processor 501 may further include an AI processor used to process computing operations related to machine learning.

Memory 502 may include one or more computer readable storage media that may be non-transitory. Memory 502 may also include non-volatile memory, such as high-speed random access memory, one or more disk storage devices, and flash memory devices. In some embodiments, the non-transitory computer readable storage medium of memory 502 is at least one used to be executed by processor 501 to train an echo cancellation model and/or implement a method of canceling echo delay of the present disclosure. Save one command.

In some embodiments, the electronic device 500 may further include a peripheral device interface 503 and at least one peripheral device. Processor 501, memory 502 and peripheral interface 503 may be connected by a bus or signal line. Each peripheral device may be connected to the peripheral device interface 503 through a bus, signal line, or circuit board. Specifically, the peripheral device includes a radio frequency (RF) circuit 504, a touch screen 505, a camera 506, an audio circuit 507, a positioning component 508 and a power supply 509.

The peripheral device interface 503 may be used to connect at least one peripheral device related to I/O (input/output) to the processor 501 and the memory 502 . In some embodiments, processor 501, memory 502 and peripheral interface 503 are integrated on the same chip or circuit board; In some other embodiments, any one or more of processor 501, memory 502, and peripheral interface 503 may be implemented on a separate chip or circuit board, which is not limited in this embodiment. .

The RF circuit 504 is used to receive and transmit RF (radio frequency) signals, also referred to as full spectrum signals. RF circuitry 504 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuitry 504 converts electrical signals to electromagnetic signals for transmission or converts received electromagnetic signals to electrical signals. Optionally, RF circuitry 504 includes an antenna system, RF transceiver, one or more amplifiers, tuners, oscillators, digital signal processors, codec chipsets, user identification module cards, and the like. The RF circuitry 504 can communicate with other terminals via at least one wireless communication protocol. Applicable wireless communication protocols include, but are not limited to, metropolitan area networks, multi-generational mobile communication networks (2G, 3G, 4G and 5G), wireless local area networks and/or WiFi networks. In some embodiments, RF circuitry 504 may also include circuitry related to NFC (Near Field Communication), without limitation in this disclosure.

The display screen 505 is used to display a UI (User Interface). The corresponding UI may include graphics, text, icons, video, and any combination thereof. When the display screen 505 is a touch screen, the display screen 505 also has the ability to collect touch signals at or on the surface of the display screen 505 . The touch signal may be input to the processor 501 as a control signal for processing. At this time, the display screen 505 can also be used to provide virtual buttons and/or a virtual keyboard, also known as soft buttons and/or a soft keyboard. In some embodiments, display screen 505 may be one disposed on the front panel of electronic device 500; In another embodiment, there may be at least two display screens 505, each disposed on a different surface of the terminal 1000 or in a folded design; In another embodiment, the display screen 505 may be a flexible display screen disposed on a curved or folded surface of the terminal 1000 . The display screen 505 may be set to have an irregular shape other than a rectangle, that is, a screen having a special shape. The display screen 505 may be made of a material such as LCD, OLED, or the like.

A camera component 506 is used to collect images or video. Optionally, camera component 506 includes a front camera and a back camera. In general, the front camera is set on the front panel of the terminal and the rear camera is set on the back of the terminal. In some embodiments, there are at least two rear cameras, each of which is any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, the main camera and the depth-of-field camera are fused to implement a background blur function, and the main camera By converging with a wide-angle camera, panoramic shooting and VR (virtual reality, virtual reality) shooting functions or other convergence shooting functions are realized. In some embodiments, camera component 506 may also include a flash. The flash may be a single color temperature flash or a dual color temperature flash. Dual color temperature flash refers to a combination of a warm light flash and a cool light flash and can be used to compensate for light at different color temperatures.

The audio circuit 507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, convert the sound waves into electrical signals and input them to the processor 501 for processing or to the RF circuitry 504 to realize voice communication. For stereo acquisition or noise reduction, a plurality of microphones may be provided, which may be disposed in different parts of the terminal 500, respectively. The microphone may be an array microphone or an omni-directional acquisition type microphone. A speaker is used to convert electrical signals from the processor 501 or RF circuit 504 into sound waves. The speaker may be a conventional thin film speaker or a piezoelectric ceramic speaker. If the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves that humans can hear, but also convert electrical signals into sound waves that humans can't hear, which can be used for distance measurement and other purposes. In some embodiments, audio circuitry 507 may further include an earphone jack.

The positioning component 508 is used to find the current geographic location of the electronic device 500 to implement navigation or location-based services (LBS). The positioning component 508 may be a positioning component based on the US GPS, the Chinese Beidou system, the Russian GLONASS system, or the EU Galileo system.

The power source 509 is used to supply power to each component of the electronic device 500 . The power source 509 can be AC, DC, disposable batteries or rechargeable batteries. If the power source 509 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

In some embodiments, electronic device 500 also includes one or more sensors 510 . The one or more sensors 510 include, but are not limited to, an acceleration sensor 511, a gyro sensor 512, a pressure sensor 513, a fingerprint sensor 514, an optical sensor 515, and a proximity sensor 516. .

The acceleration sensor 511 may detect magnitudes of acceleration with respect to three coordinate axes of the coordinate system set by the terminal 500 . For example, the acceleration sensor 511 may be used to detect gravitational acceleration components on three coordinate axes. The processor 501 may control the touch screen 505 to display the user interface horizontally or vertically according to the gravitational acceleration signal collected by the acceleration sensor 511 . The acceleration sensor 511 may also be used to collect game or user movement data.

The gyro sensor 512 may detect the direction and rotation angle of the main body of the terminal 500, and the gyro sensor 512 may cooperate with the acceleration sensor 511 to collect a user's 3D motion on the terminal 500. there is. The processor 501 may implement the following functions according to the data collected by the gyro sensor 512: motion detection (eg, UI change according to user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

The pressure sensor 513 may be disposed below the side frame of the terminal 500 and/or the touch screen 505 . When the pressure sensor 513 is set on the side frame of the terminal 500, it can detect the user's gripping signal for the terminal 500, and the processor 501 is configured according to the gripping signal collected by the pressure sensor 513. It can recognize left and right hands or perform quick operations. When the pressure sensor 513 is disposed below the touch screen 505 , the processor 501 controls operability control on the UI according to the user's pressure manipulation on the touch screen 505 . The operability control includes at least one of button control, scroll bar control, icon control, and menu control.

The fingerprint sensor 514 is used to collect the user's fingerprint, and the processor 501 identifies the user's identity according to the fingerprint collected by the fingerprint sensor 514, or the fingerprint sensor 514 uses the collected fingerprint to identify the user's identity. Once the user's identity is recognized as a trusted identity, the processor 501 authorizes the user to perform relevant sensitive tasks including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 514 may be disposed on the front, rear, or side of the electronic device 500 . If the electronic device 500 is provided with a physical key or manufacturer's logo, the fingerprint sensor 514 may be integrated with the physical key or manufacturer's logo.

An optical sensor 515 is used to collect the ambient light intensity. In one embodiment, the processor 501 may control the display brightness of the touch screen 505 according to the ambient light intensity collected by the optical sensor 515 . Specifically, when the ambient light intensity is high, the display brightness of the touch screen 505 increases; When the ambient light intensity is low, the display brightness of the touch screen 505 is also lowered. In another embodiment, processor 501 may dynamically adjust imaging parameters of camera component 506 based on ambient light intensity collected by optical sensor 515 .

Proximity sensor 516, also known as distance sensor, is typically provided on the front panel of electronic device 500. Proximity sensor 516 is used to collect the distance between the user and the front of electronic device 500 . In one embodiment, when the proximity sensor 516 detects that the distance between the user and the front of the terminal 500 gradually decreases, the processor 501 controls the touch screen 505 to turn it off in a bright screen state. switch to screen state; When the proximity sensor 516 detects that the distance between the user and the front of the electronic device 500 gradually increases, the processor 501 may control the touch screen 505 to switch from an off-screen state to a bright screen state. .

Those skilled in the art will understand that the structure shown in FIG. 5 does not constitute a limitation on the electronic device 500, includes more or fewer components than those shown, or some components. It is to be understood that combinations or different component arrangements may be employed.

According to one embodiment of the present disclosure, a computer readable storage medium may be further provided, wherein, when instructions of the computer readable storage medium are executed by at least one processor, a landmark of a face image according to the present disclosure Prompts at least one processor to execute the method of predicting coordinates. Here, examples of the computer readable storage medium include read only memory (ROM), random access programmable read only memory (PROM), electrically erasable programmable read only memory (EEPROM), random access memory (RAM), and dynamic random access memory. Access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, BLU-RAY or Optical Disc Memory, Hard Disk Drive (HDD), Solid State Hard disk (SSD), card memory (e.g. multimedia card, secure digital (SD) card or ultra-high speed digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state It may contain state disks and other devices. Any other device for storing a computer program and any associated data, data files and data structures in a non-transitory manner, and providing the computer program and any associated data, data files and data structures to a processor or computer for processing or a computer is configured to be able to execute the computer program. The computer program of the computer-readable storage medium may be executed in an environment deployed on a computer device such as a client, host, proxy device, server, etc., and in one example, the computer program and any related data, data file data structures may be connected to a network distributed on connected computer systems, whereby computer programs and all related data, data files and data structures may be stored, accessed and executed in a distributed manner by one or more processors or computers.

According to an embodiment of the present disclosure, a computer program product may be further provided, and instructions of the computer program product may be executed by a processor of a computer device to implement a method for predicting landmark coordinates of the face image.

Those skilled in the art will readily be able to conceive of other embodiments of the present disclosure after having described and practiced the invention disclosed herein. This application follows the general principles of this disclosure and is intended to cover any variations, uses or adaptations of this disclosure, including common knowledge or common technical means in the art not disclosed in this disclosure. The specification and examples are to be regarded as illustrative only, with the true scope and spirit of the disclosure being construed by the appended claims.

Claims (14)

In the method of predicting landmark coordinates of a face image,
Acquiring a multi-stage feature map of a face image through a convolutional neural network layer;
Obtaining an initial query matrix by fully connecting last stage feature maps of the multi-stage feature maps through a fully connected layer, the initial query matrix representing initial features of landmarks of the face image, The number of elements in the initial query matrix is equal to the number of landmarks of the face image -;
obtaining a memory feature matrix by flattening and concatenating the multi-stage feature maps; and
Determining the landmark coordinates of the face image by inputting the memory feature matrix and the initial query matrix to at least one cascaded decoder layer
including,
A method for predicting coordinates of landmarks in face images.
According to claim 1,
The decoder layer includes a cascaded self-attention module layer, a deformable attention module layer, and a landmark coordinate prediction layer,
Determining the landmark coordinates of the face image,
inputting an initial query matrix and an initial query matrix in which location information is embedded into a self-attention module layer of a first decoder layer as a query matrix, a key matrix, and a value matrix;
Transformable Attention obtaining an output matrix of a module layer, wherein the output matrix and the memory feature matrix of the self-attention module layer are the query matrix and the value matrix of the transformable attention module layer, and the transformation of the first decoder layer The landmark coordinates predicted in the previous decoder layer input to the possible attention module layer are initial landmark coordinates obtained based on the initial query matrix -;
The output matrix of the transformable attention module layer of the current decoder layer and the output matrix of the transformable attention module layer in which the position information is embedded are cascaded into a value matrix of a self-attention module layer of the next decoder layer. , as a query matrix and a key matrix, inputting to the self-attention module layer of the next decoder layer; and
obtaining the landmark coordinates of the face image predicted by the current decoder layer by inputting the output matrix of the deformable attention module layer of the current decoder layer to a landmark coordinate prediction layer of the current decoder layer; - The landmark coordinates of the face image predicted by the last decoder layer are the landmark coordinates of the final face image -;
including,
A method for predicting coordinates of landmarks in face images.
According to claim 2,
The output matrix of the self-attention module layer of the decoder layer,

is obtained with
From here,

denotes the i-th row vector of the output matrix,

Represents the attention weight obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module layer and the j-th row vector of the key matrix input to the self-attention module layer,

Represents the j-th row vector of the initial query matrix or the output matrix of the transformation attention layer of the previous decoder layer, N represents the number of landmarks in the face image,
A method for predicting coordinates of landmarks in face images.
According to claim 3,
The output matrix of the transformable attention module layer of the decoder layer,

is obtained as, where,

denotes the updated feature of the ith landmark of the output matrix,

represents an attention weight obtained by performing a fully connected operation and a softmax operation on the query matrix input to the transformable attention module layer,

denotes a feature corresponding to the k-th reference point coordinate in the memory feature matrix, and the location offset between the k-th reference point coordinate and the i-th landmark coordinate of the landmark coordinate predicted in the previous decoder layer is the query input to the transformable attention module layer. Obtained by performing a full-complete operation on the matrix, K is a preset value,
A method for predicting coordinates of landmarks in face images.
According to claim 2,
The landmark coordinates predicted by the decoder layer,

, where y represents the landmark coordinates predicted by the current decoder layer,

denotes the landmark coordinates predicted by the previous decoder layer or the initial landmark coordinates,

Is

Representing the output of the landmark coordinate prediction layer representing the offset of y for
A method for predicting coordinates of landmarks in face images.
According to claim 2,
The convolutional network layer, the fully connected layer, and the at least one decoder layer are obtained through training using training image samples based on the regression loss function below,

From here,

represents the regression loss,

Represents the landmark coordinates of training image samples predicted by each decoder layer,

denotes the actual landmark coordinates of the training image sample,

is the quantity of decoder layers,

is the index of the decoder layer,
A method for predicting coordinates of landmarks in face images.
An apparatus for predicting landmark coordinates of a face image,
Obtaining a multi-stage feature map of the face image through a convolutional neural network layer, and fully connecting a last stage feature map of the multi-stage feature map through a fully connected layer to obtain an initial query matrix, wherein the initial query matrix is Represents initial features of landmarks of a face image, and the number of elements in the initial query matrix is equal to the number of landmarks of the face image - an encoder configured to obtain a memory feature matrix by flattening and concatenating the multi-stage feature maps. ; and
A decoder comprising at least one cascaded decoder layer, the at least one decoder layer configured to determine the landmark coordinates of the face image based on the memory feature matrix and the initial query matrix received from the encoder. configured -;
including,
Device.
An apparatus for predicting landmark coordinates of a face image,
processor; and
memory storing one or more instructions executed by the processor
wherein the processor executes the instructions to:
obtaining a multi-stage feature map of a face image through a convolutional neural network layer;
Obtaining an initial query matrix by fully connecting last stage feature maps of the multi-stage feature maps through a fully connected layer, wherein the initial query matrix represents initial features of landmarks of the face image, and The number of elements in the initial query matrix is equal to the number of landmarks of the face image -;
obtaining a memory feature matrix by flattening and concatenating the multi-stage feature maps; and
Determining the landmark coordinates of the face image by inputting the memory feature matrix and the initial query matrix to at least one cascaded decoder layer
configured to perform
Device.
According to claim 7 or 8,
The decoder layer includes a cascaded self-attention module layer, a transformable attention module layer, and a landmark coordinate prediction layer,
The self-attention module layer of the first decoder layer is configured to obtain an initial query matrix into which location information is embedded and an output matrix of the self-attention module layer of the first decoder layer based on the initial query matrix, wherein the location information is embedded. The initial query matrix and the initial query matrix are input as a query matrix, a key matrix, and a value matrix of the self-attention module layer of the first decoder layer, respectively;
Self-attention module layers of decoder layers other than the first decoder layer include an output matrix of a deformable attention module layer of a cascaded previous decoder layer and a deformable deformable of a previous decoder layer in which the position information is embedded. Acquiring an output matrix of a transformable attention module of a current decoder layer based on an output matrix of the attention module layer, wherein the output matrix of the transformable attention module layer of the previous decoder layer and the position information are embedded. The output matrix of the deformable attention module layer of the previous decoder layer is the value matrix, query matrix, and key matrix of the self-attention module layer of the current decoder layer,
The deformable attention module layer of the decoder layer is based on the output matrix of the self-attention module layer of the current decoder layer, the memory feature matrix, and the landmark coordinates predicted in the cascaded previous decoder layer. and obtain the output matrix of the transformable attention module layer, wherein the output matrix of the self-attention module layer of the current decoder layer and a memory feature matrix are a query matrix of the transformable attention module layer of the current decoder layer. and a value matrix, wherein the landmark coordinates predicted in the previous decoder layer of the first decoder layer are initial landmark coordinates obtained based on the initial query matrix;
The landmark coordinate prediction layer is configured to obtain the landmark coordinates of the face image predicted by the decoder layer based on the output matrix of the deformable attention module layer of the current decoder layer,
The landmark coordinates of the face image predicted by the last decoder layer are the landmark coordinates of the final face image, device.
According to claim 9,
The self-attention module layer of the decoder layer,

Obtaining the output matrix with
From here,

denotes the i-th row vector of the output matrix,

denotes an attention weight obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module layer and the j-th row vector of the key matrix input to the self-attention module layer,

Represents the j-th row vector of the initial query matrix or the output matrix of the transformation attention layer of the previous decoder layer, N represents the number of landmarks in the face image,
Device.
According to claim 10,
The deformable attention module layer of the decoder layer,

Obtaining the output matrix with
From here,

denotes the updated feature of the ith landmark of the output matrix,

Represents the attention weight obtained by performing a fully connected operation and a softmax operation on the query matrix input to the transformable attention module layer,

denotes a feature corresponding to the k-th reference point coordinate in the memory feature matrix, and the location offset between the k-th reference point coordinate and the i-th landmark coordinate of the landmark coordinate predicted in the previous decoder layer is the query input to the transformable attention module layer. Obtained by performing a full-complete operation on the matrix, K is a preset value,
Device.
According to claim 9,
The landmark coordinate prediction layer of the decoder layer,

Predicting the landmark coordinates with
Here, y represents the landmark coordinates predicted by the current decoder layer,

denotes the landmark coordinates predicted by the previous decoder layer or the initial landmark coordinates,

Is

Representing the output of the landmark coordinate prediction layer representing the offset of y for
Device.
According to claim 9,
The device is trained using training image samples based on the regression loss function below,

From here,

represents the regression loss,

Represents the landmark coordinates of training image samples predicted by each decoder layer,

denotes the actual landmark coordinates of the training image sample,

is the quantity of decoder layers,

is the index of the decoder layer,
Device.
A computer-readable storage medium storing a computer program implementing a method of predicting landmark coordinates of a face image according to any one of claims 1 to 6 when executed by a processor.

Images