WO2024060909A1 - 识别表情的方法、装置、设备及介质 - Google Patents

识别表情的方法、装置、设备及介质 Download PDF

Info

Publication number
WO2024060909A1
WO2024060909A1 PCT/CN2023/114307 CN2023114307W WO2024060909A1 WO 2024060909 A1 WO2024060909 A1 WO 2024060909A1 CN 2023114307 W CN2023114307 W CN 2023114307W WO 2024060909 A1 WO2024060909 A1 WO 2024060909A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
feature
features
recognized
fused
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/114307
Other languages
English (en)
French (fr)
Inventor
武文琦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alipay Hangzhou Information Technology Co Ltd
Original Assignee
Alipay Hangzhou Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alipay Hangzhou Information Technology Co Ltd filed Critical Alipay Hangzhou Information Technology Co Ltd
Publication of WO2024060909A1 publication Critical patent/WO2024060909A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation

Definitions

  • the present application relates to the field of computer information processing technology, and in particular to methods, devices, equipment and media for recognizing expressions.
  • Human expressions such as happiness, sadness, fear, anger, disgust, surprise and contempt, etc.
  • Human expressions are expressed on the face as a kind of psychological state. Therefore, the human psychological state can be judged through the expression of the human face, thereby achieving relevant tasks.
  • Interactive applications of facial expressions such as human-computer interaction.
  • the expression recognition results can be used to determine the current emotional state of the collector, thereby inferring his or her willingness to scan the face, which can be used as one of the basis for soft-start judgment of the overall face-scanning link.
  • facial expressions are unstable and irregular in various states, making it difficult for expression recognition algorithms to achieve high robustness. Therefore, how to achieve highly robust expression recognition and improve the accuracy of expression recognition are technical issues that need to be solved urgently.
  • the embodiments of this specification provide a method, device, equipment and medium for recognizing facial expressions to solve the problems of low robustness and accuracy in existing facial expression methods.
  • embodiments of this specification provide a method for recognizing expressions, which includes: obtaining an image to be recognized that contains facial features; and based on a local binary pattern algorithm, obtaining a local binary pattern corresponding to the image to be recognized. image; obtain the first image feature corresponding to the image to be recognized and the second image feature corresponding to the local binary pattern image; fuse the first image feature and the second image feature to obtain the fused Image features: use an expression recognition model built based on the attention mechanism to process the fused image features to obtain an expression recognition result for the image to be recognized.
  • An embodiment of the present specification provides a device for recognizing expressions, including: an image acquisition module, used to acquire an image to be recognized containing facial features; a first image processing module, based on a local binary pattern algorithm, to obtain the image to be recognized. Identify the local binary pattern image corresponding to the image; the feature extraction module is used to obtain the first image feature corresponding to the image to be recognized and the second image feature corresponding to the local binary pattern image; the feature fusion module is used to obtain The first image feature and the second image feature are fused to obtain the fused image feature; the expression recognition module is used to process the fused image feature using an expression recognition model built based on the attention mechanism. , to obtain the expression recognition result for the image to be recognized.
  • An embodiment of this specification provides a device for recognizing expressions, including at least one processor and a memory communicatively connected to the at least one processor.
  • the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can: obtain an image to be recognized that contains facial features. ; Based on the local binary pattern algorithm, obtain the local binary pattern image corresponding to the image to be recognized; obtain the first image feature corresponding to the image to be recognized and the second image feature corresponding to the local binary pattern image; The first image feature and the second image feature are fused to obtain the fused image feature; the expression recognition model based on the attention mechanism is used to process the fused image feature to obtain the target image to be Expression recognition results for recognized images.
  • An embodiment of the present specification provides a computer-readable medium on which computer-readable instructions are stored.
  • the computer-readable instructions can be executed by a processor to implement a method for recognizing facial expressions.
  • One embodiment of this specification achieves the following beneficial effects: obtaining the local binary pattern image corresponding to the image to be recognized through the local binary pattern algorithm, extracting the first image feature corresponding to the image to be recognized and the third image feature corresponding to the local binary pattern image.
  • Two image features fuse the first image feature and the second image feature to obtain the fused image features.
  • Recognition results When performing expression recognition, an expression recognition model built by a self-attention mechanism is introduced based on comprehensive consideration of the first image features and the second image features, and attention is paid to the image features after the fusion of the first image features and the second image features.
  • Strength enhancement improves the expression of long-range relationships between image features, thereby making expression recognition more robust and accurate.
  • Figure 1 is a schematic diagram of an application scenario of a method for recognizing expressions provided by an embodiment of this specification.
  • FIG. 2 is a schematic flowchart of a method for recognizing expressions provided by an embodiment of this specification.
  • Figure 3 is a schematic flowchart of an overall solution for a method of recognizing expressions provided by an embodiment of this specification.
  • FIG. 4 is a schematic diagram of an expression recognition device corresponding to FIG. 2 provided by an embodiment of this specification.
  • FIG. 5 is a schematic diagram of an expression recognition device corresponding to FIG. 2 provided by an embodiment of this specification.
  • an expression recognition method based on LBP features is used for expression recognition.
  • LBP features are extracted from the original RGB image, and then end-to-end expression recognition classification is performed based only on the LBP features. This makes The color information in the original RGB image is lost and cannot be used, resulting in low accuracy and robustness of the final recognition result.
  • Figure 1 is a schematic diagram of an application scenario of a method for identifying expressions provided by the embodiment of this specification; as shown in Figure 1, this solution can include the following steps: Image 1, server 2; wherein, server 2 may contain a model for expression recognition on image 1 to be recognized.
  • the image 1 to be recognized can be input to the server 2, and the server 2 can obtain the local binary pattern image corresponding to the image to be recognized based on the local binary pattern algorithm; obtain the local binary pattern image corresponding to the image to be recognized; The first image feature and the second image feature corresponding to the local binary pattern image; fuse the first image feature and the second image feature to obtain the fused image feature; use an expression recognition model based on the attention mechanism to perform the fusion The final image features are processed to obtain the expression recognition results for the image to be recognized.
  • Figure 2 is a schematic flowchart of a method for identifying expressions provided by an embodiment of this specification. From a program perspective, the execution subject of the process can be a program installed on the application server or an application client.
  • the process may include the following steps.
  • Step 202 Obtain the image to be recognized containing facial features.
  • the image to be recognized contains facial features.
  • the image to be recognized can be a photo taken in history or an image collected in real time, and the image to be recognized can be an RGB image.
  • Step 204 Based on the local binary pattern algorithm, obtain the local binary pattern image corresponding to the image to be recognized.
  • the image to be recognized is converted into a grayscale image, and the grayscale image is processed based on the local binary pattern algorithm to obtain the corresponding local binary pattern image, that is, the corresponding LBP image is obtained.
  • Step 206 Obtain the first image feature corresponding to the image to be recognized and the second image feature corresponding to the local binary pattern image.
  • the first image feature may be a color feature of the image to be identified
  • the second image feature may be an LBP feature of the image to be identified.
  • LBP Local Binary Pattern, Local Binary Pattern
  • LBP Local Binary Pattern
  • the first image feature and the second image feature are extracted with the same dimension, so that the dimension of the first image feature and the dimension of the second image feature are the same.
  • the number of pixels of the first image feature is 128X128, and the second image feature The number of pixels represented is also 128X128.
  • Step 208 Fusion of the first image feature and the second image feature to obtain a fused image feature.
  • the fused image feature contains both the information in the first image feature and the information in the second image feature.
  • the first image features and the second image features can be comprehensively utilized to realize the complementary advantages of the two image features, so as to obtain higher robustness and accuracy of expression recognition results in the future.
  • Step 210 Use an expression recognition model built based on the attention mechanism to process the fused image features to obtain an expression recognition result for the image to be recognized.
  • the expression recognition model can be built based on the self-attention mechanism, and the expression recognition results can be obtained based on the fused image features.
  • the expression recognition model can be a classification model, which matches the corresponding expression according to the fused image features and outputs it.
  • the method in Figure 2 is based on the fused image features of the first image feature of the image to be recognized and the LBP feature, and is processed by an expression recognition model built with an attention mechanism to obtain the expression recognition result of the image to be recognized.
  • expression recognition on the one hand, the corresponding relationship between the comprehensive performance of the first image feature and LBP feature and the expression is considered to avoid the singleness of the extracted features; on the other hand, the expression recognition model built by the self-attention mechanism is introduced to fuse the Attention is enhanced on the final image features to improve the expression of long-range relationships between image features; thus making expression recognition more robust and accurate. This enables accurate recognition of facial expressions when facial expressions are unstable and irregular in various states.
  • step 202 after obtaining the image to be recognized including the face image, may further include: using a face alignment algorithm to perform face alignment processing on the image to be recognized.
  • the face alignment algorithm is an existing algorithm, which automatically locates key facial feature information based on the input face image, such as eyes, nose tip, mouth corners, eyebrows, and outline points of various parts of the face. Align these key feature information to a preset fixed position.
  • the angles of faces on different images to be recognized may be different. Some are facing the camera, and some are not. lens. In some images to be recognized, the face part is not located in the center of the image, and the useless background occupies a larger space in the image.
  • images facing the camera at various angles can be adjusted to images with faces facing the camera forward; images with faces located in different positions of the image can be adjusted so that the faces are all located in the middle of the image. image; achieve the purpose of data normalization.
  • the aligned images to be recognized for feature extraction more favorable image features can be extracted and the negative effects caused by background information and different face angles can be reduced.
  • step S204 obtaining the first image feature corresponding to the image to be recognized and the second image feature corresponding to the local binary pattern image may specifically include: using a first preset residual network model to The image to be identified is subjected to feature extraction processing to obtain the first image feature corresponding to the image to be identified; the second preset residual network model is used to perform feature extraction processing on the local binary pattern image to obtain the local binary pattern image.
  • the second image feature corresponding to the value pattern image may specifically include: using a first preset residual network model to The image to be identified is subjected to feature extraction processing to obtain the first image feature corresponding to the image to be identified; the second preset residual network model is used to perform feature extraction processing on the local binary pattern image to obtain the local binary pattern image.
  • the second image feature corresponding to the value pattern image may specifically include: using a first preset residual network model to The image to be identified is subjected to feature extraction processing to obtain the first image feature corresponding to the image to be identified; the second preset residual network model is used to perform feature extraction processing on the
  • the first preset residual network model and the second preset residual network model can be the same residual network model, or they can be residual network models with the same structure, or they can be residual networks with the same output feature dimensions. Model.
  • the first preset residual network model and the second preset residual network model are used to extract image features of the graphics to be identified and the local binary image respectively, so that higher-precision image features can be extracted.
  • the first preset residual network model and/or the second preset residual network model may include a ResNet model.
  • the first preset residual network model and/or the second preset residual network model can select a ResNet (Residual Neural Network, residual network) model, specifically a ResNet18 model, a ResNet50 model or a ResNet101 model can be selected. wait.
  • ResNet Residual Neural Network, residual network
  • other types of residual network models can also be used, and there are no specific limitations here.
  • step 208, fusing the first image features and the second image features to obtain fused image features may specifically include: based on a first feature fusion algorithm, merging the first image features into Feature fusion is performed with the second image feature to obtain a fused image feature; the feature dimension of the fused image feature is the same as the feature dimension of the first image feature and the second image feature.
  • the first feature fusion algorithm may include feature map addition and an add fusion method with a constant number of channels to combine the first image Features are fused with second image features.
  • the first image feature contains 128X128 pixels
  • the second image feature also contains 128X128 pixels.
  • Each pixel value of the first image feature is added to the corresponding pixel value of the second image feature.
  • the pixel value of the first pixel of the first image feature is 5, the value of the first pixel of the second image feature is 6, and the value of the first pixel of the fused image feature is 11.
  • the dimensionality of the fused image features itself does not increase, but the amount of information in each dimension increases. The calculation amount of the first feature fusion algorithm is small.
  • fusing the first image feature and the second image feature to obtain the fused image feature may specifically include: based on a first feature fusion algorithm, fusing the first image feature and the second image feature.
  • the first feature fusion algorithm may include a concat fusion method with an increased number of channels to fuse local features and global features to obtain the second fused image features.
  • the above method performs two feature fusions. After the first feature fusion is performed to obtain the first fused image features, global features and local features are respectively extracted from the first fused image features, and then the second feature fusion algorithm is used to fuse the global features and local features. This makes the information of the obtained image features richer, thereby making subsequent expression recognition more accurate.
  • it may also include: performing non-linear enhancement processing on the global features and the local features to obtain the processing The global features after processing and the local features after processing; wherein, the feature fusion of the local features and the global features specifically includes: feature fusion of the global features after the processing and the local features after the processing. Fusion.
  • the global features and the local features can be processed by an activation function layer ReLU to perform nonlinear enhancement, and then the nonlinearly enhanced global features and local features can be feature fused.
  • the expression recognition model based on the attention mechanism is used to identify the fused image features.
  • the processing may specifically include: dividing the image to be recognized into a preset number of sub-images; obtaining image features of each sub-image; arranging the fused image features and the image features of each sub-image in a preset arrangement order to obtain a feature input queue; and inputting the feature input queue into the expression recognition model built based on the attention mechanism.
  • the expression recognition model can include an expression recognition model built based on the Transformer network structure.
  • the image to be recognized can be divided into a preset number of sub-images, and the image features of each sub-image can be obtained, and the image features can be color features.
  • the fused image features and the color features of each sub-image are arranged in a preset order to obtain a feature input queue, and the feature input queue is input into the expression recognition model to obtain the expression recognition result.
  • the local binary pattern image can also be divided into a preset number of sub-images, and the image features of each sub-image can be obtained, and the image features can be LBP features.
  • the fused image features and the color features of each sub-image are arranged in a preset order to obtain a feature input queue, and the feature input queue is input into the expression recognition model to obtain the expression recognition result.
  • the image to be recognized and the local binary pattern image can also be divided into a preset number of sub-images, and the color characteristics of each sub-image of the image to be recognized and the local binary pattern image are obtained.
  • LBP features of each sub-image The fused image features, the color features of each sub-image, and the LBP features of each sub-image are arranged in a preset order to obtain a feature input queue.
  • the feature input queue is input into the expression recognition model to obtain the expression recognition result.
  • the expression recognition model may include a position encoder, a Transformer encoder and an expression classifier; inputting the feature input queue into the expression recognition model built using an attention mechanism specifically includes: using the The position encoder performs position encoding on the feature input queue to obtain a feature sequence; the feature sequence is input to the Transformer encoder to obtain coding features; the coding features are input to the expression classifier to obtain expression recognition result.
  • the position encoder After the feature input queue is input into the expression recognition model, the position encoder performs position encoding on each image feature to obtain a feature sequence, and the feature sequence is input into the Transformer encoder for encoding to obtain the encoded features.
  • the Transformer encoder contains multiple sub-encoders.
  • the structure of each sub-encoder is a self-attention mechanism plus a feedforward neural network.
  • the fused image features and the image features of each sub-image are enhanced by self-attention features to establish a long-range relationship between the image features, that is, to make the correlation between the fused image features and the image features of each sub-image stronger.
  • the encoded features are input into the expression classifier through a fully connected layer to obtain the expression Recognition result.
  • the expression classifier may include an expression 7 classifier. Expressions are classified into seven categories, namely, happiness, sadness, fear, anger, disgust, surprise and contempt. In practical applications, other types of expression classifiers may also be used, which are not specifically limited here.
  • the preset arrangement order may be determined based on machine learning.
  • the arrangement order of the fused image features and each sub-image feature can be determined based on machine learning, that is, the feed-forward fused features are output through machine learning to output an arrangement order that is more conducive to expression recognition.
  • a machine model for sorting sub-images can be trained according to the accuracy of expression recognition, so that the machine model can provide image sorting that is more conducive to expression recognition.
  • the image to be recognized in the embodiment of this specification may include at least one of an RGB image, a three-dimensional image, and a NIR (Near Infrared) image.
  • an RGB image a three-dimensional image
  • NIR Near Infrared
  • an end-to-end learning method is used through a multi-modal approach to perform expression recognition to further increase Robustness and accuracy of expression recognition.
  • Figure 3 is a schematic flowchart of an overall solution for a method of recognizing expressions provided by an embodiment of this specification.
  • the method for recognizing expressions provided in the embodiment of this specification may include the following steps.
  • Step 301 Obtain an image to be recognized that contains facial features; where the image to be recognized can be an RGB image.
  • Step 303 Based on the local binary pattern algorithm, obtain the local binary pattern image corresponding to the image to be recognized, that is, obtain the corresponding LBP image.
  • Step 305 Use the first preset residual network model to perform feature extraction processing on the image to be identified, and obtain the first image features corresponding to the image to be identified; wherein the first preset residual network model can use the ResNet18 network In the model, the first image feature may be a color feature.
  • Step 307 Use the second preset residual network model to perform feature extraction processing on the local binary pattern image to obtain the second image features corresponding to the local binary pattern image; wherein the second preset residual network model can Using the ResNet18 network model, the second image feature can be the LBP feature.
  • Step 309 Based on the first feature fusion algorithm, perform feature fusion on the first image feature and the second image feature to obtain a first fused image feature; the feature dimension of the first fused image feature is the same as the first fused image feature. The feature dimensions of the image feature and the second image feature are the same.
  • Step 311 Use the Local Attention (local attention mechanism) structure or model to extract the first fusion map Local features of image features; among them, the local attention mechanism structure can include Conv Layer convolution layer 11, ReLU Layer nonlinear activation layer 13, Batch Nomalization batch normalization layer 14, Conv Layer convolution layer 11 and Batch Nomalization batch Normalization layer 14.
  • the local attention mechanism structure can include Conv Layer convolution layer 11, ReLU Layer nonlinear activation layer 13, Batch Nomalization batch normalization layer 14, Conv Layer convolution layer 11 and Batch Nomalization batch Normalization layer 14.
  • Step 313 Use the Global Attention (global attention mechanism) structure or model to extract the global features of the first fused image feature; wherein, the global attention mechanism structure may include the Average Pooling layer 12 and the Conv Layer convolution layer in sequence. 11. ReLU Layer nonlinear activation layer 13, Batch Nomalization batch normalization layer 14, Conv Layer convolution layer 11 and Batch Nomalization batch normalization layer 14. Compared with the local attention structure, the global attention structure adds an average pooling operation at the front end to better extract global structural information.
  • Step 315 Perform nonlinear enhancement processing on the global features and the local features to obtain processed global features and processed local features; the global features and local features can be nonlinearly enhanced through the ReLU function.
  • Step 317 Fusing the processed global features and the processed local features to obtain a second fused image feature.
  • Step 319 Divide the image to be recognized into a preset number of sub-images; obtain the image features of each sub-image; arrange the fused image features and each sub-image feature in a preset order, Obtain the feature input queue; among them, the Class square in the figure represents the fused image feature, and the other squares represent the image features of each sub-image.
  • the divided sub-images and fused features can be processed into features of preset dimensions through processing methods such as Linear and Flat to form a feature input queue.
  • Step 321 Use position encoding (position encoder) to perform position encoding on the feature input queue to obtain a feature sequence
  • Step 323 Input the feature sequence into the Transformer Encoder (Transformer Encoder) to obtain encoding features;
  • Transformer is a A model of self-attention mechanism. That is, the feature sequence is enhanced with self-attention features to establish the long-range relationship between the image features in the feature sequence.
  • Step 325 Pass the encoding feature through a fully connected layer and then input it into the Classifier (expression classifier) to obtain the expression recognition result.
  • FIG. 4 is a schematic diagram of an expression recognition device corresponding to FIG. 2 provided by an embodiment of this specification.
  • the device may include: an image acquisition module 401, used to acquire an image to be recognized containing facial features; a first image processing module 403, using Based on the local binary pattern algorithm, the local binary pattern image corresponding to the image to be recognized is obtained; the feature extraction module 405 is used to obtain the first image feature corresponding to the image to be recognized and the corresponding local binary pattern image the second image feature; the feature fusion module 407, used to fuse the first image feature and the second image feature to obtain the fused image feature; the expression recognition module 409, used to use the attention-based mechanism to build An expression recognition model is used to process the fused image features to obtain an expression recognition result for the image to be recognized.
  • the device further comprises a face alignment module, configured to perform face alignment processing on the image to be identified using a face alignment algorithm.
  • a face alignment module configured to perform face alignment processing on the image to be identified using a face alignment algorithm.
  • the feature extraction module includes a first feature extraction module and a second extraction module; the first feature extraction module is used to perform feature extraction processing on the image to be identified using a first preset residual network model, Obtain the first image feature corresponding to the image to be recognized; the second feature extraction module is used to perform feature extraction processing on the local binary pattern image using a second preset residual network model to obtain the local binary pattern image. The second image feature corresponding to the pattern image.
  • the feature fusion module includes a first feature fusion module and a second feature fusion module; the first feature fusion module is used to combine the first image feature and the second feature fusion algorithm based on a first feature fusion algorithm.
  • the image features are feature fused to obtain the fused image features; the feature dimensions of the fused image features are the same as the feature dimensions of the first image feature and the second image feature; the second feature fusion module uses Using a local attention mechanism to extract local features of the first fused image feature; using a global attention mechanism to extract global features of the first fused image feature; based on a second feature fusion algorithm, combining the local features with the Global features are used for feature fusion to obtain the second fused image features.
  • the embodiments of this specification also provide equipment corresponding to the above method.
  • FIG. 5 is a schematic diagram of an expression recognition device corresponding to FIG. 2 provided by an embodiment of this specification.
  • the device 500 may include at least one processor 510 and a memory 530 communicatively connected with the at least one processor.
  • the memory 530 stores instructions 520 that can be executed by the at least one processor 510, and the instructions are executed by the at least one processor 510, so that the at least one processor 510 can: obtain information containing human faces.
  • embodiments of this specification also provide computer-readable media corresponding to the above method.
  • Computer-readable instructions are stored on the computer-readable medium, and the computer-readable instructions can be executed by the processor to implement the above method of recognizing expressions.
  • PLD Programmable Logic Device
  • FPGA Field Programmable Gate Array
  • HDL Hardware Description Language
  • the controller may be implemented in any suitable manner.
  • the controller may take the form of a microprocessor or processor. and computer-readable media, logic gates, switches, Application Specific Integrated Circuits (ASICs), programmable logic controllers that store computer-readable program codes (such as software or firmware) that can be executed by the (micro)processor
  • ASICs Application Specific Integrated Circuits
  • controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320.
  • the memory controller can also be implemented as part of the control logic of the memory.
  • the controller in addition to implementing the controller in the form of pure computer-readable program code, the controller can be completely programmed with logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded logic by logically programming the method steps. Microcontroller, etc. to achieve the same function. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions can be considered as structures within hardware components as well as software modules implementing the methods.
  • a typical implementation device is a computer.
  • the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device Or a combination of any of these devices.
  • embodiments of the present invention may be provided as methods, systems, or computer program products.
  • the invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.
  • the invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
  • each process and/or box in the flowchart and/or block diagram, as well as the combination of the process and/or box in the flowchart and/or block diagram can be implemented by computer program instructions.
  • These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.
  • These computer program instructions may also be stored in a computer or other programmable data processing device that cause a computer or other programmable data processing device to perform a specific manner.
  • a computer-readable memory that operates in a manner such that instructions stored in the computer-readable memory produce an article of manufacture that includes instruction means that implements a process or processes in a flowchart and/or a block or blocks in a block diagram function specified in each box.
  • These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device.
  • Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.
  • a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
  • processors CPUs
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • Memory may include non-permanent storage in computer-readable media, random access memory (RAM) and/or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
  • RAM random access memory
  • ROM read-only memory
  • flash RAM flash random access memory
  • Computer-readable media includes both persistent and non-volatile, removable and non-removable media that can be implemented by any method or technology for storage of information.
  • Information may be computer-readable instructions, data structures, modules of programs, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), and read-only memory.
  • PRAM phase change memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • RAM random access memory
  • read-only memory read-only memory
  • ROM read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • flash memory or other memory technology
  • compact disc read-only memory CD-ROM
  • DVD digital versatile disc
  • magnetic tape cartridges magnetic tape storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
  • computer-readable media does not include transitory media, such as modulated data signals and carrier waves.
  • embodiments of the present application may be provided as methods, systems or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may employ computer-usable program code embodied therein in one or more computer-usable programs. In the form of a computer program product implemented on a storage medium (including but not limited to disk storage, CD-ROM, optical storage, etc.).
  • the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules.
  • program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types.
  • the present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network.
  • program modules may be located in local and remote computer storage media, including storage devices.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

本说明书实施例公开了一种识别表情的方法，包括：获取包含人脸特征的待识别图像；基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像；获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征；将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征；利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，得到针对所述待识别图像的表情识别结果。

Description

识别表情的方法、装置、设备及介质 技术领域
本申请涉及计算机信息处理技术领域，尤其涉及识别表情的方法、装置、设备及介质。
背景技术
人类的表情，比如高兴、伤心、害怕、愤怒、厌恶、惊讶和轻蔑等，作为一种心理状态的表达从脸部表现出来，因此可以通过人脸的表情来判断人类的心理状态，从而实现有关人脸表情的交互式应用，如人机交互等。在人脸识别系统中可以通过表情识别结果来判断采集人当前的情绪状态，从而推断其刷脸的意愿，可作为整体刷脸链路的软启动判断依据之一。然而，人脸表情在多种状态下呈现不稳定、不规律的状态，使得表情识别算法难以实现较高的鲁棒性。因此，如何实现高鲁棒性的表情识别，提高表情识别准确性是亟待解决的技术问题。
发明内容
本说明书实施例提供一种识别表情的方法、装置、设备及介质，以解决现有的表情方法存在的鲁棒性和准确性较低的问题。
为解决上述技术问题，本说明书实施例提供的一种识别表情的方法，包括：获取包含人脸特征的待识别图像；基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像；获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征；将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征；利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，得到针对所述待识别图像的表情识别结果。
本说明书实施例提供的一种识别表情的装置，包括：图像获取模块，用于获取包含人脸特征的待识别图像；第一图像处理模块，用于基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像；特征提取模块，用于获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征；特征融合模块，用于将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征；表情识别模块，用于利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，得到针对所述待识别图像的表情识别结果。
本说明书实施例提供的一种识别表情的设备，包括至少一个处理器以及与所述至少一个处理器通信连接的存储器。其中，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够：获取包含人脸特征的待识别图像；基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像；获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征；将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征；利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，得到针对所述待识别图像的表情识别结果。
本说明书实施例提供的一种计算机可读介质，其上存储有计算机可读指令，所述计算机可读指令可被处理器执行以实现一种识别表情的方法。
本说明书一个实施例实现了能够达到以下有益效果：通过局部二值模式算法得到待识别图像对应的局部二值模式图像，提取待识别图像对应的第一图像特征和局部二值模式图像对应的第二图像特征，将第一图像特征和第二图像特征进行融合，得到融合后的图像特征，利用基于注意力机制搭建的表情识别模型对融合后的图像特征进行处理，得到针对待识别图像的表情识别结果。在进行表情识别时，在综合考虑第一图像特征和第二图像特征的基础上引入了自注意力机制搭建的表情识别模型，对第一图像特征和第二图像特征融合后的图像特征进行注意力增强，提高图像特征之间的长程关系表达，从而使得表情识别具有更高的鲁棒性和准确性。
附图说明
图1为本说明书实施例提供的一种识别表情的方法的应用场景示意图。
图2为本说明书实施例提供的一种识别表情的方法的流程图示意图。
图3为本说明书实施例提供的一种识别表情的方法的整体方案流程示意图。
图4为本说明书实施例提供的对应于图2的一种识别表情的装置示意图。
图5为本说明书实施例提供的对应于图2的一种识别表情的设备示意图。
具体实施方式
以下结合附图，详细说明本说明书各实施例提供的技术方案。
现有技术中，采用基于LBP特征的表情识别方法进行表情识别，该方法中从原始RGB图像中提取LBP特征，然后只基于LBP特征进行端到端的表情识别分类。这就使 得原始RGB图像中的颜色信息损失而得不到利用，造成最终识别结果准确性和鲁棒性较低。
为了解决现有技术中的缺陷，本方案给出了以下实施例：图1为本说明书实施例提供的一种识别表情的方法的应用场景示意图；如图1所示，该方案可以包括待识别图像1、服务器2；其中，服务器2中可包含对待识别图像1进行表情识别的模型。具体的，本说明书实施例中可以将待识别图像1输入到服务器2中，在服务器2中可以基于局部二值模式算法，得到待识别图像对应的局部二值模式图像；获取待识别图像对应的第一图像特征和局部二值模式图像对应的第二图像特征；将第一图像特征和第二图像特征进行融合，得到融合后的图像特征；利用基于注意力机制搭建的表情识别模型，对融合后的图像特征进行处理，得到针对待识别图像的表情识别结果。
接下来，将针对说明书实施例提供的一种识别表情的方法结合附图进行具体说明：图2为本说明书实施例提供的一种识别表情的方法的流程示意图。从程序角度而言，流程的执行主体可以为搭载于应用服务器的程序或应用客户端。
如图2所示，该流程可以包括以下步骤。
步骤202：获取包含人脸特征的待识别图像。
获取待识别图像，待识别图像包含人脸特征。其中待识别图像可以为历史拍摄的照片，也可以为实时采集的图像，待识别图像可以为RGB图像。
步骤204：基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像。
将待识别图像，转化为灰度图，基于局部二值模式算法对灰度图进行处理得到对应的局部二值模式图像，也就是得到对应的LBP图像。
步骤206：获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征。
其中，第一图像特征可以为待识别图像的颜色特征。第二图像特征可以为待识别图像的LBP特征。
LBP(Local Binary Pattern，局部二值模式)是一种用来描述图像局部纹理特征的算子，其计算简单、数据量小且具有旋转不变性和灰度不变性等显著优点。在进行特征提取时，以相同的维度对第一图像特征与第二图像特征进行提取，使得第一图像特征的维度与第二图像特征的维度相同。例如第一图像特征的像素个数为128X128，第二图像特 征的像素个数也为128X128。
步骤208：将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征。
融合后的图像特征既包含第一图像特征中的信息又包含了第二图像特征的信息。通过特征融合能够综合利用第一图像特征和第二图像特征，实现两个图像特征的优势互补，以便后续获得更高鲁棒性和准确性的表情识别结果。
步骤210：利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，得到针对所述待识别图像的表情识别结果。
表情识别模型可以是基于自注意力机制搭建的，可以根据融合后的图像特征得出表情识别结果，表情识别模型可以为分类模型，根据融合后的图像特征匹配对应的表情进行输出。
应当理解，本说明书一个或多个实施例所述的方法其中部分步骤的顺序可以根据实际需要相互交换，或者其中的部分步骤也可以省略或删除。
图2中的方法，以待识别图像的第一图像特征与LBP特征进行融合后图像特征为基础，以注意力机制搭建的表情识别模型进行处理，得到待识别图像的表情识别结果。在进行表情识别时，一方面考虑了第一图像特征和LBP特征的综合表现与表情的对应关系，避免了提取特征的单一性；另一方面引入自注意力机制搭建的表情识别模型，对融合后的图像特征进行注意力增强，提高图像特征之间的长程关系表达；从而使得表情识别具有更高的鲁棒性和准确性。从而能够适应人脸表情在多种状态下呈现不稳定、不规律的表现时，对表情的准确识别。
基于图2的方法，本说明书实施例还提供了该方法的一些具体实施方案，下面进行说明。
可选地，步骤202，所述获取包含人脸图像的待识别图像之后，还可以包括：利用人脸对齐算法，对所述待识别图像进行人脸对齐处理。
人脸对齐算法为现有算法，即根据输入的人脸图像，自动定位出面部关键特征信息，如眼睛、鼻尖、嘴角、眉毛以及人脸各部件轮廓点等。将这些关键特征信息分别对准到一个预设的固定位置上。
不同的待识别图像上的人脸的角度可能不相同，有的是正对镜头的，有的没有正对 镜头。有的待识别图像的人脸部分没有位于图像的中央，无用的背景占据图像较大的空间。通过人脸对齐算法对待识别图像进行处理，可以将处于各种角度面对镜头的图像，调整为人脸正向面对镜头的图像；将人脸位于图像不同位置的图像调整为人脸均位于图像中间处的图像；达到数据归一化的目的。在利用对齐后的待识别图像进行特征提取时，可以提取到更为有利的图像特征，减少背景信息以及人脸角度不同造成的负面影响。
可选地，步骤S204，所述获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征，具体可以包括：利用第一预设残差网络模型对所述待识别图像进行特征提取处理，得到所述待识别图像对应的第一图像特征；利用第二预设残差网络模型对所述局部二值模式图像进行特征提取处理，得到所述局部二值模式图像对应的第二图像特征。
其中，第一预设残差网络模型和第二预设残差网络模型可以为同一个残差网络模型，也可以为相同结构的残差网络模型，也可以为输出特征维度相同的残差网络模型。
深度学习网络模型在达到一定深度后再一味地增加层数不能带来进一步的性能的提高，反而会招致网络收敛变慢，准确性也可能变差，也就是说深度学习网络模型可能会出现退化。采用残差网络模型，可以避免这一问题，只要训练数据足够，逐步加深残差网络模型，可以获得更好的性能表现。采用第一预设残差网络模型和第二预设残差网络模型分别对待识别图形和局部二值图像进行图像特征提取，可以提取出较高精度的图像特征。
可选地，所述第一预设残差网络模型和/或所述第二预设残差网络模型可以包括ResNet模型。
其中，所述第一预设残差网络模型和/或所述第二预设残差网络模型可以选取ResNet(Residual Neural Network，残差网络)模型，具体可以选取ResNet18模型、ResNet50模型或ResNet101模型等。实际应用中，还可以采用其他类型的残差网络模型，这里不作具体限定。
可选地，步骤208，所述将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征，具体可以包括：基于第一特征融合算法，将所述第一图像特征和所述第二图像特征进行特征融合，得到融合后的图像特征；所述融合后的图像特征的特征维度与所述第一图像特征和所述第二图像特征的特征维度相同。
第一特征融合算法可以包括特征图相加、通道数不变的add融合方式，将第一图像 特征与第二图像特征进行融合。假设第一图像特征含有128X128个像素点，第二图像特征同样含有128X128个像素点。第一图像特征的各个像素值分别与第二图像特征的与之相对应的像素值相加。如，第一图像特征的第一个像素点的像素值为5，第二图像特征的第一个像素点的值为6，融合后的图像特征的第一个像素点的值为11。融合后的图像特征的维度本身没有增加，只是每一维度下的信息量在增加。第一特征融合算法的计算量较小。
可选地，所述将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征，具体可以包括：基于第一特征融合算法，将所述第一图像特征和所述第二图像特征进行特征融合，得到第一融合图像特征；所述第一融合图像特征的特征维度与所述第一图像特征和所述第二图像特征的特征维度相同；利用局部注意力机制提取所述第一融合图像特征的局部特征；利用全局注意力机制提取所述第一融合图像特征的全局特征；基于第二特征融合算法，将所述局部特征和所述全局特征进行特征融合，得到第二融合图像特征。
也就是先利用第一特征融合算法将第一图像特征与第二图像特征进行融合得到第一融合图像特征。再利用局部注意力机制提取第一融合图像的局部特征，利用全局注意力机制提取第一融合图像的全局特征。再利用第二特征融合算法进行融合。第二特征融合算法可以包括通道数增加的concat融合方式，将局部特征和全局特征进行特征融合，得到第二融合图像特征。
上述方法，进行了两次特征融合。在进行第一次特征融合得到第一融合图像特征后，对第一融合图像特征分别提取全局特征和局部特征，再采用第二特征融合算法对全局特征和局部特征进行融合。使得获得的图像特征的信息更加丰富，从而使得后续的表情识别的准确性更高。
可选地，所述基于第二特征融合算法，将所述局部特征和所述全局特征进行特征融合之前，还可以包括：对所述全局特征和所述局部特征进行非线性增强处理，得到处理后的全局特征和处理后的局部特征；其中，所述将所述局部特征和所述全局特征进行特征融合，具体包括：将所述处理后的全局特征和所述处理后的局部特征进行特征融合。
其中，可将所述全局特征和所述局部特征经过激活函数层ReLU处理，进行非线性增强，再将经非线性增强后的全局特征和局部特征进行特征融合。
可选地，所述利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征 进行处理，具体可以包括：将所述待识别图像划分为预设数量的子图像；获取各个子图像的图像特征；将所述融合后的图像特征与所述各个子图像的图像特征，按照预设排列顺序进行排列，得到特征输入队列；将所述特征输入队列输入所述利用基于注意力机制搭建的表情识别模型。
其中，表情识别模型可以包括基于Transformer网络结构搭建的表情识别模型。
本说明书实施例中可以将待识别图像划分为预设数量的子图像，获取各个子图像的图像特征，图像特征可以为颜色特征。将融合后的图像特征与各个子图像的颜色特征按照预设排列顺序进行排列，得到特征输入队列，将特征输入队列输入表情识别模型，得到表情识别结果。
相应的，本说明书另一个实施例中，也可以将局部二值模式图像划分为预设数量的子图像，获取各个子图像的图像特征，图像特征可以为LBP特征。将融合后的图像特征与各个子图像的颜色特征按照预设排列顺序进行排列，得到特征输入队列，将特征输入队列输入表情识别模型，得到表情识别结果。
相应的，本说明书另一个实施例中，也可以将待识别图像和局部二值模式图像均划分为预设数量的子图像，获取待识别图像的各个子图像的颜色特征以及局部二值模式图像的各个子图像的LBP特征。将融合后的图像特征、各个子图像的颜色特征、各个子图像的LBP特征按照预设排列顺序进行排列，得到特征输入队列，将特征输入队列输入表情识别模型，得到表情识别结果。
可选地，所述表情识别模型可以包括位置编码器、Transformer编码器以及表情分类器；所述将所述特征输入队列输入所述利用基于注意力机制搭建的表情识别模型，具体包括：利用所述位置编码器，对所述特征输入队列进行位置编码，得到特征序列；将所述特征序列输入所述Transformer编码器，得到编码特征；将所述编码特征输入所述表情分类器，得到表情识别结果。
将特征输入队列输入表情识别模型后，位置编码器对各个图像特征进行位置编码，得到特征序列，将特征序列输入Transformer编码器进行编码，得到编码特征。Transformer编码器包含多个子编码器，每个子编码器的结构是一个自注意力机制加上一个前馈神经网络，在编码过程中，将融合后的图像特征和各个子图像的图像特征进行了自注意力特征增强，建立图像特征间的长程关系，也就是使得融合后的图像特征和各个子图像的图像特征之间的关联性更强。将编码特征经过一个全连接层输入到表情分类器，得到表情 识别结果。表情分类器可以包括表情7分类器。将表情分为七类，分别为高兴、伤心、害怕、愤怒、厌恶、惊讶和轻蔑。实际应用中，还可以采用其他类型的表情分类器，这里不作具体限定。
可选地，所述预设排列顺序可以是基于机器学习的方式确定的。
融合后的图像特征与各个子图像特征的排列顺序可以基于机器学习的方式确定，也就是基于前馈融合特征通过机器学习的方式进行输出，来输出更有利于表情识别的排列顺序。例如，可以根据表情的识别准确率来训练用于对子图像排序的机器模型，使得该机器模型可以提供更有利于表情识别的图像排序。
可选地，本说明书实施例中的待识别图像可以包括RGB图像、三维图像、NIR(Near Infrared，近红外光)图像中的至少一种。
例如当待识别图像为三维图像时，在颜色特征和LBP特征的基础上，再加上第三维度特征信息，通过多模态的方式进行端到端的学习方式，来进行表情识别，来进一步增加表情识别的鲁棒性和准确性。
图3为本说明书实施例提供的一种识别表情的方法的整体方案流程示意图。
如图3所示，本说明书实施例提供的识别表情的方法可以包括以下步骤。
步骤301：获取包含人脸特征的待识别图像；其中，待识别图像可以为RGB图像。
步骤303：基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像，也就是得到对应的LBP图像。
步骤305：利用第一预设残差网络模型对所述待识别图像进行特征提取处理，得到所述待识别图像对应的第一图像特征；其中，第一预设残差网络模型可采用ResNet18网络模型，第一图像特征可以为颜色特征。
步骤307：利用第二预设残差网络模型对所述局部二值模式图像进行特征提取处理，得到所述局部二值模式图像对应的第二图像特征；其中第二预设残差网络模型可采用ResNet18网络模型，第二图像特征可以为LBP特征。
步骤309：基于第一特征融合算法，将所述第一图像特征和所述第二图像特征进行特征融合，得到第一融合图像特征；所述第一融合图像特征的特征维度与所述第一图像特征和所述第二图像特征的特征维度相同。
步骤311：利用Local Attention(局部注意力机制)结构或模型提取所述第一融合图 像特征的局部特征；其中，局部注意力机制结构可以依次包括Conv Layer卷积层11，ReLU Layer非线性激活层13，Batch Nomalization批量归一化层14，Conv Layer卷积层11和Batch Nomalization批量归一化层14。
步骤313：利用Global Attention(全局注意力机制)结构或模型提取所述第一融合图像特征的全局特征；其中，全局注意力机制结构可以依次包括Average Pooling平均池化层12，Conv Layer卷积层11，ReLU Layer非线性激活层13，Batch Nomalization批量归一化层14，Conv Layer卷积层11和Batch Nomalization批量归一化层14。相比于local attention结构，global attention结构在最前端增加average pooling操作可以更好的提取全局的结构信息。
步骤315：对所述全局特征和所述局部特征进行非线性增强处理，得到处理后的全局特征和处理后的局部特征；其中可以通过ReLU函数对全局特征和局部特征进行非线性增强处理。
步骤317：将所述处理后的全局特征和所述处理后的局部特征进行特征融合，得到第二融合图像特征。
步骤319：将所述待识别图像划分为预设数量的子图像；获取各个子图像的图像特征；将所述融合后的图像特征与所述各个子图像特征，按照预设排列顺序进行排列，得到特征输入队列；其中，图中Class方块代表融合后的图像特征，其他方块代表各个子图像的图像特征。
实际应用中，可以通过Linear、Flat等处理方式将划分得到的各个子图像以及融合特征处理为预设维度的特征，来构成特征输入队列。
步骤321：利用position encoding(位置编码器)，对特征输入队列进行位置编码，得到特征序列；步骤323：将所述特征序列输入所述Transformer Encoder(Transformer编码器)，得到编码特征；Transformer是一种自注意力机制的模型。也就是将特征序列进行自注意力特征增强，以建立特征序列中的各图像特征间的长程关系。
步骤325：将所述编码特征经过一个全连接层后输入Classifier(表情分类器)，得到表情识别结果。
基于同样的思路，本说明书实施例还提供了上述方法对应的装置。图4为本说明书实施例提供的对应于图2的一种识别表情的装置示意图。如图4所示，该装置可以包括：图像获取模块401，用于获取包含人脸特征的待识别图像；第一图像处理模块403，用 于基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像；特征提取模块405，用于获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征；特征融合模块407，用于将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征；表情识别模块409，用于利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，得到针对所述待识别图像的表情识别结果。
基于图4的装置，本说明书实施例还提供了该方法的一些具体实施方案，下面进行说明。
可选地，所述装置还包括人脸对齐模块，用于利用人脸对齐算法，对所述待识别图像进行人脸对齐处理。
可选地，所述特征提取模块包括第一特征提取模块和第二提取模块；所述第一特征提取模块用于利用第一预设残差网络模型对所述待识别图像进行特征提取处理，得到所述待识别图像对应的第一图像特征；所述第二特征提取模块用于利用第二预设残差网络模型对所述局部二值模式图像进行特征提取处理，得到所述局部二值模式图像对应的第二图像特征。
可选地，所述特征融合模块包括第一特征融合模块和第二特征融合模块；所述第一特征融合模块用于基于第一特征融合算法，将所述第一图像特征和所述第二图像特征进行特征融合，得到融合后的图像特征；所述融合后的图像特征的特征维度与所述第一图像特征和所述第二图像特征的特征维度相同；所述第二特征融合模块用于利用局部注意力机制提取所述第一融合图像特征的局部特征；利用全局注意力机制提取所述第一融合图像特征的全局特征；基于第二特征融合算法，将所述局部特征和所述全局特征进行特征融合，得到第二融合图像特征。
基于同样的思路，本说明书实施例还提供了上述方法对应的设备。
图5为本说明书实施例提供的对应于图2的一种识别表情的设备示意图。如图5所示，设备500可以包括至少一个处理器510以及与所述至少一个处理器通信连接的存储器530。其中，所述存储器530存储有可被所述至少一个处理器510执行的指令520，所述指令被所述至少一个处理器510执行，以使所述至少一个处理器510能够：获取包含人脸特征的待识别图像；基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像；获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的 第二图像特征；将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征；利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，得到针对所述待识别图像的表情识别结果。
基于同样的思路，本说明书实施例还提供了上述方法对应的计算机可读介质。计算机可读介质上存储有计算机可读指令，所述计算机可读指令可被处理器执行以实现上述识别表情的方法。
本说明书中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于图5所示的设备而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
在20世纪90年代，对于一个技术的改进可以很明显地区分是硬件上的改进(例如，对二极管、晶体管、开关等电路结构的改进)还是软件上的改进(对于方法流程的改进)。然而，随着技术的发展，当今的很多方法流程的改进已经可以视为硬件电路结构的直接改进。设计人员几乎都通过将改进的方法流程编程到硬件电路中来得到相应的硬件电路结构。因此，不能说一个方法流程的改进就不能用硬件实体模块来实现。例如，可编程逻辑器件(Programmable Logic Device,PLD)(例如现场可编程门阵列(Field Programmable Gate Array，FPGA))就是这样一种集成电路，其逻辑功能由用户对器件编程来确定。由设计人员自行编程来把一个数字符系统“集成”在一片PLD上，而不需要请芯片制造厂商来设计和制作专用的集成电路芯片。而且，如今，取代手工地制作集成电路芯片，这种编程也多半改用“逻辑编译器(logic compiler)”软件来实现，它与程序开发撰写时所用的软件编译器相类似，而要编译之前的原始代码也得用特定的编程语言来撰写，此称之为硬件描述语言(Hardware Description Language，HDL)，而HDL也并非仅有一种，而是有许多种，如ABEL(Advanced Boolean Expression Language)、AHDL(Altera Hardware Description Language)、Confluence、CUPL(Cornell University Programming Language)、HDCal、JHDL(Java Hardware Description Language)、Lava、Lola、MyHDL、PALASM、RHDL(Ruby Hardware Description Language)等，目前最普遍使用的是VHDL(Very-High-Speed Integrated Circuit Hardware Description Language)与Verilog。本领域技术人员也应该清楚，只需要将方法流程用上述几种硬件描述语言稍作逻辑编程并编程到集成电路中，就可以很容易得到实现该逻辑方法流程的硬件电路。
控制器可以按任何适当的方式实现，例如，控制器可以采取例如微处理器或处理器 以及存储可由该(微)处理器执行的计算机可读程序代码(例如软件或固件)的计算机可读介质、逻辑门、开关、专用集成电路(Application Specific Integrated Circuit，ASIC)、可编程逻辑控制器和嵌入微控制器的形式，控制器的例子包括但不限于以下微控制器：ARC 625D、Atmel AT91SAM、Microchip PIC18F26K20以及Silicone Labs C8051F320，存储器控制器还可以被实现为存储器的控制逻辑的一部分。本领域技术人员也知道，除了以纯计算机可读程序代码方式实现控制器以外，完全可以通过将方法步骤进行逻辑编程来使得控制器以逻辑门、开关、专用集成电路、可编程逻辑控制器和嵌入微控制器等的形式来实现相同功能。因此这种控制器可以被认为是一种硬件部件，而对其内包括的用于实现各种功能的装置也可以视为硬件部件内的结构。或者甚至，可以将用于实现各种功能的装置视为既可以是实现方法的软件模块又可以是硬件部件内的结构。
上述实施例阐明的系统、装置、模块或单元，具体可以由计算机芯片或实体实现，或者由具有某种功能的产品来实现。一种典型的实现设备为计算机。具体的，计算机例如可以为个人计算机、膝上型计算机、蜂窝电话、相机电话、智能电话、个人数字符助理、媒体播放器、导航设备、电子邮件设备、游戏控制台、平板计算机、可穿戴设备或者这些设备中的任何设备的组合。
为了描述的方便，描述以上装置时以功能分为各种单元分别描述。当然，在实施本申请时可以把各单元的功能在同一个或多个软件和/或硬件中实现。
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方 式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
在一个典型的配置中，计算设备包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。
内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。
计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字符多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带式磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
还需要说明的是，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、商品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、商品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、商品或者设备中还存在另外的相同要素。
本领域技术人员应明白，本申请的实施例可提供为方法、系统或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可 用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本申请可以在由计算机执行的计算机可执行指令的一般上下文中描述，例如程序模块。一般地，程序模块包括执行特定任务或实现特定抽象数据类型的例程、程序、对象、组件、数据结构等等。也可以在分布式计算环境中实践本申请，在这些分布式计算环境中，由通过通信网络而被连接的远程处理设备来执行任务。在分布式计算环境中，程序模块可以位于包括存储设备在内的本地和远程计算机存储介质中。
以上所述仅为本申请的实施例而已，并不用于限制本申请。对于本领域技术人员来说，本申请可以有各种更改和变化。凡在本申请的精神和原理之内所作的任何修改、等同替换、改进等，均应包含在本申请的权利要求范围之内。

Claims (15)

  1. 一种识别表情的方法，包括：
    获取包含人脸特征的待识别图像；
    基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像；
    获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征；
    将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征；
    利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，得到针对所述待识别图像的表情识别结果。
  2. 根据权利要求1所述的方法，所述获取包含人脸图像的待识别图像之后，还包括：
    利用人脸对齐算法，对所述待识别图像进行人脸对齐处理。
  3. 根据权利要求1所述的方法，所述获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征，具体包括：
    利用第一预设残差网络模型对所述待识别图像进行特征提取处理，得到所述待识别图像对应的第一图像特征；
    利用第二预设残差网络模型对所述局部二值模式图像进行特征提取处理，得到所述局部二值模式图像对应的第二图像特征。
  4. 根据权利要求3所述的方法，所述第一预设残差网络模型和/或所述第二预设残差网络模型包括ResNet模型。
  5. 根据权利要求1所述的方法，所述将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征，具体包括：
    基于第一特征融合算法，将所述第一图像特征和所述第二图像特征进行特征融合，得到融合后的图像特征；所述融合后的图像特征的特征维度与所述第一图像特征和所述第二图像特征的特征维度相同。
  6. 根据权利要求1所述的方法，所述将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征，具体包括：
    基于第一特征融合算法，将所述第一图像特征和所述第二图像特征进行特征融合，得到第一融合图像特征；所述第一融合图像特征的特征维度与所述第一图像特征和所述第二图像特征的特征维度相同；
    利用局部注意力机制提取所述第一融合图像特征的局部特征；
    利用全局注意力机制提取所述第一融合图像特征的全局特征；
    基于第二特征融合算法，将所述局部特征和所述全局特征进行特征融合，得到第二融合图像特征。
  7. 根据权利要求6所述的方法，所述基于第二特征融合算法，将所述局部特征和所述全局特征进行特征融合之前，还包括：
    对所述全局特征和所述局部特征进行非线性增强处理，得到处理后的全局特征和处理后的局部特征；
    所述将所述局部特征和所述全局特征进行特征融合，具体包括：
    将所述处理后的全局特征和所述处理后的局部特征进行特征融合。
  8. 根据权利要求1所述的方法，所述利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，具体包括：
    将所述待识别图像划分为预设数量的子图像；
    获取各个子图像的图像特征；
    将所述融合后的图像特征与所述各个子图像的图像特征，按照预设排列顺序进行排列，得到特征输入队列；
    将所述特征输入队列输入所述利用基于注意力机制搭建的表情识别模型。
  9. 根据权利要求8所述的方法，所述表情识别模型包括位置编码器、Transformer编码器以及表情分类器；
    所述将所述特征输入队列输入所述利用基于注意力机制搭建的表情识别模型，具体包括：
    利用所述位置编码器，对所述特征输入队列进行位置编码，得到特征序列；
    将所述特征序列输入所述Transformer编码器，得到编码特征；
    将所述编码特征输入所述表情分类器，得到表情识别结果。
  10. 根据权利要求8所述的方法，所述预设排列顺序是基于机器学习的方式确定的。
  11. 根据权利要求1所述的方法，所述待识别图像包括RGB图像、三维图像、近红外图像中至少一种。
  12. 根据权利要求1所述的方法，所述待识别图像为包括至少部分人脸面部特征的图像。
  13. 一种识别表情的装置，包括：
    图像获取模块，用于获取包含人脸特征的待识别图像；
    第一图像处理模块，用于基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像；
    特征提取模块，用于获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征；
    特征融合模块，用于将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征；
    表情识别模块，用于利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，得到针对所述待识别图像的表情识别结果。
  14. 一种识别表情的设备，包括：
    至少一个处理器；以及，
    与所述至少一个处理器通信连接的存储器；其中，
    所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够：
    获取包含人脸特征的待识别图像；
    基于局部二值模式算法，得到所述待识别图像对应的局部二值模式图像；
    获取所述待识别图像对应的第一图像特征和所述局部二值模式图像对应的第二图像特征；
    将所述第一图像特征和所述第二图像特征进行融合，得到融合后的图像特征；
    利用基于注意力机制搭建的表情识别模型，对所述融合后的图像特征进行处理，得到针对所述待识别图像的表情识别结果。
  15. 一种计算机可读介质，其上存储有计算机可读指令，所述计算机可读指令可被处理器执行以实现权利要求1至12中任一项所述的识别表情的方法。
PCT/CN2023/114307 2022-09-20 2023-08-22 识别表情的方法、装置、设备及介质 Ceased WO2024060909A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211149128.5A CN115439912A (zh) 2022-09-20 2022-09-20 一种识别表情的方法、装置、设备及介质
CN202211149128.5 2022-09-20

Publications (1)

Publication Number Publication Date
WO2024060909A1 true WO2024060909A1 (zh) 2024-03-28

Family

ID=84248655

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/114307 Ceased WO2024060909A1 (zh) 2022-09-20 2023-08-22 识别表情的方法、装置、设备及介质

Country Status (2)

Country Link
CN (1) CN115439912A (zh)
WO (1) WO2024060909A1 (zh)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115439912A (zh) * 2022-09-20 2022-12-06 支付宝(杭州)信息技术有限公司 一种识别表情的方法、装置、设备及介质
CN116259086A (zh) * 2022-12-09 2023-06-13 支付宝(杭州)信息技术有限公司 活体检测模型训练方法、装置、存储介质及设备
CN117197856A (zh) * 2022-12-26 2023-12-08 南昌工学院 一种融合自注意力机制的学习者在线兴趣检测方法
CN116912908A (zh) * 2023-06-20 2023-10-20 支付宝(杭州)信息技术有限公司 一种表情识别方法及装置

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112257647A (zh) * 2020-11-03 2021-01-22 徐州工程学院 基于注意力机制的人脸表情识别方法
CN112329683A (zh) * 2020-11-16 2021-02-05 常州大学 一种基于注意力机制融合的多通道卷积神经网络人脸表情识别方法
CN112836589A (zh) * 2021-01-13 2021-05-25 苏州元启创人工智能科技有限公司 基于特征融合的视频中人脸表情识别方法
CN113392766A (zh) * 2021-06-16 2021-09-14 哈尔滨理工大学 一种基于注意力机制的人脸表情识别方法
WO2021184902A1 (zh) * 2020-03-19 2021-09-23 京东方科技集团股份有限公司 图像分类方法、装置、及其训练方法、装置、设备、介质
CN113887487A (zh) * 2021-10-20 2022-01-04 河海大学 一种基于CNN-Transformer的面部表情识别方法及装置
CN113963422A (zh) * 2021-11-22 2022-01-21 上海电力大学 基于多注意力融合Transformer架构的不确定性人脸表情识别方法
JP2022063236A (ja) * 2020-10-09 2022-04-21 株式会社Ｎｔｔドコモ 画像処理装置、方法、機器及びコンピュータ読み取り可能な記憶媒体
CN115439912A (zh) * 2022-09-20 2022-12-06 支付宝(杭州)信息技术有限公司 一种识别表情的方法、装置、设备及介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3698268A4 (en) * 2017-11-22 2021-02-17 Zhejiang Dahua Technology Co., Ltd. Methods and systems for face recognition
CN108304789A (zh) * 2017-12-12 2018-07-20 北京深醒科技有限公司 脸部识别方法及装置
CN112784764B (zh) * 2021-01-27 2022-07-12 南京邮电大学 一种基于局部与全局注意力机制的表情识别方法及系统
CN113486981B (zh) * 2021-07-30 2023-02-07 西安电子科技大学 基于多尺度特征注意力融合网络的rgb图像分类方法
CN114155572B (zh) * 2021-11-04 2025-01-28 华中师范大学 一种面部表情识别方法和系统
CN114582004A (zh) * 2022-04-28 2022-06-03 中国科学技术大学 人脸表情识别方法、系统、设备及存储介质

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021184902A1 (zh) * 2020-03-19 2021-09-23 京东方科技集团股份有限公司 图像分类方法、装置、及其训练方法、装置、设备、介质
JP2022063236A (ja) * 2020-10-09 2022-04-21 株式会社Ｎｔｔドコモ 画像処理装置、方法、機器及びコンピュータ読み取り可能な記憶媒体
CN112257647A (zh) * 2020-11-03 2021-01-22 徐州工程学院 基于注意力机制的人脸表情识别方法
CN112329683A (zh) * 2020-11-16 2021-02-05 常州大学 一种基于注意力机制融合的多通道卷积神经网络人脸表情识别方法
CN112836589A (zh) * 2021-01-13 2021-05-25 苏州元启创人工智能科技有限公司 基于特征融合的视频中人脸表情识别方法
CN113392766A (zh) * 2021-06-16 2021-09-14 哈尔滨理工大学 一种基于注意力机制的人脸表情识别方法
CN113887487A (zh) * 2021-10-20 2022-01-04 河海大学 一种基于CNN-Transformer的面部表情识别方法及装置
CN113963422A (zh) * 2021-11-22 2022-01-21 上海电力大学 基于多注意力融合Transformer架构的不确定性人脸表情识别方法
CN115439912A (zh) * 2022-09-20 2022-12-06 支付宝(杭州)信息技术有限公司 一种识别表情的方法、装置、设备及介质

Also Published As

Publication number Publication date
CN115439912A (zh) 2022-12-06

Similar Documents

Publication Publication Date Title
WO2024060909A1 (zh) 识别表情的方法、装置、设备及介质
CN109522818B (zh) 一种表情识别的方法、装置、终端设备及存储介质
Abbasnejad et al. Using synthetic data to improve facial expression analysis with 3d convolutional networks
Haq et al. Enhanced real-time facial expression recognition using deep learning
CN111160350B (zh) 人像分割方法、模型训练方法、装置、介质及电子设备
Rajan et al. Facial expression recognition techniques: a comprehensive survey
US12080098B2 (en) Method and device for training multi-task recognition model and computer-readable storage medium
CN116259075A (zh) 基于提示微调预训练大模型的行人属性识别方法
CN111144348A (zh) 图像处理方法、装置、电子设备及存储介质
WO2020103700A1 (zh) 一种基于微表情的图像识别方法、装置以及相关设备
Raut Facial emotion recognition using machine learning
US20200005086A1 (en) Deep learning-based automatic gesture recognition method and system
CN111680550A (zh) 情感信息识别方法、装置、存储介质及计算机设备
CN111460782B (zh) 一种信息处理方法、装置及设备
Zerrouki et al. Deep learning for hand gesture recognition in virtual museum using wearable vision sensors
Abiram et al. Identity preserving multi-pose facial expression recognition using fine tuned VGG on the latent space vector of generative adversarial network
CN114973383A (zh) 一种微表情识别方法、装置、电子设备及存储介质
CN116704554A (zh) 一种基于深度学习的手姿估计与识别的方法、设备及介质
Chanthaphan et al. Facial emotion recognition based on facial motion stream generated by kinect
Maheswari et al. Techniques, promising directions, challenges, datasets, and representations of facial expression analysis
Olivares-Mercado et al. Analysis of hand-crafted and learned feature extraction methods for real-time facial expression recognition
CN112183155B (zh) 动作姿态库建立、动作姿态生成、识别方法及装置
CN115830633A (zh) 基于多任务学习残差神经网络的行人重识别方法和系统
Sanou et al. An extensible deep architecture for action recognition problem
Verma et al. A single-point control system for consumer devices using edge-fog computing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23867202

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23867202

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 21.08.2025)