CN112183419B - Micro-expression classification method based on optical flow generation network and reordering - Google Patents

Micro-expression classification method based on optical flow generation network and reordering Download PDF

Info

Publication number
CN112183419B
CN112183419B CN202011070119.8A CN202011070119A CN112183419B CN 112183419 B CN112183419 B CN 112183419B CN 202011070119 A CN202011070119 A CN 202011070119A CN 112183419 B CN112183419 B CN 112183419B
Authority
CN
China
Prior art keywords
feature
optical flow
frame
image
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011070119.8A
Other languages
Chinese (zh)
Other versions
CN112183419A (en
Inventor
柯逍
林艳
王俊强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fuzhou University
Original Assignee
Fuzhou University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fuzhou University filed Critical Fuzhou University
Priority to CN202011070119.8A priority Critical patent/CN112183419B/en
Publication of CN112183419A publication Critical patent/CN112183419A/en
Application granted granted Critical
Publication of CN112183419B publication Critical patent/CN112183419B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to a micro-expression classification method based on optical flow generation network and reordering. Firstly, acquiring a micro expression data set, extracting a start frame and a peak frame, and preprocessing; training an optical flow generation network, and generating optical flow characteristics according to all initial frames and peak frames; then, the obtained optical flow image is divided into a corresponding training set and a corresponding testing set according to an LOSO principle, and a residual error network is input for training; and finally, reordering results obtained by preliminary classification of the residual error network to obtain a final result with higher precision.

Description

Micro-expression classification method based on optical flow generation network and reordering
Technical Field
The invention relates to the field of pattern recognition and computer vision, in particular to a micro-expression classification method based on optical flow generation network and reordering.
Background
In the field of emotion calculation, facial expressions are often researched to judge the emotion of a human at the moment, but the human body is the highest animal and sometimes disguises or hides the emotion of the human body, and in such a case, people cannot acquire useful information from the macroscopic expression of the face. To extract useful information from camouflaged facial expressions, ekman discovered a transient, involuntary, rapid facial emotion, i.e., a micro-expression, which is provoked to involuntarily appear in the face when one tries to hide some real emotion. A standard micro-expression lasts 1/5 to 1/25 seconds and usually appears only in a specific part of the face.
Micro-expressions have great prospects in the aspects of national security, criminal inquiries and medical applications, but the subtlety and conciseness of micro-expressions form a great challenge for human eyes, so that in recent years, people put forward a lot of work of realizing automatic identification of micro-expressions by using computer vision and machine learning algorithms.
Disclosure of Invention
The invention aims to provide a micro-expression classification method based on an optical flow generation network and reordering, which can effectively classify micro-expression images.
In order to achieve the purpose, the technical scheme of the invention is as follows: a micro-expression classification method based on optical flow generation network and reordering comprises the following steps:
s1, acquiring a micro expression data set, extracting a start frame and a peak frame, and preprocessing;
step S2, training an optical flow generation network, and generating optical flow characteristics according to all initial frames and peak frames;
step S3, dividing the obtained optical flow image into a training set and a test set according to an LOSO principle, and inputting a residual error network for training;
and step S4, reordering the classification results obtained by the residual error network to obtain a final result with higher precision.
In an embodiment of the present invention, the step S1 specifically includes the following steps:
step S11, acquiring a micro expression data set, and cutting the image into 224 × 224 images after face alignment;
step S12, extracting the initial frame and the peak frame directly according to the marked content for the micro expression data set with the initial frame and the peak frame mark, and executing step S15;
step S13, extracting the initial frame and the peak frame of the video sequence by using a frame difference method for the micro expression data set which is not marked with the initial frame and the peak frame; the frame difference method comprises the following steps: let P be { P ═ PiDenotes an input image sequence, where p is 1,2iRepresenting the ith input picture with the first frame of the sequence as the start frame, i.e. pstart＝p1The gray values of pixels corresponding to the first frame and the nth frame of the video sequence are recorded as f1(x, y) and fn (x, y), the gray values of the pixels corresponding to the two frames of images are subtracted, the absolute value of the subtraction is taken to obtain a difference image Dn, Dn (x, y) ═ fn (x, y) -f1(x, y) |, and the average inter-frame difference Dnavg of the difference image is calculated, wherein the calculation method comprises the following steps:
Figure BDA0002714591380000021
wherein, Dn.shape [0 ]]Representing the height of the differential image Dn, Dn]The width of the difference image Dn is indicated. Calculating the average interframe difference of all frames except the initial frame and sequencing, wherein the frame with the maximum average interframe difference is the peak value frame p corresponding to the image sequenceapex(ii) a After extracting the start frame and the peak frame, executing step S15;
step S15, performing euler action amplification on the extracted start frame and peak frame, wherein the calculation process is as follows:
I(x,t)＝g(x+(1+α)δ(t))
where I (x, t) represents the brightness value of the image at the location x and time t, g (-) represents the mapping function of euler's motion magnification, and δ (t) represents the motion deviation, and the magnified image is generated by adjusting the motion magnification factor α.
In an embodiment of the present invention, in step S2, the optical flow generation network performs point-to-point pixel training by using a structure of two sub-networks, the two sub-networks are ranked with respect to each other, one sub-network performs optical flow estimation on large displacement, the other sub-network performs optical flow estimation on small displacement, each sub-network is composed of a feature extraction module and an optical flow estimation module, and finally, the optical flow estimates obtained by the two sub-networks are fused to obtain a final optical flow estimated image;
s21, for the sub-network of large displacement optical flow estimation, the feature extraction module is composed of nine convolution layers, the input of the feature extraction module is the superposition of the amplified input image pair, the feature mapping function of the feature extraction module is H (·), and the calculation process is as follows:
featurebig＝H(pls+pla)
wherein p islsIndicating the starting frame after enlargement, plaShowing the enlarged peak frame, featurebigRepresenting the result of motion feature extraction for large displacement;
the optical flow estimation module of the large displacement optical flow estimation sub-network consists of an upper pooling layer and a convolution layer, and the feature value feature obtained by the output of the feature extraction modulebigAnd feature at layer 5-1 of the feature extraction module, i.e., layer 4 of the feature extraction modulebig4And superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain the calculation result of the 1 st layer of the optical flow estimation module, wherein the calculation process is as follows:
featureBflow1＝estimate(featurebig+featurebig4)
wherein featureBflow1Representing the characteristics output by the layer 1 of the optical flow estimation module under the large displacement sub-network, and estimate (DEG) representing the mapping function of the layer 1 of the optical flow estimation module under the large displacement sub-network;
then, for the remaining 2-4 layers of optical flow estimation modules, the result calculated by the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureBflowi＝estimate(featurebig+featurebig(5-i)+featureBflow(i-1))
wherein featureBflow(i-1)The characteristics represent the output of the i-1 layer of the optical flow estimation module under the large displacement sub-network;
s22, for the small displacement optical flow estimation sub-network, the feature extraction module is composed of nine convolution layers, the first three convolutions are respectively to the input initial frame p without action amplificationsSum peak frame paRespectively extracting features, wherein the input of the last six convolutions is superposition of the output results of the first three convolutions on two image frames, the mapping function of the first three convolutions of the feature extraction module is first (-) and the mapping function of the last convolution is last (-), and the calculation process is as follows:
featuresmall＝last(first(ps)+first(pa))
wherein featuresmallRepresenting the result of motion feature extraction for small displacements;
the optical flow estimation module of the small displacement optical flow estimation sub-network consists of an upper pooling layer and a convolution layer, and the feature value feature obtained by the output of the feature extraction modulesmallAnd feature at layer 6-1 of the feature extraction module, i.e., layer 5 of the feature extraction modulesmall5And superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain a calculation result of a first layer of the optical flow estimation module, wherein the calculation process is as follows:
featureSflow1＝estimate(featuresmall+featuresmall5)
wherein featureSflow1The method comprises the steps of representing characteristics output by a layer 1 of an optical flow estimation module under a small displacement sub-network, and estimate (DEG) represents a mapping function of the layer 1 of the optical flow estimation module under the small displacement sub-network;
then, for the remaining 2-5 layers of optical flow estimation modules, the result calculated by the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureSflowi＝estimate(featuresmall+featuresmall(6-i)+featureSflow(i-1))
wherein featureSflow(i-1)The characteristics of the output of the i-1 layer of the optical flow estimation module under the small displacement sub-network are represented;
s23, fusing the results obtained by the large displacement optical flow estimation sub-network and the small displacement optical flow estimation sub-network to obtain the final output result, and making fusion (-) represent the final fusion operation, wherein the calculation process is as follows:
pfusion＝fusion(featureBflow4+featureSflow5)。
in an embodiment of the present invention, the step S3 specifically includes the following steps:
step S31, each optical flow image data set has a plurality of subjects, each subject represents a testee, each subject has a plurality of micro-expression sequences, which represent the micro-expression sequences generated by the testee, one subject of one data set is taken as a test set when the data sets are divided according to the principle of leave-one-subject-out, all the other subjects are merged together as a training set, the last data set obtains a subjectiA training set and a test set, wherein SubiRepresenting the number of subjects in a data set;
and step S32, inputting the divided test set and training set into a residual error network in sequence for classification to obtain a primary classification result.
In an embodiment of the present invention, the step S4 specifically includes the following steps:
step S41, for the preliminary classification result obtained by the residual error network training, there may be a case where the probabilities of two results of the same graph are very similar, so that the results need to be reordered;
step S42, selecting images under corresponding classification in the training set according to the classification results with similar classification probability of the tested images, selecting k neighbors according to the selected images, and calculating as follows:
Figure BDA0002714591380000044
Figure BDA0002714591380000041
wherein eiRepresenting the ith selected training set image, and p represents the tested image;
step S43, calculating tested image p and selected image eiThe calculation process of (a) is as follows:
Di＝1-probemax(ei)+probemax(p)
wherein probemaxRepresenting the maximum probability among the probabilities of the classification results;
step S44: for each selected training set image eiCalculating Jaccard distance D from the tested image pjThe calculation process is as follows:
Figure BDA0002714591380000042
Figure BDA0002714591380000043
weighting DiAnd DjObtaining a final distance result;
the micro-expression images are reclassified in a reordering mode, so that the situation of wrong classification caused by too close two types of probabilities possibly occurring in the micro-expression recognition process is reduced, and the accuracy of micro-expression recognition is improved.
Compared with the prior art, the invention has the following beneficial effects:
1. the constructed optical flow generation network and the reordered micro-expression classification method can effectively classify the micro-expression images and improve the classification effect of the micro-expression images.
2. The method generates the result of the optical flow estimation between two frames in a neural network mode, and has more robust effect, better performance and clearer boundary compared with the traditional optical flow generation method.
3. Aiming at the problem that two types of expression are difficult to distinguish in the conventional micro expression recognition process, the invention uses the method for reordering the test images with similar probability of classification results, better classifies the single micro expression and improves the classification effect.
Drawings
Fig. 1 is a schematic diagram of the principle of the present invention.
Detailed Description
The technical scheme of the invention is specifically explained below with reference to the accompanying drawings.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
As shown in fig. 1, the present embodiment provides a micro-representation classification method based on optical flow generation network and reordering, which specifically includes the following steps:
step S1: acquiring a micro expression data set, extracting a start frame and a peak frame, and preprocessing;
step S2: training an optical flow generation network, and generating optical flow characteristics according to all initial frames and peak frames;
step S3: dividing the obtained optical flow image into a training set and a testing set according to an LOSO principle, and inputting a residual error network for training;
step S4: and finally, reordering the classification results obtained by the residual error network to obtain a final result with higher precision.
In this embodiment, the step S1 includes the following steps:
step S11: acquiring a micro expression data set, and cutting an image into 224 × 224 images after face alignment;
step S12: for a micro expression data set with initial frame and peak frame labels, directly extracting the initial frame and the peak frame according to the label content;
step S13: extracting the initial frame and the peak frame of the video sequence by using a frame difference method for the micro expression data set which is not marked by the initial frame and the peak frame;
step S14: the specific content of the frame difference method is that P is { P ═ PiDenotes an input image sequence, where p is 1,2iRepresenting the ith input picture, let us start the first frame of the sequence, i.e. pstart＝p1The gray values of pixels corresponding to the first frame and the nth frame of the video sequence are recorded as f1(x, y) and fn (x, y), the gray values of the pixels corresponding to the two frames of images are subtracted, the absolute value of the difference is taken to obtain a difference image Dn, and Dn (x, y) ═ fn (x, y) -f1(x, y) |, the average inter-frame difference Dnavg of the difference image is calculated, and the calculating method is as follows:
Figure BDA0002714591380000061
wherein, Dn.shape [0 ]]Shape [1 ] represents the height of the difference image Dn]Indicating the width of the differential image Dn. Calculating the average interframe difference of all frames except the initial frame and sequencing, wherein the frame with the maximum average interframe difference is the peak value frame p corresponding to the image sequenceapex；
Step S15: and carrying out Euler action amplification on the processed initial frame and the processed peak frame, wherein the calculation process is as follows:
I(x,t)＝g(x+(1+α)δ(t))
wherein I (x, t) represents the brightness value of the image at position x and time t, g (-) represents the mapping function of Euler motion amplification process, and δ (t) represents the motion deviation, the method adjusts the motion amplification coefficient
Figure BDA0002714591380000062
To generate amplifiedAnd (4) an image.
In this embodiment, step S2 specifically includes the following steps:
step S21: the optical flow generation network carries out point-to-point pixel training by adopting a structure of two sub-networks, the two sub-networks are leveled with each other, one sub-network is specially used for carrying out optical flow estimation on large displacement, the other sub-network is specially used for carrying out optical flow estimation on small displacement, each sub-network is composed of a feature extraction module and an optical flow estimation module, and finally the optical flow estimation obtained by the two sub-networks are fused to obtain a final optical flow estimation image;
step S22: for a sub-network specially used for carrying out optical flow estimation on large displacement, a feature extraction module of the sub-network mainly comprises nine convolution layers, the input of the module is superposition of an amplified input image pair, a feature mapping function of the module is made to be H (·), and the calculation process is as follows:
featurebig＝H(pls+pla)
wherein p islsIndicating the starting frame after enlargement, plaShowing the enlarged peak frame, featurebigRepresenting the result of feature extraction for large displacement motion.
The optical flow estimation module of the large-displacement optical flow estimation sub-network mainly comprises an upper pooling layer and a convolution layer, and a feature value feature output by the upper layer modulebigAnd feature of upper module layer 5-1, layer 4big4And (3) superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain a calculation result of a first layer, wherein the calculation process is as follows:
featureBflow1＝estimate(featurebig+featurebig4)
wherein featureBflow1Representing the characteristics output by the layer 1 of the optical flow estimation module under the large displacement sub-network, and the estimate (DEG) represents the mapping function of the layer;
then, for the remaining 2-4 layers, the calculation result of the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureBflowi＝estimate(featurebig+featurebig(5-i)+featureBflow(i-1))
wherein featureBflow(i-1)The characteristics represent the output of the i-1 layer of the optical flow estimation module under the large displacement sub-network;
step S23: for the sub-network specially used for carrying out optical flow estimation on small displacement, the feature extraction module mainly comprises nine convolution layers, wherein the first three convolutions are respectively used for the input initial frame p which is not amplified in actionsSum peak frame paAnd respectively extracting features, wherein the input of the last six convolutions is the superposition of the first three convolutions on the output structures of the two image frames, the mapping function of the first three convolutions of the module is first (·), the mapping function of the last six convolutions is last (·), and the calculation process is as follows:
featuresmall＝last(first(ps)+first(pa))
wherein featuresmallRepresenting the result of feature extraction for small displacement motion.
The optical flow estimation module of the small displacement optical flow estimation sub-network mainly comprises an upper pooling layer and a convolution layer, and a feature value feature output by the upper layer modulesmallAnd feature of layer 6-1, layer 5 of the upper modulesmall5And (3) superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain a calculation result of a first layer, wherein the calculation process is as follows:
featureSflow1＝estimate(featuresmall+featuresmall5)
wherein featureSflow1Representing the characteristics output by the layer 1 of the optical flow estimation module under the small displacement sub-network, and the estimate (DEG) represents the mapping function of the layer;
then, for the remaining 2-5 layers, the calculation result of the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureSflowi＝estimate(featuresmall+featuresmall(6-i)+featureSflow(i-1))
wherein featureSflow(i-1)The characteristics represent the output of the i-1 layer of the optical flow estimation module under the small displacement sub-network;
step S24: and finally, fusing the results obtained by the two sub-networks to obtain a final output result, and enabling fusion (-) to represent the final fusion operation, wherein the calculation process is as follows:
pfusion＝fusion(featureBflow4+featureSflow5)
the convolution neural network is used for simulating and fusing optical flow estimation results of large displacement and small displacement, the generalization of the model is facilitated to be improved, the micro expression segments with overlarge or undersize micro expression changes can be adjusted more reasonably, and compared with the traditional method, the implementation mode of the convolution neural network reduces the edge blurring problem possibly generated in the micro expression optical flow estimation process, and the optical flow estimation result is more accurate.
In this embodiment, step S3 specifically includes the following steps:
step S31: each data set is provided with a plurality of subjects, each subject represents a testee, each subject comprises a plurality of micro-expression sequences which represent the micro-expression sequences generated by the testee, and according to the principle of leave-one-subject-out, one subject of one data set is taken as a test set at one time when the data sets are divided, and all the rest subjects are combined together to be used as a training set, so that finally, the subjects can be obtained for one data setiA training set and a test set, wherein SubiRepresenting the number of subjects in a data set.
Step S32: inputting the divided test set and training set into a residual error network in sequence for classification to obtain a primary classification result;
in this embodiment, step S4 specifically includes the following steps:
step S41: for the results of the preliminary classification, there may be a case that the probability of some two results of the same graph is very similar, which requires the results to be reordered;
step S42: selecting images under corresponding classification in a training set according to classification results with similar classification probabilities of tested images, and selecting k neighbors according to the selected images, wherein the calculation process is as follows:
Figure BDA0002714591380000082
Figure BDA0002714591380000081
wherein eiRepresenting the ith selected training set image and p representing the image under test.
Step S43: calculating the tested image p and the selected image eiThe calculation process of (a) is as follows:
Di＝1-probemax(ei)+probemax(p)
wherein probemaxRepresenting the highest probability among the probabilities of the classification results.
Step S44: for each selected training set image eiCalculating the Jaccard distance D between the measured image p and the measured imagejThe calculation process is as follows:
Figure BDA0002714591380000091
Figure BDA0002714591380000092
weighting DiAnd DjI.e. to obtain the final distance result.
The micro-expression images are reclassified in a reordering mode, so that the situation of wrong classification caused by the fact that certain two types of probabilities are too close to each other in the micro-expression recognition process is greatly reduced, and the accuracy of micro-expression recognition is improved.
The above are preferred embodiments of the present invention, and all changes made according to the technical scheme of the present invention that produce functional effects do not exceed the scope of the technical scheme of the present invention belong to the protection scope of the present invention.

Claims (3)

1. A micro-expression classification method based on optical flow generation network and reordering is characterized by comprising the following steps:
s1, acquiring a micro expression data set, extracting a start frame and a peak frame, and preprocessing;
step S2, training an optical flow generation network, and generating optical flow characteristics according to all initial frames and peak frames;
step S3, dividing the obtained optical flow image into a training set and a test set according to an LOSO principle, and inputting a residual error network for training;
step S4, reordering the classification results obtained by the residual error network to obtain a final result with higher precision;
in step S2, the optical flow generation network performs point-to-point pixel training by using a structure of two sub-networks, the two sub-networks are ranked with respect to each other, one sub-network performs optical flow estimation on large displacement, the other sub-network performs optical flow estimation on small displacement, each sub-network is composed of a feature extraction module and an optical flow estimation module, and finally the optical flow estimates obtained by the two sub-networks are fused to obtain a final optical flow estimation image;
s21, for the sub-network of large displacement optical flow estimation, the feature extraction module is composed of nine convolution layers, the input of the feature extraction module is the superposition of the amplified input image pair, the feature mapping function of the feature extraction module is H (·), and the calculation process is as follows:
featurebig＝H(pls+pla)
wherein p islsIndicating the starting frame after enlargement, plaShowing the enlarged peak frame, featurebigRepresenting the result of motion feature extraction for large displacement;
the optical flow estimation module of the large displacement optical flow estimation sub-network consists of an upper pooling layer and a convolution layer, and the feature value feature obtained by the output of the feature extraction modulebigAnd layer 5-1 of the feature extraction Module, layer 4 of the feature extraction ModuleFeature ofbig4And superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain the calculation result of the 1 st layer of the optical flow estimation module, wherein the calculation process is as follows:
featureBflow1＝estimate(featurebig+featurebig4)
wherein featureBflow1Representing the characteristics output by the layer 1 of the optical flow estimation module under the large displacement sub-network, and estimate (DEG) representing the mapping function of the layer 1 of the optical flow estimation module under the large displacement sub-network;
then, for the remaining 2-4 layers of optical flow estimation modules, the result calculated by the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureBflowi＝estimate(featurebig+featurebig(5-i)+featureBflow(i-1))
wherein featureBflow(i-1)The characteristics represent the output of the i-1 layer of the optical flow estimation module under the large displacement sub-network;
s22, for the small displacement optical flow estimation sub-network, the feature extraction module is composed of nine convolution layers, the first three convolutions are respectively to the input initial frame p without action amplificationsSum peak frame paRespectively extracting features, wherein the input of the last six convolutions is superposition of the output results of the first three convolutions on two image frames, the mapping function of the first three convolutions of the feature extraction module is first (-) and the mapping function of the last convolution is last (-), and the calculation process is as follows:
featuresmall＝last(first(ps)+first(pa))
wherein featuresmallRepresenting the result of the motion feature extraction for small displacement;
the light stream estimation module of the small-displacement light stream estimation subnetwork comprises an upper pooling layer and a convolution layer, and the feature value feature output by the feature extraction modulesmallAnd feature at layer 6-1 of the feature extraction module, i.e., layer 5 of the feature extraction modulesmall5Superposing, pooling, estimating optical flow, and recovering optical flow image resolution to obtain estimated optical flowCalculating the calculation result of the first layer of the module according to the following calculation process:
featureSflow1＝estimate(featuresmall+featuresmall5)
wherein featureSflow1The method comprises the steps of representing characteristics output by a layer 1 of an optical flow estimation module under a small displacement sub-network, and estimate (DEG) represents a mapping function of the layer 1 of the optical flow estimation module under the small displacement sub-network;
then, for the remaining 2-5 layers of optical flow estimation modules, the result calculated by the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureSflowi＝estimate(featuresmall+featuresmall(6-i)+featureSflow(i-1))
wherein featureSflow(i-1)The characteristics represent the output of the i-1 layer of the optical flow estimation module under the small displacement sub-network;
s23, fusing the results obtained by the large displacement optical flow estimation subnetwork and the small displacement optical flow estimation subnetwork to obtain the final output result, and making fusion (-) represent the final fusion operation, wherein the calculation process is as follows:
pfusion＝fusion(featureBflow4+featureSflow5)
the step S4 specifically includes the following steps:
step S41, for the preliminary classification result obtained by the residual error network training, there may be a case where the probabilities of two results of the same graph are very similar, so that the results need to be reordered;
step S42, selecting images under corresponding classification in the training set according to the classification results with similar classification probability of the tested images, selecting k neighbors according to the selected images, and calculating as follows:
Figure FDA0003579516260000021
Figure FDA0003579516260000031
wherein eiRepresenting the ith selected training set image, and p represents the tested image;
step S43, calculating tested image p and selected image eiThe calculation process of (a) is as follows:
Di＝1-probemax(ei)+probemax(p)
wherein probemaxRepresenting the maximum probability in the classification result probabilities;
step S44: for each selected training set image eiCalculating the Jaccard distance D between the measured image p and the measured imagejThe calculation process is as follows:
Figure FDA0003579516260000032
Figure FDA0003579516260000033
weighting DiAnd DjI.e. to obtain the final distance result.
2. The method for sorting micro-scenarios based on optical flow generation networking and reordering of claim 1, wherein the step S1 specifically comprises the following steps:
step S11, acquiring a micro expression data set, and cutting the image into 224 × 224 images after face alignment;
step S12, extracting the initial frame and the peak value frame directly according to the marked content for the micro expression data set with the initial frame and the peak value frame mark, and executing step S15;
step S13, extracting the start frame and the peak value frame of the video sequence by using a frame difference method for the micro expression data set which is not marked by the start frame and the peak value frame; the frame difference method comprises the following steps: let P be { P ═ PiDenotes input, i ═ 1,2Image sequence of piRepresenting the ith input picture with the first frame of the sequence as the start frame, i.e. pstart＝p1The gray values of pixels corresponding to the first frame and the nth frame of the video sequence are recorded as f1(x, y) and fn (x, y), the gray values of the pixels corresponding to the two frames of images are subtracted, the absolute value of the subtraction is taken to obtain a difference image Dn, Dn (x, y) ═ fn (x, y) -f1(x, y) |, and the average inter-frame difference Dnavg of the difference image is calculated, wherein the calculation method comprises the following steps:
Figure FDA0003579516260000034
wherein, Dn.shape [0 ]]Shape [1 ] represents the height of the difference image Dn]Representing the width of the difference image Dn, calculating the average inter-frame difference of all frames except the initial frame and the initial frame, and sequencing, wherein the frame with the maximum average inter-frame difference is the peak value frame p corresponding to the image sequenceapex(ii) a After the start frame and the peak frame are extracted, step S15 is executed;
step S15, performing euler action amplification on the extracted start frame and peak frame, wherein the calculation process is as follows:
I(x,t)＝g(x+(1+α)δ(t))
where I (x, t) represents the brightness value of the image at the location x and time t, g (-) represents the mapping function of euler's motion magnification, and δ (t) represents the motion deviation, and the magnified image is generated by adjusting the motion magnification factor α.
3. The method for sorting micro-scenarios based on optical flow generation networking and reordering of claim 1, wherein the step S3 specifically comprises the following steps:
step S31, under each optical flow image data set, there are multiple subjects, each of which represents a subject, and each of which contains multiple micro-expression sequences, which represent the multiple micro-expression sequences generated by the subject, according to the principle of leave-one-subject-out, when dividing the data set, one subject of one data set is taken as the test set, and all the other subjects are merged togetherAs a training set, the last data set gets SubiA training set and a test set, wherein SubiRepresenting the number of subjects in a data set;
and step S32, inputting the divided test set and training set into a residual error network in sequence for classification to obtain a primary classification result.
CN202011070119.8A 2020-10-09 2020-10-09 Micro-expression classification method based on optical flow generation network and reordering Active CN112183419B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011070119.8A CN112183419B (en) 2020-10-09 2020-10-09 Micro-expression classification method based on optical flow generation network and reordering

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011070119.8A CN112183419B (en) 2020-10-09 2020-10-09 Micro-expression classification method based on optical flow generation network and reordering

Publications (2)

Publication Number Publication Date
CN112183419A CN112183419A (en) 2021-01-05
CN112183419B true CN112183419B (en) 2022-06-10

Family

ID=73948334

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011070119.8A Active CN112183419B (en) 2020-10-09 2020-10-09 Micro-expression classification method based on optical flow generation network and reordering

Country Status (1)

Country Link
CN (1) CN112183419B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113591660A (en) * 2021-07-24 2021-11-02 中国石油大学(华东) Micro-expression recognition method based on meta-learning
CN114005157B (en) * 2021-10-15 2024-05-10 武汉烽火信息集成技术有限公司 Micro-expression recognition method for pixel displacement vector based on convolutional neural network
CN117392727B (en) * 2023-11-02 2024-04-12 长春理工大学 Facial micro-expression recognition method based on contrast learning and feature decoupling

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108830223A (en) * 2018-06-19 2018-11-16 山东大学 A kind of micro- expression recognition method based on batch mode Active Learning
US10423773B1 (en) * 2019-04-12 2019-09-24 Coupang, Corp. Computerized systems and methods for determining authenticity using micro expressions
CN110532950A (en) * 2019-08-29 2019-12-03 中国科学院自动化研究所 Video feature extraction method, micro- expression recognition method based on micro- expression video
CN111626179A (en) * 2020-05-24 2020-09-04 中国科学院心理研究所 Micro-expression detection method based on optical flow superposition

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108830223A (en) * 2018-06-19 2018-11-16 山东大学 A kind of micro- expression recognition method based on batch mode Active Learning
US10423773B1 (en) * 2019-04-12 2019-09-24 Coupang, Corp. Computerized systems and methods for determining authenticity using micro expressions
CN110532950A (en) * 2019-08-29 2019-12-03 中国科学院自动化研究所 Video feature extraction method, micro- expression recognition method based on micro- expression video
CN111626179A (en) * 2020-05-24 2020-09-04 中国科学院心理研究所 Micro-expression detection method based on optical flow superposition

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow;Li, QY (Li, Qiuyu) at el.;《MULTIMEDIA TOOLS AND APPLICATIONS》;20191030;全文 *
Micro-expression Analysis by Fusing Deep Convolutional Neural Network and Optical Flow;Li, QY (Li, Qiuyu) at el.;《IEEE》;20181230;全文 *
人脸面部活动单元检测及微表情分析;苏文超;《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》;20190815(第08期);全文 *
基于光流方向信息熵统计的微表情捕捉;李丹等;《工程科学学报》;20171115(第11期);全文 *
基于多运动特征融合的微表情识别算法;苏育挺;《激光与光电子学进展》;20200731;第57卷(第14期);全文 *

Also Published As

Publication number Publication date
CN112183419A (en) 2021-01-05

Similar Documents

Publication Publication Date Title
CN112183419B (en) Micro-expression classification method based on optical flow generation network and reordering
CN107808146B (en) Multi-mode emotion recognition and classification method
KR101893554B1 (en) Method and apparatus of recognizing facial expression base on multi-modal
CN114219984A (en) Improved YOLOv 3-based micro pest detection system and method
CN111046734A (en) Multi-modal fusion sight line estimation method based on expansion convolution
CN110477907B (en) Modeling method for intelligently assisting in recognizing epileptic seizures
CN116664677B (en) Sight estimation method based on super-resolution reconstruction
CN112464900B (en) Multi-template visual target tracking method based on twin network
CN114758293A (en) Deep learning population counting method based on auxiliary branch optimization and local density block enhancement
CN113191216A (en) Multi-person real-time action recognition method and system based on gesture recognition and C3D network
CN112200065B (en) Micro-expression classification method based on action amplification and self-adaptive attention area selection
CN111881818B (en) Medical action fine-grained recognition device and computer-readable storage medium
CN113221815A (en) Gait identification method based on automatic detection technology of skeletal key points
CN112488165A (en) Infrared pedestrian identification method and system based on deep learning model
CN117541574A (en) Tongue diagnosis detection method based on AI semantic segmentation and image recognition
CN116645717A (en) Microexpressive recognition method and system based on PCANet+ and LSTM
CN116030516A (en) Micro-expression recognition method and device based on multi-task learning and global circular convolution
CN113963374A (en) Pedestrian attribute identification method based on multi-level features and identity information assistance
CN113553895A (en) Multi-pose face recognition method based on face orthogonalization
CN113255666A (en) Personalized question answering system and method based on computer vision
CN111640126A (en) Artificial intelligence diagnosis auxiliary method based on medical image
CN111985437A (en) Multi-target time sequence face state recognition method based on deep learning
CN118279195B (en) Method for enhancing fundus imaging definition by Al technology
CN115965626B (en) Medical image processing method and system based on artificial intelligence algorithm
CN118542639B (en) Fundus image diagnosis analysis system and method based on pattern recognition

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant