CN112183419A - Micro-expression classification method based on optical flow generation network and reordering - Google Patents

Micro-expression classification method based on optical flow generation network and reordering Download PDF

Info

Publication number
CN112183419A
CN112183419A CN202011070119.8A CN202011070119A CN112183419A CN 112183419 A CN112183419 A CN 112183419A CN 202011070119 A CN202011070119 A CN 202011070119A CN 112183419 A CN112183419 A CN 112183419A
Authority
CN
China
Prior art keywords
optical flow
feature
frame
network
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011070119.8A
Other languages
Chinese (zh)
Other versions
CN112183419B (en
Inventor
柯逍
林艳
王俊强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fuzhou University
Original Assignee
Fuzhou University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fuzhou University filed Critical Fuzhou University
Priority to CN202011070119.8A priority Critical patent/CN112183419B/en
Publication of CN112183419A publication Critical patent/CN112183419A/en
Application granted granted Critical
Publication of CN112183419B publication Critical patent/CN112183419B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to a micro-expression classification method based on optical flow generation network and reordering. Firstly, acquiring a micro expression data set, extracting a start frame and a peak frame, and preprocessing; training an optical flow generation network, and generating optical flow characteristics according to all initial frames and peak frames; then, the obtained optical flow image is divided into a corresponding training set and a corresponding testing set according to an LOSO principle, and a residual error network is input for training; and finally, reordering results obtained by preliminary classification of the residual error network to obtain a final result with higher precision.

Description

Micro-expression classification method based on optical flow generation network and reordering
Technical Field
The invention relates to the field of pattern recognition and computer vision, in particular to a micro-expression classification method based on optical flow generation network and reordering.
Background
In the field of emotion calculation, facial expressions are often researched to judge the emotion of a human at the moment, but the human body is the highest animal and sometimes disguises or hides the emotion of the human body, and in such a case, people cannot acquire useful information from the macroscopic expression of the face. To be able to tap useful information from camouflaged facial expressions, ackerman discovered a transient, involuntary, rapid facial emotion, i.e., a micro-expression, that is provoked to involuntarily appear in the face when one tries to hide some kind of real emotion. A standard micro-expression lasts 1/5 to 1/25 seconds and usually appears only in a specific part of the face.
Micro-expressions have great prospects in the aspects of national security, criminal inquiries and medical applications, but the subtlety and conciseness of micro-expressions form a great challenge for human eyes, so that in recent years, people put forward a lot of work of realizing automatic identification of micro-expressions by using computer vision and machine learning algorithms.
Disclosure of Invention
The invention aims to provide a micro-expression classification method based on an optical flow generation network and reordering, which can effectively classify micro-expression images.
In order to achieve the purpose, the technical scheme of the invention is as follows: a micro-representation classification method based on optical flow generation network and reordering comprises the following steps:
s1, acquiring a micro expression data set, extracting a start frame and a peak frame, and preprocessing;
step S2, training an optical flow generation network, and generating optical flow characteristics according to all initial frames and peak frames;
step S3, dividing the obtained optical flow image into a training set and a test set according to an LOSO principle, and inputting a residual error network for training;
and step S4, reordering the classification results obtained by the residual error network to obtain a final result with higher precision.
In an embodiment of the present invention, the step S1 specifically includes the following steps:
step S11, acquiring a micro expression data set, and cutting the image into 224 × 224 images after face alignment;
step S12, extracting the initial frame and the peak frame directly according to the marked content for the micro expression data set with the initial frame and the peak frame mark, and executing step S15;
step S13, extracting the initial frame and the peak frame of the video sequence by using a frame difference method for the micro expression data set which is not marked with the initial frame and the peak frame; the frame difference method comprises the following steps: let P be { P ═ PiDenotes an input image sequence, where p is 1,2iRepresenting the ith input picture with the first frame of the sequence as the starting frame, i.e. pstart＝p1The gray values of pixels corresponding to the first frame and the nth frame of the video sequence are recorded as f1(x, y) and fn (x, y), the gray values of the pixels corresponding to the two frames of images are subtracted, the absolute value of the subtraction is taken to obtain a difference image Dn, Dn (x, y) ═ fn (x, y) -f1(x, y) |, and the average inter-frame difference Dnavg of the difference image is calculated, wherein the calculation method comprises the following steps:
Figure BDA0002714591380000021
wherein, Dn.shape [0 ]]Shape [1 ] represents the height of the difference image Dn]The width of the difference image Dn is indicated. Calculating the average interframe difference of all frames except the initial frame and sequencing, wherein the frame with the maximum average interframe difference is the peak value frame p corresponding to the image sequenceapex(ii) a After extracting the start frame and the peak frame, executing step S15;
step S15, performing euler action amplification on the extracted start frame and peak frame, wherein the calculation process is as follows:
I(x,t)＝g(x+(1+α)(t))
wherein I (x, t) represents the brightness value of the image at the position x and time t, g (-) represents the mapping function of euler's motion amplification process, and (t) represents the motion deviation, and the amplified image is generated by adjusting the motion amplification factor α.
In an embodiment of the present invention, in step S2, the optical flow generation network performs point-to-point pixel training by using a structure of two sub-networks, the two sub-networks are ranked with respect to each other, one sub-network performs optical flow estimation on large displacement, the other sub-network performs optical flow estimation on small displacement, each sub-network is composed of a feature extraction module and an optical flow estimation module, and finally, the optical flow estimates obtained by the two sub-networks are fused to obtain a final optical flow estimated image;
s21, for the sub-network of large displacement optical flow estimation, the feature extraction module is composed of nine convolution layers, the input of the feature extraction module is the superposition of the amplified input image pair, the feature mapping function of the feature extraction module is H (·), and the calculation process is as follows:
featurebig＝H(pls+pla)
wherein p islsIndicating the starting frame after enlargement, plaShowing the enlarged peak frame, featurebigRepresenting the result of motion feature extraction for large displacement;
the optical flow estimation module of the large displacement optical flow estimation sub-network consists of an upper pooling layer and a convolution layer, and the feature value feature obtained by the output of the feature extraction modulebigAnd feature at layer 5-1 of the feature extraction module, i.e., layer 4 of the feature extraction modulebig4And superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain the calculation result of the 1 st layer of the optical flow estimation module, wherein the calculation process is as follows:
featureBflow1＝estimate(featurebig+featurebig4)
wherein featureBflow1Representing the characteristics output by the layer 1 of the optical flow estimation module under the large displacement sub-network, and estimate (DEG) representing the mapping function of the layer 1 of the optical flow estimation module under the large displacement sub-network;
then, for the remaining 2-4 layers of optical flow estimation modules, the result calculated by the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureBflowi＝estimate(featurebig+featurebig(5-i)+featureBflow(i-1))
wherein featureBflow(i-1)The characteristics represent the output of the i-1 layer of the optical flow estimation module under the large displacement sub-network;
s22, for the small displacement optical flow estimation sub-network, the feature extraction module is composed of nine convolution layers, the first three convolutions are respectively to the input initial frame p without action amplificationsSum peak frame paRespectively extracting features, wherein the input of the last six convolutions is superposition of the output results of the first three convolutions on two image frames, the mapping function of the first three convolutions of the feature extraction module is first (-) and the mapping function of the last convolution is last (-), and the calculation process is as follows:
featuresmall＝last(first(ps)+first(pa))
wherein featuresmallRepresenting the result of motion feature extraction for small displacements;
the optical flow estimation module of the small displacement optical flow estimation sub-network consists of an upper pooling layer and a convolution layer, and the feature value feature obtained by the output of the feature extraction modulesmallAnd feature at layer 6-1 of the feature extraction module, i.e., layer 5 of the feature extraction modulesmall5And superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain a calculation result of a first layer of the optical flow estimation module, wherein the calculation process is as follows:
featureSflow1＝estimate(featuresmall+featuresmall5)
wherein featureSflow1The method comprises the steps of representing characteristics output by a layer 1 of an optical flow estimation module under a small displacement sub-network, and estimate (DEG) represents a mapping function of the layer 1 of the optical flow estimation module under the small displacement sub-network;
then, for the remaining 2-5 layers of optical flow estimation modules, the result calculated by the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureSflowi＝estimate(featuresmall+featuresmall(6-i)+featureSflow(i-1))
wherein featureSflow(i-1)Optical flow estimation model under representation of small displacement sub-networkCharacteristics of the i-1 layer output of the block;
s23, fusing the results obtained by the large displacement optical flow estimation sub-network and the small displacement optical flow estimation sub-network to obtain the final output result, and making fusion (-) represent the final fusion operation, wherein the calculation process is as follows:
pfusion＝fusion(featureBflow4+featureSflow5)。
in an embodiment of the present invention, the step S3 specifically includes the following steps:
step S31, under each optical flow image data set, there are multiple subjects, each subject represents a testee, and each subject contains multiple micro-expression sequences, which represent the multiple micro-expression sequences generated by the testee, according to the principle of leave-one-subject-out, one subject of one data set is taken as a test set when the data sets are divided, all other subjects are combined together as a training set, and the last data set obtains a subjectiA training set and a test set, wherein SubiRepresenting the number of subjects in a data set;
and step S32, inputting the divided test set and training set into a residual error network in sequence for classification to obtain a primary classification result.
In an embodiment of the present invention, the step S4 specifically includes the following steps:
step S41, for the preliminary classification result obtained by the residual error network training, there may be a case where the probabilities of two results of the same graph are very similar, so that the results need to be reordered;
step S42, selecting images under corresponding classification in the training set according to the classification results with similar classification probability of the tested images, selecting k neighbors according to the selected images, and calculating as follows:
Figure BDA0002714591380000044
Figure BDA0002714591380000041
wherein eiRepresenting the ith selected training set image, and p represents the tested image;
step S43, calculating tested image p and selected image eiThe calculation process of (a) is as follows:
Di＝1-probemax(ei)+probemax(p)
wherein probemaxRepresenting the maximum probability in the classification result probabilities;
step S44: for each selected training set image eiCalculating the Jaccard distance D between the measured image p and the measured imagejThe calculation process is as follows:
Figure BDA0002714591380000042
Figure BDA0002714591380000043
weighting DiAnd DjObtaining a final distance result;
the micro-expression images are reclassified in a reordering mode, so that the situation of wrong classification caused by too close two types of probabilities possibly occurring in the micro-expression recognition process is reduced, and the accuracy of micro-expression recognition is improved.
Compared with the prior art, the invention has the following beneficial effects:
1. the constructed optical flow generation network and the reordered micro-expression classification method can effectively classify the micro-expression images and improve the classification effect of the micro-expression images.
2. The method generates the result of the optical flow estimation between two frames in a neural network mode, and has more robust effect, better performance and clearer boundary compared with the traditional optical flow generation method.
3. Aiming at the problem that two types of expressions are difficult to distinguish in the conventional micro expression recognition process, the invention uses the method for reordering the test images with similar probability of the classification result, better classifies the single micro expression and improves the classification effect.
Drawings
Fig. 1 is a schematic diagram of the principle of the present invention.
Detailed Description
The technical scheme of the invention is specifically explained below with reference to the accompanying drawings.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
As shown in fig. 1, the present embodiment provides a micro-representation classification method based on optical flow generation network and reordering, which specifically includes the following steps:
step S1: acquiring a micro expression data set, extracting a start frame and a peak frame, and preprocessing;
step S2: training an optical flow generation network, and generating optical flow characteristics according to all initial frames and peak frames;
step S3: dividing the obtained optical flow image into a training set and a testing set according to an LOSO principle, and inputting a residual error network for training;
step S4: and finally, reordering the classification results obtained by the residual error network to obtain a final result with higher precision.
In this embodiment, the step S1 includes the following steps:
step S11: acquiring a micro expression data set, and cutting an image into 224 × 224 images after face alignment;
step S12: for a micro expression data set with initial frame and peak frame labels, directly extracting the initial frame and the peak frame according to the label content;
step S13: extracting the initial frame and the peak frame of the video sequence by using a frame difference method for the micro expression data set which is not marked by the initial frame and the peak frame;
step S14: the specific content of the frame difference method is that P is { P ═ PiDenotes an input image sequence, where p is 1,2iRepresenting the ith input picture, let us start the first frame of the sequence, i.e. pstart＝p1The gray values of pixels corresponding to the first frame and the nth frame of the video sequence are recorded as f1(x, y) and fn (x, y), the gray values of the pixels corresponding to the two frames of images are subtracted, the absolute value of the subtraction is taken to obtain a difference image Dn, Dn (x, y) ═ fn (x, y) -f1(x, y) |, and the average inter-frame difference Dnavg of the difference image is calculated, wherein the calculation method comprises the following steps:
Figure BDA0002714591380000061
wherein, Dn.shape [0 ]]Shape [1 ] represents the height of the difference image Dn]The width of the difference image Dn is indicated. Calculating the average interframe difference of all frames except the initial frame and sequencing, wherein the frame with the maximum average interframe difference is the peak value frame p corresponding to the image sequenceapex；
Step S15: and carrying out Euler action amplification on the processed initial frame and the processed peak frame, wherein the calculation process is as follows:
I(x,t)＝g(x+(1+α)(t))
wherein I (x, t) represents the brightness value of the image at position x and time t, g (-) represents the mapping function of Euler motion amplification process, and (t) represents the motion deviation, the method adjusts the motion amplification coefficient
Figure BDA0002714591380000062
To generate an enlarged image.
In this embodiment, step S2 specifically includes the following steps:
step S21: the optical flow generation network carries out point-to-point pixel training by adopting a structure of two sub-networks, the two sub-networks are leveled with each other, one sub-network is specially used for carrying out optical flow estimation on large displacement, the other sub-network is specially used for carrying out optical flow estimation on small displacement, each sub-network is composed of a feature extraction module and an optical flow estimation module, and finally the optical flow estimation obtained by the two sub-networks are fused to obtain a final optical flow estimation image;
step S22: for a sub-network specially used for carrying out optical flow estimation on large displacement, a feature extraction module of the sub-network mainly comprises nine convolution layers, the input of the module is superposition of an amplified input image pair, a feature mapping function of the module is made to be H (·), and the calculation process is as follows:
featurebig＝H(pls+pla)
wherein p islsIndicating the starting frame after enlargement, plaShowing the enlarged peak frame, featurebigRepresenting the result of feature extraction for large displacement motion.
The optical flow estimation module of the large-displacement optical flow estimation sub-network mainly comprises an upper pooling layer and a convolution layer, and a feature value feature output by the upper layer modulebigAnd feature of upper module layer 5-1, layer 4big4And (3) superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain a calculation result of a first layer, wherein the calculation process is as follows:
featureBflow1＝estimate(featurebig+featurebig4)
wherein featureBflow1Representing the characteristics output by the layer 1 of the optical flow estimation module under the large displacement sub-network, and the estimate (DEG) represents the mapping function of the layer;
then, for the remaining 2-4 layers, the calculation result of the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureBflowi＝estimate(featurebig+featurebig(5-i)+featureBflow(i-1))
wherein featureBflow(i-1)The characteristics represent the output of the i-1 layer of the optical flow estimation module under the large displacement sub-network;
step S23: for the sub-network specially used for carrying out optical flow estimation on small displacement, the feature extraction module mainly comprises nine convolution layers, wherein the first three convolutions are respectively used for the input initial frame p which is not amplified in actionsSum peak frame paAnd respectively extracting features, wherein the input of the last six convolutions is the superposition of the first three convolutions on the output structures of the two image frames, the mapping function of the first three convolutions of the module is first (·), the mapping function of the last six convolutions is last (·), and the calculation process is as follows:
featuresmall＝last(first(ps)+first(pa))
wherein featuresmallRepresenting the result of feature extraction for small displacement motion.
The optical flow estimation module of the small displacement optical flow estimation sub-network mainly comprises an upper pooling layer and a convolution layer, and a feature value feature output by the upper layer modulesmallAnd feature of layer 6-1, layer 5 of the upper modulesmall5And (3) superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain a calculation result of a first layer, wherein the calculation process is as follows:
featureSflow1＝estimate(featuresmall+featuresmall5)
wherein featureSflow1Representing the characteristics output by the layer 1 of the optical flow estimation module under the small displacement sub-network, and the estimate (DEG) represents the mapping function of the layer;
then, for the remaining 2-5 layers, the calculation result of the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureSflowi＝estimate(featuresmall+featuresmall(6-i)+featureSflow(i-1))
wherein featureSflow(i-1)The characteristics represent the output of the i-1 layer of the optical flow estimation module under the small displacement sub-network;
step S24: and finally, fusing the results obtained by the two sub-networks to obtain a final output result, and enabling fusion (-) to represent the final fusion operation, wherein the calculation process is as follows:
pfusion＝fusion(featureBflow4+featureSflow5)
the convolution neural network is used for simulating and fusing optical flow estimation results of large displacement and small displacement, the generalization of the model is facilitated to be improved, the micro expression segments with overlarge or undersize micro expression changes can be adjusted more reasonably, and compared with the traditional method, the implementation mode of the convolution neural network reduces the edge blurring problem possibly generated in the micro expression optical flow estimation process, and the optical flow estimation result is more accurate.
In this embodiment, step S3 specifically includes the following steps:
step S31: each data set is provided with a plurality of subjects, each subject represents a testee, each subject comprises a plurality of micro-expression sequences which represent the micro-expression sequences generated by the testee, and according to the principle of leave-one-subject-out, one subject of one data set is taken as a test set at one time when the data sets are divided, and all the rest subjects are combined together to be used as a training set, so that finally, the subjects can be obtained for one data setiA training set and a test set, wherein SubiRepresenting the number of subjects in a data set.
Step S32: inputting the divided test set and training set into a residual error network in sequence for classification to obtain a primary classification result;
in this embodiment, step S4 specifically includes the following steps:
step S41: for the results of the preliminary classification, there may be a case that the probability of some two results of the same graph is very similar, which requires the results to be reordered;
step S42: selecting images under corresponding classification in a training set according to classification results with similar classification probabilities of tested images, and selecting k neighbors according to the selected images, wherein the calculation process is as follows:
Figure BDA0002714591380000082
Figure BDA0002714591380000081
wherein eiRepresenting the ith selected training set image and p representing the image under test.
Step S43: calculating the tested image p and the selected image eiThe calculation process of (a) is as follows:
Di＝1-probemax(ei)+probemax(p)
wherein probemaxRepresenting the highest probability among the probabilities of the classification results.
Step S44: for each selected training set image eiCalculating the Jaccard distance D between the measured image p and the measured imagejThe calculation process is as follows:
Figure BDA0002714591380000091
Figure BDA0002714591380000092
weighting DiAnd DjI.e. to obtain the final distance result.
The micro-expression images are reclassified in a reordering mode, so that the situation of wrong classification caused by the fact that certain two types of probabilities are too close to each other in the micro-expression recognition process is greatly reduced, and the accuracy of micro-expression recognition is improved.
The above are preferred embodiments of the present invention, and all changes made according to the technical scheme of the present invention that produce functional effects do not exceed the scope of the technical scheme of the present invention belong to the protection scope of the present invention.

Claims (5)

1. A micro-expression classification method based on optical flow generation network and reordering is characterized by comprising the following steps:
s1, acquiring a micro expression data set, extracting a start frame and a peak frame, and preprocessing;
step S2, training an optical flow generation network, and generating optical flow characteristics according to all initial frames and peak frames;
step S3, dividing the obtained optical flow image into a training set and a test set according to an LOSO principle, and inputting a residual error network for training;
and step S4, reordering the classification results obtained by the residual error network to obtain a final result with higher precision.
2. The method for sorting micro-scenarios based on optical flow generation networking and reordering of claim 1, wherein the step S1 specifically comprises the following steps:
step S11, acquiring a micro expression data set, and cutting the image into 224 × 224 images after face alignment;
step S12, extracting the initial frame and the peak frame directly according to the marked content for the micro expression data set with the initial frame and the peak frame mark, and executing step S15;
step S13, extracting the initial frame and the peak frame of the video sequence by using a frame difference method for the micro expression data set which is not marked with the initial frame and the peak frame; the frame difference method comprises the following steps: let P be { P ═ PiDenotes an input image sequence, where p is 1,2iRepresenting the ith input picture with the first frame of the sequence as the starting frame, i.e. pstart＝p1Recording the gray values of pixels corresponding to the first frame and the nth frame of the video sequence as f1(x, y) and fn (x, y), subtracting the gray values of the pixels corresponding to the two frames of images, taking the absolute value of the subtraction to obtain a difference image Dn, Dn (x, y) ═ fn (x, y) -f1(x, y) |, calculating the average inter-frame difference Dnavg of the difference image, and calculating the methodThe following were used:
Figure FDA0002714591370000011
wherein, Dn.shape [0 ]]Shape [1 ] represents the height of the difference image Dn]Representing the width of the difference image Dn, calculating the average inter-frame difference of all frames except the initial frame and the initial frame, and sequencing, wherein the frame with the maximum average inter-frame difference is the peak value frame p corresponding to the image sequenceapex(ii) a After extracting the start frame and the peak frame, executing step S15;
step S15, performing euler action amplification on the extracted start frame and peak frame, wherein the calculation process is as follows:
I(x,t)＝g(x+(1+α)(t))
wherein I (x, t) represents the brightness value of the image at the position x and time t, g (-) represents the mapping function of euler's motion amplification process, and (t) represents the motion deviation, and the amplified image is generated by adjusting the motion amplification factor α.
3. The method according to claim 1, wherein in step S2, the optical flow generating network performs point-to-point pixel training by adopting a structure of two sub-networks, the two sub-networks are ranked with respect to each other, one sub-network performs optical flow estimation for large displacement, the other sub-network performs optical flow estimation for small displacement, each sub-network is composed of a feature extraction module and an optical flow estimation module, and finally the optical flow estimates obtained from the two sub-networks are fused to obtain a final optical flow estimation image;
s21, for the sub-network of large displacement optical flow estimation, the feature extraction module is composed of nine convolution layers, the input of the feature extraction module is the superposition of the amplified input image pair, the feature mapping function of the feature extraction module is H (·), and the calculation process is as follows:
featurebig＝H(pls+pla)
wherein p islsAfter showing enlargementStart frame of plaShowing the enlarged peak frame, featurebigRepresenting the result of motion feature extraction for large displacement;
the optical flow estimation module of the large displacement optical flow estimation sub-network consists of an upper pooling layer and a convolution layer, and the feature value feature obtained by the output of the feature extraction modulebigAnd feature at layer 5-1 of the feature extraction module, i.e., layer 4 of the feature extraction modulebig4And superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain the calculation result of the 1 st layer of the optical flow estimation module, wherein the calculation process is as follows:
featureBflow1＝estimate(featurebig+featurebig4)
wherein featureBflow1Representing the characteristics output by the layer 1 of the optical flow estimation module under the large displacement sub-network, and estimate (DEG) representing the mapping function of the layer 1 of the optical flow estimation module under the large displacement sub-network;
then, for the remaining 2-4 layers of optical flow estimation modules, the result calculated by the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureBflowi＝estimate(featurebig+featurebig(5-i)+featureBflow(i-1))
wherein featureBflow(i-1)The characteristics represent the output of the i-1 layer of the optical flow estimation module under the large displacement sub-network;
s22, for the small displacement optical flow estimation sub-network, the feature extraction module is composed of nine convolution layers, the first three convolutions are respectively to the input initial frame p without action amplificationsSum peak frame paRespectively extracting features, wherein the input of the last six convolutions is superposition of the output results of the first three convolutions on two image frames, the mapping function of the first three convolutions of the feature extraction module is first (-) and the mapping function of the last convolution is last (-), and the calculation process is as follows:
featuresmall＝last(first(ps)+first(pa))
wherein featuresmallRepresenting motion feature extraction for small displacementsThe result of (1);
the optical flow estimation module of the small displacement optical flow estimation sub-network consists of an upper pooling layer and a convolution layer, and the feature value feature obtained by the output of the feature extraction modulesmallAnd feature at layer 6-1 of the feature extraction module, i.e., layer 5 of the feature extraction modulesmall5And superposing, pooling, estimating the optical flow and restoring the optical flow image resolution to obtain a calculation result of a first layer of the optical flow estimation module, wherein the calculation process is as follows:
featureSflow1＝estimate(featuresmall+featuresmall5)
wherein featureSflow1The method comprises the steps of representing characteristics output by a layer 1 of an optical flow estimation module under a small displacement sub-network, and estimate (DEG) represents a mapping function of the layer 1 of the optical flow estimation module under the small displacement sub-network;
then, for the remaining 2-5 layers of optical flow estimation modules, the result calculated by the previous layer is added to the input of the next layer, and the calculation process is as follows:
featureSflowi＝estimate(featuresmall+featuresmall(6-i)+featureSflow(i-1))
wherein featureSflow(i-1)The characteristics represent the output of the i-1 layer of the optical flow estimation module under the small displacement sub-network;
s23, fusing the results obtained by the large displacement optical flow estimation sub-network and the small displacement optical flow estimation sub-network to obtain the final output result, and making fusion (-) represent the final fusion operation, wherein the calculation process is as follows:
pfusion＝fusion(featureBflow4+featureSflow5)。
4. the method for sorting micro-scenarios based on optical flow generation networking and reordering of claim 1, wherein the step S3 specifically comprises the following steps:
step S31, each optical flow image data set has multiple subjects, each subject represents a subject, and each subject has multiple micro-expression sequences, each micro-expression sequence represents the subjectAccording to the principle of leave-one-subject-out, when the data sets are divided, one subject of one data set is taken as a test set, all other subjects are combined together to be used as a training set, and the last data set obtains the subjectiA training set and a test set, wherein SubiRepresenting the number of subjects in a data set;
and step S32, inputting the divided test set and training set into a residual error network in sequence for classification to obtain a primary classification result.
5. The method for sorting micro-scenarios based on optical flow generation networking and reordering of claim 1, wherein the step S4 specifically comprises the following steps:
step S41, for the preliminary classification result obtained by the residual error network training, there may be a case where the probabilities of two results of the same graph are very similar, so that the results need to be reordered;
step S42, selecting images under corresponding classification in the training set according to the classification results with similar classification probability of the tested images, selecting k neighbors according to the selected images, and calculating as follows:
Figure FDA0002714591370000044
Figure FDA0002714591370000041
wherein eiRepresenting the ith selected training set image, and p represents the tested image;
step S43, calculating tested image p and selected image eiThe calculation process of (a) is as follows:
Di＝1-probemax(ei)+probemax(p)
wherein probemaxIn expressing the probability of classification resultThe maximum probability;
step S44: for each selected training set image eiCalculating the Jaccard distance D between the measured image p and the measured imagejThe calculation process is as follows:
Figure FDA0002714591370000042
Figure FDA0002714591370000043
weighting DiAnd DjObtaining a final distance result;
the micro-expression images are reclassified in a reordering mode, so that the situation of wrong classification caused by too close two types of probabilities possibly occurring in the micro-expression recognition process is reduced, and the accuracy of micro-expression recognition is improved.
CN202011070119.8A 2020-10-09 2020-10-09 Micro-expression classification method based on optical flow generation network and reordering Active CN112183419B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011070119.8A CN112183419B (en) 2020-10-09 2020-10-09 Micro-expression classification method based on optical flow generation network and reordering

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011070119.8A CN112183419B (en) 2020-10-09 2020-10-09 Micro-expression classification method based on optical flow generation network and reordering

Publications (2)

Publication Number Publication Date
CN112183419A true CN112183419A (en) 2021-01-05
CN112183419B CN112183419B (en) 2022-06-10

Family

ID=73948334

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011070119.8A Active CN112183419B (en) 2020-10-09 2020-10-09 Micro-expression classification method based on optical flow generation network and reordering

Country Status (1)

Country Link
CN (1) CN112183419B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113591660A (en) * 2021-07-24 2021-11-02 中国石油大学(华东) Micro-expression recognition method based on meta-learning
CN114005157A (en) * 2021-10-15 2022-02-01 武汉烽火信息集成技术有限公司 Micro-expression recognition method of pixel displacement vector based on convolutional neural network
CN117392727A (en) * 2023-11-02 2024-01-12 长春理工大学 Facial micro-expression recognition method based on contrast learning and feature decoupling

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108830223A (en) * 2018-06-19 2018-11-16 山东大学 A kind of micro- expression recognition method based on batch mode Active Learning
US10423773B1 (en) * 2019-04-12 2019-09-24 Coupang, Corp. Computerized systems and methods for determining authenticity using micro expressions
CN110532950A (en) * 2019-08-29 2019-12-03 中国科学院自动化研究所 Video feature extraction method, micro- expression recognition method based on micro- expression video
CN111626179A (en) * 2020-05-24 2020-09-04 中国科学院心理研究所 Micro-expression detection method based on optical flow superposition

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108830223A (en) * 2018-06-19 2018-11-16 山东大学 A kind of micro- expression recognition method based on batch mode Active Learning
US10423773B1 (en) * 2019-04-12 2019-09-24 Coupang, Corp. Computerized systems and methods for determining authenticity using micro expressions
CN110532950A (en) * 2019-08-29 2019-12-03 中国科学院自动化研究所 Video feature extraction method, micro- expression recognition method based on micro- expression video
CN111626179A (en) * 2020-05-24 2020-09-04 中国科学院心理研究所 Micro-expression detection method based on optical flow superposition

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
LI, QY (LI, QIUYU) AT EL.: "Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow", 《MULTIMEDIA TOOLS AND APPLICATIONS》 *
LI, QY (LI, QIUYU) AT EL.: "Micro-expression Analysis by Fusing Deep Convolutional Neural Network and Optical Flow", 《IEEE》 *
李丹等: "基于光流方向信息熵统计的微表情捕捉", 《工程科学学报》 *
苏文超: "人脸面部活动单元检测及微表情分析", 《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》 *
苏育挺: "基于多运动特征融合的微表情识别算法", 《激光与光电子学进展》 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113591660A (en) * 2021-07-24 2021-11-02 中国石油大学(华东) Micro-expression recognition method based on meta-learning
CN114005157A (en) * 2021-10-15 2022-02-01 武汉烽火信息集成技术有限公司 Micro-expression recognition method of pixel displacement vector based on convolutional neural network
CN114005157B (en) * 2021-10-15 2024-05-10 武汉烽火信息集成技术有限公司 Micro-expression recognition method for pixel displacement vector based on convolutional neural network
CN117392727A (en) * 2023-11-02 2024-01-12 长春理工大学 Facial micro-expression recognition method based on contrast learning and feature decoupling
CN117392727B (en) * 2023-11-02 2024-04-12 长春理工大学 Facial micro-expression recognition method based on contrast learning and feature decoupling

Also Published As

Publication number Publication date
CN112183419B (en) 2022-06-10

Similar Documents

Publication Publication Date Title
CN112183419B (en) Micro-expression classification method based on optical flow generation network and reordering
JP4743823B2 (en) Image processing apparatus, imaging apparatus, and image processing method
KR101893554B1 (en) Method and apparatus of recognizing facial expression base on multi-modal
CN112507617B (en) Training method of SRFlow super-resolution model and face recognition method
CN112837344B (en) Target tracking method for generating twin network based on condition countermeasure
CN110477907B (en) Modeling method for intelligently assisting in recognizing epileptic seizures
CN105373810B (en) Method and system for establishing motion recognition model
CN113869276B (en) Lie recognition method and system based on micro-expression
CN114219984A (en) Improved YOLOv 3-based micro pest detection system and method
CN114973383A (en) Micro-expression recognition method and device, electronic equipment and storage medium
CN113191216A (en) Multi-person real-time action recognition method and system based on gesture recognition and C3D network
CN114333046A (en) Dance action scoring method, device, equipment and storage medium
CN112200065B (en) Micro-expression classification method based on action amplification and self-adaptive attention area selection
CN111881818B (en) Medical action fine-grained recognition device and computer-readable storage medium
CN113506274A (en) Detection system for human cognitive condition based on visual saliency difference map
CN117576753A (en) Micro-expression recognition method based on attention feature fusion of facial key points
CN117541574A (en) Tongue diagnosis detection method based on AI semantic segmentation and image recognition
CN113221815A (en) Gait identification method based on automatic detection technology of skeletal key points
CN112149613A (en) Motion estimation evaluation method based on improved LSTM model
CN116758621A (en) Self-attention mechanism-based face expression depth convolution identification method for shielding people
CN116343100A (en) Target identification method and system based on self-supervision learning
CN116030516A (en) Micro-expression recognition method and device based on multi-task learning and global circular convolution
CN116416664A (en) Depression recognition system, medium and equipment combined with facial dynamic behavior
CN115424337A (en) Iris image restoration system based on priori guidance
CN114758293A (en) Deep learning population counting method based on auxiliary branch optimization and local density block enhancement

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant