CN112183419B - A Micro-Expression Classification Method Based on Optical Flow Generation Network and Reordering - Google Patents

Abstract

The invention relates to a micro-expression classification method based on optical flow generation network and reordering. First, acquire the micro-expression dataset, extract the start frame and peak frame, and perform preprocessing; train the optical flow generation network to generate its optical flow features according to all the start frames and peak frames; then, the obtained optical flow image, According to the LOSO principle, it is divided into the corresponding training set and test set, and input the residual network for training; finally, the results obtained by the preliminary classification of the residual network are reordered to obtain the final result with higher accuracy.

Description

A Micro-Expression Classification Method Based on Optical Flow Generation Network and Reordering

technical field

The invention relates to the field of pattern recognition and computer vision, in particular to a micro-expression classification method based on an optical flow generation network and reordering.

Background technique

In the field of affective computing, there are often studies on human facial expressions to judge the emotions of human beings at the moment, but human beings are the most advanced animals and sometimes disguise or hide their emotions. In this case, people do not No useful information can be obtained from the macroscopic expressions of the human face. In order to be able to mine useful information from disguised facial expressions, Ekman discovered a short-lived, involuntary, rapid facial emotion known as microexpressions, which occurs when people try to hide a real emotion. It will be stimulated and will appear on people's faces unconsciously. A standard microexpression lasts 1/5 to 1/25 of a second and usually appears only on a specific part of the face.

Micro-expressions have great prospects in national security, criminal interrogation, and medical applications, but the subtlety and simplicity of micro-expressions pose a great challenge to the human eye. Therefore, in recent years, many algorithms using computer vision and machine learning have been proposed. To realize the automatic recognition of micro-expressions.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a micro-expression classification method based on optical flow generation network and reordering, which can effectively classify micro-expression images.

In order to achieve the above object, the technical scheme of the present invention is: a micro-expression classification method based on optical flow generation network and reordering, comprising the following steps:

Step S1, obtaining a micro-expression data set, extracting a start frame and a peak frame, and performing preprocessing;

Step S2, train the optical flow generation network, and generate its optical flow feature according to all the start frames and peak frames;

Step S3: Divide the obtained optical flow image into a training set and a test set according to the LOSO principle, and input the residual network for training;

Step S4, reordering the classification results obtained by the residual network to obtain a final result with higher accuracy.

In an embodiment of the present invention, the step S1 specifically includes the following steps:

Step S11, obtaining a micro-expression dataset, and trimming the image into a 224*224 size image after face alignment;

Step S12, for the micro-expression data set marked with the start frame and the peak frame, directly extract the start frame and the peak frame according to the labeling content, and perform step S15;

Step S13, for the micro-expression data set without the start frame and the peak frame label, use the frame difference method to extract the start frame and the peak frame of the video sequence; the frame difference method is as follows: let P={pi _} ,i=1,2,...represents the input image sequence, where p _i represents the ith input image, let the first frame of the sequence be the start frame, that is, p _start =p ₁ , the first frame of the video sequence and the The gray value of the pixel corresponding to the nth frame is denoted as f1(x,y), fn(x,y), the gray value of the corresponding pixel of the two frames of images is subtracted, and the absolute value is taken to obtain the difference image Dn , Dn(x,y)=|fn(x,y)-f1(x,y)|, calculate the average inter-frame difference Dnavg of the difference image, the calculation method is as follows:

Among them, Dn.shape[0] represents the height of the difference image Dn, and Dn.shape[1] represents the width of the difference image Dn. Calculate and sort the average inter-frame difference of all frames except the start frame and the start frame, and the frame with the largest average inter-frame difference is the peak frame _papex corresponding to the image sequence; after extracting the start frame and the peak frame Execute step S15;

Step S15, performing Euler action amplification on the extracted start frame and peak frame, and the calculation process is as follows:

I(x,t)=g(x+(1+α)δ(t))

where I(x,t) represents the brightness value of the image at position x and time t, g( ) represents the mapping function of the Euler motion amplification process, and δ(t) represents the motion deviation, which is generated by adjusting the motion amplification coefficient α Enlarged image.

In an embodiment of the present invention, in step S2, the optical flow generation network adopts the structure of two sub-networks to perform point-to-point pixel training, the two sub-networks are level with each other, and one sub-network performs optical flow for large displacements Estimation, another sub-network performs optical flow estimation for small displacement, each sub-network is composed of a feature extraction module and an optical flow estimation module, and finally the optical flow estimates obtained by the two sub-networks are fused to obtain the final optical flow estimation image;

S21. For the large-displacement optical flow estimation sub-network, the feature extraction module is composed of nine convolutional layers. The input of the feature extraction module is the superposition of the enlarged input image pairs, and the feature mapping function of the feature extraction module is H. ( ), the calculation process is as follows:

feature _big =H(p _ls +p _la )

where p _ls represents the start frame after zooming in, p _la represents the peak frame after zooming in, and feature _big represents the result of feature extraction for large displacement motion;

The optical flow estimation module of the large displacement optical flow estimation sub-network is composed of an upper pooling layer and a convolutional layer. The feature value feature _big output by the feature extraction module and the 5-1 layer of the feature extraction module are the fourth layer of the feature extraction module. The feature _big4 of the layer is superimposed, and the upper pool is used to estimate the optical flow and restore the resolution of the optical flow image, and obtain the calculation result of the first layer of the optical flow estimation module. The calculation process is as follows:

feature _Bflow1 = estimate(feature _big +feature _big4 )

Among them, feature _Bflow1 represents the output feature of the first layer of the optical flow estimation module under the large displacement sub-network, and estimate( ) represents the mapping function of the first layer of the optical flow estimation module under the large displacement sub-network;

Then, for the remaining 2-4 layers of the optical flow estimation module, the calculation results of the previous layer are added to the input of the next layer. The calculation process is as follows:

feature _Bflowi = estimate(feature _big +feature _big(5-i) +feature _Bflow(i-1) )

Among them, feature _Bflow(i-1) represents the feature output by the i-1th layer of the optical flow estimation module under the large displacement sub-network;

S22. For the small-displacement optical flow estimation sub-network, the feature extraction module is composed of nine convolution layers, and the first three convolutions respectively perform features on the input starting frame p _s and peak frame p _a without action amplification. Extraction, the input of the last six convolutions is the superposition of the output results of the first three convolutions on the two image frames, so that the mapping function of the first three convolutions of the feature extraction module is first( ), the mapping function of the last six convolutions The function is last( ), and its calculation process is as follows:

feature _small = last(first( _ps )+first(p _a ))

Among them, feature _small represents the result of feature extraction for small displacement motion;

The optical flow estimation module of the small-displacement optical flow estimation sub-network is composed of an upper pooling layer and a convolutional layer. The feature value feature _small output by the feature extraction module and the 6-1 layer of the feature extraction module are the fifth layer of the feature extraction module. The feature _small5 of the layer is superimposed and pooled to estimate the optical flow and restore the resolution of the optical flow image, and obtain the calculation result of the first layer of the optical flow estimation module. The calculation process is as follows:

feature _Sflow1 = estimate(feature _small +feature _small5 )

Among them, feature _Sflow1 represents the feature output by the first layer of the optical flow estimation module under the small displacement sub-network, and estimate( ) represents the mapping function of the first layer of the optical flow estimation module under the small displacement sub-network;

Then, for the remaining 2-5 layers of the optical flow estimation module, the calculation results of the previous layer are added to the input of the next layer. The calculation process is as follows:

feature _Sflowi = estimate(feature _small +feature _small(6-i) +feature _Sflow(i-1) )

Among them, feature _Sflow(i-1) represents the feature output by the i-1th layer of the optical flow estimation module under the small displacement sub-network;

S23, fuse the results obtained by the large displacement optical flow estimation sub-network and the small displacement optical flow estimation sub-network to obtain the final output result, let fusion( ) represent the final fusion operation, and the calculation process is as follows:

p _fusion = fusion(feature _Bflow4 +feature _Sflow5 ).

In an embodiment of the present invention, the step S3 specifically includes the following steps:

Step S31, there are multiple subjects under each optical flow image dataset, each subject represents a subject, and each subject contains multiple micro-expression sequences, representing multiple micro-expression sequences generated by the subject. Expression sequence, according to the principle of leave-one-subject-out, when dividing the data set, one subject of one data set is taken as the test set at a time, and all the remaining subjects are combined as a training set, and the last data set gets Sub _i Training set and test set, where Sub _i represents the number of subjects in a dataset;

Step S32 , input the divided test set and training set into the residual network for classification, and obtain a preliminary classification result.

In an embodiment of the present invention, the step S4 specifically includes the following steps:

Step S41, for the preliminary classification results obtained by the residual network training, there may be situations where the probabilities of two of the results in the same graph are very similar, so the results need to be reordered;

Step S42, according to the classification results with similar classification probabilities of the tested images, select images under the corresponding classification in the training set, and select k nearest neighbors according to the selected images, and the calculation process is as follows:

where e _i represents the i-th selected training set image, and p represents the tested image;

Step S43, calculate the distance between the tested image p and the selected image e _i , and the calculation process is as follows:

D _i =1-probe _max ( _ei )+probe _max (p)

Among them, probe _max represents the maximum probability of the classification result probability;

Step S44: For each selected training set image e _i , calculate the Jaccard distance D _j from the tested image p, and the calculation process is as follows:

The final distance result is obtained by weighting D _i and D _j ;

The micro-expression images are re-classified by means of re-ranking, so that the probabilities of the two types in the micro-expression recognition process are too close to reduce misclassification, and the accuracy of micro-expression recognition is improved.

Compared with the prior art, the present invention has the following beneficial effects:

1. The optical flow generation network and the reordered micro-expression classification method constructed by the present invention can effectively classify micro-expression images, and improve the classification effect of micro-expression images.

2. The present invention generates an optical flow estimation result between two frames by means of a neural network, which is more robust, has better performance and clearer boundaries than the traditional optical flow generation method.

3. Aiming at the problem that certain two types of expressions are difficult to distinguish in the past micro-expression recognition process, the present invention uses the method of reordering test images with similar classification results, so as to better classify a single micro-expression. , to improve the classification effect.

Description of drawings

FIG. 1 is a schematic diagram of the principle of the present invention.

Detailed ways

The technical solutions of the present invention will be described in detail below with reference to the accompanying drawings.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

As shown in FIG. 1 , this embodiment provides a micro-expression classification method based on an optical flow generation network and reordering, which specifically includes the following steps:

Step S1: obtain the micro-expression data set, extract the start frame and the peak frame, and perform preprocessing;

Step S2: train the optical flow generation network, and generate its optical flow features according to all the start frames and peak frames;

Step S3: Divide the obtained optical flow image into a training set and a test set according to the LOSO principle, and input the residual network for training;

Step S4: Finally, reorder the classification results obtained by the residual network to obtain a final result with higher accuracy.

In this embodiment, including step S1 specifically includes the following steps:

Step S11: Acquire a micro-expression dataset, and cut the image into a 224*224 size image after face alignment;

Step S12: For the micro-expression data set with the start frame and the peak frame label, directly extract the start frame and the peak frame according to the labeling content;

Step S13: For the micro-expression data set without the start frame and peak frame labeling, use the frame difference method to extract the start frame and peak frame of the video sequence;

Step S14: The specific content of the frame difference method is, let P={pi _} , _i =1, 2, . is the starting frame, that is, p _start =p ₁ , the gray value of the first frame of the video sequence and the corresponding pixel of the nth frame are recorded as f1(x, y), fn(x, y), and the corresponding pixels of the two frames of images are The gray value of the point is subtracted, and its absolute value is taken to obtain the difference image Dn, Dn(x,y)=|fn(x,y)-f1(x,y)|, and the average frame interval of the difference image is calculated. Differential Dnavg, the calculation method is as follows:

Among them, Dn.shape[0] represents the height of the difference image Dn, and Dn.shape[1] represents the width of the difference image Dn. Calculate and sort the average inter-frame difference of all frames except the initial frame and the initial frame, and the frame with the largest average inter-frame difference is the peak frame _papex corresponding to the image sequence;

Step S15: Perform Euler action amplification on the processed start frame and peak frame, and the calculation process is as follows:

I(x,t)=g(x+(1+α)δ(t))

来生成放大后的图像。where I(x, t) represents the brightness value of the image at position x and time t, g( ) represents the mapping function of the Euler motion amplification process, and δ(t) represents the motion deviation. This method adjusts the motion amplification factor by adjusting the motion amplification factor.

to generate an enlarged image.

In this embodiment, step S2 specifically includes the following steps:

Step S21: The optical flow generation network adopts the structure of two sub-networks to perform point-to-point pixel training, and the two sub-networks are level with each other, one sub-network is dedicated to optical flow estimation for large displacements, and the other sub-network is dedicated to small displacements. Perform optical flow estimation, each sub-network is composed of a feature extraction module and an optical flow estimation module, and finally fuse the optical flow estimates obtained by the two sub-networks to obtain the final optical flow estimation image;

Step S22: For the sub-network specially designed for optical flow estimation for large displacement, its feature extraction module is mainly composed of nine convolutional layers. The function is H( ), and its calculation process is as follows:

feature _big =H(p _ls +p _la )

where p _ls represents the start frame after zooming in, p _la represents the peak frame after zooming in, and feature _big represents the result of feature extraction for large displacement motion.

The optical flow estimation module of the large-displacement optical flow estimation sub-network is mainly composed of an upper pooling layer and a convolutional layer. We use the feature value feature _big output by the upper module and the 5-1 layer of the upper module, that is, the feature of the fourth layer. The feature _big4 is superimposed, and the upper pool is used to estimate the optical flow and restore the resolution of the optical flow image, and obtain the calculation result of the first layer. The calculation process is as follows:

feature _Bflow1 = estimate(feature _big +feature _big4 )

Among them, feature _Bflow1 represents the feature output by the first layer of the optical flow estimation module under the large displacement sub-network, and estimate( ) represents the mapping function of this layer;

Then, for the remaining 2-4 layers, the calculation results of the previous layer are added to the input of the next layer. The calculation process is as follows:

feature _Bflowi = estimate(feature _big +feature _big(5-i) +feature _Bflow(i-1) )

Among them, feature _Bflow(i-1) represents the feature of the output of the i-1th layer of the optical flow estimation module under the large displacement sub-network;

Step S23: For the sub-network specially designed for optical flow estimation for small displacements, the feature extraction module is mainly composed of nine convolution _layers . The peak frame _pa is subjected to separate feature extraction, and the input of the last six convolutions is the superposition of the first three convolutions on the output structure of the two image frames, so that the mapping function of the first three convolutions of this module is first( ), The mapping function of the six convolutions is last( ), and the calculation process is as follows:

feature _small = last(first( _ps )+first(p _a ))

Among them, feature _small represents the result of feature extraction for small displacement motion.

The optical flow estimation module of the small displacement optical flow estimation sub-network is mainly composed of an upper pooling layer and a convolutional layer. We use the feature value feature _small output by the upper module and the 6-1 layer of the upper module, that is, the features of the fifth layer. The feature _small5 is superimposed and pooled to estimate the optical flow and restore the resolution of the optical flow image to obtain the calculation result of the first layer. The calculation process is as follows:

feature _Sflow1 = estimate(feature _small +feature _small5 )

Among them, feature _Sflow1 represents the feature output by the first layer of the optical flow estimation module under the small displacement sub-network, and estimate( ) represents the mapping function of this layer;

Then, for the remaining 2-5 layers, the calculation results of the previous layer are added to the input of the next layer. The calculation process is as follows:

feature _Sflowi = estimate(feature _small +feature _small(6-i) +feature _Sflow(i-1) )

Among them, feature _Sflow(i-1) represents the feature output by the i-1th layer of the optical flow estimation module under the small displacement sub-network;

Step S24: Finally, fuse the results obtained by the two sub-networks to obtain the final output result, let fusion( ) represent the final fusion operation, and the calculation process is as follows:

p _fusion = fusion(feature _Bflow4 +feature _Sflow5 )

Using the convolutional neural network to simulate the optical flow estimation results of large displacement and small displacement and fuse them can help to improve the generalization of the model and make it more reasonable to adjust the micro-expression fragments with too large or too small changes. And compared with the traditional method, the implementation of the convolutional neural network reduces the edge blurring problem that may occur in the process of micro-expression optical flow estimation, so that the result of the optical flow estimation is more accurate.

In this embodiment, step S3 specifically includes the following steps:

Step S31: There are multiple subjects under each dataset, each subject represents a subject, and each subject contains multiple micro-expression sequences, representing multiple micro-expression sequences generated by the subject, According to the principle of leave-one-subject-out, when we divide the data set, we take one subject of one data set as the test set at a time, and all the remaining subjects are combined together as a training set, so finally we can get one data set. Sub _i training sets and test sets, where Sub _i represents the number of subjects in a dataset.

Step S32: For the divided test set and training set, we sequentially input them into the residual network for classification, and obtain a preliminary classification result;

In this embodiment, step S4 specifically includes the following steps:

Step S41: For the results of the preliminary classification, there may be situations where the probabilities of some two results of the same graph are very similar, which requires reordering the results;

Step S42: According to the classification results with similar classification probabilities of the tested images, images under the corresponding classification are selected in the training set, and k nearest neighbors are selected according to the selected images. The calculation process is as follows:

where e _i represents the i-th selected training set image, and p represents the test image.

Step S43: Calculate the distance between the tested image p and the selected image e _i , and the calculation process is as follows:

D _i =1-probe _max ( _ei )+probe _max (p)

Among them, probe _max represents the largest probability among the classification result probabilities.

Step S44: For each selected training set image e _i , calculate the Jaccard distance D _j from the tested image p, and the calculation process is as follows:

The final distance result is obtained by weighting D _i and D _j .

The re-sorting method is used to re-classify the micro-expression images, so that the probabilities of certain two categories that may occur in the process of micro-expression recognition are too close to cause misclassification to be greatly reduced, and the accuracy of micro-expression recognition is improved.

The above are the preferred embodiments of the present invention, all changes made according to the technical solutions of the present invention, when the resulting functional effects do not exceed the scope of the technical solutions of the present invention, belong to the protection scope of the present invention.

Claims (3)

1. a micro-expression classification method based on optical flow generation network and reordering, is characterized in that, comprises the steps: Step S1, obtaining a micro-expression data set, extracting a start frame and a peak frame, and performing preprocessing; Step S2, train the optical flow generation network, and generate its optical flow feature according to all the start frames and peak frames; Step S3: Divide the obtained optical flow image into a training set and a test set according to the LOSO principle, and input the residual network for training; Step S4, reordering the classification results obtained by the residual network to obtain a final result with higher accuracy; In the step S2, the optical flow generation network adopts the structure of two sub-networks to perform point-to-point pixel training, and the two sub-networks are level with each other. The displacement is used to estimate the optical flow. Each sub-network is composed of a feature extraction module and an optical flow estimation module. Finally, the optical flow estimation obtained by the two sub-networks is fused to obtain the final optical flow estimation image; S21. For the large-displacement optical flow estimation sub-network, the feature extraction module is composed of nine convolutional layers. The input of the feature extraction module is the superposition of the enlarged input image pairs, and the feature mapping function of the feature extraction module is H. ( ), the calculation process is as follows: feature _big =H(p _ls +p _la ) where p _ls represents the start frame after zooming in, p _la represents the peak frame after zooming in, and feature _big represents the result of feature extraction for large displacement motion; The optical flow estimation module of the large displacement optical flow estimation sub-network is composed of an upper pooling layer and a convolutional layer. The feature value feature _big output by the feature extraction module and the 5-1 layer of the feature extraction module are the fourth layer of the feature extraction module. The feature _big4 of the layer is superimposed, and the upper pool is used to estimate the optical flow and restore the resolution of the optical flow image, and obtain the calculation result of the first layer of the optical flow estimation module. The calculation process is as follows: feature _Bflow1 = estimate(feature _big +feature _big4 ) Among them, feature _Bflow1 represents the output feature of the first layer of the optical flow estimation module under the large displacement sub-network, and estimate( ) represents the mapping function of the first layer of the optical flow estimation module under the large displacement sub-network; Then, for the remaining 2-4 layers of the optical flow estimation module, the calculation results of the previous layer are added to the input of the next layer. The calculation process is as follows: feature _Bflowi = estimate(feature _big +feature _big(5-i) +feature _Bflow(i-1) ) Among them, feature _Bflow(i-1) represents the feature of the output of the i-1th layer of the optical flow estimation module under the large displacement sub-network; S22. For the small-displacement optical flow estimation sub-network, the feature extraction module is composed of nine convolution layers, and the first three convolutions respectively perform features on the input starting frame p _s and peak frame p _a without action amplification. Extraction, the input of the last six convolutions is the superposition of the output results of the first three convolutions on the two image frames, so that the mapping function of the first three convolutions of the feature extraction module is first( ), the mapping function of the last six convolutions The function is last( ), and its calculation process is as follows: feature _small = last(first( _ps )+first(p _a )) Among them, feature _small represents the result of feature extraction for small displacement motion; The optical flow estimation module of the small-displacement optical flow estimation sub-network is composed of an upper pooling layer and a convolutional layer. The feature value feature _small output by the feature extraction module and the 6-1 layer of the feature extraction module are the fifth layer of the feature extraction module. The feature _small5 of the layer is superimposed and pooled to estimate the optical flow and restore the resolution of the optical flow image, and obtain the calculation result of the first layer of the optical flow estimation module. The calculation process is as follows: feature _Sflow1 = estimate(feature _small +feature _small5 ) Among them, feature _Sflow1 represents the feature output by the first layer of the optical flow estimation module under the small displacement sub-network, and estimate( ) represents the mapping function of the first layer of the optical flow estimation module under the small displacement sub-network; Then, for the remaining 2-5 layers of the optical flow estimation module, the calculation results of the previous layer are added to the input of the next layer. The calculation process is as follows: feature _Sflowi = estimate(feature _small +feature _small(6-i) +feature _Sflow(i-1) ) Among them, feature _Sflow(i-1) represents the feature output by the i-1th layer of the optical flow estimation module under the small displacement sub-network; S23, fuse the results obtained by the large displacement optical flow estimation sub-network and the small displacement optical flow estimation sub-network to obtain the final output result, let fusion( ) represent the final fusion operation, and the calculation process is as follows: p _fusion = fusion(feature _Bflow4 +feature _Sflow5 ) The step S4 specifically includes the following steps: Step S41, for the preliminary classification results obtained by the residual network training, there may be situations where the probabilities of two of the results in the same graph are very similar, so the results need to be reordered; Step S42, according to the classification results with similar classification probabilities of the tested images, select images under the corresponding classification in the training set, and select k nearest neighbors according to the selected images, and the calculation process is as follows:

where e _i represents the i-th selected training set image, and p represents the tested image; Step S43, calculate the distance between the tested image p and the selected image e _i , and the calculation process is as follows: D _i =1-probe _max ( _ei )+probe _max (p) Among them, probe _max represents the maximum probability of the classification result probability; Step S44: For each selected training set image e _i , calculate the Jaccard distance D _j from the tested image p, and the calculation process is as follows:

The final distance result is obtained by weighting D _i and D _j . 2. a kind of micro-expression classification method based on optical flow generation network and reordering according to claim 1, is characterized in that, described step S1 specifically comprises the steps: Step S11, obtaining a micro-expression dataset, and trimming the image into a 224*224 size image after face alignment; Step S12, for the micro-expression data set marked with the start frame and the peak frame, directly extract the start frame and the peak frame according to the labeling content, and perform step S15; Step S13, for the micro-expression data set without the start frame and the peak frame label, use the frame difference method to extract the start frame and the peak frame of the video sequence; the frame difference method is as follows: let P={pi _} ,i=1,2,...represents the input image sequence, where p _i represents the ith input image, let the first frame of the sequence be the start frame, that is, p _start =p ₁ , the first frame of the video sequence and the The gray value of the pixel corresponding to the nth frame is denoted as f1(x,y), fn(x,y), the gray value of the corresponding pixel of the two frames of images is subtracted, and the absolute value is taken to obtain the difference image Dn , Dn(x,y)=|fn(x,y)-f1(x,y)|, calculate the average inter-frame difference Dnavg of the difference image, the calculation method is as follows:

Among them, Dn.shape[0] represents the height of the difference image Dn, Dn.shape[1] represents the width of the difference image Dn, calculate the average inter-frame difference of all frames except the start frame and the start frame and sort, The frame with the largest difference between the average frames is the peak frame _papex corresponding to the image sequence; Step S15 is performed after extracting the start frame and the peak frame; Step S15, performing Euler action amplification on the extracted start frame and peak frame, and the calculation process is as follows: I(x,t)=g(x+(1+α)δ(t)) where I(x,t) represents the brightness value of the image at position x and time t, g( ) represents the mapping function of the Euler motion amplification process, and δ(t) represents the motion deviation, which is generated by adjusting the motion amplification coefficient α Enlarged image. 3. a kind of micro-expression classification method based on optical flow generation network and reordering according to claim 1, is characterized in that, described step S3 specifically comprises the steps: Step S31, there are multiple subjects under each optical flow image dataset, each subject represents a subject, and each subject contains multiple micro-expression sequences, representing multiple micro-expression sequences generated by the subject. Expression sequence, according to the principle of leave-one-subject-out, when dividing the data set, one subject of one data set is taken as the test set at a time, and all the remaining subjects are combined as a training set, and the last data set gets Sub _i Training set and test set, where Sub _i represents the number of subjects in a dataset; Step S32 , input the divided test set and training set into the residual network for classification, and obtain a preliminary classification result.

Images