CN117456578A

CN117456578A - Method, device, processor and computer readable storage medium for realizing rapid micro-expression recognition processing based on bidirectional optical flow

Info

Publication number: CN117456578A
Application number: CN202311437097.8A
Authority: CN
Inventors: 费子翔; 张玉坤; 周文举; 费敏锐
Original assignee: University of Shanghai for Science and Technology
Current assignee: University of Shanghai for Science and Technology
Priority date: 2023-10-31
Filing date: 2023-10-31
Publication date: 2024-01-26

Abstract

The invention relates to a method for realizing rapid micro-expression recognition processing based on a bidirectional optical flow, which comprises the following steps: collecting micro-expression video clip information of the face of a tester according to a vision system; extracting emotion video fragments from an emotion memory bank, and capturing facial muscle movement conditions of micro expressions in the emotion video fragments through positive and negative bidirectional optical flows; the extraction method is used for extracting key frames in the emotion video fragments and eliminating redundant frames in continuous sequence images; and retrieving optical flow information between key frames in the optical flow information memory. The invention also relates to a device, a processor and a storage medium for realizing rapid micro-expression recognition in the bidirectional optical flow. The method, the device, the processor and the computer readable storage medium for realizing the rapid micro-expression recognition processing based on the bidirectional optical flow are adopted, the facial muscle movement condition of the micro-expression is captured through the positive and negative bidirectional optical flow, the micro-expression of a tester is recognized by utilizing the muscle movement trend, and the micro-expression recognition accuracy is improved.

Description

Method, device, processor and computer readable storage medium for realizing rapid micro-expression recognition processing based on bidirectional optical flow

Technical Field

The invention relates to the lie detection field, in particular to the technical field of microexpressive recognition, and specifically relates to a method, a device, a processor and a computer readable storage medium for realizing rapid microexpressive recognition processing based on bidirectional optical flow.

Background

The micro-expression is a facial expression which is generated by stimulating the true emotion of the heart and can accurately reflect the true emotion of the heart. Standard micro-expressions last between 1/5 second and 1/25 second, usually only occur at specific locations on the face, and the intensity of the motion is extremely weak, being difficult to perceive by the naked eye only. In the past, the analysis of the micro-expression needs to be performed by a quite experienced professional for a long time and careful observation. In recent years, with the development of artificial intelligence technology and the continuous breakthrough of computer vision and pattern recognition technology in the field of microexpressive analysis, intelligent real-time rapid microexpressive recognition becomes a challenging and valuable research field.

In the prior art, the existing mature micro-expression recognition methods are mainly divided into two types, namely a micro-expression recognition method based on facial feature points and a micro-expression recognition method based on facial texture features. The former is mainly identified by using local feature points of the face, and the latter is mainly identified according to the change trend of the texture features of the face. The micro-expression recognition method based on the feature points provides a complete realization framework for micro-expression recognition research, however, the micro-expression recognition method is easily influenced by the selection of the feature points and has strong limitation. The micro-expression recognition method based on the facial texture features can capture the micro-variation of the face through the global features of the face and recognize the micro-expression through the running state of the facial texture, but the prior art is difficult to accurately extract the texture running state information of the face, and the accuracy of the micro-expression recognition is affected.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provide a method, a device, a processor and a computer readable storage medium thereof for realizing rapid micro-expression recognition processing based on bidirectional optical flow, which have the advantages of high accuracy, simple and convenient operation and wider application range.

In order to achieve the above object, a method, an apparatus, a processor and a computer readable storage medium thereof for implementing a fast micro-expression recognition process based on a bidirectional optical flow according to the present invention are as follows:

the method for realizing rapid micro-expression recognition processing based on the bidirectional optical flow is mainly characterized by comprising the following steps of:

(1) Collecting micro-expression video clip information of the face of a tester according to a vision system, and storing the collected video clip information into an emotion memory bank;

(2) Extracting emotion video fragments in an emotion memory bank, capturing facial muscle movement conditions of micro expressions in the emotion video fragments through positive and negative bidirectional optical flows, extracting optical flow information through positive indexes, checking sequence optical flows through reverse indexes, and inhibiting error distance and direction information; the emotion video clips are characterized through the facial optical flow information, and the extracted two-way optical flow information is stored in an optical flow information memory bank; evaluating the accuracy of optical flow information in an optical flow information base through a coordinate error evaluation rule;

(3) Extracting key frames in the emotion video fragments by a key frame extraction method based on an optical flow field mutation effect, and eliminating redundant frames in continuous sequence images;

(4) Retrieving optical flow information between key frames in an optical flow information memory library, and storing the optical flow information between the key frames into a perception template library; the texture running state of the facial micro-expression is characterized by optical flow information, and the optical flow information among key frames is used as the input of a support vector machine to classify the facial emotion condition of the tester.

Preferably, the step (1) specifically includes:

and acquiring micro-expression video clip information of the face of the tester according to the vision system, converting the acquired video clip information into a continuous image sequence, performing de-distortion treatment on the obtained continuous image by adopting a Gaussian smoothing filter, and processing the micro-expression video clip information by the vision system into an emotion memory bank.

Preferably, the step (2) of extracting the emotion video segment in the emotion memory bank captures the facial muscle movement condition of the micro expression in the emotion video segment through the positive and negative bidirectional optical flow, specifically:

when the reverse optical flow is carried out, the forward frame spacing and the reverse frame spacing time are the same, and the forward and reverse optical flows are solved according to the following formula:

Where n represents the nth frame image of the sequence of consecutive images,optical flow vector along x-axis for the n+1th to n-th images in reverse order, +.>Optical flow vector along x-axis for positive sequence nth frame to n+1th frame image, +.>Optical flow vector along y-axis for the n+1th to n-th images in reverse order, +.>Optical flow vectors along the y-axis for the n-th to n + 1-th images of the positive sequence,for the velocity component along the x-axis of the n+1th to n-th frame images in reverse order, +.>Velocity component along x-axis for positive sequence n-th to n+1-th frame images,/>For the velocity components along the y-axis of the n +1 to n-th frame images in reverse order,velocity component along y-axis for positive sequence nth frame to n+1th frame image, +.>Is the gray value of the nth frame image, t ⁿ Is the time of the nth frame image.

Preferably, the forward index in the step (2) extracts optical flow information, and the reverse index collates sequential optical flow to suppress error distance and direction information, specifically:

assuming that the optical flow velocity components resolved in the forward and reverse directions are the same, i.e Then

Constructing a linear method set by a plurality of optical flow information on a single frame image, and solving the constructed linear equation set by a singular value decomposition method to obtainAnd->

Wherein,for the velocity component along the x-axis of the n+1th to n-th frame images in reverse order, +. >Optical flow vector along x-axis for the n+1th to n-th images in reverse order, +.>Optical flow vector along x-axis for positive sequence nth frame to n+1th frame image, +.>Velocity component along y-axis for positive sequence nth frame to n+1th frame image, +.>Optical flow vector along y-axis for the n+1th to n-th images in reverse order, +.>Optical flow vector along y-axis for n-th to n+1-th frame images of positive order, +.>Is the gray value of the nth frame image, t ⁿ Is the time of the nth frame image.

Preferably, in the step (2), the accuracy of the optical flow information in the optical flow information base is evaluated by a coordinate error evaluation rule, which specifically includes:

the offset between the optical flow calculated value and the optical flow true value is evaluated by adopting a coordinate error evaluation rule, and the offset between the optical flow calculated value and the optical flow true value is evaluated according to the following formula:

wherein I is _p (x, y, n) and I _t (x, y, n) represents the predicted optical flow and the true optical flow at the nth frame image coordinates (x, y), which are a two-dimensional vector, W and H are the width and height of the optical flow field, respectively.

Preferably, the step (3) specifically includes the following steps:

acquiring continuous image frame data by taking a time axis as a coordinate, determining a first frame image as a first key frame, storing the first key frame image into a key frame library, calculating vector change information (delta u, delta v) of optical flow between a current key frame and a next ordinary frame, calculating and checking optical flow change quantity, checking whether the optical flow change quantity reaches a set threshold value, and setting the current ordinary frame as an important key frame if the optical flow change quantity reaches the set threshold value;

The calculation and inspection optical flow variable quantity is specifically as follows:

calculating the test optical flow variation according to the following formula:

kf _j ＝{f _i |([Δu>τ]∪[Δv>τ])}；

wherein f _i Representing a normal frame, kf _j Representing a key frame, deltau is the cumulative sum of the distance variations of the optical flow along the x-axis, deltav is the cumulative sum of the distance variations of the optical flow along the y-axis,and->Distance variation along x-axis and y-axis of optical flow at (m, n) position on normal frame, +.>And->The distance variation along the x-axis and the y-axis of the optical flow of the key frame at the (m, n) position is l, the number of the optical flow information on the single frame image is l, and τ is a constant value for setting a threshold value for the optical flow variation.

Preferably, in the step (4), optical flow information between key frames is stored in a perception template library, and the method specifically includes the following steps:

if deltau or deltav between the current common frame and the last key frame is larger than the set optical flow change threshold tau, setting the current frame as a new key frame and storing the new key frame in a key frame library, otherwise, if deltau and deltav between the current common frame and the last key frame are smaller than the set optical flow change threshold tau, the similarity between the current common frame and the last key frame is higher, and the last key frame represents the current common frame; discarding the current common frame, comparing the common frame of the next frame with the key frame of the previous frame, and checking whether the key frame meets the requirement; when the key frame is successfully selected, storing the current key frame and the corresponding optical flow field information into a key frame library;

Store in the keyframe library according to the following formula:

U＝{(F ₁ ,kf ₁ ,t ₁ ),(F _2, kf ₂ ,t ₂ ),(F ₃ ,kf ₃ ,t ₃ )…(F _j ,kf _j ,t _j )}

wherein U is a corresponding key frame library, F _j To correspond to kf _j Optical flow field information of frame, t _j Indicating the recording time of the j-th key frame.

Preferably, in the step (4), the texture running state of the facial micro-expression is characterized by optical flow information, and the optical flow information between key frames is used as the input of a support vector machine to classify the facial emotion condition of the tester, specifically:

taking information in a perception template library as input of a support vector machine, carrying out dimension reduction on data by optical flow information in the perception template library through a principal component analysis method, inputting feature vectors after dimension reduction into the support vector machine, and constructing and solving a constraint optimization problem; determining through optimization in the training process, obtaining an optimal solution and obtaining a separation hyperplane; and identifying the micro-expression by using a classification decision function constructed by the separation hyperplane.

The device for realizing the rapid micro-expression recognition processing based on the bidirectional optical flow is mainly characterized by comprising the following components:

a processor configured to execute computer-executable instructions;

and the memory stores one or more computer executable instructions which, when executed by the processor, implement the steps of the method for implementing the rapid micro-expression recognition processing based on the bidirectional optical flow.

The processor for realizing the rapid micro-expression recognition processing based on the bidirectional optical flow is mainly characterized in that the processor is configured to execute computer executable instructions, and when the computer executable instructions are executed by the processor, the steps of the method for realizing the rapid micro-expression recognition processing based on the bidirectional optical flow are realized.

The computer readable storage medium is characterized in that the computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method for implementing the rapid micro-expression recognition processing based on the bidirectional optical flow.

The invention discloses a method, a device, a processor and a computer readable storage medium for realizing rapid micro-expression recognition processing based on bidirectional optical flow, which provide a bidirectional optical flow tracking strategy model, capture facial muscle movement conditions of micro-expressions through positive and negative bidirectional optical flow, recognize micro-expressions of a tester by utilizing muscle movement trend, and improve micro-expression recognition accuracy. Aiming at the problem that the traditional optical flow method is easy to have errors of distance and angle during extraction, the invention provides a reverse optical flow tracking method for correcting forward optical flow, inhibiting error distance and direction information, improving the accuracy of optical flow information and better acquiring facial texture operation information. Aiming at the problems that the information quantity of video data is large, the similarity between continuous image frames is high, redundant frames are more, and system computing resources are occupied, the invention provides a key frame extraction method based on the optical flow field mutation effect, which extracts effective image frames in video and improves the real-time performance of a system. The invention adopts a coordinate error evaluation rule to evaluate the offset between the optical flow calculated value and the optical flow true value. The coordinate error evaluation rule can take both distance errors and angle errors into account when evaluating errors. The overall error and the angle error level of the optical flow in the optical flow field can be reflected. The invention can characterize the texture running state information of the facial micro-expression through the optical flow information, and classify the facial emotion condition of the tester by taking the optical flow information among key frames as the input of a support vector machine. The invention constructs a new micro-expression recognition algorithm framework, and the method not only improves the accuracy of the system, but also improves the real-time performance of the system. The method can accurately and rapidly identify the facial micro-expression of the tester.

Drawings

Fig. 1 is a flow chart of a method for implementing rapid micro-expression recognition processing based on a bidirectional optical flow.

Fig. 2 is a schematic diagram of a bidirectional optical flow tracking strategy of the method for implementing rapid micro-expression recognition processing based on bidirectional optical flow according to the present invention.

Fig. 3 is a schematic diagram of a key frame selection process of the method for implementing rapid micro-expression recognition processing based on a bidirectional optical flow.

Fig. 4 is a flowchart of a key frame selection strategy of the method for implementing the rapid micro-expression recognition processing based on the bidirectional optical flow.

Fig. 5 is an effect diagram and a partial enlargement of the LK algorithm, the FlowNet2 algorithm, and the algorithms herein extraction under the SMIC dataset.

Fig. 6 is a graph of the feature extraction effect of the LBP algorithm, LK algorithm, flowNet2 algorithm, and the algorithms herein under the CASME2 dataset.

FIG. 7 is a graph of a recognition confusion matrix under CASME2 dataset for a comparison algorithm.

Fig. 8 is a graph of identification times of LBP-TOP algorithm, LK algorithm, flowNet2 algorithm, and the algorithms herein under CASMEII dataset.

Detailed Description

In order to more clearly describe the technical contents of the present invention, a further description will be made below in connection with specific embodiments.

The method for realizing rapid micro-expression recognition processing based on the bidirectional optical flow comprises the following steps:

As a preferred embodiment of the present invention, the step (1) specifically includes:

As a preferred embodiment of the present invention, the step (2) of extracting the emotion video segment in the emotion memory bank captures the facial muscle movement condition of the micro-expression in the emotion video segment by the positive and negative bidirectional optical flow, specifically:

where n represents the nth frame image of the sequence of consecutive images,optical flow vector along x-axis for the n+1th to n-th images in reverse order, +.>Optical flow vector along x-axis for positive sequence nth frame to n+1th frame image, +.>Optical flow vector along y-axis for the n+1th to n-th images in reverse order, +.>Optical flow vectors along the y-axis for the n-th to n + 1-th images of the positive sequence,for the velocity component along the x-axis of the n+1th to n-th frame images in reverse order, +. >Velocity component along x-axis for positive sequence n-th to n+1-th frame images,/>For the velocity component along the y-axis of the n+1th to n-th frame images in reverse order, +.>Velocity component along y-axis for positive sequence nth frame to n+1th frame image, +.>Is the gray value of the nth frame image, t ⁿ Is the time of the nth frame image.

As a preferred embodiment of the present invention, the forward index in the step (2) extracts optical flow information, and the reverse index collates sequential optical flow, and suppresses erroneous distance and direction information, specifically:

Wherein,for the velocity component along the x-axis of the n+1th to n-th frame images in reverse order, +.>Optical flow vector along x-axis for the n+1th to n-th images in reverse order, +.>Optical flow vector along x-axis for positive sequence nth frame to n+1th frame image, +.>Velocity component along y-axis for positive sequence nth frame to n+1th frame image, +.>Optical flow vector along y-axis for the n+1th to n-th images in reverse order, +.>Optical flow vector along y-axis for n-th to n+1-th frame images of positive order, +. >Is the gray value of the nth frame image, t ⁿ Is the time of the nth frame image.

As a preferred embodiment of the present invention, the evaluating accuracy of the optical flow information in the optical flow information base by the coordinate error evaluation rule in the step (2) specifically includes:

As a preferred embodiment of the present invention, the step (3) specifically includes the following steps:

Calculating the test optical flow variation according to the following formula:

kf _j ＝{f _i |([Δu>τ]∪[Δv>τ])}；

As a preferred embodiment of the present invention, the storing the optical flow information between the key frames in the step (4) into the perception template library specifically includes the following steps:

Store in the keyframe library according to the following formula:

As a preferred embodiment of the present invention, in the step (4), the texture running state of the facial micro-expression is characterized by optical flow information, and the optical flow information between key frames is used as the input of a support vector machine to classify the facial emotion condition of the tester, specifically:

The device for realizing the rapid micro-expression recognition processing based on the bidirectional optical flow comprises:

a processor configured to execute computer-executable instructions;

The processor for implementing the fast micro-expression recognition processing based on the bidirectional optical flow according to the invention, wherein the processor is configured to execute computer executable instructions, and the computer executable instructions implement the steps of the method for implementing the fast micro-expression recognition processing based on the bidirectional optical flow when being executed by the processor.

The computer readable storage medium of the present invention has a computer program stored thereon, the computer program being executable by a processor to perform the steps of the method for implementing a fast micro-expression recognition process based on a bi-directional optical flow as described above.

Considering that the micro-expression motion strength is weaker, the motion has locality, and the existing algorithm is difficult to quickly and accurately identify. The invention provides a rapid micro-expression recognition method based on bidirectional optical flow, which can recognize micro-expressions by capturing facial movement conditions through positive and negative bidirectional optical flow, and improves the accuracy of micro-expression recognition. And a key frame extraction method based on an optical flow field effect is introduced, redundant frames in video clips are removed, and the real-time performance of the system is improved. The invention realizes the key frame extraction of the emotion video fragment and the capturing of the running state of the facial texture, and improves the real-time performance and accuracy of facial micro-expression recognition.

In a specific embodiment of the present invention, the following two examples are provided:

as shown in fig. 1-8, the invention provides a mobile robot map construction method based on closed loop detection correction, which comprises the following specific steps:

step S1, acquiring micro-expression video clip information of the face of a tester according to a vision system, and storing the acquired video clip information into an emotion memory bank;

the method specifically comprises the following steps: and (2) acquiring micro-expression video clip information of the face of the tester according to the vision system, converting the acquired video clip information into a continuous image sequence, and performing de-distortion treatment on the continuous images obtained in the step (S1) by adopting a Gaussian smoothing filter so as to eliminate graphic distortion caused by external environment change. According to the working principle of the vision system, the vision system collects the micro-expression video clip information and processes the micro-expression video clip information to be stored in the emotion memory bank.

In step S2, the emotion video segments in the emotion memory bank are extracted, the facial muscle movement condition of the micro-expressions in the emotion video segments is captured through positive and negative bidirectional optical flows, the optical flow information is extracted through positive indexes, the sequential optical flows are calibrated through reverse indexes, and the error distance and direction information is restrained. The concrete mode is as follows:

assuming that the gray value at a pixel point (x, y) in the image at the moment t is I (x, y, t); at the point (x+dx, y+dy, t+dt) where the pixel runs to point (x+dx, y+dy, t+dt), the optical flow tracking satisfies dI (x, y, t)/dt=0 according to the gray level invariant assumption theorem, i.e. when the image time interval is short, the gray level value in the image remains unchanged, and according to the image pixel gray level conservation principle, the optical flow tracking can be expressed as:

I(x,y,t)＝I(x+dx,y+dy,t+dt)

Assuming that the amount of motion is small, the right side of the equation is developed according to the taylor formula:

where τ is a higher order infinitesimal, neglecting the infinitesimal term in equation 2, and substituting equation 2 into equation 1 yields:

wherein,and->The optical flow vectors of I (x, y, t) have horizontal components x and vertical components y, respectively.

Respectively usingTo represent the horizontal component x and the vertical component y of the optical flow vector of I (x, y, t). Will G _x ,G _y Substitution into

Equation 2 and taylor expansion, ignoring higher-order infinite terms above the second order, can be obtained:

wherein,is the gray value of the nth frame image, t ⁿ Is the time of the nth frame image.

In the case of reverse optical flow, the forward inter-frame distance and the reverse inter-frame distance are the same, and the method for solving the forward optical flow and the reverse optical flow is as follows:

where n represents the nth frame image of the sequence of consecutive images,optical flow vector along x-axis for the n+1th to n-th images in reverse order, +.>Optical flow vector along x-axis for positive sequence nth frame to n+1th frame image, +.>Optical flow vector along y-axis for the n+1th to n-th images in reverse order, +.>Optical flow vectors along the y-axis for the n-th to n + 1-th images of the positive sequence,for the velocity component along the x-axis of the n+1th to n-th frame images in reverse order, +.>Velocity component along x-axis for positive sequence n-th to n+1-th frame images,/ >For the velocity component along the y-axis of the n+1th to n-th frame images in reverse order, +.>Velocity component along y-axis for positive sequence nth frame to n+1th frame image, +.>Is the gray value of the nth frame image, t ⁿ Is the time of the nth frame image.

The method comprises the steps of constructing a linear method set through a plurality of optical flow information on a single frame image, and solving the constructed linear equation set by utilizing a singular value decomposition method to obtainAnd->

The design scheme of the invention can correct the forward optical flow through reverse optical flow tracking, inhibit the error distance and direction information, improve the accuracy of optical flow information and better capture the muscle movement condition of the micro-expression of the face.

Preferably, in the step S2, the accuracy of the optical flow information in the optical flow information base is evaluated by the coordinate error evaluation rule constructed by the present patent. The method comprises the following steps:

the accuracy of the optical flow is affected by the distance error and the angle error, and the coordinate error evaluation rule is adopted to evaluate the offset between the optical flow calculated value and the optical flow true value. The coordinate error evaluation rule can take both distance errors and angle errors into account when evaluating errors. The integral error and the angle error level of the optical flow in the optical flow field can be reflected, and the calculation formula is as follows:

Wherein I is _p (x, y, n) and I _t (x, y, n) represents the predicted optical flow and the true optical flow at the n-th frame image coordinates (x, y), and is a two-dimensional vector. W and H are the width and height of the optical flow field, respectively.

In step S3, weight learning is performed according to the self-sensing anti-mapping network learning rule, and a response value of the grid cell is obtained, which is specifically as follows:

the key frames in the emotion video fragments are extracted by a key frame extraction method based on the optical flow field mutation effect, redundant frames in continuous sequence images are removed, and the key frames are obtained by the following steps:

aiming at the problems that the information quantity of video data is large, the similarity between continuous image frames is high, redundant frames are more, and system computing resources are occupied, the key frame extraction method based on the optical flow field mutation effect is provided, effective image frames in the video are extracted, the system instantaneity is improved, and a key frame selection flow diagram is shown in figure 3. The method comprises the steps of acquiring continuous image frame data by taking a time axis as a coordinate, firstly determining a first frame image as a first key frame, storing the first key frame image into a key frame library, calculating vector change information (delta u, delta v) of optical flow between a current key frame and a next ordinary frame, checking whether the optical flow change amount reaches a set threshold value, and if the optical flow change amount reaches the set threshold value, setting the current ordinary frame as an important key frame, wherein the calculation mode of the optical flow change amount is as follows:

kf _j ＝{f _i |([Δu>τ]∪[Δv>τ])}

Wherein f _i Representing a normal frame, kf _j Representing a key frame, deltau is the cumulative sum of the distance variations of the optical flow along the x-axis, deltav is the cumulative sum of the distance variations of the optical flow along the y-axis,and->Distance variation along x-axis and y-axis of optical flow at (m, n) position on normal frame, +.>And->For key frames at (m, n) The distance variation of the optical flow of the position along the x axis and the y axis is l, and the number of the optical flow information on the single frame image is shown. The method is characterized in that tau is a constant value, the threshold value is set to be 8.5, if the threshold value is set to be smaller, more similar frame images exist in the extracted key frames, more redundant frames are generated, the instantaneity of the micro-expression recognition system is affected, and if the threshold value is set to be larger, ordinary frames of the key frame accessories are difficult to characterize by some key frames, and the micro-expression recognition accuracy is affected.

In step S4, the optical flow information between the key frames in the optical flow information memory is retrieved, and the optical flow information between the key frames is stored in the perception template library. The texture running state information of the facial micro-expressions is characterized by optical flow information, and the optical flow information among key frames is used as the input of a support vector machine to classify the facial emotion condition of the tester. The step of storing the optical flow information between the key frames in the perception template library is specifically as follows:

If Deltau or Deltav between the current common frame and the last key frame is larger than the set optical flow change threshold tau, the current frame is set as a new key frame and stored in a key frame library, otherwise, if Deltau and Deltav between the current common frame and the last key frame are smaller than the set optical flow change threshold tau, the similarity between the current common frame and the last key frame is higher, the last key frame can represent the current common frame, the current common frame is abandoned for saving the calculated amount, and the next common frame and the last key frame are compared, so that whether the key frame requirements are met is checked. When the key frame is successfully selected, the current key frame and the corresponding optical flow field information are stored in a key frame library. The storage mode is shown in the following formula, and a key frame extraction strategy flow chart is shown in fig. 4.

Wherein U is a corresponding key frame library, F _j To correspond to kf _j Optical flow field information of frame, t _j Represent the firstRecording time of j key frames.

The design scheme of the invention can extract effective image frames in video by a key frame extraction method based on the optical flow field mutation effect aiming at the large information quantity of video data and high similarity between continuous image frames, and eliminates redundant information in a system, thereby improving the real-time performance of the system.

In the step S4, the texture running state information of the facial micro-expression is represented by the optical flow information, and the optical flow information between key frames is used as the input of the support vector machine to classify the facial emotion condition of the tester. The method comprises the following steps: the information in the perception template library is used as the input of a support vector machine, the optical flow information in the perception template library is subjected to dimension reduction on data through a principal component analysis method, and the feature vectors after dimension reduction are input into the support vector machine to construct and solve a constraint optimization problem. And determining through optimization in the training process, obtaining an optimal solution and acquiring a separation hyperplane. And finally, identifying the micro-expression by using a classification decision function constructed by the separation hyperplane.

The invention constructs a new micro-expression recognition algorithm framework, which not only improves the accuracy of the system, but also improves the real-time performance of the system. The method can accurately and rapidly identify the facial micro-expression of the tester.

The rapid micro-expression recognition method based on the bidirectional optical flow is described below with reference to a specific experiment, and the process will be described below with reference to the specific experiment.

Computer configuration for use in the experiments herein: the CPU is an i5-9400F, 8-core processor, the main frequency is 2.9GHz, and the memory is 8GB. The accuracy of microexpressive recognition was verified using SMIC, CASME, and CASMEII microexpressive datasets, respectively.

FIG. 5 is the LK algorithm, flowNet2 algorithm, and acquired optical flow field information on SMIC data for the algorithms herein. The image micro-expression is positive micro-expression, the mouth is slightly closed, the chin is slightly lifted upwards, and other parts of the face do not have larger texture motions, so that the current scene or the current event is not satisfied. The diagrams (a), (e) and (i) are global light flow diagrams obtained by an LK algorithm, a FlowNet2 algorithm and the proposed algorithm, and the area A, the area B and the area C of the right three columns are partial enlarged diagrams of the LK algorithm, (e) the FlowNet2 algorithm and (i) the proposed algorithm respectively. The area a is an enlarged view of the vicinity of the mouth, the area B is an enlarged view of the vicinity of the eyebrow, and the area C is an enlarged area of the background attachment. It can be seen from the graph (b) that the conventional LK algorithm fails to capture the texture movement of the mouth well at the mouth part, and at the same time, in the graph (c), there is more chaotic light flow information around the eyes, and in the graph (d), there is an enlarged view of the background, the area is not changed in movement, however, the LK algorithm extracts the wrong homogeneous light flow information, and the background is mistakenly considered to have smaller amplitude movement. Compared with the traditional LK algorithm, the algorithm has better texture running track extraction capability, and as can be seen from the graphs (j), (k) and (l), the algorithm can better capture texture movement at the positions of the mouth and the chin, and has no disordered wrong optical flow information at the positions of the eyebrows and the background of the region C, because the wrong optical flow of the sequence is corrected by introducing reverse optical flow. The deep learning-based FlowNet2 network has better texture extraction capability in the mouth and chin, but has more wrong optical flow information in the B region and the C region.

Fig. 6 is a feature extraction diagram of LBP algorithm, LK algorithm, flowNet2 algorithm, and herein algorithm on CASME2 data, wherein four rows of pictures from top to bottom are sequentially selected from the sequence of ep01_1f, ep04_02f, ep02_31, ep06_01 in the CASMEII dataset. The LBP characteristic can be used for representing the texture running condition of the micro expression through the distance and the angle of the optical flow movement, however, the LK algorithm still has more wrong optical flow information in a static area (such as the background of a rear wall) when capturing the texture running condition, while the FlowNet2 optical flow method has better results compared with the LK algorithm, but in the EP04_02f sequence, more wrong information still exists when capturing the optical flow information of the eyebrow part, and the algorithm is improved, so that the feature extraction capability is better on each sequence of a CASME data set.

Table 1 shows the comparison of various algorithms under SMIC, CASME and CASMEII data sets, from which it can be derived that the UAR (average accuracy) and UF1 on the three data sets are highest for the method presented herein, the URA and UF1 on the CASME and CASMEII data sets for the algorithms presented herein are slightly lower than on the SMIC data sets due to the classification of the three expressions positive, negative and normal on the SMIC data sets, whereas classification on the CASME and CASMEII data sets is five, resulting in a lower degree of confusion of classification, but the algorithms presented herein perform still optimally on the SMIC and CASMEII data sets. The algorithm provided by the method constructs a bidirectional optical flow tracking model, so that error optical flow information generated by a forward index can be repaired through reverse index tracking, the motion texture information of the facial micro-expression is better extracted, and the recognition accuracy of the micro-expression is improved. It can be seen from the table that the process presented herein is improved by about 11.2% in UAR and about 10.6% in UF1 compared to BDCNN process. It can be seen that the algorithm has better micro-expression recognition capability.

Table 2 = comparison table of results of various algorithms under SMIC, CASME and CASMEII dataset

The recognition confusion matrix diagram of various algorithms on the CASME2 data set is shown in fig. 7, the columns of the matrix represent the prediction category, the rows represent the prediction result of the category, and the diagonal lines of the matrix represent the proportion of the emotion to the number of correctly classified emotion. Wherein hap.=dig.=sur.=rep.=oth..represent happiness, aversion, surprise, sadness and other microexpressions, respectively. As can be seen from the figure, the traditional LBP algorithm and the traditional LBP-TOP algorithm are poor in recognition, because the two algorithms can extract the texture condition of the micro expression, but cannot represent the motion condition of the micro expression, and the LK algorithm can represent the texture running condition of the micro expression, but can extract more wrong optical flow information to influence the recognition accuracy, so that the recognition accuracy of the micro expression recognition algorithm based on the LK mode is low. The spareMDMO algorithm divides the human face into 36 regions of interest through 66 facial feature points of the human face, and recognizes micro-expressions according to the main direction optical flow features of each region of interest, so that the influence of facial redundancy features on the micro-expressions is reduced, and the recognition accuracy is improved. The FlowNet, the STSTNet and the RCA-N are deep learning-based micro-expression recognition methods, and the methods have higher micro-expression recognition rate, especially the RCA-N algorithm, the recognition rate of the happy expression can reach 0.86, however, the deep learning-based micro-expression recognition method has poor real-time performance, real-time online operation is difficult to realize, and the recognition rate of certain single expression is poor, for example, the recognition rate of the RCA-N algorithm in the averse expression is only 0.57, the recognition rate of the FlowNet2 algorithm in the happy expression is only 0.58, and the recognition rate is poor. The algorithm has the advantages that the real-time performance of the algorithm is improved, the accuracy of the algorithm is improved through the bidirectional optical flow tracking model and the key frame selection strategy, and compared with other algorithms, the algorithm has higher recognition rate in the recognition accuracy of single expression, wherein the micro expression with the highest recognition rate is sad, and the recognition rate reaches 0.87.

Fig. 8 is a graph showing the recognition time of LBP algorithm, LK algorithm, flowNet2 algorithm and the present algorithm under the CASMEII dataset, wherein the present algorithm reduces redundant frames due to the introduction of a key frame selection strategy based on bidirectional optical flow, so that the present algorithm is reduced in recognition time by about 25.9% compared with the micro-expression recognition algorithm based on LK algorithm, and by about 36.5% compared with the micro-expression recognition algorithm based on FlowNet2 network.

From the above experiments, it can be seen that: the method reduces the error of optical flow capturing and improves the accuracy of micro expression recognition by introducing a bidirectional optical flow strategy model. And the algorithm provides a key frame selection strategy model, reduces redundant frames in video clips and improves the real-time performance of the system. In terms of accuracy, the method is improved by about 12.5% compared with a spark=MDMO algorithm, and in terms of operation speed, the algorithm is reduced by about 36.5% compared with a micro-expression recognition algorithm based on FlowNet2, so that the method has good micro-expression recognition capability.

Embodiment two:

in accordance with a first aspect of the present invention, a second aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, the program when executed by a processor performing the steps of:

Step S1: collecting micro-expression video clip information of the face of a tester according to a vision system, and storing the collected video clip information into an emotion memory bank;

step S2: extracting emotion video fragments from an emotion memory bank, and extracting optical flow information of the emotion video fragments by using a bidirectional optical flow method constructed by the patent;

step S3: extracting key frames in the emotion video fragments by a key frame extraction method based on an optical flow field mutation effect, and eliminating redundant frames in continuous sequence images;

step S4: and calling bidirectional optical flow information among the key frames, describing the texture running state of the facial micro-expression through the bidirectional optical flow vector information, and classifying the facial emotion condition of the tester by taking the optical flow among the key frames as the input of a support vector machine.

The storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), an optical disk, or other various media capable of storing program codes.

The specific limitation concerning the implementation steps after the program execution in the computer readable storage medium is referred to as the first embodiment, and will not be described in detail herein.

Example III

In accordance with a first aspect of the present invention, a third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the program:

The above specific limitation concerning the implementation steps of the computer device may be referred to as embodiment one, and will not be described in detail herein.

It is to be understood that the same or similar parts in the above embodiments may be referred to each other, and that in some embodiments, the same or similar parts in other embodiments may be referred to.

It should be noted that in the description of the present invention, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Furthermore, in the description of the present invention, unless otherwise indicated, the meaning of "plurality" means at least two.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.

It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

Those of ordinary skill in the art will appreciate that all or part of the steps carried out in the method of the above embodiments may be implemented by a program to instruct related hardware, and the corresponding program may be stored in a computer readable storage medium, where the program when executed includes one or a combination of the steps of the method embodiments.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented as software functional modules and sold or used as a stand-alone product.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

In this specification, the invention has been described with reference to specific embodiments thereof. It will be apparent, however, that various modifications and changes may be made without departing from the spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. The method for realizing the rapid micro-expression recognition processing based on the bidirectional optical flow is characterized by comprising the following steps of:

2. The method for implementing rapid micro-expression recognition processing based on bidirectional optical flow according to claim 1, wherein the step (1) specifically comprises:

3. The method for realizing rapid micro-expression recognition processing based on bidirectional optical flow according to claim 1, wherein the extracting of the emotion video segments in the emotion memory bank in the step (2) captures the facial muscle movement condition of the micro-expression in the emotion video segments through the forward and reverse bidirectional optical flow, specifically comprises:

where n represents the nth frame image of the sequence of consecutive images,optical flow vector along x-axis for the n+1th to n-th images in reverse order, +.>Optical flow vector along x-axis for positive sequence nth frame to n+1th frame image, +.>Optical flow vector along y-axis for the n+1th to n-th images in reverse order, +.>Optical flow vectors along the y-axis for the n-th to n + 1-th images of the positive sequence,for the velocity component along the x-axis of the n+1th to n-th frame images in reverse order, +.>Velocity component along x-axis for positive sequence n-th to n+1-th frame images,/>For the velocity component along the y-axis of the n+1th to n-th frame images in reverse order, +.>Velocity component along y-axis for positive sequence nth frame to n+1th frame image, +.>Is the gray value of the nth frame image, t ⁿ Is the time of the nth frame image.

4. The method for implementing rapid micro-expression recognition processing based on bidirectional optical flow according to claim 1, wherein the forward index in the step (2) extracts optical flow information, and the reverse index collates sequential optical flow, and suppresses error distance and direction information, specifically includes:

Wherein,for the velocity component along the x-axis of the n+1th to n-th frame images in reverse order, +.>Optical flow vector along x-axis for the n+1th to n-th images in reverse order, +.>Optical flow vectors along the x-axis for the n-th to n + 1-th images of the positive sequence,velocity component along y-axis for positive sequence nth frame to n+1th frame image, +.>Optical flow vector along y-axis for the n+1th to n-th images in reverse order, +.>Optical flow vector along y-axis for n-th to n+1-th frame images of positive order, +.>Is the gray value of the nth frame image, t ⁿ Is the time of the nth frame image.

5. The method for implementing rapid micro-expression recognition processing based on bidirectional optical flow according to claim 1, wherein the evaluating accuracy of optical flow information in the optical flow information base by the coordinate error evaluation rule in the step (2) is specifically as follows:

wherein I is _p (x, y, n) and I _t (x, y, n) represents the coordinates (x, y) of the nth frame image The predicted optical flow and the real optical flow are two-dimensional vectors, and W and H are the width and height of the optical flow field respectively.

6. The method for implementing rapid micro-expression recognition processing based on bidirectional optical flow according to claim 1, wherein the step (3) specifically comprises the following steps:

calculating the test optical flow variation according to the following formula:

kf _j ＝{f _i ([Δu＞τ]∪[Δu＞τ])}；

7. The method for implementing rapid micro-expression recognition processing based on bi-directional optical flow according to claim 1, wherein in the step (4), optical flow information between key frames is stored in a perception template library, and specifically comprises the following steps:

store in the keyframe library according to the following formula:

U＝{(F ₁ ，kf ₁ ，t ₁ )，(F ₂ ，kf ₂ ，t ₂ )，(F ₃ ，kf ₃ ，t ₃ )…(F _j ，kf _j ,t _j )}

8. The method for implementing rapid micro-expression recognition processing based on bidirectional optical flow according to claim 1, wherein the step (4) is characterized in that the texture running state of the facial micro-expression is characterized by optical flow information, and the optical flow information between key frames is used as the input of a support vector machine to classify the facial emotion condition of the tester, specifically:

9. An apparatus for implementing a fast micro-expression recognition process based on bidirectional optical flow, the apparatus comprising:

a processor configured to execute computer-executable instructions;

a memory storing one or more computer-executable instructions which, when executed by the processor, perform the steps of the method of any one of claims 1 to 8 for performing a fast micro-expression recognition process based on bi-directional optical flow.

10. A processor for implementing a bi-directional optical flow based rapid micro-expression recognition process, wherein the processor is configured to execute computer-executable instructions that, when executed by the processor, implement the steps of the bi-directional optical flow based rapid micro-expression recognition process method of any one of claims 1 to 8.

11. A computer-readable storage medium, having stored thereon a computer program executable by a processor to perform the steps of the method of any one of claims 1 to 8 for implementing a fast micro-expression recognition process based on bi-directional optical flow.