KR102503885B1 - Apparatus and method for predicting human depression level using multi-layer bi-lstm with spatial and dynamic information of video frames - Google Patents

Abstract

An apparatus for predicting a human depression level by analyzing fine facial expressions according to an embodiment of the present invention includes a data storage unit for storing video data, a spatial information generation unit for generating spatial information from video data, and three consecutive data streams from video data. Extracted frames, create a volume local directional number (VLDN) feature map for analyzing facial dynamics based on consecutive frames, and input to a Deep Convolutional Neural Network (CNN) model to generate dynamic information about facial movements. It includes a VLDN feature map generation unit that generates spatial information and dynamic information as output values through a Temporal Median Pooling (TMP) method, and a prediction unit that predicts the level of human depression based on the output value through a recursive neural network.

Description

APPARATUS AND METHOD FOR PREDICTING HUMAN DEPRESSION LEVEL USING MULTI-LAYER BI-LSTM WITH SPATIAL AND DYNAMIC INFORMATION OF VIDEO FRAMES }

The present invention relates to an apparatus and method for predicting a level of depression in a human by analyzing video data. More specifically, it relates to an apparatus and method for predicting human depression level using multi-layer BI-LSTM considering spatial information of video data and dynamic information of video data.

Recently, psychiatric analysis of human mental health is increasing in society. One of the most well-known human mental health disorders is Major Depressive Disorder (MDD), also known as depression. Depression is a mental illness that negatively affects the patient's overall life, such as the patient's family, work life, eating habits, and sleeping habits, and also adversely affects society. Therefore, if the onset of depression can be detected early, it can help solve the problem of depression both personally and socially.

Conventionally, a test to confirm depression was performed by a psychiatric expert's evaluation. In addition, a psychiatrist consulted with the patient face-to-face to determine the presence of depression and the level of depression. However, there is a problem in that expert judgment of depression is labor-intensive and highly dependent on the expert's subjective perception.

Therefore, if it is possible to check whether a patient is depressed and the level of depression through a more convenient method, many problems caused by depression can be identified and resolved at an early stage. Accordingly, various studies have been conducted to predict the level of depression by analyzing video data recorded of human facial expressions.

Korean Patent Publication No. 10-2020-0061016 (title of invention: method for measuring and diagnosing depression index using facial skin images; publication date: June 2, 2020)

SUMMARY OF THE INVENTION The present invention is to solve the above problems, and an object of the present invention is to provide a device capable of predicting whether a patient has depression and the level thereof based on video data including the face of the patient.

An object of the present invention is to provide an apparatus and method for predicting the level of depression in a human through deep learning analysis by extracting spatial features of video data and temporal features from video frames.

An object of the present invention is to provide a depression level prediction device that can solve the noise problem according to the length of an input sequence by using the median value of spatial information and dynamic information as an input value through deep learning analysis through a TMP method. do.

An object of the present invention is to provide a depression level prediction device capable of providing a more accurate algorithm model for depression level prediction by utilizing a Bi-LSTM model composed of two layers.

An apparatus for predicting a human depression level by analyzing fine facial expressions according to an embodiment of the present invention includes a data storage unit for storing video data, a spatial information generation unit for generating spatial information from video data, and three consecutive data streams from video data. Extracted frames, create a volume local directional number (VLDN) feature map for analyzing facial dynamics based on consecutive frames, and input to a Deep Convolutional Neural Network (CNN) model to generate dynamic information about facial movements. It includes a VLDN feature map generation unit that generates spatial information and dynamic information as output values through a Temporal Median Pooling (TMP) method, and a prediction unit that predicts the level of human depression based on the output value through a recursive neural network.

According to an embodiment of the present invention, the level of human depression can be predicted by analyzing spatio-temporal features in a video frame including facial expressions using a deep learning technique.

According to an embodiment of the present invention, the deep learning technique can perform effective human depression level prediction in terms of performance using multi-layer LSTM.

According to an embodiment of the present invention, there is an effect of predicting the level of depression according to facial expressions and providing appropriate drug treatment and psychological treatment.

According to an embodiment of the present invention, it can be applied to the field of human emotion recognition through a computer, and can be applied to a wide range of fields related to human psychology in the future.

1 is a block diagram illustrating an apparatus for predicting a level of depression in a human by analyzing minute facial expressions according to an embodiment of the present invention.
2 is a detailed diagram illustrating an apparatus for predicting a level of human depression by analyzing minute facial expressions according to an embodiment of the present invention.
3 is a diagram for explaining a detailed configuration of a VLDN feature map generator.
4 is an exemplary view of a VLDN feature map according to an embodiment of the present invention.
5 is an exemplary view of a CNN model for explaining dynamic information generated by a VLDN feature map generator according to an embodiment of the present invention.
6 is a flowchart illustrating a method of predicting a level of depression in a human by analyzing minute facial expressions according to an embodiment of the present invention.
7 is a diagram showing result values of experiments on the AVEC2013 and AVEC2014 data sets.
8 is a diagram showing results obtained by analyzing AVEC2013 and AVEC2014 data sets in consideration of spatial feature extraction of the entire face image and spatial feature extraction of fragments divided into an arbitrary number when generating spatial information.
9 is a diagram showing result values of an experiment in which a model analyzing only dynamic information and other models are compared with respect to the AVEC2013 and AVEC2014 data sets.
10 is a diagram showing experimental results of the TMP method for the AVEC2013 and AVEC2014 data sets.

The above objects, features and advantages will be described later in detail with reference to the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs will be able to easily implement the technical spirit of the present invention. In describing the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

In the drawings, the same reference numerals are used to indicate the same or similar elements, and all combinations described in the specification and claims may be combined in any manner. And unless otherwise specified, it should be understood that references to the singular may include one or more, and references to the singular may also include plural.

Terms used herein are only for the purpose of describing specific exemplary embodiments and are not intended to be limiting. Singular expressions as used herein may also be intended to include plural meanings unless the context clearly dictates otherwise. The term “and/or,” “and/or” includes all combinations and any one of the associated listed items. The terms "comprises", "comprising", "including", "including", "having", "having" and the like are meant to be inclusive, and thus such terms shall be construed as having a recited feature, integer, Specifies steps, operations, elements, and/or components, and does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. The method steps, processes, and actions described herein should not be construed as requiring their performance in the specific order discussed or illustrated, unless such order of performance is specifically established. . It should also be understood that additional or alternative steps may be used.

In addition, each component may be implemented as a hardware processor, and the above components may be integrated and implemented as one hardware processor, or the above components may be combined with each other and implemented as a plurality of hardware processors.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an apparatus for predicting a level of depression in a human by analyzing minute facial expressions according to an embodiment of the present invention.

Depression level prediction apparatus 100 includes a data storage unit 110 for storing video data including facial expressions, a spatial information generator 120 for generating spatial information by extracting spatial features from video data, and dynamic dynamics in video data. A VLDN feature map generator 130 that generates a VLDN feature map corresponding to information, an information processor 150 that processes spatial and dynamic information through a TMP method, and a multi-layer Bi-LSTM, one of the recursive neural networks, for depression. It consists of a level prediction unit 160 that predicts the level of. The VLDN feature map generator 130 includes an edge response calculator (see reference numeral 135 in FIG. 3 ), a direction number checker (see reference numeral 136 in FIG. 3 ), and a VLDN generator (reference numeral 135 in FIG. 3 ). See '137') may be further included. A detailed description of the device will be described with reference to FIG. 2 .

2 is a detailed diagram illustrating an apparatus for predicting a level of human depression by analyzing minute facial expressions according to an embodiment of the present invention.

In the depression level predicting device 100 , the data storage unit 110 may store face video data 111 . The face video data 111 stored in the data storage unit 110 corresponds to data including human facial expressions. For example, the face video data 111 may be data stored by capturing a video image of a patient's face for which the level of depression needs to be determined. Alternatively, it may be video data taken by a patient who needs to determine the depression level. The video data 111 stored in the data storage unit 110 is not limited and interpreted, and all video data 111 including a human face may be included regardless of format and size.

The spatial information generator 120 may generate spatial information by extracting spatial features of the face image 121 of the video data stored in the data storage 110 . The face image 121 may be extracted as a sample RGB frame for the face image from the face video data 111 stored in the data storage 110 . The spatial information generator 120 may use the Inception-Resnet-v2 network convolution model to effectively generate spatial information by extracting spatial features from the face image 121 . The Inception-Resnet-v2 network convolution model can learn general features such as texture, color, and edge information of images. The Inception-Resnet-v2 network convolution model can be a pre-trained model using the ImageNet data set.

Spatial information in the present invention may be spatial information obtained by extracting spatial features of the face image from the entire face image 121 . According to another embodiment of the present invention, the spatial information generator 120 may divide the face image 121 into an arbitrary number of pieces and extract spatial features of the face image 121 from the divided pieces. For example, the facial image 121 may be divided into four pieces 122 and spatial features for each piece may be extracted. Spatial features for each slice can also utilize the Inception-Resnet-v2 network convolution model. The spatial information generating unit 120 may extract and aggregate spatial features of face images of all video data. Spatial information can be generated by aggregating features extracted from all image data and using a CNN (Convolutional Neural Network) model using a Euclidean loss function.

The information processing unit 150 may utilize spatial information about spatial features of the entire face image 121 and spatial features of the divided pieces 122 as input data. Hereinafter, the VLDN feature map generator 130 that generates dynamic information will be described.

The VLDN feature map generator 130 may extract three consecutive previous, current, and next frames 131 from the face video data 111 stored in the data storage unit 110 . The VLDN feature map generation unit 130 may generate a Volume Local Directional Number (VLDN) feature map for analyzing facial dynamics for consecutive frames 131 . The generated VLDN feature map is generated as a VLDN gray image and can be input to a CNN with a channel size of 1. The resulting value output by using the CNN convolution model can be used as dynamic information of video data. Hereinafter, VLDN feature map generation will be described in detail.

3 is a diagram for explaining a detailed configuration of a VLDN feature map generator.

The VLDN feature map generator 130 may include an edge response calculator 135, a direction number checker 136, and a VLDN generator 137. The edge response calculation unit 135 may generate a VLDN feature map that is an extension of a local directional number (LDN).

More specifically, an intermediate value may be generated through processing of the edge response calculator 135 for pixel values of the previous, current, and next frames 131 . The edge response calculator 135 may perform a function of calculating edge responses of pixels adjacent to the center pixel based on the Kirsch mask. The edge response calculator 135 may calculate edge responses for three consecutive frames 131 through Equation (1). Here, PR, CR, and PO correspond to pixel values of the previous, current, and next frames 131, respectively.

The direction number confirmation unit 136 may check the numbers of the highest positive and negative directions. Equation (2) may be referred to for the equation for checking the highest positive and negative direction numbers in the direction number check unit 136.

For example, in FIG. 3, the most significant positive number is 620 in three consecutive frames 131, and the most significant positive direction number for 620 corresponds to 6. The most negative number is -740, and the most negative number toward -740 is 1. The VLDN generation unit 137 may generate VLDN values for three consecutive frames 131 using Equation (3).

Here, the value of MPx,y corresponds to the most significant positive-direction numeric value, and the MNx,y value corresponds to the most significant negative-direction numeric value. For example, according to an embodiment of the present invention, the value of MPx,y is 6 and the value of MNx,y may correspond to 1. The VLDNx,y value for three consecutive frames 131 corresponds to 49, and when returned based on the binary method, it corresponds to 110001 (2).

The VLDN generation unit 137 may generate all VLDN values for all three consecutive frames 131 of video data stored in the data storage unit 110 . A VLDN feature map may be generated based on the generated VLDN value. Below, you can see an example of the generated VLDN feature map.

4 is an exemplary view of a VLDN feature map according to an embodiment of the present invention.

A VLDN feature map may be generated as an image using the VLDN value calculated by the edge response calculator 135, the direction number checker 136, and the VLDN generator 137. Referring to FIG. 4 , it can be confirmed that a part of a VLDN feature map is generated for face video data stored in the data storage unit 110 . A CNN model is used to generate effective dynamic information through such a VLDN feature map. A detailed description of the CNN model will be described with reference to FIG. 5 .

5 is an exemplary CNN model diagram for explaining dynamic information generated by the VLDN feature map generator 130 according to an embodiment of the present invention.

The VLDN feature map generator 130 may generate dynamic information about facial motion by using the generated VLDN feature map as a CNN model. CNN models can be pre-trained to model facial movements. CNN models can use 5x5 filters in the first and second convolutional layers instead of 3x3 filters. A CNN model can use one convolution instead of three convolutions for the first, second, and layer. A CNN model might consist of 10 convolutions, 5 max poolings, and 3 fully connected layers. CNN models can use the Euclidean loss function instead of the softmax loss function. Through the CNN model, dynamic information about facial motion can be generated by using the VLDN feature map as an input value. Hereinafter, returning to FIG. 2, a detailed description of the information processing unit 150 and the level prediction unit 160 will be given.

The information processing unit 150 may receive the spatial information 151 and the dynamic information 152 and generate them as input values of the level prediction unit 160 through information processing. The level prediction unit 160 may predict the level of depression by using the Bi-LSTM model.

Recurrent Neural Network (RNN), one of the recurrent neural networks, corresponds to a model that can effectively learn sequential information by mapping inputs to outputs through hidden states. However, RNN-based approaches have gradient explosion and gradient vanishing problems when the input is a long sequence.

LSTM (Long Short Term Memory), one of the recursive neural networks, solves the gradient loss problem through input, forget, and output gates, and can learn long-length sequences. According to an embodiment of the present invention, the level predictor 160 may learn spatial information and dynamic information using a multi-layer Bi-LSTM model 161 instead of a single LSTM layer. Therefore, the second Bi-LSTM layer of the Bi-LSTM has an effect of learning from the previous state of the same layer as the first Bi-LSTM layer. However, the current multilayer Bi-LSTM model 161 can learn long temporal features by providing inter-frame features as inputs, but directly using frame-level features as inputs to the LSTM model can be vulnerable to noise.

Therefore, before inputting the information to the multilayer Bi-LSTM model 161 of the level predictor 160, the information processing unit 150 can solve the noise-vulnerable problem through the TMP (Temporal Median Pooling) method. The TMP method refers to a method of temporally dividing spatial information 151 and dynamic information 152 into an arbitrary number. For example, the spatial information 151 and the dynamic information 152 may be grouped into five units and divided into one unit. Any number is not interpreted as being limited, and it is reasonable to interpret it as a number that can be divided at the level of a person skilled in the art.

After division, it is possible to return the median value of the divided information based on an arbitrary number as one unit. The information processing unit 150 aggregates the median value 153 of the spatial information 151 and the median value 154 of the dynamic information 152 through the TMP method, and input values related to the multilayer Bi-LSTM model of the level prediction unit 160. can be used as The multilayer Bi-LSTM used by the level prediction unit 160 is one of the recursive neural networks, and includes LSTM, RNN, and the like. Hereinafter, a basic explanation of the recursive neural network LSTM will be given.

where σ denotes the logistic function and tanh denotes the hyperbolic tangent function.

i _t , f _t , and O _t denote input, hidden, and output gates. W _i , W _f , W _O and b _i , b _f , b _O correspond to weight matrices and bias terms for the input, hidden and output states, respectively. X _t is the input at moment t, e _t represents the cell input state, and C _t represents the cell output state. Finally, the hidden layer state is denoted by h _t .

Similar to the LSTM model, the multilayer Bi-LSTM model 161 consists of input, hidden, output gates and memory units. In the multilayer Bi-LSTM model 161, sequential data is processed with forward and backward sense using two different hidden layers connected to the same output layer. The hidden states of the anterior and posterior layers are estimated by h _f = O _f tanh(C _f ) and h _b = O _b tanh (C _b ). The final hidden state is denoted by H=(h _f ,h _b ). The multilayer Bi-LSTM model 161 consists of stacking two Bi-LSTM cells to learn spatial and dynamic information. The multilayer Bi-LSTM model 161 learns by supplying frame-to-frame functions as inputs, but by directly employing frame-level functions that can be input to the LSTM model, there is an effect that allowable tone-noise intervention is possible. Therefore, in the present invention, the problem of noise interference can be solved by predicting the level of depression through the multilayer Bi-LSTM model 161.

In the multilayer Bi-LSTM model 161, two layers may be an output layer and a regression layer. Layers can have different numbers of hidden units. For example, hidden units can be 512 or 256. The median value 153 of the spatial information 151 and the median value 154 of the dynamic information 152 processed by the TMP method in the information processing unit 150 can be used as input values to the multilayer Bi-LSTM model 161. Through the multi-layer Bi-LSTM model 161, the average value of the resulting values of the spatial information 151 and the dynamic information 152 can be used as the prediction value of the final depression level prediction. Hereinafter, a method of predicting a level of depression in a human through deep learning analysis of minute facial expressions according to an embodiment of the present invention will be described using FIG. 7 .

6 is a flowchart illustrating a method of predicting a level of depression in a human by analyzing minute facial expressions according to an embodiment of the present invention.

In relation to the depression level prediction method, detailed embodiments overlapping with the above-described depression level prediction apparatus 100 may be omitted. The apparatus 100 for predicting the depression level may be implemented as a server, and hereinafter, the apparatus will be named and described as a server. The server may store face images and video data. The server may generate spatial information from the face image (S101). The server may extract three consecutive frames 131 from face video data and generate a volume local directional number (VLDN) feature map for analyzing facial dynamics based on the consecutive frames 131 (S103 ).

Dynamic information about facial motion may be generated by inputting the VLDN feature map to a Deep Convolutional Neural Network (CNN) model (S105). The server may generate the spatial information 151 and the dynamic information 152 as output values through a Temporal Median Pooling (TMP) method (S107). The server can predict the level of human depression based on the output value of the recursive neural network, and can use the multilayer Bi-LSTM model 161 as the recursive neural network (S109).

The VLDN feature map may be generated through the steps of calculating an edge response, checking a direction number, and generating a VLDN. Detailed embodiments overlap with those described in the depression level predicting apparatus 100, and thus will be omitted.

Hereinafter, experiments and results for comparing the performance of the apparatus 100 for predicting depression levels according to an embodiment of the present invention with those of conventional methods will be described using FIGS. 7 to 10 .

7 is a diagram showing result values of an experiment on the AVEC2013 data set and the AVEC2014 data set.

Before explaining the experimental results on the performance of the depression level prediction apparatus 100 of the present invention, the experimental design and experimental data will be described. Two datasets of the Audio/Visual Emotion Challenge (AVEC) 2013 and 2014 depression sub-challenge datasets are used to test the performance of the device 100 for predicting the level of depression, and the method of the proposed device is compared with the existing method.

The AVEC2013 dataset consists of 150 videos of 82 people, and the dataset is divided into three datasets: training, development, and test. Each dataset has 50 videos, 100 videos were used to train the model, and the remaining 50 videos can be used to evaluate the performance of the depression level prediction apparatus 100 of the present invention. On average, each video is 25 minutes long, and in each video, participants can respond to specific questions that are recorded via a microphone and webcam. Additionally, each video was labeled with a level of depression, and these labels were defined through the BDI-II questionnaire.

The AVEC2014 depression dataset consists of 150 videos, which are divided into training, development, and test datasets each consisting of 50 videos. Similar to the previous dataset, in this dataset 100 videos are used for training and 50 videos are used to evaluate the performance of the proposed method, these videos are recorded with a webcam and microphone, each video length is about 2 minutes on average. applies to

For better video analysis, the sample frame rate was reduced from 30 frames per second to 6 frames per second using a subsampling strategy, and the video clips were sub-frames of 40 frames length with an overlap of 5 frames between two consecutive videos. split into sequences. To obtain the face information, we use the face detection and face landmark position for each frame, and the face region is extracted from each frame and resized to 299*299.

In the present invention, the Inception-ResNet-v2 network is trained in MATLAB R2018b. Initialize the parameters of Inception-ResNet-v2 from the pre-trained ImageNet model. In the model related to spatial information, the stochastic gradient descent (SGD) algorithm is used with a batch size of 32, momentum is set to 0.9, and weight decay is set to 0.0002. Here, the learning rate is fixed at 0.001. In contrast, the proposed CNN model for dynamic information is trained from scratch. Here, the CNN model is trained with the SGD algorithm with a batch size of 32 and the learning rate is set to 0.001. Also, in both models, the Euclidean loss function is applied instead of the softmax loss function. To train the multilayer Bi-LSTM model 161, we use the Adam optimizer with 512 and 256 hidden units to learn spatiotemporal information, the batch size is set to 10, and the learning rate is 0.001.

The depression stage is measured by taking the average of the predicted values from spatial information (151) and dynamic information (152) for each video clip, and the overall performance uses MAE (Mean Absolute Error) and RMSE (root Mean Square Error) is evaluated by The smaller the values of MAE and RMSE, the higher the accuracy of predicting the level of depression.

MAE and RMSE are defined as follows.

N is the total number of samples, xj represents the predicted value, and xj corresponds to the actual measured value of the jth sample.

In the experiment, the performance of spatial information analysis and dynamic information analysis is measured using the InceptionResNet-v2 model using multi-layer Bi-LSTM and VLDN-CNN using multi-layer Bi-LSTM, respectively. We also estimate the MAE and RMSE by fusing the spatial and temporal networks performed by taking the average of the outputs of the two informational analyzes. The experimental results are described below.

It can be seen that the depression level predicting apparatus 100 of the present invention analyzed using the spatial information 151 and the dynamic information 152 is superior to the existing approach with values of MAE 7.04 and RMSE 8.93. However, in the AVEC2013 data set, Zhou et al. [31] shows better performance with MAE 6.20 and RMSE 8.28. Furthermore, in the AVEC2014 dataset, Zhou et al. [31] shows better performance with MAE 6.21 and RMSE 8.39. However, an approach based on deep learning, Zhu et al. [6] and Jazaery et al. It can be seen that [7] has better performance. Zhu et al. [6] is a deep learning analysis model for both static frames and optical flow images, and Jazaery et al. In the case of [7], it is a model based on RNN, but it can be confirmed that the motion capture part is relatively noisy.

8 is a diagram showing results obtained by analyzing AVEC2013 and AVEC2014 data sets in consideration of spatial feature extraction of the entire face image and spatial feature extraction of fragments divided into an arbitrary number when generating spatial information.

When generating spatial information, it can be seen that analyzing the spatial information for spatial feature extraction of the fragment by dividing it into an arbitrary number produces a better effect than analyzing the entire face image through only spatial feature extraction. When spatial features are extracted by segmentation and spatial information is analyzed, the MAE value is 7.22 and the RMSE value is 9.02 in the AVEC2013 data set. In the AVEC2014 data set, the MAE value is 6.96 and the RMSE value is 8.91.

9 is a diagram showing result values of an experiment in which a model analyzing only dynamic information and other models are compared with respect to the AVEC2013 and AVEC2014 data sets.

In the case of a model that analyzes only the dynamic information 152 using the VLDN of the present invention, a time model based on MHH [11] and a time model based on optical flow image [6] can be compared. In both the AVEC2013 and AVEC2014 data sets, it can be seen that the model that analyzes only the dynamic information 152 using the VLDN of the present invention has better performance than other models.

10 is a diagram showing experimental results of the TMP method for the AVEC2013 and AVEC2014 data sets.

Through the experimental results of FIG. 10, the device for predicting depression level according to the number of times for the TMP method in which the information processing unit 150 divides the spatial information 151 and the dynamic information 152 into an arbitrary number of times and measures the median value. performance can be checked. In addition, the experimental results of the TMP method and other temporary maximum pooling methods and other pooling methods can be confirmed.

In both AVEC2013 and AVEC2014 data sets, it can be seen that the TMP method has smaller MAE and RMSE values than other pooling methods. Furthermore, it can be confirmed that the random time number part is the best when the TMP method is performed based on 5. When the depression level predicting apparatus 100 of the present invention performed the TMP method with 5 time counts, values of MAE 7.04 and RMSE 9.08 were measured in the AVEC2013 data set. In the AVEC2014 data set, MAE 6.86 and RMSE 8.78 values were measured.

The embodiments of the present invention disclosed in the present specification and drawings are only presented as specific examples to easily explain the technical content of the present invention and help understanding of the present invention, and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein.

Claims (10)

a data storage unit for storing video data;
a spatial information generation unit generating spatial information from the video data;
Extract three consecutive frames from the video data, create a volume local directional number (VLDN) feature map for analyzing facial dynamics based on the consecutive frames, and input to a Deep Convolutional Neural Network (CNN) model to a VLDN feature map generating unit generating dynamic information about facial motion;
an information processing unit generating the spatial information and the dynamic information as output values through a Temporal Median Pooling (TMP) method;
a level prediction unit that predicts a level of depression of a human based on the output value through a recursive neural network;
containing
A device that predicts the level of depression in humans by analyzing microscopic facial expressions.
According to claim 1,
The VLDN feature map generator
an edge response calculation unit calculating edge responses of pixels adjacent to the center pixel based on the kirsch masks of the three consecutive frames;
a direction number checking unit that checks a highest positive direction number and a highest negative direction number among the adjacent pixels;
a VLDN generator configured to generate a VLDN value using the highest positive direction number and the highest negative direction number, and to generate the VLDN feature map by generating all VLDN values for three sequentially consecutive video frames;
characterized in that it includes
A device that predicts the level of depression in humans by analyzing microscopic facial expressions.
According to claim 1
The spatial information generating unit
Characterized in that an image is arbitrarily divided into four regions from the video data, spatial features are extracted from the entire image, and spatial information is generated by extracting spatial features from each of the four regions.
A device that predicts the level of depression in humans by analyzing microscopic facial expressions.
According to claim 1,
The level prediction unit
Characterized in that the recursive neural network is a Bi-LSTM composed of two layers
A device that predicts the level of depression in humans by analyzing microscopic facial expressions.
According to claim 1,
The information processing unit
After dividing the spatial information and the dynamic information into an arbitrary number of pieces, a median value for each piece is returned and generated as an output value.
A device that predicts the level of depression in humans by analyzing microscopic facial expressions.
In the method of predicting the level of human depression by analyzing fine facial expressions,
storing video data by a data storage unit;
generating spatial information from a face image of the video data by a spatial information generator;
a VLDN feature map generating unit extracting three consecutive frames from the video data and generating a volume local directional number (VLDN) feature map for analyzing facial dynamics based on the consecutive frames;
generating dynamic information about facial motion by inputting the VLDN feature map to a Deep Convolutional Neural Network (CNN) model by the VLDN feature map generation unit;
generating, by an information processing unit, the spatial information and the dynamic information as output values through a Temporal Median Pooling (TMP) method;
predicting, by a level prediction unit, a level of human depression based on the output value through a recursive neural network; containing
A method for predicting levels of depression in humans by analyzing micro-facial expressions.
According to claim 6,
The step of generating the VLDN feature map
calculating edge responses of adjacent pixels adjacent to the central pixel based on the kirsch masks of the three consecutive frames;
checking a highest positive direction number and a highest negative direction number among the adjacent pixels;
generating the VLDN feature map by generating a VLDN value using the highest positive direction number and the highest negative direction number, and generating all VLDN values for three sequentially consecutive video frames;
characterized in that it further comprises
A method for predicting levels of depression in humans by analyzing micro-facial expressions.
According to claim 6,
The step of generating the spatial information is
randomly dividing an image from the video data into four regions;
extracting spatial features from the entire image; and
And generating spatial information by extracting spatial features from each of the four regions.
A method for predicting levels of depression in humans by analyzing micro-facial expressions.
According to claim 6,
Characterized in that the recursive neural network is a Bi-LSTM composed of two layers
A method for predicting levels of depression in humans by analyzing micro-facial expressions.
According to claim 6,
The step of generating the output value is
After dividing the spatial information and the dynamic information into an arbitrary number of pieces, returning a median value for each piece and generating it as an output value.
A method for predicting levels of depression in humans by analyzing micro-facial expressions.

Images