CN111938670A - Depression identification method and system based on human skeleton kinematics characteristic information - Google Patents
Depression identification method and system based on human skeleton kinematics characteristic information Download PDFInfo
- Publication number
- CN111938670A CN111938670A CN202010608130.9A CN202010608130A CN111938670A CN 111938670 A CN111938670 A CN 111938670A CN 202010608130 A CN202010608130 A CN 202010608130A CN 111938670 A CN111938670 A CN 111938670A
- Authority
- CN
- China
- Prior art keywords
- data
- depression
- neural network
- human skeleton
- kinematic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000003062 neural network model Methods 0.000 claims abstract description 40
- 238000012549 training Methods 0.000 claims abstract description 40
- 238000007781 pre-processing Methods 0.000 claims abstract description 30
- 238000004364 calculation method Methods 0.000 claims abstract description 24
- 238000013075 data extraction Methods 0.000 claims abstract description 16
- 238000003745 diagnosis Methods 0.000 claims abstract description 16
- 208000020401 Depressive disease Diseases 0.000 claims abstract description 8
- 238000012545 processing Methods 0.000 claims abstract description 8
- 238000013480 data collection Methods 0.000 claims abstract description 6
- 238000013528 artificial neural network Methods 0.000 claims description 35
- 230000006870 function Effects 0.000 claims description 34
- 230000000994 depressogenic effect Effects 0.000 claims description 28
- 230000009471 action Effects 0.000 claims description 26
- 238000013527 convolutional neural network Methods 0.000 claims description 26
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 25
- 238000002372 labelling Methods 0.000 claims description 22
- 230000000638 stimulation Effects 0.000 claims description 22
- 238000000605 extraction Methods 0.000 claims description 19
- 238000012216 screening Methods 0.000 claims description 11
- 238000012360 testing method Methods 0.000 claims description 11
- 230000004913 activation Effects 0.000 claims description 6
- 239000006185 dispersion Substances 0.000 claims description 6
- 238000012854 evaluation process Methods 0.000 claims description 6
- 239000012634 fragment Substances 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 6
- 230000009467 reduction Effects 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 6
- 238000011156 evaluation Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 4
- 230000002123 temporal effect Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000003759 clinical diagnosis Methods 0.000 description 2
- 206010010144 Completed suicide Diseases 0.000 description 1
- 206010012374 Depressed mood Diseases 0.000 description 1
- 208000006083 Hypokinesia Diseases 0.000 description 1
- 206010024642 Listless Diseases 0.000 description 1
- 206010027940 Mood altered Diseases 0.000 description 1
- 208000019022 Mood disease Diseases 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 230000003920 cognitive function Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013401 experimental design Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 239000005556 hormone Substances 0.000 description 1
- 229940088597 hormone Drugs 0.000 description 1
- 230000003483 hypokinetic effect Effects 0.000 description 1
- 208000017971 listlessness Diseases 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 208000020016 psychiatric disease Diseases 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 230000008448 thought Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1126—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique
- A61B5/1128—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique using image analysis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Public Health (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- Veterinary Medicine (AREA)
- Psychiatry (AREA)
- Surgery (AREA)
- Artificial Intelligence (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Physiology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Child & Adolescent Psychology (AREA)
- Fuzzy Systems (AREA)
- Evolutionary Computation (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Mathematical Physics (AREA)
- Signal Processing (AREA)
- Dentistry (AREA)
- Developmental Disabilities (AREA)
- Educational Technology (AREA)
- Hospice & Palliative Care (AREA)
- Psychology (AREA)
- Social Psychology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a depression recognition method and system based on human skeleton kinematics characteristic information, belonging to the field of image processing and mode recognition, aiming at solving the technical problem of how to establish an objective and effective index, and diagnosing depression according to the prepared index parameters, and adopting the technical scheme that: the method comprises the following specific steps: s1, collecting human skeleton kinematic characteristic data; s2, extracting human skeleton kinematic characteristic data; s3, preprocessing the characteristic data and making a data set; s4, constructing and training a depression recognition neural network model; and S5, inputting the human skeleton kinematic data to be recognized into the trained depression recognition neural network model, completing the recognition and diagnosis of depression, obtaining a prediction result and calculating the accuracy. The system comprises a data collection module, a data extraction module, a preprocessing and data set making module, a model building and training module and a result prediction and accuracy calculation module.
Description
Technical Field
The invention relates to the field of image processing and pattern recognition, in particular to a depression recognition method and system based on human skeleton kinematics characteristic information.
Background
Depression (depression) refers to a group of affective disorders characterized clinically by marked and persistent mood depression, hypomotility, and mental and cognitive retardation. Patients with depression have serious psychological disorder and bad mood, often lack sleep, listlessness and even suicide behavior in patients with severe depression.
At present, the diagnosis of depression is still in the stage of symptom deduction, and there is no objective physicochemical examination and diagnosis index. The existing depression identification method has more defects, which mainly comprise the following steps:
firstly, clinical diagnosis measures are judged in a subjective mode, objective and effective indexes are lacked, and misdiagnosis and missed diagnosis are easily caused;
(II) identifying the reference index to be single: the existing method for identifying the depression mainly takes hormone level and various biochemical indexes in a patient as identification reference indexes, and lacks of index parameters of bioelectricity signals, so that the depression identification has certain limitation.
Therefore, how to establish an objective and effective index is a technical problem to be solved urgently at present in the diagnosis of depression prepared according to the index parameters.
Disclosure of Invention
The technical task of the invention is to provide a depression identification method and system based on human skeleton kinematic characteristic information, so as to solve the problem of how to establish objective and effective indexes and prepare depression diagnosis according to index parameters.
The technical task of the invention is realized in the following way, the method for identifying the depression based on the human skeleton kinematic characteristic information comprises the steps of extracting the human joint kinematic characteristic information according to the human skeleton characteristic information captured by Kinect equipment, carrying out data preprocessing on the spatial position data of each joint based on time dimension, constructing a data set by utilizing the preprocessed data, and carrying out training and testing of a depression identification neural network model by using the data set; the method comprises the following specific steps:
s1, collecting human skeleton kinematic characteristic data;
s2, extracting human skeleton kinematic characteristic data;
s3, preprocessing the characteristic data and making a data set;
s4, constructing and training a depression recognition neural network model;
and S5, inputting the human skeleton kinematic data to be recognized into the trained depression recognition neural network model, completing the recognition and diagnosis of depression, obtaining a prediction result and calculating the accuracy.
Preferably, the collection of the human skeleton kinematic feature data in step S1 is specifically as follows:
s101, starting a Kinect device to collect kinematic characteristic data of a human skeleton;
s102, guiding the experimental subject to enter a designated position, and playing a pre-designed stimulation action task instruction; wherein the stimulation action comprises a total of five kinematic segments, respectively: lifting and resetting a left arm, lifting and resetting a right arm, lifting and resetting two arms, turning left and resetting, and turning right and resetting; the time interval between each kinematic segment is 5s, and the time required for completing the whole stimulation action is 60 s;
s103, opening Kinect studio V2 software to complete the recording work of human skeleton kinematic data;
s104, after the data recording is finished, the psychologist evaluates the depression state of the current experimental subject, and the evaluation process records the Hamilton depression scale (24 versions) scores of the experimental subject; wherein, the experimental subjects are divided into a depression group and a non-depression group;
and S105, screening effective experimental data when the score of the depression group is more than 20 and the score of the control group is less than 8 according to the Hamilton depression scale score standard.
Preferably, the extraction of the human skeleton kinematic feature data in step S2 is specifically as follows:
s201, traversing xef recording files in the folder, acquiring file names and obtaining recording file paths;
s202, executing a script command to run SkeletonExtractor.exe in batch to open xef record files and read original record data;
s203, extracting time series data based on the xef recorded files;
s204, extracting the spatial positions (x, y, z) of the human skeleton joint points, and eliminating noise data; the method specifically comprises the following steps: for any Kinect detected target i 1,2,3 …, each human body skeleton joint point detects a spatial position (x, y, z), and for any time t, the target object i to be locked has the following relation:
s205, extracting quaternion ((Rx, Ry, Rz), Rw) of the spatial coordinate system of the data subjected to the noise reduction processing, and the formula is as follows:
each captured human skeletal joint point is composed of (x, y, z) and ((Rx, Ry, Rz), Rw), that is, each human skeletal joint point n at the time t of the kinematic segment is composed of seven dimensional data expressing the spatial position relationship:
djt＝[xjt，yjt，zjt，Rxjt，Ryjt，Rzjt，Rwjt]；
and for 25 human skeleton joint points captured by the Kinect, the human skeleton data extracted within the kinematic fragment time T is as follows:
wherein t represents time; j represents a human skeletal joint point; t is more than or equal to 1 and less than or equal to T, j is more than or equal to 1 and less than or equal to 25;
s206, obtaining the extraction result (time, the space position coordinate value based on the time dimension of each joint and quaternion) of the experimental object, and saving the extraction result as a file in the csv format by the name of the experimental object.
Preferably, the characteristic data preprocessing and data set creation in step S3 are specifically as follows:
s301, reading the extracted csv file of the experimental object;
s302, reading the time sequence and rewriting the time sequence into a csv file with the same name in another folder;
s303, reading data of the csv file except the time sequence, and preprocessing the adopted standardized data, wherein the method specifically comprises the following steps: for human skeleton data time series x1，x2，...，xtThe linear transformation of the raw data using a dispersion normalization method, with the result mapped to [0,1]Interval, the formula is as follows:
where n denotes the sequence length, the new sequence y1，y2，...，yn∈[0，1]And is dimensionless;
s304, writing the standardized data into the csv file in the step S302;
s305, labeling the experimental subjects of a depressed group and a non-depressed group respectively, labeling the depressed group as 0, and labeling the non-depressed group as 1;
s306, making a data set through a new data sequence preprocessed by a pythonnnumpy library, and storing data extracted by all experimental objects in a binary data format as a npy format file;
and S307, adopting a random division mode for the data set, setting 70% of the data set as a training set, and using 30% of the data set as a test set.
Preferably, the construction and training of the depression recognition neural network model in step S4 are as follows:
s401, inputting training set data into a time Convolution Neural Network (Temporal Convolution Neural Network) to obtain an output result of the time Convolution Neural Network; the time convolution neural network comprises two time hole convolution residual blocks which are sequentially connected, each layer of each residual block is a one-dimensional hole convolution network, the random discarding rate is 0.5, the activation function is ReLU, and the output of each layer is directly used as the input of the next layer;
s402, inputting the obtained time convolution neural Network output result as characteristic information into a bottleneck convolution neural Network (BottleneckConvolationneural Network) to obtain an output result of the bottleneck convolution neural Network; wherein the bottleneck convolutional neural network comprises a bottleneck network residual block; the bottleneck network residual block comprises three layers of one-dimensional convolution networks, and the kernel function size K of the one-dimensional convolution networks is 1,3 and 1;
and S403, converting the output result of the bottleneck convolutional neural network into a one-dimensional vector, connecting the one-dimensional vector and the one-dimensional vector, inputting the one-dimensional vector into a full connection layer, and then identifying the depression through a softmax classifier.
Preferably, the identification and diagnosis of the depression are completed in step S5, and the prediction result is obtained and the calculation accuracy is as follows:
s501, identifying and diagnosing the depression by using softmax as a classifier, wherein the formula is as follows:
wherein S isiRepresenting a softmax predicted value of the ith class; i and j are category serial numbers;
s502, inputting the human skeleton kinematic feature data into a depression recognition neural network model, wherein when depression category prediction is carried out, each category (depression and non-depression) corresponds to a calculation value smaller than 1, the sum is 1, and the category corresponding to the maximum calculation value is a prediction category;
s503, comparing the prediction categories with the real categories, calculating the proportion of the number of the depression categories of the correct prediction categories in the training data set to the total number of the data, and outputting the accuracy of the depression recognition neural network model;
s504, calculating the Loss of the prediction error category by adopting a Loss function, setting the Loss function of the depression recognition neural network model as a cross entropy function Loss, and adopting the following formula:
wherein M represents the number of categories; c represents a category number; y iscRepresenting a real tag; p is a radical ofcIndicating the output of softmax.
A depression recognition system based on human skeleton kinematics characteristic information comprises,
the data collection module is used for collecting human skeleton kinematic characteristic data;
the data extraction module is used for extracting human skeleton kinematic characteristic data;
the preprocessing and data set making module is used for preprocessing the characteristic data and making a data set;
the model building and training module is used for building and training a depression recognition neural network model;
and the result prediction and accuracy rate calculation module is used for inputting the human skeleton kinematic data to be recognized into the trained depression recognition neural network model to complete the recognition diagnosis of the depression, obtain the prediction result and calculate the accuracy rate.
Preferably, the data collection module includes,
the device starting and data collecting submodule is used for starting the Kinect device to collect the kinematic characteristic data of the human skeleton;
the instruction playing submodule is used for guiding the experimental subject to enter a designated position and playing a pre-designed stimulation action task instruction; wherein the stimulation action comprises a total of five kinematic segments, respectively: lifting and resetting a left arm, lifting and resetting a right arm, lifting and resetting two arms, turning left and resetting, and turning right and resetting; the time interval between each kinematic segment is 5s, and the time required for completing the whole stimulation action is 60 s;
the data recording submodule is used for opening Kinect studio V2 software to finish the recording work of human skeleton kinematic data;
a depression state evaluation submodule for the psychiatrist to evaluate the depression state of the current subject after completing the data recording, wherein the evaluation process records the Hamilton depression scale (24 edition) scores of the subject; wherein, the experimental subjects are divided into a depression group and a non-depression group;
the screening submodule is used for screening effective experimental data when the score of a depression group is larger than 20 and the score of a control group is smaller than 8 according to the Hamilton depression scale scoring standard;
the data extraction module comprises a data extraction module and a data extraction module,
the file path recording submodule is used for traversing xef recorded files in the folder, acquiring file names and obtaining recording file paths;
an original record data reading submodule, configured to execute a script command to run skeletonextractor in batch to open.exe, xef record a file, and read original record data;
a time-series data extraction sub-module for extracting time-series data based on the xef record file;
the noise data removing submodule is used for extracting the spatial positions (x, y, z) of the human skeleton joint points and removing the noise data; the method specifically comprises the following steps: for any Kinect detected target i 1,2,3 …, each human body skeleton joint point detects a spatial position (x, y, z), and for any time t, the target object i to be locked has the following relation:
and the quaternion extraction submodule is used for extracting quaternion ((Rx, Ry, Rz), Rw) of a space coordinate system of the data subjected to the noise reduction processing, and the formula is as follows:
each captured human skeletal joint point is composed of (x, y, z) and ((Rx, Ry, Rz), Rw), that is, each human skeletal joint point n at the time t of the kinematic segment is composed of seven dimensional data expressing the spatial position relationship:
djt＝[xjt，yjt，zjt，Rxjt，Ryjt，Rzjt，Rwjt]；
and for 25 human skeleton joint points captured by the Kinect, the human skeleton data extracted within the kinematic fragment time T is as follows:
wherein t represents time; j represents a human skeletal joint point; t is more than or equal to 1 and less than or equal to T, j is more than or equal to 1 and less than or equal to 25;
the extraction result obtaining submodule is used for obtaining the extraction result (time, the space position coordinate value and the quaternion of each joint based on the time dimension) of the experimental object and saving the extraction result as a file in the csv format by the name of the experimental object;
the preprocessing and data set creation module includes,
the file reading submodule is used for reading the extracted csv file of the experimental object;
the time sequence reading submodule is used for reading the time sequence and rewriting the time sequence into a csv file with the same name in another folder;
the standardized data preprocessing submodule is used for reading the data of the csv file except the time sequence and preprocessing the adopted standardized data, and specifically comprises the following steps: for human skeleton data time series x1，x2，...，xtThe linear transformation of the raw data using a dispersion normalization method, with the result mapped to [0,1]Interval, the formula is as follows:
where n denotes the sequence length, the new sequence y1，y2，...，yn∈[0，1]And is dimensionless;
the file writing submodule is used for writing the standardized data into the csv file;
a subject labeling submodule for labeling the experimental subjects of the depressed group and the non-depressed group, respectively, labeling the depressed group as 0 and labeling the non-depressed group as 1;
the data set making submodule is used for making a data set through a new data sequence preprocessed by a pythonnnumpy library, and the data extracted by all experimental objects are stored as a. npy format file in a binary data format;
the data set dividing submodule is used for adopting a random dividing mode for the data set, setting 70% of the data set as a training set, and using 30% of the data set as a testing set;
the model building and training module comprises a model building and training module,
the time Convolution Neural Network result output submodule is used for inputting the training set data into a time Convolution Neural Network (Temporal Convolution Neural Network) to obtain an output result of the time Convolution Neural Network; the time convolution neural network comprises two time hole convolution residual blocks which are sequentially connected, each layer of each residual block is a one-dimensional hole convolution network, the random discarding rate is 0.5, the activation function is ReLU, and the output of each layer is directly used as the input of the next layer;
a bottleneck convolutional neural Network result output submodule, configured to input the obtained time convolutional neural Network output result as feature information into a bottleneck convolutional neural Network (bottleneckconvolutional convolutional neural Network), so as to obtain an output result of the bottleneck convolutional neural Network; wherein the bottleneck convolutional neural network comprises a bottleneck network residual block; the bottleneck network residual block comprises three layers of one-dimensional convolution networks, and the kernel function size K of the one-dimensional convolution networks is 1,3 and 1;
the depression recognition submodule is used for converting an output result of the bottleneck convolutional neural network into a one-dimensional vector, connecting the one-dimensional vector and the one-dimensional vector, inputting the one-dimensional vector into the full connection layer, and then recognizing the depression through a softmax classifier;
the result prediction and accuracy calculation module comprises,
the identification and diagnosis submodule is used for identifying and diagnosing the depression by adopting softmax as a classifier, and the formula is as follows:
wherein S isiSoftmax pre-representing class iMeasuring; i and j are category serial numbers;
the prediction class acquisition submodule is used for inputting the human skeleton kinematic feature data into the depression recognition neural network model, when depression class prediction is carried out, each class (depression and non-depression) corresponds to a calculation value smaller than 1, the sum is 1, and the class corresponding to the maximum calculation value is a prediction class;
the comparison submodule is used for comparing the prediction category with the real category, calculating the proportion of the number of depression categories of the correct prediction category in the training data set to the total number of data, and outputting the accuracy of the depression recognition neural network model;
the Loss calculation submodule is used for calculating the Loss of the prediction error category by adopting a Loss function, setting the Loss function of the depression recognition neural network model as a cross entropy function Loss, and the formula is as follows:
wherein M represents the number of categories; c represents a category number; y iscRepresenting a real tag; p is a radical ofcIndicating the output of softmax.
An electronic device, comprising: a memory and a processor;
wherein the memory stores computer-executable instructions;
the one processor executes the computer-executable instructions stored by the memory, so that the one processor executes the depression identification method based on the human skeleton kinematic feature information.
A computer readable storage medium, wherein computer executable instructions are stored in the computer readable storage medium, and when a processor executes the computer, the depression identification method based on human skeleton kinematic feature information is realized.
The depression identification method and system based on the human skeleton kinematics characteristic information have the following advantages:
according to the clinical manifestations of hypokinesia, thought and cognitive function retardation of depression patients, a high-efficiency depression identification method is constructed by designing specific instruction actions and using human skeleton kinematic characteristic information acquired by Kinect equipment, and a convenient and reliable screening means is provided for early identification and auxiliary clinical diagnosis and treatment of depression;
the method comprises the following steps of (II) completing capture of human body kinematic characteristic skeleton information through Kinect equipment, combining with data extraction and preprocessing technologies, and constructing a depression recognition model based on deep learning related theories to recognize depression, so that the accuracy of depression recognition is improved;
the invention is based on the time convolution bottleneck network recognition method, construct human skeleton kinematic characteristic learning network model, withdraw the key kinematic characteristic information, has avoided the interference that the irrelevant characteristic discerns depression effectively, improve the reliability that the depression discerns, the test set recognition rate accuracy rate to depression of the invention is up to more than 75%;
the human skeleton recognition task is completed by using Kinect equipment, and depression recognition work is performed on the basis of human body kinematic characteristic information of time dimension and space position; the stimulation task provided by the invention is originally designed according to the suggestion of psychiatrists, and a noise data removing and kinematic segment dividing and data extracting method is provided according to the action of the set stimulation task and by combining with experimental design; meanwhile, the identification of the depression by the time convolution and bottleneck neural network model more suitable for the method is improved based on the original time convolution neural network model.
Drawings
The invention is further described below with reference to the accompanying drawings.
FIG. 1 is a flow chart of a depression identification method based on human skeleton kinematic feature information;
FIG. 2 is a diagram of a human skeleton action key action frame;
FIG. 3 is a schematic diagram of an experimental environment layout of a Kinect device;
FIG. 4 is a schematic diagram of a neural network model structure;
fig. 5 is a schematic diagram of a confusion matrix of depression recognition results.
Detailed Description
The depression identification method and system based on human skeleton kinematics characteristic information of the invention are explained in detail with reference to the drawings and the specific embodiments in the specification.
Example 1:
as shown in the attached drawing 1, the method for identifying depression based on human skeleton kinematic feature information extracts 25 joint kinematic feature information of a human body according to the human skeleton feature information captured by a Kinect device, performs data preprocessing on spatial position data of each joint based on time dimension, constructs a data set by using the preprocessed data, and performs training and testing of a depression identification neural network model by using the data set; the method comprises the following specific steps:
s1, collecting human skeleton kinematic characteristic data; the method comprises the following specific steps:
s101, starting a Kinect device to collect kinematic characteristic data of a human skeleton;
s102, guiding the experimental subject to enter a designated position, and playing a pre-designed stimulation action task instruction, as shown in the attached figure 3;
and (3) stimulation action design: the invention combines the actual physical state of the depression patients according to the suggestion of psychiatrists, and simultaneously reduces the difference of the perception and action abilities of the depression patients caused by non-critical factors such as gender, occupation and the like; therefore, a set of stimulation task actions which are simple, relatively obvious in motion amplitude and convenient to capture by the Kinect is designed.
As shown in fig. 2, the stimulation action comprises a total of five kinematic segments, respectively: lifting and resetting a left arm, lifting and resetting a right arm, lifting and resetting two arms, turning left and resetting, and turning right and resetting; the time interval between each kinematic segment is 5s, and the time required for completing the whole stimulation action is 60 s;
s103, opening Kinect studio V2 software to complete the recording work of human skeleton kinematic data;
s104, after the data recording is finished, the psychologist evaluates the depression state of the current experimental subject, the evaluation process records the Hamilton depression scale (24 versions) scores of the experimental subject, and reference basis is provided for screening depression group data and non-depression group data and labeling data of subsequent experiments; wherein, the experimental subjects are divided into a depression group and a non-depression group;
and S105, screening effective experimental data when the score of the depression group is more than 20 and the score of the control group is less than 8 according to the Hamilton depression scale score standard.
S2, extracting human skeleton kinematic characteristic data; the method comprises the following specific steps:
s201, traversing xef recording files in the folder, acquiring file names and obtaining recording file paths;
s202, executing a script command to run SkeletonExtractor.exe in batch to open xef record files and read original record data; among them, the. xef record file is based on the. net core3.0 framework, which has developed for Kinect V2.0 records;
s203, extracting time series data based on the xef recorded files;
s204, extracting the spatial positions (x, y, z) of the human skeleton joint points, and eliminating noise data; since there may be non-target objects appearing in the video recording environment during the process of recording the target object by the Kinect V2, noise data such as non-experimental objects need to be eliminated and captured possibly. Kinect can detect out 6 human skeleton targets at most simultaneously, and according to experimental environment design, target object and amazing action home range all are in fixed detection area, specifically are: for any Kinect detected target i 1,2,3 …, each human body skeleton joint point detects a spatial position (x, y, z), and for any time t, the target object i to be locked has the following relation:
s205, extracting quaternion ((Rx, Ry, Rz), Rw) of the spatial coordinate system of the data subjected to the noise reduction processing, and the formula is as follows:
each captured human skeletal joint point is composed of (x, y, z) and ((Rx, Ry, Rz), Rw), that is, each human skeletal joint point n at the time t of the kinematic segment is composed of seven dimensional data expressing the spatial position relationship:
djt＝[xjt，yjt，zjt，Rxjt，Ryjt，Rzjt，Rwjt]；
and for 25 human skeleton joint points captured by the Kinect, the human skeleton data extracted within the kinematic fragment time T is as follows:
wherein t represents time; j represents a human skeletal joint point; t is more than or equal to 1 and less than or equal to T, j is more than or equal to 1 and less than or equal to 25;
s206, obtaining the extraction result (time, the space position coordinate value based on the time dimension of each joint and quaternion) of the experimental object, and saving the extraction result as a file in the csv format by the name of the experimental object.
S3, preprocessing the characteristic data and making a data set; the method comprises the following specific steps:
s301, reading the extracted csv file of the experimental object;
s302, reading the time sequence and rewriting the time sequence into a csv file with the same name in another folder;
s303, reading data of the csv file except the time sequence, and preprocessing the adopted standardized data, wherein the method specifically comprises the following steps: for human skeleton data time series x1，x2，...，xtThe linear transformation of the raw data using a dispersion normalization method, with the result mapped to [0,1]Interval, the formula is as follows:
where n denotes the sequence length, the new sequence y1，y2，...，yn∈[0，1]And is dimensionless;
and the depressed group data and the non-depressed group data are respectively positioned in the two folders, the two folders are traversed, and the csv files in the folders are read.
S304, writing the standardized data into the csv file in the step S302;
s305, labeling the experimental subjects of a depressed group and a non-depressed group respectively, labeling the depressed group as 0, and labeling the non-depressed group as 1;
s306, in order to improve the loading efficiency of experimental data, data set production is carried out through a new data sequence preprocessed by a pythonnnumpy library, and data extracted by all experimental objects are stored as a npy format file in a binary data format;
and S307, adopting a random division mode for the data set, setting 70% of the data set as a training set, and using 30% of the data set as a test set.
S4, constructing and training a depression recognition neural network model; as shown in fig. 4, the following is detailed:
s401, inputting training set data into a time Convolution Neural Network (Temporal Convolution Neural Network) to obtain an output result of the time Convolution Neural Network; the time convolution neural network comprises two time hole convolution residual blocks which are sequentially connected, each layer of each residual block is a one-dimensional hole convolution network, the random discarding rate is 0.5, the activation function is ReLU, and the output of each layer is directly used as the input of the next layer;
s402, inputting the obtained time convolution neural Network output result as characteristic information into a bottleneck convolution neural Network (BottleneckConvolationneural Network) to obtain an output result of the bottleneck convolution neural Network; wherein the bottleneck convolutional neural network comprises a bottleneck network residual block; the bottleneck network residual block comprises three layers of one-dimensional convolution networks, and the kernel function size K of the one-dimensional convolution networks is 1,3 and 1;
and S403, converting the output result of the bottleneck convolutional neural network into a one-dimensional vector, connecting the one-dimensional vector and the one-dimensional vector, inputting the one-dimensional vector into a full connection layer, and then identifying the depression through a softmax classifier.
The human skeleton kinematics feature learning network comprises a time convolution network, a bottleneck network, a full connection layer and a softmax classifier. Based on the existing time convolution neural network, the advantages of various networks are fully utilized, a learning network with efficient depression recognition is constructed, the kinematic skeleton characteristics of a human body are extracted, the characteristics output by the network are aggregated and then input into a full connection layer for characteristic fusion, and depression recognition is carried out based on a classical softmax classifier after the full connection layer.
For the kinematics skeleton characteristic learning, the invention designs a standard convolution filter to fix time understanding, and can ideally model the whole kinematics skeleton characteristic sequence by modeling a proper time window. Important kinematic characteristics are learned, and the scheme uses extended convolution to realize exponential time sequence scale on different layers of the network. For input event time index length and filter {0,1, … n } for some filters, the expanding convolution operation of the elements of the skeleton sequence is:
wherein d represents an expansion coefficient; n denotes an index length.
The bottleneck network adopts a one-dimensional convolution bottle network structure, the size of the three-layer kernel filter is matched to be K which is 1,3 and 1, and the compatibility of the length between layers is ensured. The bottleneck feature may be generated from a multi-layer perceptron, with an internal layer having a small number of hidden elements, relative to the relative sizes of other layers. Once the inner layer estimate and the fully connected layer estimate are available, the classifier block can track the exact source of relevant information through compressed sensing. Due to the forward augmented time convolution residual block feed, the bottleneck structure can better identify the relevant body event index sequence and key body joint movement mechanics characteristics.
S5, inputting the human skeleton kinematic data to be recognized into the trained depression recognition neural network model to complete the recognition and diagnosis of depression, obtaining a prediction result and calculating the accuracy; the method comprises the following specific steps:
s501, identifying and diagnosing the depression by using softmax as a classifier, wherein the formula is as follows:
wherein S isiRepresenting a softmax predicted value of the ith class; i and j are category serial numbers; the invention provides a human skeleton kinematic feature sequence model with efficient calculation on the basis of a given group of sequence data.
S502, inputting the human skeleton kinematic feature data into a depression recognition neural network model, wherein when depression category prediction is carried out, each category (depression and non-depression) corresponds to a calculation value smaller than 1, the sum is 1, and the category corresponding to the maximum calculation value is a prediction category;
s503, comparing the prediction categories with the real categories, calculating the proportion of the number of the depression categories of the correct prediction categories in the training data set to the total number of the data, and outputting the accuracy of the depression recognition neural network model;
s504, calculating the Loss of the prediction error category by adopting a Loss function, setting the Loss function of the depression recognition neural network model as a cross entropy function Loss, and adopting the following formula:
wherein M represents the number of categories; c represents a category number; y iscRepresenting a real tag; p is a radical ofcIndicating the output of softmax.
For example, as shown in fig. 5, in the confusion matrix, the total number of samples contained in the test set is 62, 23 samples in the depression group, and 39 samples in the normal group; wherein the number of predicted depressed groups is 16 and the number of predicted normal groups is 7; the number of normal groups predicted to be normal groups was 31, and the number of predicted depressed groups was 8; the confusion matrix shows 75.8% accuracy as (31+ 16)/62.
Example 2:
the invention relates to a depression recognition system based on human skeleton kinematics characteristic information, which comprises,
the data collection module is used for collecting human skeleton kinematic characteristic data; the data collection module comprises a data acquisition module,
the device starting and data collecting submodule is used for starting the Kinect device to collect the kinematic characteristic data of the human skeleton;
the instruction playing submodule is used for guiding the experimental subject to enter a designated position and playing a pre-designed stimulation action task instruction; wherein the stimulation action comprises a total of five kinematic segments, respectively: lifting and resetting a left arm, lifting and resetting a right arm, lifting and resetting two arms, turning left and resetting, and turning right and resetting; the time interval between each kinematic segment is 5s, and the time required for completing the whole stimulation action is 60 s;
the data recording submodule is used for opening Kinect studio V2 software to finish the recording work of human skeleton kinematic data;
a depression state evaluation submodule for the psychiatrist to evaluate the depression state of the current subject after completing the data recording, wherein the evaluation process records the Hamilton depression scale (24 edition) scores of the subject; wherein, the experimental subjects are divided into a depression group and a non-depression group;
and the screening submodule is used for screening effective experimental data when the score of the depression group is more than 20 and the score of the control group is less than 8 according to the Hamilton depression scale scoring standard.
The data extraction module is used for extracting human skeleton kinematic characteristic data; the data extraction module comprises a data extraction module,
the file path recording submodule is used for traversing xef recorded files in the folder, acquiring file names and obtaining recording file paths;
an original record data reading submodule, configured to execute a script command to run skeletonextractor in batch to open.exe, xef record a file, and read original record data;
a time-series data extraction sub-module for extracting time-series data based on the xef record file;
the noise data removing submodule is used for extracting the spatial positions (x, y, z) of the human skeleton joint points and removing the noise data; the method specifically comprises the following steps: for any Kinect detected target i 1,2,3 …, each human body skeleton joint point detects a spatial position (x, y, z), and for any time t, the target object i to be locked has the following relation:
and the quaternion extraction submodule is used for extracting quaternion ((Rx, Ry, Rz), Rw) of a space coordinate system of the data subjected to the noise reduction processing, and the formula is as follows:
each captured human skeletal joint point is composed of (x, y, z) and ((Rx, Ry, Rz), Rw), that is, each human skeletal joint point n at the time t of the kinematic segment is composed of seven dimensional data expressing the spatial position relationship:
djt＝[xjt，yjt，zjt，Rxjt，Ryjt，Rzjt，Rwjt]；
and for 25 human skeleton joint points captured by the Kinect, the human skeleton data extracted within the kinematic fragment time T is as follows:
wherein t represents time; j represents a human skeletal joint point; t is more than or equal to 1 and less than or equal to T, j is more than or equal to 1 and less than or equal to 25;
and the extraction result obtaining submodule is used for obtaining the extraction result (time, the space position coordinate value based on the time dimension of each joint and the quaternion) of the experimental object and saving the extraction result as a file in the csv format by the name of the experimental object.
The preprocessing and data set making module is used for preprocessing the characteristic data and making a data set; the preprocessing and data set generation module comprises a preprocessing module,
the file reading submodule is used for reading the extracted csv file of the experimental object;
the time sequence reading submodule is used for reading the time sequence and rewriting the time sequence into a csv file with the same name in another folder;
the standardized data preprocessing submodule is used for reading the data of the csv file except the time sequence and preprocessing the adopted standardized data, and specifically comprises the following steps: for human skeleton data time series x1，x2，...，xtThe linear transformation of the raw data using a dispersion normalization method, with the result mapped to [0,1]Interval, the formula is as follows:
where n denotes the sequence length, the new sequence y1，y2，...，yn∈[0，1]And is dimensionless;
the file writing submodule is used for writing the standardized data into the csv file;
a subject labeling submodule for labeling the experimental subjects of the depressed group and the non-depressed group, respectively, labeling the depressed group as 0 and labeling the non-depressed group as 1;
the data set making submodule is used for making a data set through a new data sequence preprocessed by a pythonnnumpy library, and the data extracted by all experimental objects are stored as a. npy format file in a binary data format;
and the data set division submodule is used for adopting a random division mode on the data set, setting 70% of the data set as a training set, and using 30% of the data set as a test set.
The model building and training module is used for building and training a depression recognition neural network model; the model building and training module comprises a model building and training module,
the time Convolution Neural Network result output submodule is used for inputting the training set data into a time Convolution Neural Network (Temporal Convolution Neural Network) to obtain an output result of the time Convolution Neural Network; the time convolution neural network comprises two time hole convolution residual blocks which are sequentially connected, each layer of each residual block is a one-dimensional hole convolution network, the random discarding rate is 0.5, the activation function is ReLU, and the output of each layer is directly used as the input of the next layer;
a bottleneck convolutional neural Network result output submodule, configured to input the obtained time convolutional neural Network output result as feature information into a bottleneck convolutional neural Network (bottleneckconvolutional convolutional neural Network), so as to obtain an output result of the bottleneck convolutional neural Network; wherein the bottleneck convolutional neural network comprises a bottleneck network residual block; the bottleneck network residual block comprises three layers of one-dimensional convolution networks, and the kernel function size K of the one-dimensional convolution networks is 1,3 and 1;
and the depression identification submodule is used for converting the output result of the bottleneck convolutional neural network into a one-dimensional vector, connecting the one-dimensional vector and the one-dimensional vector, inputting the one-dimensional vector into the full connection layer, and then identifying the depression through a softmax classifier.
The result prediction and accuracy rate calculation module is used for inputting the human skeleton kinematic data to be recognized into the trained depression recognition neural network model to complete the recognition diagnosis of the depression, obtain the prediction result and calculate the accuracy rate; the result prediction and accuracy calculation module comprises,
the identification and diagnosis submodule is used for identifying and diagnosing the depression by adopting softmax as a classifier, and the formula is as follows:
wherein S isiRepresenting a softmax predicted value of the ith class; i and j are category serial numbers;
the prediction class acquisition submodule is used for inputting the human skeleton kinematic feature data into the depression recognition neural network model, when depression class prediction is carried out, each class (depression and non-depression) corresponds to a calculation value smaller than 1, the sum is 1, and the class corresponding to the maximum calculation value is a prediction class;
the comparison submodule is used for comparing the prediction category with the real category, calculating the proportion of the number of depression categories of the correct prediction category in the training data set to the total number of data, and outputting the accuracy of the depression recognition neural network model;
the Loss calculation submodule is used for calculating the Loss of the prediction error category by adopting a Loss function, setting the Loss function of the depression recognition neural network model as a cross entropy function Loss, and the formula is as follows:
wherein M represents the number of categories; c represents a category number; y iscRepresenting a real tag; p is a radical ofcIndicating the output of softmax.
Example 3:
an embodiment of the present invention further provides an electronic device, including: a memory and at least one processor;
wherein the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform a depression identification method based on human skeletal kinematics characteristic information as in embodiment 1.
Example 4:
the embodiment of the invention also provides a computer-readable storage medium, wherein a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the depression identification method based on the human skeleton kinematic feature information in any embodiment of the invention. Specifically, a system or an apparatus equipped with a storage medium on which software program codes that realize the functions of any of the above-described embodiments are stored may be provided, and a computer (or a CPU or MPU) of the system or the apparatus is caused to read out and execute the program codes stored in the storage medium.
In this case, the program code itself read from the storage medium can realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
Examples of the storage medium for supplying the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-R depression recognition method based on human skeleton kinematics characteristic information and system M, DVD-RW, DVD + RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer via a communications network.
Further, it should be clear that the functions of any one of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform a part or all of the actual operations based on instructions of the program code.
Further, it is to be understood that the program code read out from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then causes a CPU or the like mounted on the expansion board or the expansion unit to perform part or all of the actual operations based on instructions of the program code, thereby realizing the functions of any of the above-described embodiments.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A depression recognition method based on human skeleton kinematic feature information is characterized in that the method comprises the steps of extracting human joint kinematic feature information according to the human skeleton feature information captured by Kinect equipment, carrying out data preprocessing on spatial position data of each joint based on time dimension, constructing a data set by utilizing the preprocessed data, and carrying out training and testing on a depression recognition neural network model by using the data set; the method comprises the following specific steps:
s1, collecting human skeleton kinematic characteristic data;
s2, extracting human skeleton kinematic characteristic data;
s3, preprocessing the characteristic data and making a data set;
s4, constructing and training a depression recognition neural network model;
and S5, inputting the human skeleton kinematic data to be recognized into the trained depression recognition neural network model, completing the recognition and diagnosis of depression, obtaining a prediction result and calculating the accuracy.
2. The method for identifying depression based on human skeleton kinematic feature information according to claim 1, wherein the step S1 is to collect human skeleton kinematic feature data as follows:
s101, starting a Kinect device to collect kinematic characteristic data of a human skeleton;
s102, guiding the experimental subject to enter a designated position, and playing a pre-designed stimulation action task instruction; wherein the stimulation action comprises a total of five kinematic segments, respectively: lifting and resetting a left arm, lifting and resetting a right arm, lifting and resetting two arms, turning left and resetting, and turning right and resetting; the time interval between each kinematic segment is 5s, and the time required for completing the whole stimulation action is 60 s;
s103, opening Kinect studio V2 software to complete the recording work of human skeleton kinematic data;
s104, after the data recording is finished, the psychologist evaluates the depression state of the current experimental subject, and the evaluation process records the Hamilton depression scale score of the experimental subject; wherein, the experimental subjects are divided into a depression group and a non-depression group;
and S105, screening effective experimental data when the score of the depression group is more than 20 and the score of the control group is less than 8 according to the Hamilton depression scale score standard.
3. The method for identifying depression based on human skeleton kinematic feature information according to claim 1, wherein the step S2 is to extract human skeleton kinematic feature data as follows:
s201, traversing xef recording files in the folder, acquiring file names and obtaining recording file paths;
s202, executing a script command to run SkeletonExtractor.exe in batch to open xef record files and read original record data;
s203, extracting time series data based on the xef recorded files;
s204, extracting the spatial positions (x, y, z) of the human skeleton joint points, and eliminating noise data; the method specifically comprises the following steps: for any Kinect detected target i 1,2,3 …, each human body skeleton joint point detects a spatial position (x, y, z), and for any time t, the target object i to be locked has the following relation:
s205, extracting quaternion ((Rx, Ry, Rz), Rw) of the spatial coordinate system of the data subjected to the noise reduction processing, and the formula is as follows:
each captured human skeletal joint point is composed of (x, y, z) and ((Rx, Ry, Rz), Rw), that is, each human skeletal joint point n at the time t of the kinematic segment is composed of seven dimensional data expressing the spatial position relationship:
djt＝[xjt，yjt，zjt，Rxjt，Ryjt，Rzjt，Rwjt]；
and for the human skeleton joint points captured by the Kinect, the human skeleton data extracted within the kinematic fragment time T is as follows:
wherein t represents time; j represents a human skeletal joint point; t is more than or equal to 1 and less than or equal to T, j is more than or equal to 1 and less than or equal to 25;
s206, obtaining the extraction result of the experimental object, and saving the extraction result as a file in the csv format by the name of the experimental object.
4. The method for identifying depression based on human skeleton kinematic feature information as claimed in claim 1, wherein the feature data preprocessing and making data set in step S3 are as follows:
s301, reading the extracted csv file of the experimental object;
s302, reading the time sequence and rewriting the time sequence into a csv file with the same name in another folder;
s303, reading data of the csv file except the time sequence, and preprocessing the adopted standardized data, wherein the method specifically comprises the following steps: for human skeleton data time series x1，x2，...，xtThe linear transformation of the raw data using a dispersion normalization method, with the result mapped to [0,1]Interval, the formula is as follows:
where n denotes the sequence length, the new sequence y1，y2，...，yn∈[0，1]And is dimensionless;
s304, writing the standardized data into the csv file in the step S302;
s305, labeling the experimental subjects of a depressed group and a non-depressed group respectively, labeling the depressed group as 0, and labeling the non-depressed group as 1;
s306, making a data set through a new data sequence preprocessed by a pythonnnumpy library, and storing data extracted by all experimental objects in a binary data format as a npy format file;
and S307, adopting a random division mode for the data set, setting 70% of the data set as a training set, and using 30% of the data set as a test set.
5. The method for identifying depression based on human skeleton kinematic feature information of claim 1, wherein the step S4 of constructing and training a depression identification neural network model is as follows:
s401, inputting training set data into a time convolution neural network to obtain a time convolution neural network output result; the time convolution neural network comprises two time hole convolution residual blocks which are sequentially connected, each layer of each residual block is a one-dimensional hole convolution network, the random discarding rate is 0.5, the activation function is ReLU, and the output of each layer is directly used as the input of the next layer;
s402, inputting the obtained time convolution neural network output result into a bottleneck convolution neural network as characteristic information to obtain an output result of the bottleneck convolution neural network; wherein the bottleneck convolutional neural network comprises a bottleneck network residual block; the bottleneck network residual block comprises three layers of one-dimensional convolution networks, and the kernel function size K of the one-dimensional convolution networks is 1,3 and 1;
and S403, converting the output result of the bottleneck convolutional neural network into a one-dimensional vector, connecting the one-dimensional vector and the one-dimensional vector, inputting the one-dimensional vector into a full connection layer, and then identifying the depression through a softmax classifier.
6. The method for identifying depression based on human skeleton kinematic feature information according to any one of claims 1 to 5, wherein the step S5 is implemented to identify and diagnose depression, obtain the prediction result and calculate the accuracy specifically as follows:
s501, identifying and diagnosing the depression by using softmax as a classifier, wherein the formula is as follows:
wherein S isiRepresenting a softmax predicted value of the ith class; i and j are category serial numbers;
s502, inputting human skeleton kinematic feature data into a depression recognition neural network model, wherein each category corresponds to a calculation value smaller than 1 and the sum is 1 when depression category prediction is carried out, and the category corresponding to the maximum calculation value is a prediction category;
s503, comparing the prediction categories with the real categories, calculating the proportion of the number of the depression categories of the correct prediction categories in the training data set to the total number of the data, and outputting the accuracy of the depression recognition neural network model;
s504, calculating the Loss of the prediction error category by adopting a Loss function, setting the Loss function of the depression recognition neural network model as a cross entropy function Loss, and adopting the following formula:
wherein M represents the number of categories; c represents a category number; y iscRepresenting a real tag; p is a radical ofcIndicating the output of softmax.
7. A depression recognition system based on human skeleton kinematics characteristic information is characterized by comprising,
the data collection module is used for collecting human skeleton kinematic characteristic data;
the data extraction module is used for extracting human skeleton kinematic characteristic data;
the preprocessing and data set making module is used for preprocessing the characteristic data and making a data set;
the model building and training module is used for building and training a depression recognition neural network model;
and the result prediction and accuracy rate calculation module is used for inputting the human skeleton kinematic data to be recognized into the trained depression recognition neural network model to complete the recognition diagnosis of the depression, obtain the prediction result and calculate the accuracy rate.
8. The system for identifying depression according to claim 7, wherein the data collecting module comprises,
the device starting and data collecting submodule is used for starting the Kinect device to collect the kinematic characteristic data of the human skeleton;
the instruction playing submodule is used for guiding the experimental subject to enter a designated position and playing a pre-designed stimulation action task instruction; wherein the stimulation action comprises a total of five kinematic segments, respectively: lifting and resetting a left arm, lifting and resetting a right arm, lifting and resetting two arms, turning left and resetting, and turning right and resetting; the time interval between each kinematic segment is 5s, and the time required for completing the whole stimulation action is 60 s;
the data recording submodule is used for opening Kinect studio V2 software to finish the recording work of human skeleton kinematic data;
the depression state evaluation submodule is used for evaluating the depression state of the current experimental subject by a psychologist after data recording is finished, and the evaluation process records the Hamilton depression scale score of the experimental subject; wherein, the experimental subjects are divided into a depression group and a non-depression group;
the screening submodule is used for screening effective experimental data when the score of a depression group is larger than 20 and the score of a control group is smaller than 8 according to the Hamilton depression scale scoring standard;
the data extraction module comprises a data extraction module and a data extraction module,
the file path recording submodule is used for traversing xef recorded files in the folder, acquiring file names and obtaining recording file paths;
an original record data reading submodule, configured to execute a script command to run skeletonextractor.exe in batch to open a xef record file, and read original record data;
a time-series data extraction sub-module for extracting time-series data based on the xef record file;
the noise data removing submodule is used for extracting the spatial positions (x, y, z) of the human skeleton joint points and removing the noise data; the method specifically comprises the following steps: for any Kinect detected target i 1,2,3 …, each human body skeleton joint point detects a spatial position (x, y, z), and for any time t, the target object i to be locked has the following relation:
and the quaternion extraction submodule is used for extracting quaternion ((Rx, Ry, Rz), Rw) of a space coordinate system of the data subjected to the noise reduction processing, and the formula is as follows:
each captured human skeletal joint point is composed of (x, y, z) and ((Rx, Ry, Rz), Rw), that is, each human skeletal joint point n at the time t of the kinematic segment is composed of seven dimensional data expressing the spatial position relationship:
djt＝[xjt，yjt，zjt，Rxjt，Ryjt，Rzjt，Rwjt]；
and for the human skeleton joint points captured by the Kinect, the human skeleton data extracted within the kinematic fragment time T is as follows:
wherein t represents time; j represents a human skeletal joint point; t is more than or equal to 1 and less than or equal to T, j is more than or equal to 1 and less than or equal to 25;
the extraction result obtaining submodule is used for obtaining the extraction result of the experimental object and saving the extraction result as a file in a csv format by the name of the experimental object;
the preprocessing and data set creation module includes,
the file reading submodule is used for reading the extracted csv file of the experimental object;
the time sequence reading submodule is used for reading the time sequence and rewriting the time sequence into a csv file with the same name in another folder;
the standardized data preprocessing submodule is used for reading the data of the csv file except the time sequence and preprocessing the adopted standardized data, and specifically comprises the following steps: for human skeleton data time series x1，x2，...，xtThe linear transformation of the raw data using a dispersion normalization method, with the result mapped to [0,1]Interval, the formula is as follows:
where n denotes the sequence length, the new sequence y1，y2，...，yn∈[0，1]And is dimensionless;
the file writing submodule is used for writing the standardized data into the csv file;
a subject labeling submodule for labeling the experimental subjects of the depressed group and the non-depressed group, respectively, labeling the depressed group as 0 and labeling the non-depressed group as 1;
the data set making submodule is used for making a data set through a new data sequence preprocessed by a pythonnnumpy library, and the data extracted by all experimental objects are stored as a. npy format file in a binary data format;
the data set dividing submodule is used for adopting a random dividing mode for the data set, setting 70% of the data set as a training set, and using 30% of the data set as a testing set;
the model building and training module comprises a model building and training module,
the time convolution neural network result output submodule is used for inputting the training set data to the time convolution neural network to obtain an output result of the time convolution neural network; the time convolution neural network comprises two time hole convolution residual blocks which are sequentially connected, each layer of each residual block is a one-dimensional hole convolution network, the random discarding rate is 0.5, the activation function is ReLU, and the output of each layer is directly used as the input of the next layer;
the bottleneck convolutional neural network result output submodule is used for inputting the obtained time convolutional neural network output result into the bottleneck convolutional neural network as characteristic information to obtain an output result of the bottleneck convolutional neural network; wherein the bottleneck convolutional neural network comprises a bottleneck network residual block; the bottleneck network residual block comprises three layers of one-dimensional convolution networks, and the kernel function size K of the one-dimensional convolution networks is 1,3 and 1;
the depression recognition submodule is used for converting an output result of the bottleneck convolutional neural network into a one-dimensional vector, connecting the one-dimensional vector and the one-dimensional vector, inputting the one-dimensional vector into the full connection layer, and then recognizing the depression through a softmax classifier;
the result prediction and accuracy calculation module comprises,
the identification and diagnosis submodule is used for identifying and diagnosing the depression by adopting softmax as a classifier, and the formula is as follows:
wherein S isiRepresenting a softmax predicted value of the ith class; i and j are category serial numbers;
the prediction class acquisition submodule is used for inputting the human skeleton kinematic feature data into the depression recognition neural network model, when depression class prediction is carried out, each class corresponds to a calculation value smaller than 1, the sum is 1, and the class corresponding to the maximum calculation value is a prediction class;
the comparison submodule is used for comparing the prediction category with the real category, calculating the proportion of the number of depression categories of the correct prediction category in the training data set to the total number of data, and outputting the accuracy of the depression recognition neural network model;
the Loss calculation submodule is used for calculating the Loss of the prediction error category by adopting a Loss function, setting the Loss function of the depression recognition neural network model as a cross entropy function Loss, and the formula is as follows:
wherein M represents the number of categories; c represents a category number; y iscRepresenting a real tag; p is a radical ofcIndicating the output of softmax.
9. An electronic device, comprising: a memory and at least one processor;
wherein the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method for identifying depression based on human skeletal kinematics feature information according to any of claims 1 to 6.
10. A computer-readable storage medium having stored thereon computer-executable instructions, which when executed by a processor, implement the method for identifying depression based on human skeletal kinematics characteristic information as claimed in claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010608130.9A CN111938670B (en) | 2020-06-30 | 2020-06-30 | Depression identification method and system based on human skeleton kinematics characteristic information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010608130.9A CN111938670B (en) | 2020-06-30 | 2020-06-30 | Depression identification method and system based on human skeleton kinematics characteristic information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111938670A true CN111938670A (en) | 2020-11-17 |
| CN111938670B CN111938670B (en) | 2022-07-05 |
Family
ID=73337907
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010608130.9A Active CN111938670B (en) | 2020-06-30 | 2020-06-30 | Depression identification method and system based on human skeleton kinematics characteristic information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111938670B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113111760A (en) * | 2021-04-07 | 2021-07-13 | 同济大学 | Lightweight graph convolution human skeleton action identification method based on channel attention |
| CN113793662A (en) * | 2021-08-16 | 2021-12-14 | 北京百度网讯科技有限公司 | Correlation identification method, correlation identification device, electronic equipment and computer-readable storage medium |
| CN113827243A (en) * | 2021-11-29 | 2021-12-24 | 江苏瑞脑启智医疗科技有限公司 | Attention assessment method and system |
| CN116434908A (en) * | 2023-06-14 | 2023-07-14 | 中国科学院自动化研究所 | Method and device for quantitatively and hierarchically evaluating bradykinesia based on time convolution network |
| CN117530691A (en) * | 2024-01-09 | 2024-02-09 | 南通大学 | Depression tendency detection system, method and related equipment based on indoor network |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012075708A (en) * | 2010-10-01 | 2012-04-19 | Sharp Corp | Stress state estimation device, stress state estimation method, program, and recording medium |
| CN104615983A (en) * | 2015-01-28 | 2015-05-13 | 中国科学院自动化研究所 | Behavior identification method based on recurrent neural network and human skeleton movement sequences |
| CN106175800A (en) * | 2016-07-10 | 2016-12-07 | 北京工业大学 | A kind of Mood State quantitative estimation method based on human motion state data |
| US20170251985A1 (en) * | 2016-02-12 | 2017-09-07 | Newton Howard | Detection Of Disease Conditions And Comorbidities |
| US20180039745A1 (en) * | 2016-08-02 | 2018-02-08 | Atlas5D, Inc. | Systems and methods to identify persons and/or identify and quantify pain, fatigue, mood, and intent with protection of privacy |
| CN111178312A (en) * | 2020-01-02 | 2020-05-19 | 西北工业大学 | Face expression recognition method based on multi-task feature learning network |
-
2020
- 2020-06-30 CN CN202010608130.9A patent/CN111938670B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012075708A (en) * | 2010-10-01 | 2012-04-19 | Sharp Corp | Stress state estimation device, stress state estimation method, program, and recording medium |
| CN104615983A (en) * | 2015-01-28 | 2015-05-13 | 中国科学院自动化研究所 | Behavior identification method based on recurrent neural network and human skeleton movement sequences |
| US20170251985A1 (en) * | 2016-02-12 | 2017-09-07 | Newton Howard | Detection Of Disease Conditions And Comorbidities |
| CN106175800A (en) * | 2016-07-10 | 2016-12-07 | 北京工业大学 | A kind of Mood State quantitative estimation method based on human motion state data |
| US20180039745A1 (en) * | 2016-08-02 | 2018-02-08 | Atlas5D, Inc. | Systems and methods to identify persons and/or identify and quantify pain, fatigue, mood, and intent with protection of privacy |
| CN111178312A (en) * | 2020-01-02 | 2020-05-19 | 西北工业大学 | Face expression recognition method based on multi-task feature learning network |
Non-Patent Citations (1)
| Title |
|---|
| JOSHI JYOTI，ET AL: "Relative Body Parts Movement for Automatic Depression Analysis", 《2013 HUMAINE ASSOCIATION CONFERENCE ON AFFECTIVE COMPUTING AND INTELLIGENT INTERACTION》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113111760A (en) * | 2021-04-07 | 2021-07-13 | 同济大学 | Lightweight graph convolution human skeleton action identification method based on channel attention |
| CN113793662A (en) * | 2021-08-16 | 2021-12-14 | 北京百度网讯科技有限公司 | Correlation identification method, correlation identification device, electronic equipment and computer-readable storage medium |
| CN113793662B (en) * | 2021-08-16 | 2023-07-28 | 北京百度网讯科技有限公司 | Correlation identification method, apparatus, electronic device, and computer-readable storage medium |
| CN113827243A (en) * | 2021-11-29 | 2021-12-24 | 江苏瑞脑启智医疗科技有限公司 | Attention assessment method and system |
| CN116434908A (en) * | 2023-06-14 | 2023-07-14 | 中国科学院自动化研究所 | Method and device for quantitatively and hierarchically evaluating bradykinesia based on time convolution network |
| CN117530691A (en) * | 2024-01-09 | 2024-02-09 | 南通大学 | Depression tendency detection system, method and related equipment based on indoor network |
| CN117530691B (en) * | 2024-01-09 | 2024-04-09 | 南通大学 | Depression tendency detection system, method and related equipment based on indoor network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111938670B (en) | 2022-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111938670B (en) | Depression identification method and system based on human skeleton kinematics characteristic information | |
| CN111160367B (en) | Image classification method, apparatus, computer device, and readable storage medium | |
| US20220051405A1 (en) | Image processing method and apparatus, server, medical image processing device and storage medium | |
| US11715198B2 (en) | Medical use artificial neural network-based medical image analysis apparatus and method for evaluating analysis results of medical use artificial neural network | |
| CN103177179A (en) | Diagnostic factor set determination apparatus and method | |
| Garcia Caballero et al. | V‐awake: a visual analytics approach for correcting sleep predictions from deep learning models | |
| CN116864140A (en) | Intracardiac branch of academic or vocational study postoperative care monitoring data processing method and system thereof | |
| Amiri et al. | Feature selection for bleeding detection in capsule endoscopy images using genetic algorithm | |
| CN116452592B (en) | Method, device and system for constructing brain vascular disease AI cognitive function evaluation model | |
| CN113658697A (en) | Psychological assessment system based on video fixation difference | |
| US11817216B2 (en) | Search method and information processing system | |
| CN111081374B (en) | Autism auxiliary diagnosis device based on common attention paradigm | |
| CN113888663B (en) | Reconstruction model training method, anomaly detection method, device, equipment and medium | |
| US11571156B2 (en) | Renal function assessment method, renal function assessment system and kidney care device | |
| WO2022047043A1 (en) | Surgical phase recognition with sufficient statistical model | |
| Yoo et al. | Residual one-dimensional convolutional neural network for neuromuscular disorder classification from needle electromyography signals with explainability | |
| Huang et al. | Parkinson's severity diagnosis explainable model based on 3D multi-head attention residual network | |
| JP7506943B2 (en) | Signal processing method, signal processing device, and signal processing program | |
| TWI764233B (en) | Alzheimer's disease assessment system | |
| CN113393940B (en) | Control method and medical system | |
| CN117893528B (en) | Method and device for constructing cardiovascular and cerebrovascular disease classification model | |
| CN117831789B (en) | Cancer treatment response prediction system and control method thereof | |
| CN117503056A (en) | Multi-scale respiratory event automatic positioning method and device based on bimodal feature learning | |
| EP3926540A1 (en) | Method for filtering normal medical image, method for interpreting medical image, and computing device implementing the methods | |
| Irfan et al. | Machine learning in neurological disorders: A multivariate LSTM and AdaBoost approach to Alzheimer's disease time series analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20201117 Assignee: Jinan Hualang Electronic Technology Co.,Ltd. Assignor: Qilu University of Technology Contract record no.: X2023980048589 Denomination of invention: A Depression Recognition Method and System Based on Human Skeleton Kinematic Feature Information Granted publication date: 20220705 License type: Common License Record date: 20231204 |
|
| EE01 | Entry into force of recordation of patent licensing contract |