CN116864128A

CN116864128A - Psychological state assessment system and method based on physical activity behavior pattern monitoring

Info

Publication number: CN116864128A
Application number: CN202310858960.0A
Authority: CN
Inventors: 董高峰; 徐峰; 柯小剑; 周智君; 来忠; 凌懿文; 周卉; 柯陆安; 戴捷; 诸葛田野
Original assignee: Zhejiang Gongshang University
Current assignee: Zhejiang Gongshang University
Priority date: 2023-07-13
Filing date: 2023-07-13
Publication date: 2023-10-10

Abstract

A mental state evaluation system based on physical activity behavior mode monitoring and a method thereof are disclosed, which adopts an artificial intelligent monitoring technology based on deep learning to identify action label characteristics of students in each monitoring key frame through characteristic extraction of physical activity monitoring videos of psychological normal students and students to be monitored, further carries out context semantic understanding on time sequence distribution of the action label characteristics in each monitoring key frame to obtain action mode understanding characteristic representation, and then represents differential characteristic distribution between the psychological normal students and the students to be monitored based on transfer matrixes of the action mode understanding characteristics of the psychological normal students and the students to be monitored, thereby judging the mental state of the students to be monitored. Thus, the psychological health state of the student can be intelligently and accurately monitored and evaluated.

Description

Psychological state assessment system and method based on physical activity behavior pattern monitoring

Technical Field

The application relates to the technical field of health monitoring, in particular to a psychological state assessment system and a psychological state assessment method based on physical activity behavior pattern monitoring.

Background

The psychological health of college students means that the psychological of college students has many characteristics of middle-aged young, but as a special group, college students cannot be completely equivalent to young in society.

The psychological health problems of college students are serious, wherein the academic problems, the emotion problems, the interpersonal relationship problems, the anxiety problems, the emotion problems, the sexual health, the psychological health problems of special groups and the living adaptation problems of the college students are the psychological health problems commonly existing in the college students at present, so that the college education process often needs to periodically carry out psychological health examination and psychological dispersion on the students. In addition, in the process of checking the psychological health of the college students, the college students are usually measured only by adopting a scale, so that the evaluation mode is single, the accuracy is difficult to grasp, and the judgment standard is not fixed. Psychological health standards change with age transition and cultural background changes.

Thus, an optimized mental state assessment system based on physical activity behavioral pattern monitoring is desired.

Disclosure of Invention

The present application has been made to solve the above-mentioned technical problems. The embodiment of the application provides a psychological state assessment system and a psychological state assessment method based on physical activity behavior pattern monitoring.

Accordingly, according to one aspect of the present application, there is provided a mental state assessment system based on physical activity behavioral pattern monitoring, comprising:

the behavior data acquisition module is used for acquiring physical activity video data of the students marked as psychological normal and physical activity video data of the students to be monitored;

the key frame extraction module is used for respectively extracting a plurality of reference physical activity key frames and a plurality of detection physical activity key frames from the physical activity video data of the student marked as psychology normal and the physical activity video data of the student to be monitored based on a difference frame method;

the first action recognition module is used for enabling the plurality of reference physical activity key frames to pass through the human action recognizer so as to obtain a plurality of first action tag feature vectors;

the second action recognition module is used for enabling the plurality of key frames for detecting physical activities to pass through the human action recognizer so as to obtain a plurality of second action tag feature vectors;

the motion understanding module is used for inputting the first motion label feature vectors and the second motion label feature vectors into a two-way long-short-term memory neural network model to obtain reference motion mode understanding feature vectors and detection motion mode understanding feature vectors;

The difference representation module is used for calculating a transfer matrix of the detection action mode understanding characteristic vector relative to the reference action mode understanding characteristic vector as a classification characteristic matrix; and

and the evaluation result generation module is used for passing the classification feature matrix through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the psychological state of the student to be monitored is normal or not.

According to another aspect of the present application, there is also provided a mental state assessment method based on physical activity behavior pattern monitoring, including:

acquiring physical activity video data of students marked as psychologically normal and physical activity video data of students to be monitored;

extracting a plurality of reference physical activity key frames and a plurality of detection physical activity key frames from the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored respectively based on a difference frame method;

passing the plurality of reference physical activity key frames through a human body action identifier to obtain a plurality of first action tag feature vectors;

passing the plurality of detected physical activity key frames through the human motion identifier to obtain a plurality of second motion label feature vectors;

Inputting the plurality of first motion tag feature vectors and the plurality of second motion tag feature vectors into a two-way long-short-term memory neural network model to obtain a reference motion pattern understanding feature vector and a detection motion pattern understanding feature vector;

calculating a transfer matrix of the detected motion mode understanding feature vector relative to the reference motion mode understanding feature vector as a classification feature matrix;

and the classification feature matrix is passed through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the psychological state of the student to be monitored is normal or not.

Compared with the prior art, the psychological state assessment system and the psychological state assessment method based on physical activity behavior pattern monitoring, provided by the application, adopt an artificial intelligent monitoring technology based on deep learning to identify the action label characteristics of students in each monitoring key frame through extracting the characteristics of physical activity monitoring videos of psychological normal students and students to be monitored, further perform context semantic understanding on the time sequence distribution of the action label characteristics in each monitoring key frame to obtain action pattern understanding characteristic representation, and further represent the differential characteristic distribution between the psychological normal students and the students to be monitored based on the transfer matrix of the action pattern understanding characteristics of the psychological normal students and the students to be monitored, so as to judge the psychological state of the students to be monitored. Thus, the psychological health state of the student can be intelligently and accurately monitored and evaluated.

Drawings

The above and other objects, features and advantages of the present application will become more apparent by describing embodiments of the present application in more detail with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of embodiments of the application and are incorporated in and constitute a part of this specification, illustrate the application and together with the embodiments of the application, and not constitute a limitation to the application. In the drawings, like reference numerals generally refer to like parts or steps.

Fig. 1 is a block diagram of a mental state assessment system based on physical activity behavior pattern monitoring according to an embodiment of the present application.

Fig. 2 is a schematic diagram of the architecture of a mental state assessment system based on physical activity behavior pattern monitoring according to an embodiment of the present application.

Fig. 3 is a block diagram of an evaluation result generation module in the psychological state evaluation system based on physical activity behavior pattern monitoring according to an embodiment of the present application.

Fig. 4 is a flowchart of a mental state assessment method based on physical activity behavior pattern monitoring according to an embodiment of the present application.

Fig. 5 is a schematic view of a psychological state assessment system and a psychological state assessment method based on physical activity behavior pattern monitoring according to an embodiment of the application.

Detailed Description

Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.

Fig. 1 is a block diagram of a mental state assessment system based on physical activity behavior pattern monitoring according to an embodiment of the present application. Fig. 2 is a schematic diagram of the architecture of a mental state assessment system based on physical activity behavior pattern monitoring according to an embodiment of the present application. As shown in fig. 1 and 2, a mental state estimation system 100 based on physical activity behavior pattern monitoring according to an embodiment of the present application includes: a behavioral data acquisition module 110 for acquiring physical activity video data of a student marked as psychologically normal and physical activity video data of a student to be monitored; a key frame extraction module 120, configured to extract a plurality of reference physical activity key frames and a plurality of detected physical activity key frames from the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored, respectively, based on a difference frame method; a first motion recognition module 130, configured to pass the plurality of reference physical activity key frames through a human motion recognizer to obtain a plurality of first motion label feature vectors; a second motion recognition module 140, configured to pass the plurality of key frames of detected physical activity through the human motion recognizer to obtain a plurality of second motion label feature vectors; the motion understanding module 150 is configured to input the plurality of first motion tag feature vectors and the plurality of second motion tag feature vectors into a two-way long-short-term memory neural network model to obtain a reference motion pattern understanding feature vector and a detection motion pattern understanding feature vector; a difference representation module 160 for calculating a transition matrix of the detected motion pattern understanding feature vector with respect to the reference motion pattern understanding feature vector as a classification feature matrix; and an evaluation result generating module 170, configured to pass the classification feature matrix through a classifier to obtain a classification result, where the classification result is used to indicate whether the psychological state of the student to be monitored is normal.

In the above-mentioned mental state assessment system 100 based on physical activity behavior pattern monitoring, the behavior data acquisition module 110 is configured to acquire physical activity video data of a student marked as psychologically normal and physical activity video data of the student to be monitored. As described above, the psychological health of college students means that the college students' psychology has many characteristics in the middle of young, but as a special group, college students cannot be completely equivalent to young in society. The psychological health problems of college students are serious, wherein the academic problems, the emotion problems, the interpersonal relationship problems, the anxiety problems, the emotion problems, the sexual health, the psychological health problems of special groups and the living adaptation problems of the college students are the psychological health problems commonly existing in the college students at present, so that the college education process often needs to periodically carry out psychological health examination and psychological dispersion on the students. In addition, in the process of checking the psychological health of the college students, the college students are usually measured only by adopting a scale, so that the evaluation mode is single, the accuracy is difficult to grasp, and the judgment standard is not fixed. Psychological health standards change with age transition and cultural background changes. Therefore, an optimized mental state assessment system is desired.

It should be understood that the psychological health problem of college students is serious, and the serious problem of the psychological state of the students (students without physical disabilities) can be observed to show the abnormality of the activity behavior pattern, so that the psychological state of the students can be evaluated through the monitoring of the physical activity behavior pattern of the students for psychological problem screening and dredging.

At present, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, speech signal processing, and the like. In addition, deep learning and neural networks have also shown levels approaching and even exceeding humans in the fields of image classification, object detection, semantic segmentation, text translation, and the like. In recent years, the development of deep learning and neural networks has provided new solutions and solutions for intelligent assessment of mental state based on physical activity behavioral pattern monitoring.

Specifically, in the technical scheme of the application, an artificial intelligent monitoring technology based on deep learning is adopted to identify the action tag characteristics of students in each monitoring key frame through extracting the characteristics of physical activity monitoring videos of psychology normal students and students to be monitored, and further context semantic understanding is carried out on the time sequence distribution of the action tag characteristics in each monitoring key frame to obtain action mode understanding characteristic representation. And then, representing the differential feature distribution between the psychological normal student and the student to be monitored based on the transfer matrix of the action mode understanding features of the student to be monitored, so as to judge the psychological state of the student to be monitored. Therefore, the psychological health state of the students can be intelligently and accurately monitored and evaluated, and then early warning is sent out to conduct psychological dispersion when abnormal psychological conditions of the students are monitored, so that psychological health of the students is guaranteed.

Specifically, in the technical scheme of the application, firstly, physical activity video data of a student marked as psychologically normal and physical activity video data of the student to be monitored are acquired. The physical activity video data of the student marked as psychologically normal is existing data, and the physical activity video data of the student to be monitored can be acquired by a camera.

In the above-mentioned psychological state assessment system 100 based on physical activity behavior pattern monitoring, the key frame extraction module 120 is configured to extract a plurality of reference physical activity key frames and a plurality of detected physical activity key frames from the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored, respectively, based on a difference frame method. In the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored, physical activity behavior pattern characteristics of the student can be represented by differences between adjacent monitoring frames in the video, that is, by image representations of adjacent image frames. Specifically, a plurality of reference physical activity key frames and a plurality of detection physical activity key frames are respectively extracted from physical activity video data of the student marked as psychologically normal and physical activity video data of the student to be monitored based on a difference frame method.

In another embodiment of the present application, a plurality of reference physical activity key frames and a plurality of detected physical activity key frames may be extracted from the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored, respectively, based on the color histogram similarity. It should be appreciated that color histogram similarity is an indicator that measures the degree of similarity in color distribution between two images or image frames. The color histogram is a statistical graph representing the frequency of each color in the image. By comparing the color histograms of the two images, the similarity between them can be calculated. In calculating color histogram similarity, histogram distance measurement methods such as the Papanicolaou distance (Bhattacharyya distance) or the Chi-square distance (Chi-square distance) are typically used. These distance measures can measure the degree of difference between two color histograms, reflecting the similarity of the color distribution between the two images. By calculating the color histogram similarity, the color distribution differences between different images or image frames can be compared to determine whether or not they are similar or how similar. In key frame extraction, color histogram similarity may be used to select key frames with representative and informative amounts for further analysis and processing.

Specifically, in one sub-example of this embodiment, extracting a plurality of reference physical activity key frames and a plurality of detected physical activity key frames from the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored, respectively, based on the color histogram similarity, includes: decomposing the physical activity video of the student marked as psychological normal and the physical activity video of the student to be monitored into a series of continuous frames to obtain physical activity image frames of the student with psychological normal and physical activity image frames of the student to be monitored; respectively carrying out color histogram calculation on each frame to obtain color characteristics of the frame; calculating the similarity of color histograms between adjacent frames, various similarity measurement methods such as the Babbitt distance, chi-square distance, etc. can be used; the plurality of reference physical activity key frames and the plurality of detected physical activity key frames are determined according to a threshold of similarity.

In yet another embodiment of the present application, a plurality of reference physical activity key frames and a plurality of detected physical activity key frames may be extracted from the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored, respectively, based on the movement energy. It should be understood that motion energy refers to a measure representing the intensity or degree of change of motion in a video or image sequence. It is used to measure the change of pixel values in an image or video over time, thus reflecting motion information in the image. In calculating the motion energy, a method such as an optical flow method (optical flow) or an inter-frame difference method (frame differencing) is generally used. These methods can calculate the pixel value variation between image frames to obtain motion information. The optical flow method is a method based on a change in brightness of pixels, which calculates a motion vector by tracking the movement of feature points in an image between successive frames. The optical flow method can provide a motion vector for each pixel point, thereby calculating the motion energy of the whole image. Inter-frame difference is a simple method to obtain motion information by calculating the difference between successive frames. By comparing the pixel value differences of adjacent frames, the motion areas occurring in the image can be obtained. The calculated motion energy may be used to identify and analyze key frames or motion features in the video. Thus, frames with significant motions or changes can be selected as the plurality of reference physical activity key frames and the plurality of detected physical activity key frames from the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored based on the motion energy.

In the psychological state assessment system 100 based on physical activity behavior pattern monitoring, the first action recognition module 130 is configured to pass the plurality of reference physical activity key frames through a human action recognizer to obtain a plurality of first action tag feature vectors. That is, passing the plurality of reference physical activity key frames through a human motion identifier to identify motion label features of the respective key frames to obtain a plurality of first motion label feature vectors.

Specifically, first, the plurality of reference physical activity key frames are encoded using an image encoder of the human body motion recognizer to extract motion behavior implicit features of the respective reference physical activity key frames, thereby obtaining a plurality of reference physical activity feature maps, where the image encoder includes a convolutional neural network model as a feature extractor and a non-local neural network model cascaded with the convolutional neural network model. That is, feature mining is performed on the plurality of reference physical activity key frames using a convolutional neural network model as a feature extractor having excellent performance in image feature extraction to extract physical activity behavior local hidden features of students, thereby obtaining a plurality of local physical activity feature maps. And, the correlation between the student's different local physical activity behavior features for each of the individual reference physical activity key frames creates a foreground object considering that the different local physical activity features for each of the individual key frames are not isolated for the student physical activity behavior features in the plurality of reference physical activity key frames due to the convolution being a typical local operation. Therefore, in the technical scheme of the application, in order to extract the relevance of different local physical activity behavior characteristics of students in the reference physical activity key frame, a non-local neural network is used for further extracting the characteristics of the images. That is, each of the plurality of local physical activity signatures is passed through a non-local neural network to obtain a plurality of reference physical activity signatures.

In particular, here, the non-local neural network captures hidden dependency information by calculating the similarity of different local physical activity behavior characteristics of students in the reference physical activity key frame, so as to model context characteristics, so that the network focuses on the overall content among the different local physical activity behavior characteristic data of the students in the reference physical activity key frame, and further, the main network characteristic extraction capability is improved in classification and detection tasks.

And then inputting the plurality of reference physical activity feature graphs into a classifier of the human body action recognizer to perform action recognition and label classification of the human body, so as to obtain a plurality of first action label feature vectors.

More specifically, in the embodiment of the present application, the first action recognition module 130 is further configured to: encoding, by a first image feature extraction unit, the plurality of reference physical activity key frames using an image encoder of the human motion identifier to obtain a plurality of reference physical activity feature maps; and inputting the plurality of reference physical activity feature maps into a classifier of the human body motion identifier by a first motion label vector generation unit to obtain a plurality of first motion label feature vectors.

In the psychological state assessment system 100 based on physical activity behavior pattern monitoring, the second motion recognition module 140 is configured to pass the plurality of key frames for detecting physical activity through the human motion recognizer to obtain a plurality of second motion label feature vectors. And similarly, for the physical activity key frames of the students to be monitored, performing motion feature recognition and label classification of the human body on the plurality of detected physical activity key frames through the human body motion recognizer so as to obtain a plurality of second motion label feature vectors.

More specifically, in an embodiment of the present application, the second action recognition module 140 is further configured to: encoding, by a second image feature extraction unit, the plurality of detected physical activity key frames using an image encoder of the human motion identifier to obtain a plurality of detected physical activity feature maps; and inputting the plurality of detected body activity feature maps into a classifier of the human body action recognizer by a second action tag vector generation unit to obtain a plurality of second action tag feature vectors. Wherein the image encoder comprises a convolutional neural network model as a feature extractor and a non-local neural network model cascaded with the convolutional neural network model.

In the above mental state estimation system 100 based on physical activity behavior pattern monitoring, the motion understanding module 150 is configured to input the first motion label feature vectors and the second motion label feature vectors into a two-way long-short-term memory neural network model to obtain a reference motion pattern understanding feature vector and a detection motion pattern understanding feature vector. Considering that the plurality of reference physical activity key frames and the plurality of detected physical activity key frames have dynamic motion pattern feature distribution information in a time sequence dimension, the plurality of first motion label feature vectors and the plurality of second motion label feature vectors are input into a two-way long-short-term memory neural network model to be processed so as to perform context semantic understanding on the time sequence distribution of the motion label vectors based on the two-way long-short-term memory model to obtain motion pattern understanding feature representation, thereby obtaining reference motion pattern understanding feature vectors and detected motion pattern understanding feature vectors. It should be understood that the two-way Long Short-Term Memory neural network model (LSTM) is a time-cycled neural network, which enables the weight of the neural network to be self-updated by adding an input gate, an output gate and a forgetting gate, and the weight scale at different moments can be dynamically changed under the condition of fixed parameters of the network model, so that the problems of gradient disappearance or gradient expansion can be avoided. The bidirectional long-short-term memory neural network model is formed by combining a forward LSTM and a backward LSTM, the forward LSTM can learn the physical activity behavior pattern information of the student under the current frame, and the backward LSTM can learn the physical activity behavior pattern information of the student after the current frame, so that the reference motion pattern understanding feature vector and the detection motion pattern understanding feature vector obtained through the bidirectional long-short-term memory neural network model learn the information of the global context.

In the above-described mental state estimation system 100 based on physical activity behavior pattern monitoring, the difference representing module 160 is configured to calculate a transition matrix of the detected motion pattern understanding feature vector with respect to the reference motion pattern understanding feature vector as a classification feature matrix. That is, the transfer matrix is used to represent the differential feature information between the action mode understanding features of the psychology normal students and the action mode understanding features of the students to be monitored, so that the accuracy of classification can be improved by using the transfer matrix as the classification feature matrix.

Specifically, in the embodiment of the present application, the difference representing module 160 is further configured to: calculating the detection action pattern understanding characteristics according to the following formulaThe transfer matrix of the eigenvector is understood as a classification eigenvector relative to the reference motion pattern; wherein, the formula is:wherein->Representing the detected motion pattern understanding feature vector, < >>Representing the reference motion pattern understanding feature vector, < >>Representing the transfer matrix->Representing matrix multiplication.

In the above mental state estimation system 100 based on physical activity behavior pattern monitoring, the estimation result generation module 170 is configured to pass the classification feature matrix through a classifier to obtain a classification result, where the classification result is used to indicate whether the mental state of the student to be monitored is normal. Therefore, the psychological health state of the students can be intelligently and accurately monitored and evaluated, and then early warning is sent out to conduct psychological dispersion when abnormal psychological conditions of the students are monitored, so that psychological health of the students is guaranteed.

Fig. 3 is a block diagram of an evaluation result generation module in the psychological state evaluation system based on physical activity behavior pattern monitoring according to an embodiment of the present application. As shown in fig. 3, the evaluation result generation module 170 includes: a developing unit 171 for developing the classification feature matrix into classification feature vectors according to row vectors or column vectors; a full-connection encoding unit 172, configured to perform full-connection encoding on the classification feature vector by using a full-connection layer of the classifier to obtain an encoded classification feature vector; and a classification result generating unit 173, configured to input the encoded classification feature vector into a Softmax classification function of the classifier to obtain the classification result.

In another embodiment of the application, the optimized classification feature matrix is passed through a decision tree model to obtain classification results, wherein the classification results are used for indicating whether the psychological states of students to be monitored are normal or not. It should be appreciated that the decision tree model is a classification and regression model based on a tree structure. It classifies or predicts data through a series of decision rules. The decision tree model can be regarded as a tree structure consisting of nodes and edges, wherein each node represents a feature or attribute and each edge represents a value of a feature or decision result. In the decision tree model, starting from the root node, the features are judged or decided, gradually downwards along the branches of the tree, and finally reach the leaf nodes, wherein the leaf nodes represent the final classification or prediction result. The decision tree construction process is to determine the division mode of each node according to the characteristics and the label information in the training data set, so that the division selected on each node can improve the classification accuracy or the regression prediction accuracy to the greatest extent.

Further, the psychological state assessment system based on physical activity behavior pattern monitoring further comprises: the training module is used for training the human body action identifier, the two-way long-short-term memory neural network model and the classifier; wherein, training module includes: the training data acquisition unit is used for acquiring training physical activity video data of the students marked as psychological normal and training physical activity video data of the students to be monitored; the training key frame extraction unit is used for respectively extracting a plurality of training reference physical activity key frames and a plurality of training detection physical activity key frames from the training physical activity video data of the student marked as psychological normal and the training physical activity video data of the student to be monitored based on a difference frame method; the training first action recognition unit is used for enabling the training reference physical activity key frames to pass through the human action recognizer so as to obtain a plurality of training first action label feature vectors; the training second action recognition unit is used for enabling the training detection physical activity key frames to pass through the human action recognizer so as to obtain training second action label feature vectors; the training action understanding unit is used for inputting the plurality of training first action label feature vectors and the plurality of training second action label feature vectors into the two-way long-short-term memory neural network model to obtain training reference action mode understanding feature vectors and training detection action mode understanding feature vectors; the training difference representing unit is used for calculating a transfer matrix of the training detection action mode understanding characteristic vector relative to the training reference action mode understanding characteristic vector as a training classification characteristic matrix; the classification loss function calculation unit is used for passing the training classification characteristic matrix through the classifier to obtain a classification loss function value; a loss function value calculation unit for calculating a line-column saliency consistency factor of the training classification feature matrix to obtain a line-column saliency consistency loss function value according to the following formula; and the model training unit is used for training the human body action identifier, the two-way long-short-term memory neural network model and the classifier by taking the weighted sum of the classification loss function value and the row-column convex expression consistency loss function value as the loss function value.

According to the technical scheme, through the human body action identifier and the two-way long-short-term memory neural network model, physical activity video data of the student with normal psychology and source domain differences possibly existing in the physical activity video data of the student to be monitored can be enlarged through extraction of local action image semantics and cross-time sequence characteristic context correlation, so that the classification feature matrix enhances expression of the differences between the two. On the other hand, a large amount of redundant pixel semantics which are not used for image semantic feature extraction and associated coding exist in the physical activity video data of the student with normal psychology and the physical activity video data of the student to be monitored, and the redundant pixel semantics are amplified by the human motion identifier and the two-way long-short-term memory neural network model, so that distribution divergence in a high-dimensional feature semantic space is formed in the classification feature matrix, and the classification feature matrix is caused to have induction divergence when passing through a classifier, so that the training speed of the classifier and the accuracy of classification results are influenced.

Based on the above, the applicant of the present application makes manifold expressions of the classification feature matrix in a high-dimensional feature space consistent in different distribution dimensions corresponding to a row direction and a column direction, so as to compensate for distribution divergence of the classification feature matrix. Therefore, the applicant of the present application further introduces a rank-convex expression consistency factor of the classification feature matrix as a loss function in addition to the classification loss function.

Specifically, in an embodiment of the present application, the loss function value calculation unit is configured to: calculating a row-column salience consistency factor of the training classification feature matrix according to the following formula to obtain a row-column salience consistency loss function value;

wherein, the formula is:wherein the method comprises the steps ofRepresenting the training classification feature matrix, +.>A +.o representing the training classification feature matrix>Line->Characteristic value of column>And->Respectively is a matrix->Mean vector and diagonal vector of corresponding row vector, +.>Representing a norm of the vector,/->Frobenius norms of the matrix are represented, < >>And->Is a matrix->Width and height of>、/>And->Is a weight superparameter,/->Representing an activation function->Representing the rank convex expression consistency loss function value.

That is, considering the feature distribution characteristics of the row and column dimensions of the classification feature matrix, the row-column convex expression consistency factor flattens the set of finite convex polyhedrons of the manifold in different dimensions through the geometric convex decomposition of the feature manifold represented by the classification feature matrix with respect to the distribution differences of the classification feature matrix in the sub-dimensions represented by the row and column, and constrains the geometric convex decomposition in the form of the shape weights associated with the sub-dimensions, thereby promoting the consistency of the convex geometric representation of the feature manifold of the classification feature matrix in the decomposable dimensions represented by the row and column, so that the manifold expression of the classification feature matrix in the high-dimensional feature space remains consistent in the different distribution dimensions corresponding to the row direction and the column direction, thereby compensating the distribution divergence of the classification feature matrix.

In summary, the mental state evaluation system 100 based on physical activity behavior pattern monitoring according to the embodiment of the present application is illustrated, which adopts an artificial intelligence monitoring technology based on deep learning to identify the action tag feature of the student in each monitoring key frame through feature extraction of physical activity monitoring videos of the psychological normal student and the student to be monitored, further performs context semantic understanding on the time sequence distribution of the action tag feature in each monitoring key frame to obtain action pattern understanding feature representation, and further represents the differential feature distribution between the psychological normal student and the student to be monitored based on the transfer matrix of the action pattern understanding feature of the psychological normal student and the student to be monitored, so as to determine the mental state of the student to be monitored. Thus, the psychological health state of the student can be intelligently and accurately monitored and evaluated.

In another example of the present application, there is also provided a social network-based mental state estimation system, including: firstly, acquiring social network data of students to be monitored through crawler software, wherein the social network data comprise posts, comments and praise of the students. Such data may be obtained through an API of the social network or a web crawler. It should be appreciated that students 'behavior and language on social networks tend to be more natural and realistic, and that data on social networks more reflects the student's real mental state than traditional questionnaires or face-to-face interviews. The amount of data on a social network is often very large, involving multiple social interactions of students over a long period of time, which allows meaningful features to be extracted from the large amount of data and a more accurate model to be built. The data sources on the social network are wide, and the data sources comprise various forms of interaction information such as posts, comments, praise and the like. These diversified data can provide a more comprehensive perspective to assess the mental state of the student. The data on the social network is generated in real time, and the psychological state change of the students can be monitored in real time. This is important for timely finding and intervening in psychological problems for students.

And then preprocessing the social network data of the student to be monitored to obtain preprocessed social network data. It should be appreciated that the quality and consistency of the data may be ensured by pre-processing the social network data of the student to be monitored, including cleaning the data, removing noise, normalizing, to improve the accuracy and reliability of subsequent analysis and modeling. Specifically, invalid or erroneous portions of the data may be removed by flushing the data. In a specific embodiment, the cleaning data includes: delete duplicate data, repair missing values, handle outliers, etc. Purging the data may improve the consistency and integrity of the data. Noise may be present in the social network data, such as misspellings, grammatical errors, irrelevant information, and the like. Removing these noise can reduce interference to subsequent analysis, improving accuracy of the data. Social networking data may come from different sources and formats, which need to be unified into a consistent format and unit. In a specific embodiment, the date and time are unified into a specific format, the text data are converted into a unified encoding format, and so on. The normalized data may facilitate subsequent feature extraction and modeling.

And then, extracting the characteristics of the preprocessed social network data to obtain the context feature vectors of the social text. It should be appreciated that contextual information in text may be captured by feature extraction of social text data to better understand and analyze content in social media. In particular, text in social media is typically short and fragmented, lacking contextual information in traditional languages. Through feature extraction, some key features such as word frequency, part of speech, emotion tendency and the like can be extracted from the text, so that the context environment in which the text is positioned is better understood. Text in social media is typically massive, and feature extraction may facilitate subsequent classification and clustering of the text. The text is conveniently classified by a machine learning algorithm by extracting the feature vector of the text, so that the psychological health state of the student is accurately monitored and evaluated. The amount of information in social media is huge, and feature extraction can filter out irrelevant or low-quality information. By extracting the feature vectors of the user and the text, the relationship among students, the structure of the social network and the evolution rule can be researched, so that the psychological health state of the students can be better monitored and evaluated. By extracting these features, social text data can be converted into numeric feature vectors for use in constructing a machine learning model for student mental state assessment. Such feature extraction methods may more accurately understand the psychological state of the student, thereby providing better support and intervention.

Specifically, in one embodiment of the present example, performing feature extraction on the preprocessed social network data to obtain a social text context feature vector includes: and carrying out feature extraction on the preprocessed social network data based on the word bag model to obtain the context feature vector of the social text. It should be appreciated that the bag of words model is a text representation method commonly used in natural language processing. It represents text as a fixed length vector, where each dimension represents a particular word or feature, and counts how often each word or feature appears in the text. In particular, the bag of words model treats each word or feature in the text as a separate element, ignoring their order and grammatical structure in the text. The text is then represented as a vector by counting the number of occurrences of each word or feature in the text or using other weight calculation methods (e.g., TF-IDF). Each dimension of the vector corresponds to a word or feature whose value represents the importance of the word or feature in the text.

Specifically, in another embodiment in the present example, performing feature extraction on the preprocessed social network data to obtain a social text context feature vector includes: and carrying out feature extraction on the preprocessed social network data based on the word embedding model to obtain the context feature vector of the social text. It should be appreciated that the word embedding model is a technique for mapping vocabulary to a low-dimensional continuous vector space. It represents each word as a dense vector, also called word embedding, by learning semantic and grammatical relations between the words. The conventional bag-of-words model represents each vocabulary as an independent element, and cannot capture semantic similarity between the vocabularies. The word embedding model can map words into a continuous vector space through training a neural network algorithm, so that words with similar semantics are closer in the vector space. The word embedding model has the advantage of capturing semantic relationships and context information between words, providing a richer word representation.

And finally, processing the upper and lower feature vectors of the social text to obtain a classification result, wherein the classification result is used for indicating whether the psychological state of the student to be monitored is normal or not. It should be appreciated that by categorizing the mental state of the student, potential psychological problems or abnormalities can be discovered early, thereby taking appropriate intervention. For example, if students are determined to be mental state abnormal, psychological consultation can be provided, supported or guided to seek professional help in time. The classification of student mental states can be used to monitor their mental health on a regular or real-time basis. Through continuous monitoring, psychological problems can be found and solved in time, and psychological health development of students is promoted. Based on the classification results of the psychological states of the students, personalized education and support can be provided for each student. According to the psychological state of students, teaching methods, contents and resources can be adjusted to meet their specific needs and to improve learning effects. By counting and analyzing the classification result of the psychological states of the students, the general trend and rule of psychological problems can be revealed, and data support is provided for psychological research. This helps to understand the cause and influencing factors of psychological health problems in students in depth and provides guidance for psychological intervention and prevention. In a word, by classifying the psychological states of the students to be monitored, the psychological health and personalized education can be intervened in early time, and data support is provided for psychological research, so that psychological health development and academic success of the students are promoted.

Specifically, in another embodiment in the present example, processing the social text context feature vector to obtain a classification result includes: and passing the upper and lower feature vectors of the social text through a random forest model to obtain a classification result. It should be appreciated that the random forest model is an ensemble learning algorithm that is made up of a plurality of decision trees. Each decision tree is an independent classifier that classifies input samples by performing feature selection and decision making. Random forests obtain the final classification result by voting or averaging the results of each decision tree. The main characteristics of the random forest model include: when each decision tree is constructed, the random forest performs a put-back random sampling on the training samples, so that the training set of each decision tree is slightly different. At the same time, on each node, the random forest also randomly selects the characteristics so as to increase the diversity of the model. The random forest is composed of a plurality of decision trees, each of which is an independent classifier. By voting or averaging the results of each decision tree, a random forest can obtain more stable and accurate classification results. Random forests are more robust to noise and outliers because they reduce the risk of overfitting of a single decision tree by integrating multiple decision trees. Random forests can evaluate the importance of features by calculating the frequency of use of each feature in the decision tree and the contribution to the prediction accuracy.

In summary, a social network-based mental state evaluation system has been elucidated, which crawls social network data of the students to be monitored and analyzes the social network data to obtain a classification result for indicating whether the mental states of the students to be monitored are normal, so as to accurately evaluate the mental states of the students, so as to find potential mental problems or abnormalities early, and take corresponding intervention measures. Compared with the psychological state assessment system based on physical activity behavior pattern monitoring, the psychological state assessment system based on the social network may have the following defects that firstly, the psychological state assessment system based on the social network often depends on content and expression issued by students to be monitored, and subjective problems may exist. Then, the student to be monitored may selectively display own emotional state, or there may be misleading expression, thereby affecting the accuracy of the evaluation result. Content on a social network is often self-published by students to be monitored, and inaccuracy or falsification of information may exist. Students to be monitored may exaggerate or narrow their own emotion expressions or issue contents inconsistent with the true emotion, thereby affecting the reliability of the evaluation result. Second, social network-based mental state estimation systems have difficulty considering differences between individuals. Different people may have differences in the expression patterns of the same emotion, some may prefer to express emotion, and some may be more inward or conservative. Therefore, the evaluation result of the system may not accurately reflect the true psychological state of the individual. In contrast, the physical activity behavior mode is difficult to change, and the psychological state of the student to be monitored can be reflected more truly.

Fig. 4 is a flowchart of a mental state assessment method based on physical activity behavior pattern monitoring according to an embodiment of the present application. As shown in fig. 4, a mental state estimation method based on physical activity behavior pattern monitoring according to an embodiment of the present application includes: s110, acquiring physical activity video data of students marked as psychologically normal and physical activity video data of students to be monitored; s120, respectively extracting a plurality of reference physical activity key frames and a plurality of detection physical activity key frames from the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored based on a difference frame method; s130, passing the plurality of reference physical activity key frames through a human body action identifier to obtain a plurality of first action tag feature vectors; s140, enabling the plurality of detected physical activity key frames to pass through the human body action identifier so as to obtain a plurality of second action tag feature vectors; s150, inputting the first action tag feature vectors and the second action tag feature vectors into a two-way long-short-term memory neural network model to obtain reference action mode understanding feature vectors and detection action mode understanding feature vectors; s160, calculating a transfer matrix of the detection action mode understanding feature vector relative to the reference action mode understanding feature vector as a classification feature matrix; and S170, passing the classification feature matrix through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the psychological state of the student to be monitored is normal or not.

It will be appreciated by those skilled in the art that the specific operation of the respective steps in the above-described mental state estimation method based on physical activity behavior pattern monitoring has been described in detail in the above description of the mental state estimation system based on physical activity behavior pattern monitoring with reference to fig. 1 to 3, and thus, repetitive descriptions thereof will be omitted.

Fig. 5 is a schematic view of a psychological state assessment system and a psychological state assessment method based on physical activity behavior pattern monitoring according to an embodiment of the application. As shown in fig. 5, in an application scenario of the physical activity behavior pattern monitoring-based mental state assessment system, physical activity video data of a student marked as psychologically normal, which is existing data, and physical activity video data of a student to be monitored (e.g., P as illustrated in fig. 5), which is acquired by a camera (e.g., C as illustrated in fig. 5), are first acquired. Further, the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored are input to a server (e.g., S as illustrated in fig. 5) deployed with a mental state evaluation algorithm based on physical activity behavior pattern monitoring, wherein the server is capable of processing the physical activity video data of the student marked as psychologically normal and the physical activity video data of the student to be monitored with the mental state evaluation algorithm based on physical activity behavior pattern monitoring to obtain an evaluation result for indicating whether the mental state of the student to be monitored is normal.

The embodiment of the application also provides a chip system, which comprises at least one processor, and when the program instructions are executed in the at least one processor, the method provided by the embodiment of the application is realized.

The embodiment of the application also provides a computer storage medium, on which a computer program is stored, which when executed by a computer causes the computer to perform the method of the above-described method embodiment.

The present application also provides a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of the method embodiment described above.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.

Claims

1. A mental state assessment system based on physical activity behavioral pattern monitoring, comprising:

2. The mental state evaluation system based on physical activity behavior pattern monitoring of claim 1, wherein the first action recognition module comprises:

A first image feature extraction unit for encoding the plurality of reference physical activity key frames using an image encoder of the human motion identifier to obtain a plurality of reference physical activity feature maps; and

and the first action tag vector generation unit is used for inputting the plurality of reference physical activity feature images into a classifier of the human action recognizer to obtain a plurality of first action tag feature vectors.

3. The mental state evaluation system based on physical activity behavior pattern monitoring of claim 2, wherein the second action recognition module comprises:

a second image feature extraction unit, configured to encode the plurality of key frames of detected physical activity using an image encoder of the human motion identifier to obtain a plurality of feature images of detected physical activity; and

a second motion label vector generation unit for inputting the plurality of detected body motion feature maps into a classifier of the human motion identifier to obtain a plurality of second motion label feature vectors.

4. The mental state evaluation system based on physical activity behavior pattern monitoring according to claim 3, wherein the image encoder comprises a convolutional neural network model as a feature extractor and a non-local neural network model cascaded with the convolutional neural network model.

5. The physical activity behavior pattern monitoring-based mental state assessment system of claim 4, wherein the discrepancy-representation module is further configured to:

calculating the transfer matrix of the detected motion pattern understanding feature vector relative to the reference motion pattern understanding feature vector as a classification feature matrix with the following formula;

wherein, the formula is:wherein->Representing the detected motion pattern understanding feature vector,representing the reference motion pattern understanding feature vector, < >>Representing the transfer matrix->Representing matrix multiplication.

6. The physical activity behavior pattern monitoring-based mental state assessment system according to claim 5, wherein the assessment result generation module comprises:

the unfolding unit is used for unfolding the classification characteristic matrix into a classification characteristic vector according to a row vector or a column vector;

the full-connection coding unit is used for carrying out full-connection coding on the classification characteristic vectors by using a full-connection layer of the classifier so as to obtain coded classification characteristic vectors; and

and the classification result generating unit is used for inputting the coding classification feature vector into a Softmax classification function of the classifier to obtain the classification result.

7. The physical activity behavior pattern monitoring-based mental state assessment system of claim 6, further comprising: the training module is used for training the human body action identifier, the two-way long-short-term memory neural network model and the classifier;

wherein, training module includes:

the training data acquisition unit is used for acquiring training physical activity video data of the students marked as psychological normal and training physical activity video data of the students to be monitored;

the training key frame extraction unit is used for respectively extracting a plurality of training reference physical activity key frames and a plurality of training detection physical activity key frames from the training physical activity video data of the student marked as psychological normal and the training physical activity video data of the student to be monitored based on a difference frame method;

the training first action recognition unit is used for enabling the training reference physical activity key frames to pass through the human action recognizer so as to obtain a plurality of training first action label feature vectors;

the training second action recognition unit is used for enabling the training detection physical activity key frames to pass through the human action recognizer so as to obtain training second action label feature vectors;

The training action understanding unit is used for inputting the plurality of training first action label feature vectors and the plurality of training second action label feature vectors into the two-way long-short-term memory neural network model to obtain training reference action mode understanding feature vectors and training detection action mode understanding feature vectors;

the training difference representing unit is used for calculating a transfer matrix of the training detection action mode understanding characteristic vector relative to the training reference action mode understanding characteristic vector as a training classification characteristic matrix;

the classification loss function calculation unit is used for passing the training classification characteristic matrix through the classifier to obtain a classification loss function value;

a loss function value calculation unit for calculating a line-column saliency consistency factor of the training classification feature matrix to obtain a line-column saliency consistency loss function value according to the following formula;

and the model training unit is used for training the human body action identifier, the two-way long-short-term memory neural network model and the classifier by taking the weighted sum of the classification loss function value and the row-column convex expression consistency loss function value as the loss function value.

8. The psychological state evaluation system based on physical activity behavior pattern monitoring according to claim 7, wherein the loss function value calculation unit is configured to:

calculating a row-column salience consistency factor of the training classification feature matrix according to the following formula to obtain a row-column salience consistency loss function value; wherein, the formula is:wherein->Representing the training classification feature matrix, +.>A +.o representing the training classification feature matrix>Line->Characteristic value of column>And->Respectively a matrixMean vector and diagonal vector of corresponding row vector, +.>Representing a norm of the vector,/->Frobenius norms of the matrix are represented, < >>And->Is a matrix->Width and height of>、/>And->Is the weight of the parameter to be exceeded,representing an activation function->Representing the rank convex expression consistency loss function value.

9. A mental state assessment method based on physical activity behavior pattern monitoring, comprising:

calculating a transfer matrix of the detected motion mode understanding feature vector relative to the reference motion mode understanding feature vector as a classification feature matrix; and

10. The method of mental state assessment based on physical activity behavior pattern monitoring of claim 9, wherein passing the plurality of reference physical activity key frames through a human motion identifier to obtain a plurality of first motion label feature vectors, comprising:

encoding the plurality of reference physical activity key frames using an image encoder of the human motion identifier to obtain a plurality of reference physical activity feature maps; and

The plurality of reference physical activity feature maps are input into a classifier of the human motion identifier to obtain the plurality of first motion label feature vectors.