LU505456B1 - Method for recognizing pain expression in elderly patients - Google Patents

Abstract

The invention discloses a method for recognizing pain expression in elderly patients, comprising the following steps: obtaining a face image, using MTCNN for face detection to obtain required face region; performing image preprocessing on the face region; establishing and training Resnet50 network model; and inputting the image into the Resnet50 network model for pain level prediction of the pain expression. The method for recognizing pain expression in elderly patients of the invention integrates MTCNN and Resnet50 network, and realizes the classification of pain expressions of elderly patients through establishing dataset, using MTCNN to realize face detection, using ResNet50 as the basic network framework, and using transfer learning to realize model function, thus demonstrating the feasibility of applying the computer-assisted assessment to the actual pain assessment, and providing effective guidance for pain expression management.

Description

METHOD FOR RECOGNIZING PAIN EXPRESSION IN ELDERLY PATIENTS

1. Technical Field

The invention relates to the technical field of computer vision, in particular to a method for recognizing pain expression in elderly patients, 2. Background Art

Clinically, patient’s pain level needs to be assessed before the analgesic intervention is carried out on the patient, and the commonly used clinical pain assessment methods at present are mainly the patient’s self-reported assessment and the observer’s assessment. Pain is a subjective feeling, so the self-reported assessment of the pain sufferer is generally more valuable. However, due to the decline of proprioceptive function and cognitive competence, etc., the elderly are often unable to accurately describe their own pain sensation and symptom performance; clinical tools that can be used to assess pain in the elderly mainly rely on human judgment, and the patient’s pain level is assessed by experienced and professionally trained healthcare professionals. On the one hand, this method requires much time and effort, not suitable for promotion, and the assessment results are likely to be affected by the subjectivity of the assessor; on the other hand, usually the pain assessment not only evaluates the patient’s pain level at that time, but also learns about the general and the most serious level of pain in the past 24 hours to understand the change of the pain, which is extremely inefficient in the human assessment and inconvenient to record the statistics in this case.

The core of traditional face recognition technology is to calculate feature description factors describing the individual identity based on the pixel distribution of face image, and use these feature description factors for identity matching. The field of face recognition has grown significantly with the rise of deep learning, which has led to a significant increase in the speed and accuracy of face recognition compared to traditional methods. The main advantage of deep learning is that they utilize large amounts of data for training, gradually adapting to different situations and learning the best features of the characterization data during training. Although deep learning methods are better able to cope with complex external environments than traditional methods, they still can’t ignore changing scenarios 1 completely. Face recognition under unrestricted conditions has huge challenges due to the conditions of characters with unfixed postures, many changes in expressions, complex lighting conditions, and faces prone to be blocked. 3. Summary of the Invention

The invention aims to provide a method for recognizing pain expression in elderly patients, and to accurately and quickly judge the classification of pain expression without manual intervention.

The technical scheme adopted by the invention is a method for recognizing pain expression in elderly patients, comprising the following steps:

Step 1. obtaining a face image, using MTCNN for face detection to obtain required face region;

Step 2. performing image preprocessing on the face region;

Step 3. establishing and training Resnet50 network model;

Step 4. inputting the image obtained in step 2 into the Resnet50 network model obtained in step 3 for pain level prediction of the pain expression.

The invention is also characterized in that,

MTCNN used for face detection in step 1 comprising the following steps:

Step 1.1 inputting the face image into the first layer network P-Net, firstly, the input image is scaled in multiple levels, the sliding window of 12x12 with step size of 2 is used to perform sliding detection for the image at each scaling scale, non-maximum suppression is performed on all detected face frames, then the face frames are converted to the original size, and the short sides are filled in to convert them to 24x24 square face frames;

Step 1.2 inputting a plurality of 24x24 square face frames obtained from step 1.1 into the second layer network R-Net for processing, then removing face frames below the threshold by non-maximum suppression, and finally changing size to 48x48 square face frame; 2

. . . . . . LU505456

Step 1.3 inputting a plurality of 48x48 square face frames obtained in step 1.2 into the third layer network O-Net for processing to obtain the accurate face frame, five face location markers, and the confidence level, and then performing non-maximum suppression on the accurate face frames to obtain the optimal face region location.

The image preprocessing in step 2 is as follows: cropping the face region and normalize the image after cropping to the size of 224xx 224 pixels.

Step 3 specifically includes the following steps:

Step 3.1 firstly, sequentially capturing different degrees of pain expressions of the elderly population by recording videos, and extracting key frames of the desired expressions from the captured videos, then evaluating and classifying the key frame images, initially separating the pain and non-pain images based on the state at the time of capturing, and then marking and labeling the captured images according to the criteria of 4 categories of pain, namely, mild, obvious, severe, and acute;

Step 3.2 scale normalization is performed on the image data obtained in step 3.1 to obtain the dataset;

Step 3.3 dividing the dataset obtained from step 3.2 into training set, validation set, and test set by 6:2:2 after the whole dataset is mixed and disorganized,

Step 3.4 performing enhancement process of rotation, cropping and panning on the training set obtained in step 3.3;

Step 3.5 building Resnet50 network model, setting the learning rate fixed at 0.0001, the batch size at 32, the optimizer choosing the adam optimizer, and carrying out model training using the training set obtained from step 3.4 after parameters are set;

Step 3.6 using the accuracy ACC as evaluation index for the model obtained in Step 3.5, assuming that Nan represents the number of images in the validation set obtained in all step 3.3 and N, represents the number of correctly classified images, the expression for ACC is

ACC = N,/Nan, with the larger the ACC value, the closer the model’s predictions are to the real situation, and the better the model’s performance; 3

. . . . LU505456

Step 3.7 when the model obtained in step 3.6 begins to converge with the number of iterations and the accuracy of the validation set remains dynamically stable, the training of the Resnet50 network model is completed;

Step 3.8, using the test set obtained in Step 3.3 to test the model obtained in Step 3.7.

Step 4 specifically includes the following steps:

Step 4.1 the normalized image obtained in step 2 is input into the Resnet50 network model obtained in step 3, and Softmax classifier is used to receive the characteristic matrix input from the fully connected layer and output the probability value of each classification corresponding to the input target, assuming that there are N input targets XE, reach with a label yi € {1,2 ,... ,k}, and k is the type of the model output classification, k > 2; classify the pain expressions into 4 categories (1, 2, 3, 4), k takes the value of 4; input Xi, using the hypothesis function fe(Xi) to estimate the probability value P (y = j/Xi) of the corresponding classification j, and the hypothesis function is:

Fr > as ’ {ee ses 1 Ù

CoE a . ss to À $Y CY x $ = 5 st es ç sg î 3

Loss function for Softmax classifier is:

Step 4.2, the label classification with the maximum probability output from Softmax is used as the pain level of the pain expression, and if the pain level is the same for many consecutive times, it will be regarded as the predicted pain level and output the result.

The advantageous effects of the invention are as follows: the method for recognizing pain expression in elderly patients of the invention integrates MTCNN and Resnet50 network, 4 and realizes the classification of pain expressions of elderly patients through establishing dataset, using MTCNN to realize face detection, using ResNet50 as the basic network framework, and using transfer learning to realize model function, thus demonstrating the feasibility of applying the computer-assisted assessment to the actual pain assessment, and providing effective guidance for pain expression management. 4. Brief Description of Accompany Drawings

FIG. 1 is the flow chart of the method for recognizing pain expression in elderly patients provided by the invention;

FIG. 2 is the structural diagram of MTCNN model of the method for recognizing pain expression in elderly patients provided by the invention;

FIG. 3 is the flow chart of Resnet50 network transfer learning of the method for recognizing pain expression in elderly patients provided by the invention;

FIG. 4 is the diagram illustrating the detection accuracy of MTCNN model of the method for recognizing pain expression in elderly patients provided by the invention;

FIG. 5 is the the diagram illustrating the detection result of using method for recognizing pain expression in elderly patients provided by the invention to test;

S. Specific Embodiment of the invention

The invention is further described in detail hereinafter with reference to the drawings and specific embodiments.

The invention provides a method for recognizing pain expression in elderly patients, as shown in FIG. 1, comprising the following steps:

Step 1. turning on the camera to obtain the face image of the elder to be detected, and using MTCNN for face detection to obtain required face region, which is implemented according to the following specific steps:

Step 1.1 as shown in FIG. 2, MTCNN consists of three networks, namely P-Net, R-Net,

and 0-Net, the first layer network P-Net: firstly, the input image is scaled in multiple levels, the sliding window of 12x12 with step size of 2 is used to perform sliding detection for the image at each scaling scale wherein large face can be detected through small image and small face can be detected through large image, non-maximum suppression is performed on all detected face frames to obtain face frames, then the face frames are converted to the original size, and the short sides are filled in to convert them to 24x24 square;

Step 1.2 the second layer network R-Net: a plurality of 24 X24 faces obtained from the previous level are input into the network and into the convolutional neural network for processing to get more accurate face frames, then removing face frames below the threshold by non-maximum suppression, and finally changing size to 48x48 square;

Step 1.3 the third layer network O-Net: inputting a plurality of 48x48 faces obtained from the previous level into network to obtain more accurate frames, five face location markers, and the confidence level, and then performing non-maximum suppression on the face frames to obtain the final desired face frame.

Step 2. image preprocessing is performed according to following steps:

Step 2.1 cropping the face region detected by MTCNN, laying the groundwork for further face region analysis.

Step 2.2 Performing data normalization on the cropped face region to unify the data size to 224x224 pixel.

Step 3. performing classification prediction of pain expression images in the elder based on Resnet50 network, as shown in FIG. 3, and is performed according to following steps:

Step 3.1 data collection: firstly, sequentially capturing different degrees of pain expressions of the elderly population by recording videos and extracting key frames of the desired expressions from the captured videos, then experienced nurses and doctors evaluates and classifies the key frame images, initially separating the pain and non-pain images based on the state at the time of capturing, and then marking and labeling the captured images according to the criteria of mild, obvious, severe, and acute; in order to eliminate the 6 influence of subjectivity in human assessment and to ensure the reliability of the database, the 7505456 images selected into the database were mainly those with high consistency in the ratings of multiple assessors. After a series of evaluation and screening, the invention selected 4538 images from the intercepted video key frames, including 2247 images of mild pain, 735 images of obvious pain, 729 images of severe pain, and 827 images of acute pain, and established a set of database of facial pain expressions of elderly patients, as shown in Table

L;

Table 1 Shooting scenarios for different levels of pain expression states

Patient’s facial expression changes in the process of turning over and pat on the back before and after the operation

Turn over and move

Patient’s facial expression changes in the process of moving to the bed

Patient’s facial expression changes in the process of flexion and extension training of lower limbs

Patient’s facial expression changes in the process of straight leg raise

Functional exercise oo training

Patient’s facial expression changes in the process of standing beside the bed and walking training

Step 3.2 In order to focus on the research of expression recognition algorithm and reduce the influence of other factors, the original images were normalized, mainly including three parts: rotation correction, image cropping and scale normalization, which aims to correct the influence of background interference and angular offset caused by the shooting environment and the change of face posture, so as to make the patient’s face upright and the eyes horizontal in the image, and at the same time, removing the redundant background information as much as possible so as to retain only effective facial regions containing expressions. The centers of both eyes were manually calibrated, and the image was rotated on the basis of the axis connecting the centers of both eyes, so that the centers of both eyes were 7

. . . 2e LU505456 adjusted to the same horizontal line, thus eliminating the effect of angular offset; second, the patient’s facial region was manually cropped from the rotated and corrected image;

Step 3.3 data classification, after the whole dataset is mixed and disorganized, dividing the dataset into three parts, namely training set(60%), validation set(20%), and test set(20%) ;

Step 3.4 data enhancement, performing enhancement process of rotation, cropping and panning on the training set to expand number of training dataset and improve robustness and generalization performance of the model,

Step 3.5 training parameters setting, setting the learning rate fixed at 0.0001, the batch size at 32, the optimizer choosing the adam optimizer, and carrying out model training after parameters are set; the model training uses transfer learning method, and Resnet50 network model parameter is used to replace traditional Random initialization model parameter, initializing all layers except for the Softmax layer, and then add Softmax layer targeted at processing the invention; at the same time, in order to ensure that the net weights will not be distorted too quickly due to the large differences in the target dataset during the fine-tuning process, set a relative small learning rate, which effectively prevents overfitting, accelerates the convergence speed, and improves the accuracy of the model, and effectively solves the problem that application of deep learning in classification of painful facial expressions lacking large-scale training dataset;

Step 3.6 model evaluation, using the accuracy ACC as evaluation index for the model obtained, assuming that Nan represents the number of all the validated images, and N; represents the number of correctly classified images, the expression for ACC is ACC = Ny/Nai, with the larger the ACC value, the closer the model’s predictions are to the real situation, and the better the model’s performance; the model of the invention has converged at 1063 iterations and the final final validation is stabilized at around 98.7%.

Step 4 pain level prediction is implemented according to the following steps:

Step 4.1 The normalized image obtained in step 2.2 is input into the trained Resnet50 network model, and Softmax classifier is used to receive the characteristic matrix input from the fully connected layer and output the probability value of each classification corresponding 8 to the input target, assuming that there are N input targets {X Yak, seach with a label yi € {1 ,2 ,.. ,k}, and k is the type of the model output classification, k > 2; classify the pain expressions into 4 categories (1, 2, 3, 4), k takes the value of 4; input Xi, using the hypothesis function fe(Xi) to estimate the probability value P (y = j/Xi) of the corresponding classification j, and the hypothesis function is:

Frey tt sem 4, So Aus = He La

Loss function for Softmax classifier 1s:

Step 4.2, the label classification with the maximum probability output from Softmax 1s used as the pain level of the pain expression, and in order to avoid error of system detection and improve stability of the system, inputting video frame to detect face, making judgment and labeling, when label after detection achieves stable continuous number of frames, it will be used as output results of pain level determined by the system. When the output result 1s satisfied, voice prompt and data record can be made.

By the above means, the method for recognizing pain expression in elderly patients of the invention is based on deep learning network model of ResNet50, realizing high-precision recognition and classification of pain expressions of elderly patients; the residual module in the model effectively overcomes problems such as network degradation, and ensures that the performance does not decline while the depth of the network is deepened; in response to the difficult problem of small number of dataset for publicly labeled pain expression images, the means of transfer learning is used to avoid the overfitting problem; this model has the advantages of greater depth, faster convergence, higher accuracy, easy generalization, etc., 9 which demonstrates the feasibility of applying computer-aided assessment to actual pain 7505456 assessment, and provides an effective guide for pain expression management.

In the actual application of the method for recognizing pain expression in elderly patients of the invention:

Step 1 plays the role of quickly and accurately detect face and obtain region of pain expression.

Step 2 adopts MTCNN face detection network with the principal that: imputing image to be detected to MTCNN face detection network, face classification, face frame regression, and facial feature extraction are performed by P-Net, R-Net and 0-Net respectively. The advantages of using MTCNN face detection network lies in fast detection speed, high accuracy, and easy to deploy when applied to clinic. In order to verify the effectiveness of the

MTCNN model for face detection in the method of the invention, detection accuracy and detection speed of the detection algorithm is evaluated, 32 patients were selected for face testing, as shown in FIG. 4, a detection precision of 0.97 was obtained, and the use of the

Core(TM)i5-12400, with the detection speed of 0.023s/image, indicates that the face detection algorithm proposed in the invention has high detection speed and accuracy.

Step 3 serves to discriminate the pain expression level and adopts transfer learning method based on Resnet50, which is based on the principle that by pre-training a CNN on an existing large dataset, and then using the weights of this pre-trained CNN as initialized weights, transferring it to the target dataset for training and fine-tuning, the model is pre-trained on ImageNet with order of magnitude of TB to obtain a model. Using this model parameters to replace traditional Random initialization model parameter, initializing all layers except for the Softmax layer, and then add Softmax layer targeted at processing the invention; since pre-training source data, ImageNet dataset, and the dataset of the invention are quite different, the target data is used to fine-tune whole network; at the same time, in order to ensure that the net weights will not be distorted too quickly due to the large differences in the target dataset during the fine-tuning process, set a relative small learning rate, which effectively prevents overfitting, accelerates the convergence speed, and improves the accuracy of the model, and effectively solves the problem that application of deep learning in classification of painful facial expressions lacking large-scale training dataset, The advantages of using the transfer learning method based on Resnet50 are: good model generalization performance, large depth, high accuracy, good convergence, etc.

The trained model was tested in clinical trials in step 4 and the results of the test statistics are shown in Table 2

Table 2 Results of the test statistics

Manual evaluation Resnet50 predict

Testing object Consistent or not result result

As can be seen from Table 2, by evaluating the pain expression of four elderly fracture patients in the clinic, firstly, the experts will evaluate and discriminate, as shown in FIG. 5, and then the Resnet50 network is used to capture the patient’s face through the camera for evaluation and detection, when the system continuously stabilizes 20 frames for the same label, then it means that the pain level of this patient is of this label; the detection algorithm of the invention, in the absence of human intervention, can quickly and accurately discriminate the pain expression classification, and the system has high stability.

As can be seen from the above results: (1) MTCNN-based face detection achieves an average detection precision of 0.97 and detection speed of 0.023s/image for the test samples, so it can be illustrated that the model has high detection speed and detection accuracy, laying good groundwork for further pain expression recognition. (2) The invention achieves a validation accuracy of 98.7% by building pain dataset of 11 elderly patients using algorithm based on Resnet50 transfer learning, which provides high stability and testing accuracy in real-world testing. 12

Claims (5)

CLAIMS LU505456

1. A method for recognizing pain expression in elderly patients, comprising the following steps: Step 1. obtaining a face image, using MTCNN for face detection to obtain required face region; Step 2. performing image preprocessing on the face region; Step 3. establishing and training Resnet50 network model; Step 4. inputting the image obtained in step 2 into the Resnet50 network model obtained in step 3 for pain level prediction of the pain expression.

2. The method for recognizing pain expression in elderly patients of claim 1 wherein MTCNN used for face detection in step 1 comprising the following steps: Step 1.1 inputting the face image into the first layer network P-Net, firstly, the input image is scaled in multiple levels, the sliding window of 12x12 with step size of 2 is used to perform sliding detection for the image at each scaling scale, non-maximum suppression is performed on all detected face frames, then the face frames are converted to the original size, and the short sides are filled in to convert them to 24x24 square face frames; Step 1.2 inputting a plurality of 24x24 square face frames obtained from step 1.1 into the second layer network R-Net for processing, then removing face frames below the threshold by non-maximum suppression, and finally changing size to 48x48 square face frame; Step 1.3 inputting a plurality of 48x48 square face frames obtained in step 1.2 into the third layer network O-Net for processing to obtain the accurate face frame, five face location markers, and the confidence level, and then performing non-maximum suppression on the accurate face frames to obtain the optimal face region location.

3. The method for recognizing pain expression in elderly patients of claim 1 wherein the image preprocessing in step 2 is as follows: cropping the face region and normalize the image after cropping to the size of 224xx 224 pixels.

4. The method for recognizing pain expression in elderly patients of claim 3 wherein 17905656 Step 3 specifically includes the following steps: Step 3.1 firstly, sequentially capturing different degrees of pain expressions of the elderly population by recording videos, and extracting key frames of the desired expressions from the captured videos, then evaluating and classifying the key frame images, initially separating the pain and non-pain images based on the state at the time of capturing, and then marking and labeling the captured images according to the criteria of 4 categories of pain, namely, mild, obvious, severe, and acute; Step 3.2 scale normalization 1s performed on the image data obtained in step 3.1 to obtain the dataset; Step 3.3 dividing the dataset obtained from step 3.2 into training set, validation set, and test set by 6:2:2 after the whole dataset is mixed and disorganized; Step 3.4 performing enhancement process of rotation, cropping and panning on the training set obtained in step 3.3; Step 3.5 building Resnet50 network model, setting the learning rate fixed at 0.0001, the batch size at 32, the optimizer choosing the adam optimizer, and carrying out model training using the training set obtained from step 3.4 after parameters are set; Step 3.6 using the accuracy ACC as evaluation index for the model obtained in Step

3.5, assuming that Nau represents the number of images in the validation set obtained in all step 3.3 and Nr; represents the number of correctly classified images, the expression for ACC is ACC = N,/Nau, with the larger the ACC value, the closer the model’s predictions are to the real situation, and the better the model’s performance; Step 3.7 when the model obtained in step 3.6 begins to converge with the number of iterations and the accuracy of the validation set remains dynamically stable, the training of the Resnet50 network model is completed; Step 3.8, using the test set obtained in Step 3.3 to test the model obtained in Step

3.7.

5. The method for recognizing pain expression in elderly patients of claim 4 wherein Step 4 specifically includes the following steps:

. . . . Lo . LU505456 Step 4.1 the normalized image obtained in step 2 is input into the Resnet50 network model obtained in step 3, and Softmax classifier is used to receive the characteristic matrix input from the fully connected layer and output the probability value of each classification corresponding to the input target, assuming that there are N input targets {X;¥i}F h with a label yi € {1,2,... k}, and kis th f th input targets {Æ;7;5;; > each with a label yi € {1,2,... k}, and k is the type of the model output classification, k > 2; classify the pain expressions into 4 categories (1, 2, 3, 4), k takes the value of 4; input Xi, using the hypothesis function fo(Xi) to estimate the probability value P (y = j/Xi) of the corresponding classification j, and the hypothesis function is: PS 40 À ee WA 5 * : + § Foy oy IS X SE ooo $ FIRE Ÿ © $ = = $ & . + § ERS LEE us Saad ae À Loss function for Softmax classifier is: on x 3 GROS 5 Le NEN 1 = 3 x rues dt TE oF Sn x Sr S i ANE gee} pes $ EET NS Step 4.2, the label classification with the maximum probability output from Softmax is used as the pain level of the pain expression, and if the pain level is the same for many consecutive times, it will be regarded as the predicted pain level and output the result.