CN113257440A - ICU intelligent nursing system based on patient video identification - Google Patents

ICU intelligent nursing system based on patient video identification Download PDF

Info

Publication number
CN113257440A
CN113257440A CN202110682391.XA CN202110682391A CN113257440A CN 113257440 A CN113257440 A CN 113257440A CN 202110682391 A CN202110682391 A CN 202110682391A CN 113257440 A CN113257440 A CN 113257440A
Authority
CN
China
Prior art keywords
nursing
patient
human
posture
monitoring
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110682391.XA
Other languages
Chinese (zh)
Inventor
徐新明
曾向荣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Jinxianlian Technology Co ltd
Original Assignee
Hangzhou Jinxianlian Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Jinxianlian Technology Co ltd filed Critical Hangzhou Jinxianlian Technology Co ltd
Priority to CN202110682391.XA priority Critical patent/CN113257440A/en
Publication of CN113257440A publication Critical patent/CN113257440A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16HHEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H80/00ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161Detection; Localisation; Normalisation
    • G06V40/166Detection; Localisation; Normalisation using acquisition arrangements
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168Feature extraction; Face representation
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174Facial expression recognition
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20Movements or behaviour, e.g. gesture recognition

Abstract

An ICU intelligent nursing system based on patient video identification belongs to the technical field of intelligent nursing. The system comprises an intelligent monitoring device, an image collector, a body index monitoring device and a nursing end, wherein the image collector, the body index monitoring device and the nursing end are respectively connected with the intelligent monitoring device; the intelligent monitoring device comprises a face detection module, a facial expression recognition module, a head gesture recognition module, a human body gesture recognition module, an activity evaluation module and a monitoring decision module; the monitoring decision module is used for judging whether to send a nursing instruction to a nursing end according to the facial expression information identified by the facial expression identification module or the head posture information identified by the head posture identification module; and comprehensively evaluating the state of the patient according to the facial expression information, the head posture information, the activity data, the physiological data and the vital sign data, and sending a comprehensive nursing instruction to the nursing end according to the state of the patient. The invention can monitor, analyze and evaluate the patient in real time, improve the timely intervention ability of medical care personnel and reduce the nursing workload.

Description

ICU intelligent nursing system based on patient video identification
Technical Field
The invention relates to the technical field of intelligent nursing, in particular to an ICU intelligent nursing system based on patient video identification.
Background
Currently, many intensive care indicators are not automatically captured, but are repeatedly evaluated by nurses, which not only brings a large amount of high-intensity workload to medical care personnel, but also hardly guarantees the quality of service for severe patients, especially unconscious severe patients. Intensive Care Unit (ICU) is the largest component of the medical systems in the United states. Approximately 55000 patients are cared for in the ICU each day, with a daily cost between $ 3000 and $ 10000. The cumulative cost exceeds $ 800 million per year. ICU becomes more and more important as the baby boomer enters the elderly stage. Today, over half of all patients in the U.S. ICU are over 65 years old. China also has the same trend, which creates a worldwide problem. To meet the growing demand for acute care, training more intensive care professionals is one of the solutions, but with artificial intelligence.
Patients with severe respiratory diseases may wear mechanical ventilators to assist in breathing. These machines, when they push air into the lungs of a patient, may have a breathing rhythm that does not correspond to the patient's natural breathing rate (which, for the most part, varies with physiological conditions), which can lead to the patient actually struggling with the ventilator, which is often adjusted by an experienced caregiver, with serious consequences if the adjustment is not timely, or if an emergency situation occurs. The intelligent control system can detect the expression of the patient and implement recognition of the patient's dyspnea condition. Once the patient suffers from the pain on the expression, the medical staff can be timely warned.
The invention patent application CN108234956A discloses a medical care monitoring method, which comprises the steps of receiving a current video frame sent by a monitoring video acquisition end; analyzing the current video frame to obtain the identity information of the target object; analyzing the current video frame to obtain the state information of the target object; when the state of the target object is judged to be abnormal according to the state information, generating a state abnormality notification according to the identity information and the state information; and sending the state abnormity notification to a processing end so that the processing end carries out corresponding early warning according to the state abnormity notification. The analyzing the current video frame to obtain the state information of the target object specifically includes: performing action recognition on the current video frame by adopting a preset action analysis model to obtain action information of the target object; and/or performing expression recognition on the current video frame by adopting a preset expression analysis model to obtain facial expression information of the target object; and/or performing semantic recognition on the audio information in the current video frame by adopting a preset semantic analysis model to obtain semantic information of the target object; and/or performing emotion analysis on the audio information in the current video frame by adopting a preset emotion analysis model to obtain the emotion information of the target object. The invention utilizes the preset action analysis model, the expression analysis model, the semantic analysis model and the emotion analysis model to monitor the target object in multiple aspects, monitors whether tumble, painful expression, uncomfortable language and painful emotion exist, and then informs medical care personnel to take care in time. The scheme solves the problem that the existing nursing is not timely to a certain extent, and can monitor the state of the patient in real time. However, the invention only monitors the state of the patient through video recognition, does not carry out comprehensive monitoring and evaluation in combination with monitoring data of the medical device, only carries out emergency nursing on the abnormity, and does not form a long-term specific nursing scheme. The action monitoring is only limited to the monitoring that the patient has a large action abnormity, and other action monitoring is not carried out on the patient, so that the recovery activity capability of the patient cannot be known; repeated analysis exists in expression and emotion analysis, both the expression and emotion analysis are obtained through image information analysis of videos, certain subjective judgment exists in emotion analysis, monitoring reliability is not high, and the situation that medical staff are mistakenly warned for many times possibly occurs; repeated analysis also exists in semantic analysis and expression analysis, although the former is analyzed through audio information and the latter is analyzed through image information, pain and symptoms are monitored, differences may exist in expression analysis and semantic analysis results, the differences can cause medical staff to be warned for many times, and the situation that one of the expression analysis and the semantic analysis is wrong is caused.
The invention patent application CN104506809B discloses a severe patient monitoring system based on intelligent video, and specifically discloses a system comprising two image acquisition modules, a medical staff identification module, a patient monitoring module and a personThe device comprises a machine interaction module, an alarm module and a data storage module. The method for calculating the current state of the patient by the patient monitoring module comprises the following steps: (1) the patient monitoring module processes the acquired image for 1 time at intervals of T; (2) carrying out gray level equalization processing on the pixel point matrix, compensating the influence of the light intensity on pixel point coordinate values, and obtaining a gray level equalization image pixel point pixel value matrix; (3) performing front and rear frame difference calculation on the pixel value matrix of the pixel point; (4) the pixel values in the differential gray level equilibrium image matrix are subjected to square summation to obtain a scalar function f (k); (5) and performing Gaussian low-pass filtering on the f (k) to obtain a low-pass filtered scalar function fLP(k) (ii) a (6) A first step of mixingLP(k) Comparing the obtained data with preset experience threshold values a and b, wherein a is less than b; if f isLP(k) If < a, the patient shows slight action or normal state; if a < fLP(k) If the amplitude of the movement is larger than the preset threshold, judging whether the patient has dangerous movement or not by using a mode identification method, and if the patient does not have dangerous movement, continuing to detect; if the patient has dangerous activity, the patient monitoring module begins timing, e.g., at T1A is always kept less than f within the timeLP(k) If the number b is less than the preset threshold, generating alarm information and sending the alarm information to an alarm module; if f isLP(k) If the alarm signal is greater than b, the patient is indicated to have violent action, and alarm information is immediately generated and sent to an alarm module. Although the invention also discloses a system for intelligently monitoring by using videos, the invention only detects dangerous actions of limbs of a patient, does not consider face monitoring, patient activity monitoring and body index detection of the patient, and even does not comprehensively evaluate comprehensive monitoring data to obtain a proper nursing scheme.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides an ICU intelligent nursing system based on patient video identification, which identifies the facial expression of a patient and detects the movement of limbs from a video through computer vision so as to improve the autonomous detection rate of the activity of the patient, comprehensively and automatically monitors the patient through fusion with real-time vital sign data and physiological data, improves the timely intervention capability of medical personnel and reduces the nursing workload.
The invention is realized by the following technical scheme:
an ICU intelligent nursing system based on patient video identification comprises an image collector, an intelligent monitoring device, a body index monitoring device and a nursing end; the image collector is used for collecting video data in the ICU; the physical index monitoring device is used for monitoring physiological data and vital sign data of a patient; the intelligent monitoring device is respectively connected with the image collector, the body index monitoring device and the nursing end; its characterized in that, intelligent monitoring device includes:
the face detection module is used for identifying a face position area from video data and obtaining a plurality of facial feature point position vectors in the face position area;
the facial expression recognition module is used for recognizing facial expression information based on the position vectors of the facial feature points and the facial expression comparison vectors;
the head posture identification module is used for converting the position vectors of the facial feature points into three-dimensional coordinates and identifying head posture information by utilizing a random sampling consistency algorithm;
the human body posture identification module is used for identifying a human body position area from the video data, obtaining a plurality of joint vectors in the human body position area, and identifying human body posture information based on the plurality of joint vectors and a human body posture comparison condition;
the activity evaluation module is used for evaluating the activity condition of the patient on the human posture information obtained in the monitoring period to obtain activity data;
the monitoring decision module is used for judging whether to send a nursing instruction to a nursing end according to the facial expression information or the head posture information; and comprehensively evaluating the state of the patient according to the facial expression information, the head posture information, the activity data, the physiological data and the vital sign data, and sending a comprehensive nursing instruction to the nursing end according to the state of the patient.
The invention utilizes an image collector (such as a camera) to collect video data and monitors the condition in the intensive care unit in real time. And based on the video data, carrying out facial feature point recognition to recognize facial expression information and head posture information and carry out facial condition monitoring on the patient. And the posture change condition of the human body are identified based on the video data, and the condition of four limbs of the patient is monitored. The whole body monitoring of the patient is realized by the face condition monitoring and the four-limb condition monitoring, the condition of the patient can be comprehensively evaluated by combining the physiological data and the vital sign data, and a comprehensive nursing scheme is given, so that the daily monitoring of emergency rescue and gradual recovery of the self-weight symptom state of the patient to a mild disease/rehabilitation state is realized.
This intelligent monitoring system can prompt medical personnel to intervene in time and treat under emergency, also can carry out reasonable arrangement to medical personnel's daily nursing, provides medical care management efficiency, reduces unnecessary nursing work load, provides the corresponding reasonable nurse for the patient.
Preferably, the face detection module includes:
the face recognition unit is used for recognizing a face position area from the video data by utilizing computer vision and deep learning technology;
the characteristic point identification unit is used for identifying a plurality of facial characteristic points of the face position area according to the standard face model and determining a plurality of facial characteristic point position vectors;
the standard face model comprises a plurality of standard face feature points set based on standard face contours.
Preferably, the facial expression recognition module includes:
the facial motion extraction unit is used for extracting facial motion parameter vectors of all parts of the face from the position vectors of the plurality of facial feature points; the facial motion parameter vector comprises an eyebrow motion parameter vector, an eye motion parameter vector, a cheek motion parameter vector, a mouth motion parameter vector, a nose motion parameter vector and a palate motion parameter vector;
and the facial expression recognition unit is used for contrasting the facial action parameter vector with the facial expression judgment condition and recognizing the facial expression information of each part of the face.
Preferably, the head pose recognition module includes:
the three-dimensional modeling unit is used for converting the position vectors of the facial feature points into three-dimensional coordinates in a three-dimensional world coordinate system and three-dimensional coordinates in an image collector coordinate system, and solving a relational formula of the image coordinates and the world coordinates:
to obtain a rotation matrix R and a translation matrix T;
wherein the content of the first and second substances,is a facial feature point P in a three-dimensional world coordinate systemiIs determined by the three-dimensional coordinates of (a),is a facial feature point PiThree-dimensional coordinates in the image collector coordinate system;as facial feature points PiThe physical coordinates on the imaging plane are,as facial feature points PiPixel coordinates on an imaging plane, i is a natural number; the rotation matrix R represents angles of the face in three directions, including a head raising and lowering angle, a head shaking angle and a horizontal deflection angle;
and the head posture identification unit is used for obtaining the optimal angle estimation value by utilizing the fitting of a random sampling consistency algorithm and identifying the head posture information.
Preferably, the human body posture recognition module includes:
the human body joint identification unit is used for identifying a human body position area from the video data and identifying key feature points of joints and limbs in the human body position area by utilizing a deep convolutional neural network;
the human body posture identification unit is used for acquiring a plurality of joint vectors of a human body by utilizing the deep convolutional neural network, calculating the bending angle of the joint vectors, comparing the bending angle with the human body posture comparison condition and identifying human body posture information; the human body posture comparison conditions comprise a human body posture comparison condition lying on a bed, a human body posture comparison condition sitting on the bed, a human body posture comparison condition standing, and a human body posture comparison condition sitting on a chair.
Preferably, the activity assessment module comprises:
an activity evaluation unit for monitoring the statistical time for the patient to maintain the recognized body posture and monitoring the amount of movement of the patient to change from one body posture to another body posture within a certain period;
and the activity ability evaluation unit is used for judging the activity ability of the patient based on the statistical time, the movement amount and the heart rate data in the physiological data.
Preferably, the monitoring decision module includes:
the pain nursing unit is used for comparing facial expression information of each part of the face with a pain expression judgment condition and identifying whether a patient has a pain expression; if not, the nursing instruction is not sent to the nursing end; if so: if the time of the appearance of the pain expression is required to be identified to be greater than the pain threshold time, or if the frequency of the appearance of the pain expression in a certain period is required to be identified to be greater than the pain threshold frequency, if the condition is met, sending a nursing instruction to a nursing end, otherwise, not sending the instruction;
the head shaking nursing unit is used for identifying whether the frequency of the head shaking gesture in the head gesture information in a certain period is greater than the head shaking threshold frequency, if so, sending a nursing instruction to a nursing end, and otherwise, not sending the instruction;
and the comprehensive decision unit is used for comprehensively evaluating the patient condition and sending a comprehensive nursing instruction to the nursing end according to the patient condition according to the facial expression information, the head posture information, the activity data, the physiological data and the vital sign data.
Preferably, the system further comprises a light source emitter disposed in the ICU for providing a light source to assist the image collector in operating in a dark environment.
Preferably, the system further comprises a database for storing facial expression information, head pose information, activity data, physiological data and vital sign data for the corresponding patient.
Preferably, the system further comprises a plurality of accelerometers worn at the patient's joints for detecting the motion of the patient's limbs. The monitoring decision module further comprises a dangerous nursing unit for identifying the joint movement angle, judging whether the patient falls down or moves violently when the joint movement angle is larger than the safe movement angle threshold value, and sending a nursing instruction to a nursing end.
The invention has the following beneficial effects:
the invention discloses an Artificial Intelligence system (AI) for automatically identifying the expression and the state of an illness of a patient by identifying the facial expression and the body posture of the patient through videos. The technique aims to provide some precautionary and meticulous care to provide more comprehensive data about the condition of a patient by accurately and precisely quantifying the patient's actions, as an expert is accompanied by the patient at all times, carefully calibrating the treatment. The invention can carry out comprehensive automatic monitoring on the patient by combining with the real-time vital sign data and the physiological data, the system can relieve the pressure of the overload of the personnel in the intensive care unit, help to manage the repeated patient evaluation in real time, and can realize more accurate prediction and detection of the emergency. More importantly, if the technology is applied to an Intensive Care Unit (ICU), the technology can provide better service for critically ill patients, reduce the pain of the patients and improve the life safety. The invention does not replace manpower, but can be a part of a medical system, so that doctors and nurses can exert their skills when they are needed most, and the utilization rate of medical resources is improved.
Drawings
Fig. 1 is a block diagram of an ICU intelligent nursing system based on patient video identification according to the present invention;
FIG. 2 is an example of a convolutional neural network employed by the face detection module to identify face location regions and facial feature points;
FIG. 3 is a schematic diagram of a standard face model with 68 standard facial feature points identified;
FIG. 4 is a schematic representation of facial expressions, illustrating a painful expression;
FIG. 5 is a diagram of a deep convolutional network structure for human pose recognition;
FIG. 6 is a schematic diagram showing the identification of characteristic points of the whole body when the human body is in a standing state and the hands are lifted and held up to the two ears; the points in the figure are key points of joints and limbs, and the connecting lines link the joints and the limbs together to form the body framework in the current state.
Detailed Description
The following are specific embodiments of the present invention and are further described with reference to the drawings, but the present invention is not limited to these embodiments.
Referring to fig. 1, an ICU intelligent nursing system based on patient video identification includes an image collector, an intelligent monitoring device, a body index monitoring device, and a nursing end. The image collector is used for collecting video data in the ICU, is arranged in the ICU and is opposite to the sickbed, and for example, a high-resolution and wide-field camera is adopted. The body index monitoring device is used for monitoring physiological data and vital sign data of patients, the physiological data comprises data such as blood pressure, red blood cell number, blood fat and euglobulin, and the vital sign data comprises electrocardio and arterial blood pressure data. The body index monitoring device is a medical instrument for realizing the data monitoring, such as a heart rate monitor and the like. The intelligent monitoring device is respectively connected with the image collector, the body index monitoring device and the nursing end. The image collector collects video data and inputs the video data to the intelligent monitoring device, and the body index monitoring device is used for inputting monitored physiological data and vital sign data to the intelligent monitoring device. The intelligent monitoring device carries out monitoring, analysis and judgment, sends a monitoring instruction to a nursing end for the condition needing monitoring, and medical personnel carry out nursing actions according to the received monitoring instruction.
The intelligent monitoring device comprises a face detection module, a facial expression recognition module, a head gesture recognition module, a human body gesture recognition module, an activity evaluation module and a monitoring decision module. The face detection module is used for identifying a face position area from video data and obtaining a plurality of facial feature point position vectors in the face position area. The facial expression recognition module is used for recognizing facial expression information based on the facial feature point position vectors and the facial expression contrast vectors. The head posture identification module is used for converting the position vectors of the facial feature points into three-dimensional coordinates and identifying head posture information by using a random sampling consistency algorithm. The human body posture identification module is used for identifying a human body position area from the video data, obtaining a plurality of joint vectors in the human body position area, and identifying human body posture information based on the plurality of joint vectors and a human body posture comparison condition. And the activity evaluation module is used for evaluating the activity condition of the patient on the human posture information obtained in the monitoring period to obtain activity data. The monitoring decision module is used for judging whether to send a nursing instruction to a nursing end according to the facial expression information or the head posture information; and comprehensively evaluating the state of the patient according to the facial expression information, the head posture information, the activity data, the physiological data and the vital sign data, and sending a comprehensive nursing instruction to the nursing end according to the state of the patient.
Specifically, the face detection module includes a face recognition unit and a feature point recognition unit. And the face recognition unit is used for recognizing a face position area from the video data by utilizing computer vision and deep learning technology. The feature point identification unit is used for identifying a plurality of facial feature points of the face position area according to the standard face model and determining a plurality of facial feature point position vectors. The standard face model comprises a plurality of standard face feature points set based on standard face contours. For example, referring to fig. 2, the method of multi-task convolutional neural network detection based on deep learning is adopted to complete face detection and face feature point extraction. And sorting the output results of the first layer and the second layer of convolutional neural networks according to the size of the face, and quickly acquiring a face region according to the approximate pixel value of the face. The third layer of convolutional neural network outputs several facial feature points (generally, 68 facial feature points are output, and more facial feature points can be output according to needs). The 68 feature points are determined by a standard face model, see fig. 3. The 68 feature points include 17 feature points at the cheek contours, 5 x 2 feature points at the eyebrows, 6 x 2 feature points at the eyes, 4 feature points at the bridge of the nose, 5 feature points at the nose head, 20 feature points at the mouth. Based on the output facial feature points, facial feature point position vectors are obtained.
Specifically, the facial expression recognition module comprises a facial action extraction unit and a facial expression recognition unit. The facial action extraction unit is used for extracting facial action parameter vectors of all parts of the face from the position vectors of the plurality of facial feature points. The facial motion parameter vector comprises an eyebrow motion parameter vector, an eye motion parameter vector, a cheek motion parameter vector, a mouth motion parameter vector, a nose motion parameter vector and a palate motion parameter vector. For example, each eyebrow is provided with 5 feature points, and based on the 5 feature points, motion parameter vectors of the overall morphology of the eyebrow are obtained, such as a vector containing a plurality of adjacent feature points, a vector between the 18 th feature point and the 19 th feature point, a vector between the 19 th and 20 th feature points, a vector between the 20 th and 21 st feature points, and a vector between the 21 st and 22 nd feature points. The mouth is provided with 20 characteristic points, wherein the upper lip is provided with 10 characteristic points, the lower lip is provided with 10 characteristic points, motion parameter vectors of the integral shapes of the upper arc surface and the lower arc surface of the upper lip and the lower lip respectively are obtained, such as a vector comprising a plurality of adjacent characteristic points, taking the upper lip as an example, a vector comprising 49 th characteristic point and 50 th characteristic point, a vector comprising 50 th characteristic point and 51 st characteristic point, a vector comprising 51 st characteristic point and 52 th characteristic point, a vector comprising 52 th characteristic point and 53 th characteristic point, a vector comprising 53 th characteristic point and 54 th characteristic point, a vector comprising 61 st characteristic point and 62 th characteristic point, a vector comprising 62 th characteristic point and 63 th characteristic point, a vector comprising 63 th characteristic point and 64 th characteristic point, and a vector comprising 64 th characteristic point and 65 th characteristic point; the nose is provided with 9 characteristic points, and the motion parameter vector of the integral shape of the nose bridge and the motion vector of the integral shape of the nose head are obtained, such as the vectors between a plurality of adjacent characteristic points. The motion parameter vector of the entire shape of the other part is also determined based on the contour formed by the feature points (specifically, referring to the number in fig. 3, the contour of the part can be roughly formed by connecting the feature points of the same part in consecutive numbers).
And the facial expression recognition unit is used for contrasting the facial action parameter vectors with facial expression judgment conditions and recognizing facial expression information of each part of the face. The facial expression judgment condition is also determined based on a plurality of standard facial feature points set by a standard face model, for example, 5 feature points at the eyebrows have specific standard motion parameter vectors when representing the raised state of the eyebrows; when representing the state of eyebrow drop (see 111 in fig. 4), there is also its specific standard motion parameter vector; when the bending or creasing state of the eyebrows is represented, specific standard motion parameter vectors are provided, the motion vectors recognized by a patient in real time are compared with the standard motion parameter vectors, and when the motion vectors recognized in real time are approximately matched with the standard motion parameter vectors, the facial expression information is determined to contain the raised eyebrows or the dropped eyebrows or the bent eyebrows or the creases of the eyebrows; or, the facial expression judgment condition may adopt a plurality of vector form judgment conditions, and if the position relationship among a plurality of vectors is judged to meet the conditions of two sides being low and the middle being high, it is determined that the facial expression information includes eyebrow bending; when the position relation among the vectors is judged to meet the conditions that one side close to the nose bridge is high and one side close to the ears is low, determining that the facial expression information contains eyebrow sagging; when the position relation among the vectors is judged to meet the conditions that one side close to the nose bridge is low and one side close to the ears is high, determining that the facial expression information contains the raised eyebrows; and when the position relations among the vectors are judged to be almost consistent, and the distances between the feature points at the two ends of the eyebrows are shortened, or the position relations of high, low and high appear, determining that the facial expression information contains eyebrow creases. For another example, 20 feature points at the lips have their specific standard motion parameter vectors when they indicate that the lips are too high (see 114 in fig. 4), closed, left-falling (see 117 in fig. 4), and raised mouth corners, and the facial expression information is determined to include too high lips, closed lips, left-falling, and raised mouth corners when the motion vectors recognized in real time by comparing the motion vectors recognized by the patient with the standard motion parameter vectors and the motion vectors recognized in real time substantially match with the standard motion parameter vectors. Or determining that the positions of the vectors are higher than the positions of the vectors in the standard lip opening and closing state, and determining that the facial expression information contains lip elevation; judging that the distance between the vectors on the upper and lower arc surfaces of the upper lip is shorter than the distance between the vectors on the upper and lower arc surfaces of the upper lip in the opening and closing state of the lips, and determining that the facial expression information contains tight lip closure; when the position relation among the vectors on the upper arc surface of the upper lip is judged to present a low, high, low, high and low similar relation, determining that the facial expression information contains a left-falling lip; and when the positions of the vectors at the two sides of the lips are higher than the position of the vector in the middle, determining that the facial expression information contains the mouth corner rising. For another example, when the 9 feature points at the nose indicate the states of the nose wrinkle, the nostril opening, etc., particularly the specific standard motion parameter vector, the motion vector recognized in real time for the patient is compared with the standard motion parameter vector, and when the motion vector recognized in real time is approximately matched with the standard motion parameter vector, the facial expression information is determined to include the nose wrinkle (see 116 in fig. 4) and the nostril opening. Or when the vector distance at the nose bridge is judged to be shorter than the vector distance at the nose bridge in the normal state of the standard nose, determining that the facial expression information contains the nose wrinkle; and when the vector distance of the nose is judged to be larger than the vector distance of the nose under the normal state of the standard nose, determining that the facial expression information contains nostril opening.
In addition to the above-described facial expressions, for the eyes, eye closure (see 112 in fig. 4), eyelid locking (see 115 in fig. 4), eye opening, and the like are included; for cheek, cheek elevation (see 113 in fig. 4), maxilla elevation (see 116 in fig. 4), mandible retraction, etc. are also included. The expression recognition can be realized by referring to the above manner, for example, by comparing the motion vector recognized in real time with the standard motion parameter vector, or by determining the conditions such as position, distance, length relationship, etc. of the motion vector recognized in real time.
Specifically, the head gesture recognition module comprises a three-dimensional modeling unit and a head gesture recognition unit. And the three-dimensional modeling unit is used for converting the position vectors of the facial feature points into three-dimensional coordinates in a three-dimensional world coordinate system and three-dimensional coordinates in an image collector coordinate system to construct a three-dimensional formula (1).
Formula (1)
And solving a relational formula of the image coordinates and the world coordinates to obtain a rotation matrix R and a translation matrix T:
formula (2)
Wherein the content of the first and second substances,is a facial feature point P in a three-dimensional world coordinate systemiIs determined by the three-dimensional coordinates of (a),is a facial feature point PiThree-dimensional coordinates in the image collector coordinate system;as facial feature points PiThe physical coordinates on the imaging plane are,as facial feature points PiPixel coordinates on an imaging plane, i is a natural number; the rotation matrix R represents the angles of the face in three directions, including a head raising and lowering angle, a head shaking angle and a horizontal deflection angle.
The head posture identifying unit is used for obtaining the optimal angle estimation value by utilizing random sampling consensus RANSAC algorithm fitting and identifying the head posture information.
Specifically, the human body posture recognition module comprises a human body joint recognition unit and a human body posture recognition unit. The human body joint identification unit is used for identifying a human body position area from the video data and identifying key feature points of joints and limbs in the human body position area by utilizing a deep convolutional neural network. According to the method, a human body position area is firstly identified through a deep convolutional neural network, and then key feature points of joints and limbs in the area are identified based on real-time human body 3D gestures of the deep convolutional neural network. Fig. 5 is a structure of a MobileNet posture recognition network, and points in fig. 6 are skeleton points, namely, key feature points of joints and limbs are identified. There are 18 skeletal points in fig. 6. And processing the input video image by utilizing a multilayer convolution network and outputting an identification result. The human body posture identification unit is used for acquiring a plurality of joint vectors of a human body by utilizing a deep convolutional neural network, calculating the bending angle of the joint vectors, comparing the bending angle with the human body posture comparison condition and identifying human body posture information; the human body posture comparison conditions comprise a human body posture comparison condition lying on a bed, a human body posture comparison condition sitting on the bed, a human body posture comparison condition standing, and a human body posture comparison condition sitting on a chair.
The human body posture recognition will be specifically described by taking fig. 6 as an example. Draw a straight line along the length edge direction of the bed, and draw a straight line artificially along the length of the bed in combination with the drawing of figure 6The angle between the waist point 121 as the origin and the skeleton point 120 on the neck and the knee point 122 is defined as the originThe knee point 122 is used as the origin, and the angle between the waist point 121 and the foot point 123 is used as the angleStraight line and straight line formed by point 120 on the neck and foot point 123Is at an included angle of. Lying on a bedIs =180 ° ±And is and =0°± (ii) a Standing upIs =180 ° ±And is and =90°±2* (ii) a Sit on the bedIs =90 ° ±And is and=180°±(ii) a Sitting on a chairIs =90 ° ±And is and =90°± whereinTo allow for the deviation angle, it may be 15 °. It can be seen that the above-mentioned angle range conditions are the human body posture collation condition of lying on the bed, the human body posture collation condition of sitting on the bed, the human body posture collation condition of standing, and the human body posture collation condition of sitting on the chair. By calculating correspondences in real timeThe human body posture can be judged by respectively comparing the angle values with the angle range conditions.
Specifically, the activity evaluation module comprises an activity amount evaluation unit and an activity capacity evaluation unit. The activity evaluation unit is used for monitoring the statistical time of the patient for keeping the recognized human body posture and monitoring the moving amount of the patient for changing from one human body posture to another human body posture in a certain period. According to the human body posture information identified by the human body posture identification module, the posture of a patient in a certain period and the posture of the patient in the next period can be confirmed. Determining the movement amount according to the posture change, and considering that the movement amount is about 0.4 m if the user sits on the bed after lying on the bed; if the patient sits on the bed to stands, the movement amount is about 0.8 m; the amount of movement is considered to be about 1 meter from standing to sitting on the chair. The movement trajectory of the patient in the ICU generally follows the movement of lying, sitting, standing, and sitting on a bed, and when a plurality of movements are completed, the movements are superimposed on each other to obtain a total movement. The activity ability evaluation unit is used for judging the activity ability of the patient based on the statistical time, the movement amount and the heart rate data in the physiological data. For example, in the counting time, the total movement amount is counted to obtain the total movement amount, and the average value of the heart rate data in the time is combined to determine whether the patient keeps the normal heart rate level in the movement amount, if so, the activity of the patient is good, no additional alarm prompt is needed, and if not, an alarm prompt is needed to be given to a nursing end for nursing, or a prompt device (sound or display) in the system is used for giving an alarm prompt to the patient to pause.
Specifically, the monitoring decision module comprises a pain nursing unit, a head shaking nursing unit and a comprehensive decision unit. The pain nursing unit is used for comparing facial expression information of each part of the face with a pain expression judgment condition and identifying whether a patient has a pain expression; if not, the nursing instruction is not sent to the nursing end; if so: and if the time of the appearance of the pain expression is required to be identified to be greater than the pain threshold time, or the frequency of the appearance of the pain expression in a certain period is required to be identified to be greater than the pain threshold frequency, if the condition is met, sending a nursing instruction to a nursing end, otherwise, not sending the instruction. Fig. 4 shows a schematic of pain expression. In the event of pain, facial expressions typically present a condition of drooping eyebrows, closed eyes, raised cheeks, too high lips, tight eyelids, wrinkled nose, or left-falling mouth. For example, if the pain expression determination condition is set to be that facial expression information has four states of eyebrow dropping, eye closing, eyelid locking and mouth skimming, the patient is identified to have a pain expression. Alternatively, the pain expression determination condition may be set such that any one of the states in fig. 4 exists, and then the pain expression is recognized. However, the recognition is too single, and the possibility of false recognition exists. The pain expression determination condition may be set as needed to enable recognition and the recognition state is not limited to one. The head shaking nursing unit is used for identifying whether the frequency of the head shaking gesture in the head gesture information in a certain period is greater than the head shaking threshold frequency, if so, sending a nursing instruction to a nursing end, and otherwise, not sending the instruction. The patient is determined to have the shaking head gesture by obtaining the shaking head angle in the head gesture information, the times of recognizing all gestures are counted in a certain period, and the frequency of the actual shaking head gesture is calculated. If the counting time is one minute, the shaking threshold frequency is set to be 2hz, and when the frequency is exceeded, the medical staff is required to take urgent care. The comprehensive decision unit is used for comprehensively evaluating the state of the patient according to the facial expression information, the head posture information, the activity data, the physiological data and the vital sign data and sending a comprehensive nursing instruction to the nursing end according to the state of the patient. The comprehensive decision unit can be realized by utilizing a two-layer convolutional neural network, and facial expression information, head posture information, activity data, physiological data and vital sign data are input into the convolutional neural network as input parameters. The network comprises convolution layers of 3 x 2 and 3 x 4, fully connected layers adopt 1 x 4, and finally, a Softmax layer is adopted to output the probability grade of the patient condition. Obtaining the disease condition according to the highest probability. The condition of the patient can be indicated as healthy by 0, mild by 1, moderate by 2 and critical by 3. For healthy patients, the comprehensive nursing instruction is to perform routine nursing every day; for mild patients, the comprehensive nursing instruction is to perform mild nursing every day; for moderate patients, the comprehensive nursing instruction is to perform moderate nursing every day; for dangerous patients, the comprehensive nursing instruction is to nurse the patients every day, such as polling every 30 minutes to 1 hour. The nursing principle conforms to the nursing principle in the ICU of the hospital.
The system also comprises a light source emitter arranged in the ICU and used for providing a light source to assist the image collector to work in a dark environment. The light source emitter can adopt invisible infrared light for human eyes, and when light needs to be dimmed or turned off at night, the image collector can still normally collect video data in the ICU.
The system of the invention further comprises a database for storing facial expression information, head posture information, activity data, physiological data and vital sign data corresponding to the patient. The personal information of the patient can be determined by the house number and the medical record number of the ICU where the patient is located. Data monitored during the ICU is bundled with the patient for later medical record query. Or, the face detection module can be used for face recognition and memory, recording the face form, and comparing the patient pictures stored in the database to recognize the personal information of the patient. And then, the data monitored in the ICU period is bundled and stored with the patient so as to be convenient for inquiring medical records in subsequent diagnosis and treatment.
The system also comprises a plurality of accelerometers which are worn at the joints of the patient and used for detecting the actions of the limbs of the patient. The number of the accelerometers is 4, for example, 2 are arranged at the wrist and 2 are arranged at the ankle, and other joints can be arranged according to the requirement. The monitoring decision module further comprises a dangerous nursing unit for identifying the joint movement angle, judging whether the patient falls down or moves violently when the joint movement angle is larger than the safe movement angle threshold value, and sending a nursing instruction to a nursing end. Medical personnel drive to the ICU for emergency treatment at the first time.
It will be appreciated by persons skilled in the art that the embodiments of the invention described above and shown in the drawings are given by way of example only and are not limiting of the invention. The objects of the present invention have been fully and effectively accomplished. The functional and structural principles of the present invention have been shown and described in the examples, and any variations or modifications of the embodiments of the present invention may be made without departing from the principles.

Claims (10)

1. An ICU intelligent nursing system based on patient video identification comprises an image collector, an intelligent monitoring device, a body index monitoring device and a nursing end; the image collector is used for collecting video data in the ICU; the physical index monitoring device is used for monitoring physiological data and vital sign data of a patient; the intelligent monitoring device is respectively connected with the image collector, the body index monitoring device and the nursing end; its characterized in that, intelligent monitoring device includes:
the face detection module is used for identifying a face position area from video data and obtaining a plurality of facial feature point position vectors in the face position area;
the facial expression recognition module is used for recognizing facial expression information based on the position vectors of the facial feature points and the facial expression comparison vectors;
the head posture identification module is used for converting the position vectors of the facial feature points into three-dimensional coordinates and identifying head posture information by utilizing a random sampling consistency algorithm;
the human body posture identification module is used for identifying a human body position area from the video data, obtaining a plurality of joint vectors in the human body position area, and identifying human body posture information based on the plurality of joint vectors and a human body posture comparison condition;
the activity evaluation module is used for evaluating the activity condition of the patient on the human posture information obtained in the monitoring period to obtain activity data;
the monitoring decision module is used for judging whether to send a nursing instruction to a nursing end according to the facial expression information or the head posture information; and comprehensively evaluating the state of the patient according to the facial expression information, the head posture information, the activity data, the physiological data and the vital sign data, and sending a comprehensive nursing instruction to the nursing end according to the state of the patient.
2. The ICU intelligent nursing system based on patient video recognition of claim 1, wherein the face detection module comprises:
the face recognition unit is used for recognizing a face position area from the video data by utilizing computer vision and deep learning technology;
the characteristic point identification unit is used for identifying a plurality of facial characteristic points of the face position area according to the standard face model and determining a plurality of facial characteristic point position vectors;
the standard face model comprises a plurality of standard face feature points set based on standard face contours.
3. The ICU smart nursing system based on patient video recognition of claim 1 wherein the facial expression recognition module comprises:
the facial motion extraction unit is used for extracting facial motion parameter vectors of all parts of the face from the position vectors of the plurality of facial feature points; the facial motion parameter vector comprises an eyebrow motion parameter vector, an eye motion parameter vector, a cheek motion parameter vector, a mouth motion parameter vector, a nose motion parameter vector and a palate motion parameter vector;
and the facial expression recognition unit is used for contrasting the facial action parameter vector with the facial expression judgment condition and recognizing the facial expression information of each part of the face.
4. The system of claim 1, wherein the head pose recognition module comprises:
the three-dimensional modeling unit is used for converting the position vectors of the facial feature points into three-dimensional coordinates in a three-dimensional world coordinate system and three-dimensional coordinates in an image collector coordinate system, and solving a relational formula of the image coordinates and the world coordinates:
to obtain a rotation matrix R and a translation matrix T;
wherein the content of the first and second substances,is a facial feature point P in a three-dimensional world coordinate systemiIs determined by the three-dimensional coordinates of (a),is a facial feature point PiThree-dimensional coordinates in the image collector coordinate system;as facial feature points PiThe physical coordinates on the imaging plane are,as facial feature points PiPixel coordinate on the imaging plane, i is naturalCounting; the rotation matrix R represents angles of the face in three directions, including a head raising and lowering angle, a head shaking angle and a horizontal deflection angle;
and the head posture identification unit is used for obtaining the optimal angle estimation value by utilizing the fitting of a random sampling consistency algorithm and identifying the head posture information.
5. The ICU intelligent nursing system based on patient video recognition of claim 1, wherein the body posture recognition module comprises:
the human body joint identification unit is used for identifying a human body position area from the video data and identifying key feature points of joints and limbs in the human body position area by utilizing a deep convolutional neural network;
the human body posture identification unit is used for acquiring a plurality of joint vectors of a human body by utilizing the deep convolutional neural network, calculating the bending angle of the joint vectors, comparing the bending angle with the human body posture comparison condition and identifying human body posture information; the human body posture comparison conditions comprise a human body posture comparison condition lying on a bed, a human body posture comparison condition sitting on the bed, a human body posture comparison condition standing, and a human body posture comparison condition sitting on a chair.
6. The system of claim 1, wherein the activity assessment module comprises:
an activity evaluation unit for monitoring the statistical time for the patient to maintain the recognized body posture and monitoring the amount of movement of the patient to change from one body posture to another body posture within a certain period;
and the activity ability evaluation unit is used for judging the activity ability of the patient based on the statistical time, the movement amount and the heart rate data in the physiological data.
7. The ICU intelligent nursing system based on patient video identification as claimed in claim 1, wherein the monitoring decision module comprises:
the pain nursing unit is used for comparing facial expression information of each part of the face with a pain expression judgment condition and identifying whether a patient has a pain expression; if not, the nursing instruction is not sent to the nursing end; if so: if the time of the appearance of the pain expression is required to be identified to be greater than the pain threshold time, or if the frequency of the appearance of the pain expression in a certain period is required to be identified to be greater than the pain threshold frequency, if the condition is met, sending a nursing instruction to a nursing end, otherwise, not sending the instruction;
the head shaking nursing unit is used for identifying whether the frequency of the head shaking gesture in the head gesture information in a certain period is greater than the head shaking threshold frequency, if so, sending a nursing instruction to a nursing end, and otherwise, not sending the instruction;
and the comprehensive decision unit is used for comprehensively evaluating the patient condition and sending a comprehensive nursing instruction to the nursing end according to the patient condition according to the facial expression information, the head posture information, the activity data, the physiological data and the vital sign data.
8. The system of claim 1, further comprising a light source transmitter disposed in the ICU for providing a light source to assist the image collector in operating in a dark environment.
9. The system of claim 1, further comprising a database for storing facial expression information, head pose information, activity data, physiological data, and vital sign data corresponding to the patient.
10. The ICU intelligent nursing system based on patient video identification as claimed in claim 7, further comprising a plurality of accelerometers worn on joints of the patient for detecting the limb movements of the patient, wherein the monitoring decision module further comprises a dangerous nursing unit for identifying the joint movement angle, and when the joint movement angle is greater than the threshold value of the safe movement angle, judging that the patient falls down or moves violently, sending a nursing instruction to a nursing end.
CN202110682391.XA 2021-06-21 2021-06-21 ICU intelligent nursing system based on patient video identification Pending CN113257440A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110682391.XA CN113257440A (en) 2021-06-21 2021-06-21 ICU intelligent nursing system based on patient video identification

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110682391.XA CN113257440A (en) 2021-06-21 2021-06-21 ICU intelligent nursing system based on patient video identification

Publications (1)

Publication Number Publication Date
CN113257440A true CN113257440A (en) 2021-08-13

Family

ID=77188672

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110682391.XA Pending CN113257440A (en) 2021-06-21 2021-06-21 ICU intelligent nursing system based on patient video identification

Country Status (1)

Country Link
CN (1) CN113257440A (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106355147A (en) * 2016-08-26 2017-01-25 张艳 Acquiring method and detecting method of live face head pose detection regression apparatus
CN109806113A (en) * 2019-03-14 2019-05-28 郑州大学 A kind of ward ICU horizontal lower limb rehabilitation intelligent interaction robot group system based on ad hoc network navigation
US20210121121A1 (en) * 2019-10-28 2021-04-29 International Business Machines Corporation Parkinson?s disease treatment adjustment and rehabilitation therapy based on analysis of adaptive games

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106355147A (en) * 2016-08-26 2017-01-25 张艳 Acquiring method and detecting method of live face head pose detection regression apparatus
CN109806113A (en) * 2019-03-14 2019-05-28 郑州大学 A kind of ward ICU horizontal lower limb rehabilitation intelligent interaction robot group system based on ad hoc network navigation
US20210121121A1 (en) * 2019-10-28 2021-04-29 International Business Machines Corporation Parkinson?s disease treatment adjustment and rehabilitation therapy based on analysis of adaptive games

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
张鑫 等: "基于计算机视觉的患者面部偏转角度对面瘫评定的影响分析", 《深圳中西医结合杂志》 *
徐建萍 等: "《疾病观察与护理技能丛书 风湿免疫科疾病观察与护理技能》", 30 April 2019, 北京:中国医药科技出版社 *
林强 等: "《行为识别与智能计算》", 30 November 2016, 西安:西安电子科技大学出版社 *
王崇仁 等: "《医事困学录 津门大全王士相学术经验集》", 31 January 2012, 北京:中国中医药出版社 *
邓开发: "《人工智能与艺术设计》", 30 September 2019, 上海:华东理工大学出版社 *
郭杰荣 等: "《光电信息技术实验教程》", 31 December 2015, 西安:西安电子科技大学出版社 *

Similar Documents

Publication Publication Date Title
Sathyanarayana et al. Vision-based patient monitoring: a comprehensive review of algorithms and technologies
Zhang et al. Activity monitoring using a smart phone's accelerometer with hierarchical classification
US10213152B2 (en) System and method for real-time measurement of sleep quality
Wang et al. Unconstrained video monitoring of breathing behavior and application to diagnosis of sleep apnea
US7420472B2 (en) Patient monitoring apparatus
US7502498B2 (en) Patient monitoring apparatus
CN109558865A (en) A kind of abnormal state detection method to the special caregiver of need based on human body key point
CN109394247B (en) Multi-feature fusion diagnosis user emotion monitoring method
Zamzmi et al. An approach for automated multimodal analysis of infants' pain
WO2017193497A1 (en) Fusion model-based intellectualized health management server and system, and control method therefor
WO2001088836A1 (en) Method and apparatus for remote medical monitoring incorporating video processing and system of motor tasks
CN110477925A (en) A kind of fall detection for home for the aged old man and method for early warning and system
CN105792732A (en) Apnea safety control
CN109863561A (en) Equipment, system and method for patient-monitoring to predict and prevent bed from falling
CN108784669A (en) A kind of contactless heartbeat and disordered breathing monitor system and method
CN112022096A (en) Sleep state monitoring method and device
CN106205001A (en) A kind of hospital security warning system
Werth et al. Deep learning approach for ECG-based automatic sleep state classification in preterm infants
Pogorelc et al. Home-based health monitoring of the elderly through gait recognition
CN113257440A (en) ICU intelligent nursing system based on patient video identification
CN113143223A (en) Edge artificial intelligence infant monitoring method
Yi et al. Home interactive elderly care two-way video healthcare system design
Ahmed et al. Internet of health things driven deep learning-based system for non-invasive patient discomfort detection using time frame rules and pairwise keypoints distance feature
Lyu et al. Skeleton-Based Sleep Posture Recognition with BP Neural Network
CN113762085B (en) Artificial intelligence-based infant incubator system and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination