CN113762085B - Artificial intelligence-based infant incubator system and method - Google Patents

Artificial intelligence-based infant incubator system and method Download PDF

Info

Publication number
CN113762085B
CN113762085B CN202110917475.7A CN202110917475A CN113762085B CN 113762085 B CN113762085 B CN 113762085B CN 202110917475 A CN202110917475 A CN 202110917475A CN 113762085 B CN113762085 B CN 113762085B
Authority
CN
China
Prior art keywords
baby
key point
sound
key points
incubator
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110917475.7A
Other languages
Chinese (zh)
Other versions
CN113762085A (en
Inventor
汤福南
羊月祺
耿向南
管亚飞
任义梅
张晖
汪缨
张作恒
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu Province Hospital First Affiliated Hospital With Nanjing Medical University
Original Assignee
Jiangsu Province Hospital First Affiliated Hospital With Nanjing Medical University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu Province Hospital First Affiliated Hospital With Nanjing Medical University filed Critical Jiangsu Province Hospital First Affiliated Hospital With Nanjing Medical University
Priority to CN202110917475.7A priority Critical patent/CN113762085B/en
Publication of CN113762085A publication Critical patent/CN113762085A/en
Application granted granted Critical
Publication of CN113762085B publication Critical patent/CN113762085B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61G—TRANSPORT, PERSONAL CONVEYANCES, OR ACCOMMODATION SPECIALLY ADAPTED FOR PATIENTS OR DISABLED PERSONS; OPERATING TABLES OR CHAIRS; CHAIRS FOR DENTISTRY; FUNERAL DEVICES
    • A61G11/00—Baby-incubators; Couveuses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61G—TRANSPORT, PERSONAL CONVEYANCES, OR ACCOMMODATION SPECIALLY ADAPTED FOR PATIENTS OR DISABLED PERSONS; OPERATING TABLES OR CHAIRS; CHAIRS FOR DENTISTRY; FUNERAL DEVICES
    • A61G2203/00—General characteristics of devices
    • A61G2203/30—General characteristics of devices characterised by sensor means
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61G—TRANSPORT, PERSONAL CONVEYANCES, OR ACCOMMODATION SPECIALLY ADAPTED FOR PATIENTS OR DISABLED PERSONS; OPERATING TABLES OR CHAIRS; CHAIRS FOR DENTISTRY; FUNERAL DEVICES
    • A61G2203/00—General characteristics of devices
    • A61G2203/30—General characteristics of devices characterised by sensor means
    • A61G2203/44—General characteristics of devices characterised by sensor means for weight
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61G—TRANSPORT, PERSONAL CONVEYANCES, OR ACCOMMODATION SPECIALLY ADAPTED FOR PATIENTS OR DISABLED PERSONS; OPERATING TABLES OR CHAIRS; CHAIRS FOR DENTISTRY; FUNERAL DEVICES
    • A61G2203/00—General characteristics of devices
    • A61G2203/30—General characteristics of devices characterised by sensor means
    • A61G2203/46—General characteristics of devices characterised by sensor means for temperature

Abstract

The invention relates to an artificial intelligence-based infant incubator system and method. The main controller comprises a baby limb action detection module, a baby face detection module and a sound detection module. The baby limb motion detection module: analyzing the video stream file, judging whether the baby in the incubator has abnormal limb movement, and if so, sending related alarm information to a computer; the infant face detection module: analyzing the video stream file, detecting the facial expressions of the infant, including whether eyes and mouth are opened or not and the respective angles under the opening condition, and if the expressions are judged to be crying, sending related alarm information to a computer; the sound detection module: and analyzing the audio stream file by a deep learning method, and judging whether the sound in the incubator is normal or not, whether the sound of crying and crying of the baby exists or not, or whether the sound of alarming of the incubator exists or not.

Description

Artificial intelligence-based infant incubator system and method
Technical Field
The invention relates to the technical field of infant incubators, in particular to a method for designing an infant incubator based on deep learning and computer vision and computer hearing technologies.
Background
The infant incubator integrates the advanced technologies of clinical medicine, machinery, computer automatic control, sensors and other subjects, and provides a good environment with clean air, proper temperature and humidity and similar to the mother uterus for premature infants and sick infants. The humiture in the infant incubator can be set for according to the doctor's advice, and air temperature and humidity, infant skin temperature all have digital display, surpass normal value and can report to the police with the reputation mode, for example overtemperature alarm, fan warning etc..
A plurality of infant incubators are placed in a concentrated mode in a newborn room of a hospital, parents cannot enter into the visiting room, nurses need to enter the room regularly to patrol, and whether infants are abnormal in each infant incubator or not is checked in a short distance. If during two rounds the baby has abnormal behaviour such as crying, struggling etc. a certain risk is created. However, in the infant incubator in the prior art, parameters such as internal temperature, humidity, oxygen concentration and the like are only detected and an alarm is given, so that medical care personnel cannot avoid time delay when getting over. Moreover, the prior art can not carry out intellectual detection system to baby's self state through the environment in the conventional sensor detection incubator only.
The infant has high requirements on the environment, such as certain requirements on conditions of temperature, humidity, noise and the like. However, when the environmental conditions are satisfied, the infant may still experience discomfort by the movement of the body due to the disease or external causes. In addition, the abnormal behavior of the infant in the incubator is usually accompanied by some facial states, such as eyes being closed, mouth being opened and crying, etc. In particular, in the field of neonatal clinical care, great attention is paid to checking whether the infant has convulsion, which is an important index for judging whether the nervous system of the infant is normal. Tics are rapid, repetitive clonic or tonic involuntary motor impulses that start at the face, especially around the eyes and mouth, and extend to the extremities and trunk. Convulsions are manifested as blinking convulsions in the eyes, arm tremors and foot-pedaling convulsions in the limbs. Convulsions are likely to occur during two rounds of visits by medical personnel, and if the convulsions are not discovered and treated early, serious consequences can be caused.
Therefore, how to provide a system and a method capable of detecting and judging the state of the infant in the infant incubator in time is a technical problem to be urgently solved by technical personnel in the field, and the system and the method comprise the steps of judging whether the infant is in a normal state or not by detecting the limb movement, the facial features and the sound of the infant, and detecting whether the infant generates abnormality such as twitch or the like in real time.
Disclosure of Invention
In view of the above problems, the present invention provides an artificial intelligence based infant incubator system and method, which can determine whether an infant has behavior similar to twitching or struggling by detecting the limb movement of the infant in the infant incubator. Whether the infant is in a quiet sleeping state is judged by detecting facial features of the infant in the infant incubator. Whether the baby cries or not and the alarm sound of the incubator are judged by detecting the sound in the baby incubator.
In order to solve the above problems, the present invention provides an artificial intelligence based infant incubator system, which includes an infant incubator main body, a main controller, a camera module, a microphone module, a conventional sensor module, and a computer. The microphone module collects sound, and can accurately identify whether the sound inside the incubator is normal or whether the baby cries or gives an alarm. The camera module collects video streams, and can identify the limb actions and the facial expressions of the infants.
When no anomalies are detected by the data of the conventional sensor module, the baby can still feel sick or otherwise untimely, often crying and following some limb movement of greater magnitude. Therefore, the invention is based on an artificial intelligence mode, the camera module is used for collecting the internal picture of the infant incubator, and the facial expression and the limb movement of the infant are detected to judge whether the infant state is abnormal; meanwhile, the microphone module is used for collecting the sound inside the infant incubator, the sound is classified in a deep learning mode, and the existence of crying and the alarm of the incubator are detected.
Wherein conventional sensor modules include, but are not limited to, air temperature sensors, skin temperature sensors, humidity sensors, oxygen sensors, load cells, noise sensors.
The camera module is used for collecting video pictures in the infant incubator and transmitting the video pictures to the main controller in a video stream mode; the microphone module is used for collecting sound in the infant incubator and transmitting the sound to the main controller. The computer is used for simultaneously displaying the video pictures of a plurality of infant incubators, and is beneficial to centralized management. Meanwhile, the computer receives the data of the conventional sensors of the respective infant incubators transmitted from the main controller, and collectively displays and processes the data.
The main controller comprises a baby limb action detection module, a baby face detection module, a sound detection module and a sensor data acquisition module. The baby limb motion detection module: analyzing the video stream file, judging whether the baby in the incubator has abnormal limb movement, and if so, sending related alarm information to a computer; the infant face detection module: analyzing the video stream file, detecting the facial expressions of the infant, including whether eyes and mouth are opened or not and the respective angles under the opening condition, and if the expressions are judged to be crying, sending related alarm information to a computer; the sound detection module: analyzing the audio stream file by a deep learning method, and judging whether the sound in the incubator is normal or not, whether the sound of crying and crying of the baby exists or not, or whether the sound of alarming of the incubator is normal or not; the sensor data acquisition module: data of conventional sensors of the infant incubator are collected.
The baby limb motion detection module:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame of image into the trained neural network model in sequence, and detecting the positions and confidence degrees of key points of the limbs of the baby, wherein the key points of the limbs comprise left and right wrists, left and right elbows, left and right ankles, left and right knees, left and right shoulders and left and right hips;
3) by connecting specific limb key pointsConnecting into line segments, and marking the left arm, the left elbow, the right arm, the right elbow, the left thigh, the right thigh and the calf; the number of the key points of the limbs is more than or equal to 12, and when the number of the key points of the limbs is 12, (x)1，y1) Is the key point coordinate of the left wrist; (x)2，y2) Coordinates of key points of the left elbow; (x)3，y3) Is the key point coordinate of the right wrist; (x)4，y4) Coordinates of key points of the right elbow; (x)5，y5) Is the key point coordinate of the left ankle; (x)6，y6) Coordinates of key points of the left knee; (x)7，y7) Is the coordinate of the key point of the right ankle; (x)8，y8) Is the key point coordinate of the right knee; (x)9，y9) Is the key point coordinate of the left shoulder; (x)10，y10) Is the key point coordinate of the right shoulder; (x)11，y11) The coordinates of key points of the left hip; (x)12，y12) Is the coordinate of the key point of the right hip; when the number of the limb key points is more than 12, the coordinates of the limb key points can be randomly selected among the 12 limb key points;
4) the angle formed between the significant segments, including the angle θ between the left arm and the left elbow, is calculated and recorded1Angle theta of right arm and right elbow2Angle between left elbow and body θ3Angle between right elbow and body θ4Angle θ between left thigh and calf5Angle θ between right thigh and lower leg6Angle theta formed by the line segment connecting the left thigh and the left and right buttocks7The angle theta formed by the line segment connecting the right thigh and the right and left buttocks8。θ1～θ8The calculation method of (2) is as follows:
at theta1For example, the sum θ is calculated first1The lengths a, b and c of three line segments connected by three related key points, wherein a represents the length of the line segment connected by the key points of the left elbow and the left shoulder, b represents the length of the line segment connected by the key points of the left elbow and the left wrist, and c represents the length of the line segment connected by the key points of the left shoulder and the left wrist, and then theta is calculated according to the cosine theorem1；
Figure GDA0003254009020000031
Figure GDA0003254009020000032
Figure GDA0003254009020000033
Figure GDA0003254009020000034
5) Combining the key point position in the continuous N frames of images with theta1～θ8Recording, and carrying out correlation calculation and comparison by taking the first frame image as an initial position state; n can be set manually, the value of N is related to the frame rate and time, and when the frame rate is g, if the change of pictures for f seconds is required to be checked, the N is g multiplied by f; for example: when the frame rate is 30, if the change of the continuous 5-second picture is to be checked, taking N as 30 × 5 as 150; when the frame rate is 60, if the change of the continuous 10-second picture is to be checked, taking N as 60 × 10 as 600; the value of N is preferably 30-600.
6) In the continuous N frames of images, calculating the coordinates of key points corresponding to the left and right wrists or the left and right ankles in the nth frame of image (N is more than 1 and less than or equal to N) in real time according to the following formula to obtain epsilon, taking the key point of the left wrist as an example, and the coordinate of the key point of the left wrist in the 1 st frame of image is
Figure GDA0003254009020000035
The coordinates of the key point of the left wrist in the nth frame image are
Figure GDA0003254009020000036
And calculating according to a formula to obtain the epsilon corresponding to the frame. When epsilon is more than or equal to epsilon0Judging that the key point moves greatly; when epsilon < epsilon1When the key point is detected to be in the preset position, the key point is judged to have only small movement. E is changed from e within a preset second (preferably 0.5 second)1Up to epsilon0Then rapidly drops to epsilon1The limb associated with this keypoint is represented by a large amplitude back and forth movement of the infant. Epsilon0、ε1Is a preset value, can be freely selected, epsilon0Preferably 0.2 to 0.5,. epsilon1Preferably 0.01 to 0.1. μ in the formula1、μ2Can be set manually, mu1Preferably 0.5 to 0.8 mu2Preferably 0.2 to 0.5.
Figure GDA0003254009020000041
7) If theta is greater than theta1～θ8Wherein the variation Delta theta of one angle theta is larger than the percentage K of the angle thetaθI.e. Delta theta > KθAt x θ, (K)θThe decimal value is 0-1), the body part related to the theta is considered to have larger actions, such as swinging arms, pedaling feet and the like.
8) If at a period of time t1Within seconds (t)1The value is preferably 3-10), and the number of times of continuously recording the larger movement of the hands or feet of the baby is more than N1If the moving frequency of the hands or the feet is high, the infant is judged to have abnormal behaviors (E1 is automatically set to be 1), and related alarm content needs to be sent possibly because special conditions are struggling or twitch behaviors and the like occur; n is a radical of1The preset value can be freely selected, and is preferably 3-15.
9) If at a period of time t2Within seconds (t)2The value is preferably 3-10), and the number of times of larger actions of the limb part of the baby is continuously recorded to be more than N2If the swing frequency of the limb part is higher, it is determined that the infant has an abnormal behavior (E1 is automatically set to 1), and related alarm content needs to be sent because a special condition is struggling or a twitch behavior occurs; n is a radical of2The preset value can be freely selected, and is preferably 3-15.
10) If limb movement abnormality is not detected, E1 is set to 0 by default;
the infant face detection module:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame image into a trained neural network model for detecting the face, detecting the position and confidence of key points of the face of the baby, and selecting S around the left eye and the right eye1A key point (S)1Even number), S is selected around the mouth2A key point (S)2An even number).
3) The present invention uses α to represent the degree to which the eye is open; when the value of S1 is n, n is more than or equal to 4, and alpha is calculated according to the following formula; p1～PnIs a key point around the eye, where P1And PnIs a left and a right end points of an eye, P2、P4、P6To Pn-2Is the key point of the upper half of the eye contour, P3、P5、P7、Pn-1Is the lower half of the eye contour, and P2And P3Is a key point of upper and lower symmetry, P4And P5Is a key point of upper and lower symmetry, P6And P7Is the key point of the upper and lower symmetry, … …, and so on, Pn-2And Pn-1Is a key point of upper and lower symmetry.
Figure GDA0003254009020000042
4) When alpha remains a constant alpha greater than 00When the eyes of the baby are in an open state; when α remains a constant α close to 01When the infant is in the eye-closing state, the infant is in the eye-closing state; when alpha is from alpha within a preset second (preferably 0.5 second)0Rapidly drops to alpha1Indicating that the infant has performed a blinking action. Wherein alpha is0And alpha1Can be respectively preset artificially according to the actual conditions of the infants, alpha0The value range is 0.2-0.3, alpha1The value range is 0-0.1.
5) When the eye is open at alpha0And alpha1And may have blinked twitches (E2 automatically set to 1) and associated alert content needs to be sent.
The baby mouth detection module comprises:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame image into a trained neural network model for detecting the mouth, detecting the position and confidence of key points of the mouth of the baby, and selecting S around the mouth2Selecting a plurality of key points to represent the opening degree of the mouth, and using beta to represent the key points; when S is2When the value is n, n is more than or equal to 4 and is an even number, and beta is calculated by the following formula; q1～QnIs a key point around the mouth, where Q1And QnIs a mouth with two left and right end points, Q2、Q4、Q6To Qn-2Is the key point of the upper half of the mouth contour, Q3、Q5、Q7、Qn-1Is the lower half of the mouth contour, and Q2And Q3Is a key point of upper and lower symmetry, Q4And Q5Is a key point of upper and lower symmetry, Q6And Q7Is the key point of the upper and lower symmetry, … …, and so on, Qn-2And Qn-1Is a key point of upper and lower symmetry.
Figure GDA0003254009020000051
3) Likewise, when β remains a constant β greater than 00When the baby is in the open state, the baby mouth is in the open state; when beta remains a constant beta close to 01When the baby is in a closed state, the baby stands for that the mouth of the baby is in a closed state; wherein beta is0And beta1Can be respectively preset artificially according to the actual conditions of the infants, beta0The value range is 0.2-0.5, beta1The value range is 0-0.1.
4) When the mouth opening degree is beta0And beta1And if the change is repeated, the state is abnormal (E2 is automatically set to 1), and related alarm content needs to be sent.
5) E2 is set to 0 by default if no blink tic abnormalities or mouth region abnormalities are detected;
the sound detection module:
1) acquiring an audio stream through a microphone in a baby incubator;
2) selecting sound segments with the time length within preset seconds from the audio stream;
3) inputting the sound segments into a trained sound classification model, and outputting sound classification and confidence;
4) when the output sound is classified as baby crying and the confidence is greater than X1When the alarm is received, the main controller sends a relevant alarm to the computer;
X1the preset value is set between 80 percent and 90 percent;
5) when the output sound is classified as an incubator alarm sound and the confidence is greater than X2When the alarm is received, the main controller sends a relevant alarm to the computer; x2The preset value is set between 80 percent and 90 percent;
6) when the output sound is classified as normal environmental sound, the sound in the incubator is considered to be abnormal;
7) further, the training process of the sound classification model specifically includes:
8) firstly, collecting an audio data set related to a baby incubator, wherein the data set comprises 3 or more audio categories and sound clips including artificially marked baby crying, incubator alarm sound and other categories;
9) then establishing a sound classification neural network model which comprises C1A convolutional layer and C2A full connection layer; c1The value is preferably 4-12, C2The value is preferably 2.
10) Setting training parameters of the sound classification neural network model, wherein the training parameters comprise a training time Epoch and a Learning rate Learning _ rate;
11) and importing the audio data set into a sound classification neural network model, finishing training and storing the trained model.
The invention also provides an artificial intelligence-based infant incubator method, which comprises
The method for detecting the limb movement of the infant specifically comprises the following steps:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame of image into the trained neural network model in sequence, and detecting the positions and confidence degrees of key points of the limbs of the baby, wherein the key points of the limbs comprise left and right wrists, left and right elbows, left and right ankles, left and right knees, left and right shoulders and left and right hips;
3) marking the left arm, the left elbow, the right arm, the right elbow, the left thigh, the right thigh and the lower leg by connecting specific limb key points into line segments; the number of the key points of the limbs is more than or equal to 12, and when the number of the key points of the limbs is 12, (x)1，y1) Is the key point coordinate of the left wrist; (x)2，y2) Coordinates of key points of the left elbow; (x)3，y3) Is the key point coordinate of the right wrist; (x)4，y4) Coordinates of key points of the right elbow; (x)5，y5) Is the key point coordinate of the left ankle; (x)6，y6) Coordinates of key points of the left knee; (x)7，y7) Is the coordinate of the key point of the right ankle; (x)8，y8) Is the key point coordinate of the right knee; (x)9，y9) Is the key point coordinate of the left shoulder; (x)10，y10) Is the key point coordinate of the right shoulder; (x)11，y11) The coordinates of key points of the left hip; (x)12，y12) Is the coordinate of the key point of the right hip; when the number of the limb key points is more than 12, the coordinates of the limb key points can be randomly selected among the 12 limb key points;
4) the angle formed between the significant segments, including the angle θ between the left arm and the left elbow, is calculated and recorded1Angle theta of right arm and right elbow2Angle between left elbow and body θ3Angle between right elbow and body θ4Angle θ between left thigh and calf5Angle θ between right thigh and lower leg6Angle theta formed by the line segment connecting the left thigh and the left and right buttocks7The angle theta formed by the line segment connecting the right thigh and the right and left buttocks8。θ1～θ8The calculation method of (2) is as follows:
at theta1For example, the sum θ is calculated first1The lengths a, b and c of three line segments connected by three related key points, wherein a represents the length of the line segment connected by the key points of the left elbow and the left shoulder, b represents the length of the line segment connected by the key points of the left elbow and the left wrist, and c represents the length of the line segment connected by the key points of the left shoulder and the left wrist, and then theta is calculated according to the cosine theorem1；
Figure GDA0003254009020000071
Figure GDA0003254009020000072
Figure GDA0003254009020000073
Figure GDA0003254009020000074
5) Combining the key point position in the continuous N frames of images with theta1～θ8Recording, and carrying out correlation calculation and comparison by taking the first frame image as an initial position state; n can be set manually, the value of N is related to the frame rate and time, and when the frame rate is g, if the change of pictures for f seconds is required to be checked, the N is g multiplied by f; for example: when the frame rate is 30, if the change of the continuous 5-second picture is to be checked, taking N as 30 × 5 as 150; when the frame rate is 60, if the change of the continuous 10-second picture is to be checked, taking N as 60 × 10 as 600; the value of N is preferably 30-600.
6) In the continuous N frames of images, calculating the coordinates of key points corresponding to the left and right wrists or the left and right ankles in the nth frame of image (N is more than 1 and less than or equal to N) in real time according to the following formula to obtain epsilon, taking the key point of the left wrist as an example, and the coordinate of the key point of the left wrist in the 1 st frame of image is
Figure GDA0003254009020000075
The coordinates of the key point of the left wrist in the nth frame image are
Figure GDA0003254009020000076
And calculating according to a formula to obtain the epsilon corresponding to the frame. When epsilon is more than or equal to epsilon0Judging that the key point moves greatly; when epsilon < epsilon1When the key point is detected to be in the preset position, the key point is judged to have only small movement. E is changed from e within a preset second (preferably 0.5 second)1Up to epsilon0Then rapidly drops to epsilon1The limb associated with this keypoint is represented by a large amplitude back and forth movement of the infant. Epsilon0、ε1Is a preset value, can be freely selected, epsilon0Preferably 0.2 to 0.5,. epsilon1Preferably 0.01 to 0.1. μ in the formula1、μ2Can be set manually, mu1Preferably 0.5 to 0.8 mu2Preferably 0.2 to 0.5.
Figure GDA0003254009020000077
7) If theta is greater than theta1～θ8Wherein the variation Delta theta of one angle theta is larger than the percentage K of the angle thetaθI.e. Delta theta > KθAt x θ, (K)θThe decimal value is 0-1), the body part related to the theta is considered to have larger actions, such as swinging arms, pedaling feet and the like.
8) If at a period of time t1Within seconds (t)1The value is preferably 3-10), and the number of times of continuously recording the movement of hands or feet of the baby is more than N1If the moving frequency of the hands or the feet is high, the infant is judged to have abnormal behaviors, and the infant struggles possibly due to special conditions; n is a radical of1The preset value can be freely selected, and is preferably 3-15.
9) If at a period of time t2Within seconds (t)2The value is preferably 3-10), and the number of times of larger actions of the limb part of the baby is continuously recorded to be more than N2That is, the frequency of the swing of the limb part is high, it is determined that the baby has abnormal behavior, possibly because ofSpecial cases are struggling; n is a radical of2The preset value can be freely selected, and is preferably 3-15.
Infant face detection comprising the steps of:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame image into a trained neural network model for detecting the face, detecting the position and confidence of key points of the face of the baby, and selecting S around the left eye and the right eye1A key point (S)1Even number), S is selected around the mouth2A key point (S)2An even number).
3) The present invention uses α to represent the degree to which the eye is open; when the value of S1 is n, n is more than or equal to 4, and alpha is calculated according to the following formula; p1～PnIs a key point around the eye, where P1And PnIs a left and a right end points of an eye, P2、P4、P6To Pn-2Is the key point of the upper half of the eye contour, P3、P5、P7、Pn-1Is the lower half of the eye contour, and P2And P3Is a key point of upper and lower symmetry, P4And P5Is a key point of upper and lower symmetry, P6And P7Is the key point of the upper and lower symmetry, … …, and so on, Pn-2And Pn-1Is a key point of upper and lower symmetry.
Figure GDA0003254009020000081
4) When alpha remains a constant alpha greater than 00When the eyes of the baby are in an open state; when α remains a constant α close to 01When the infant is in the eye-closing state, the infant is in the eye-closing state; when alpha is from alpha within a preset second (preferably 0.5 second)0Rapidly drops to alpha1Indicating that the infant has performed a blinking action. Wherein alpha is0And alpha1Can be respectively preset artificially according to the actual conditions of the infants, alpha0The value range is 0.2-0.3, alpha1Value rangeThe circumference is 0-0.1.
Infant mouth detection comprising the steps of:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame image into a trained neural network model for detecting the mouth, detecting the position and confidence of key points of the mouth of the baby, and selecting S around the mouth2Selecting a plurality of key points to represent the opening degree of the mouth, and using beta to represent the key points; when S is2When the value is n, n is more than or equal to 4 and is an even number, and beta is calculated by the following formula; q1～QnIs a key point around the mouth, where Q1And QnIs a mouth with two left and right end points, Q2、Q4、Q6To Qn-2Is the key point of the upper half of the mouth contour, Q3、Q5、Q7、Qn-1Is the lower half of the mouth contour, and Q2And Q3Is a key point of upper and lower symmetry, Q4And Q5Is a key point of upper and lower symmetry, Q6And Q7Is the key point of the upper and lower symmetry, … …, and so on, Qn-2And Qn-1Is a key point of upper and lower symmetry.
Figure GDA0003254009020000082
3) Likewise, when β remains a constant β greater than 00When the baby is in the open state, the baby mouth is in the open state; when beta remains a constant beta close to 01When the baby is in a closed state, the baby stands for that the mouth of the baby is in a closed state; wherein beta is0And beta1Can be respectively preset artificially according to the actual conditions of the infants, beta0The value range is 0.2-0.5, beta1The value range is 0-0.1.
4) When the mouth opening degree is beta0And beta1If the difference is repeated, it is abnormal.
Sound detection, comprising the steps of:
1) acquiring an audio stream through a microphone in a baby incubator;
2) selecting sound segments with the time length within preset seconds from the audio stream;
3) inputting the sound segments into a trained sound classification model, and outputting sound classification and confidence;
4) when the output sound is classified as baby crying and the confidence is greater than X1When the alarm is received, the main controller sends a relevant alarm to the computer; x1The preset value is set between 80 percent and 90 percent;
5) when the output sound is classified as an incubator alarm sound and the confidence is greater than X2When the alarm is received, the main controller sends a relevant alarm to the computer; x2The preset value is set between 80 percent and 90 percent;
6) when the output sound is classified as normal environmental sound, the sound in the incubator is considered to be abnormal;
7) further, the training process of the sound classification model specifically includes:
8) firstly, collecting an audio data set related to a baby incubator, wherein the data set comprises 3 or more audio categories and sound clips including artificially marked baby crying, incubator alarm sound and other categories;
9) then establishing a sound classification neural network model which comprises C1A convolutional layer and C2A full connection layer; c1The value is preferably 4-12, C2The value is preferably 2.
10) Setting training parameters of the sound classification neural network model, wherein the training parameters comprise a training time Epoch and a Learning rate Learning _ rate;
11) and importing the audio data set into a sound classification neural network model, finishing training and storing the trained model.
The software process of the artificial intelligent incubator system comprises the following steps:
reading the video stream from the camera module, and detecting the limb movement and the face area of the baby;
reading the audio stream from the microphone module, and carrying out sound classification detection in the infant incubator;
after the detection module is called, if the abnormity is found, the corresponding alarm content is sent to a computer end for displaying; if no exception is found, the above flow continues from the beginning.
And after receiving the corresponding alarm content, the computer scores the infant state in the infant incubator. The scoring formula is Score equal to E1 × w1+ E2 × w2+ E3 × w 3. Wherein E1, E12 and E3 are Boolean values, the value of E1 represents whether the abnormal limb movement occurs, the value of E1 is 0 represents that the abnormal limb movement is not detected, and the value of E1 is 1 represents that the abnormal limb movement is detected; the value of E2 represents whether the face area is detected to be abnormal or not, the value of E2 represents that the face area is not detected to be abnormal, and the value of E2 represents that the face area is detected to be abnormal; the value of E3 represents whether the sound detection is abnormal or not, the value of E3 is 0 which represents that the abnormal sound is not detected, and the value of E3 is 1 which represents that the abnormal sound is detected; w1, w2 and w3 represent the weight of the three types of events respectively, the value ranges of w1, w2 and w3 are preferably (0, 0.3), (0, 0.3) and (0, 0.4) respectively, Score represents the attention degree of the current infant, and the higher Score value represents the attention degree of the infant in the corresponding incubator.
In the embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways.
The apparatus and method embodiments described above are illustrative only, as the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s).
It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
It should also be noted that for some implementations of the required links, for example, labeling of critical points of the limb, may be based on a plane (2D) or in three-dimensional space (3D).
In addition, each functional module in the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part. For example, the software flow diagram of fig. 2 may be executed in the main controller or migrated to the computer for execution.
It should be emphasized that, in terms of infant expression recognition, limb movement recognition and sound recognition in the infant incubator, a model stored after training by using self data or an existing model is called, which are considered as the protection scope of the present invention.
It will be understood that the above embodiments are merely exemplary embodiments taken to illustrate the principles of the present invention, which is not limited thereto. It will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the spirit and substance of the invention, and these modifications and improvements are also considered to be within the scope of the invention.
Advantageous effects
The invention can accurately acquire the state of the baby in real time by adopting the baby limb action detection technology, the baby eye detection technology, the baby mouth detection technology and the sound detection technology.
The prior art relies primarily on data changes from various conventional sensors to monitor the infant from the side in an incubator. For example, it is not appropriate to uniformly specify a comfortable range because the adaptability of each infant to the environment is different, by judging whether the temperature in the incubator is suitable for the survival of the infant or not through the air temperature sensor, or judging whether the air in the incubator is suitable for the breathing of the infant or not through the oxygen concentration sensor. The infant itself is intuitive to the environment within the incubator, and the infant will show through body language as well as crying and facial expressions. In particular, in the case that all conventional sensors have correct values, the infant may also present discomfort due to diseases or other reasons, through body language, crying and facial expressions, which requires real-time detection of the infant itself, so that the infant can be noticed in the incubator in time and better attended by medical staff.
The invention focuses more on the state of the baby, can judge whether the baby is in a healthy and comfortable state under the condition that the conventional sensor cannot detect the abnormality, and has strong clinical practicability.
The system can reduce the patrol pressure of clinical medical staff, and particularly can continuously detect the self state of the baby and carry out intelligent judgment under the condition that the number of the staff participating in nursing and patrolling at night is reduced by half;
aiming at the current situation that parents of infants cannot enter a room for visiting, the system can rebroadcast the real-time picture of the camera to the parents through the computer, so that the visiting requirement is met;
the invention discovers whether the baby has convulsion as soon as possible. Some infants are admitted because of convulsion caused by nervous system problems, and medical staff need to patrol the infants in the infant incubator for at least half an hour, but cannot perform one-to-one observation for 24 hours, and if the convulsion occurs in the period of two patrol, the infants are probably overlooked, thereby causing more serious consequences. The system can detect whether the baby twitches in real time, record corresponding pictures and send an alarm to inform medical staff to perform targeted treatment on the baby.
Drawings
FIG. 1 is a general block diagram of an artificial intelligence based infant incubator system and method according to the present invention.
FIG. 2 is a flow chart of an artificial intelligence based infant incubator system and method according to the present invention.
Fig. 3 is a schematic diagram of extraction of key points of limbs of an infant according to the present invention.
Fig. 4 is a diagram of the coordinates and the angle marks of the key points of the limbs of the baby according to the invention.
FIG. 5 is a schematic diagram of the calculation of key angles according to the present invention.
Fig. 6 is a schematic diagram of the eye key points and calculation according to the present invention.
Fig. 7 is a schematic diagram of key points and calculation of the mouth according to the present invention.
FIG. 8 is a flow chart of the training process of the infant incubator sound classification deep learning model according to the present invention.
Fig. 9 is a diagram of 68 keypoint identifiers of a human face according to an embodiment.
Detailed Description
The invention is described in further detail below:
the COCO dataset is an existing public dataset; dat is a public human face feature point library; PANNs are models of sound classification trained on Audioset data, and CNN14 is one of the pre-trained models.
The preferable embodiment of the infant incubator system and method based on artificial intelligence is as follows:
the method comprises the following steps in the aspect of detecting the limb movement of the baby:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) and sequentially inputting each frame of image into a trained neural network model Posenet, wherein Posenet is a real-time human body posture model based on deep learning, and the model can realize real-time human body pose estimation by using an authoritative public COCO data set. Since the COCO data set represents the key points of the human body as 17 joints including the nose, the left and right eyes, the left and right ears, the left and right shoulders, the left and right elbows, the left and right wrists, the left and right hips, the left and right knees, and the left and right ankles, in this embodiment, the infant limb is detected using PoseNet, 12 key points are selected and the position and confidence thereof are saved, and the 12 key points includeLeft and right wrists, left and right elbows, left and right shoulders, left and right ankles, left and right knees, left and right hips; wherein (x)1，y1) The key point coordinates of the left wrist; (x)2，y2) Coordinates of key points of the left elbow; (x)3，y3) The key point coordinates of the right wrist; (x)4，y4) Coordinates of key points of the right elbow; (x)5，y5) Is the key point coordinate of the left ankle; (x)6，y6) Coordinates of key points of the left knee; (x)7，y7) Is the coordinate of the key point of the right ankle; (x)8，y8) Is the key point coordinate of the right knee; (x)9，y9) Is the key point coordinate of the left shoulder; (x)10，y10) Is the key point coordinate of the right shoulder; (x)11，y11) The coordinates of key points of the left hip; (x)12，y12) Is the coordinate of the key point of the right hip;
3) marking the left arm, the left elbow, the right arm, the right elbow, the left thigh, the right thigh and the lower leg by connecting specific key points into line segments;
4) the angles formed between some important line segments, including the angle theta between the left arm and the left elbow, are calculated and recorded1Angle theta of right arm and right elbow2Angle between left elbow and body θ3Angle between right elbow and body θ4Angle θ between left thigh and calf5Angle θ between right thigh and lower leg6Angle theta formed by the line segment connecting the left thigh and the left and right buttocks7The angle theta formed by the line segment connecting the right thigh and the right and left buttocks8。θ1～θ8The calculation method of (2) is as follows:
at theta1For example, the sum θ is calculated first1The lengths a, b and c of three line segments connected by three related key points, wherein a represents the length of the line segment connected by the key points of the left elbow and the left shoulder, b represents the length of the line segment connected by the key points of the left elbow and the left wrist, and c represents the length of the line segment connected by the key points of the left shoulder and the left wrist, and then theta is calculated according to the cosine theorem1。
Figure GDA0003254009020000121
Figure GDA0003254009020000122
Figure GDA0003254009020000123
Figure GDA0003254009020000124
5) Combining the key point position in the continuous N frames of images with theta1～θ8Recording, and carrying out correlation calculation and comparison by taking the first frame image as an initial position state; n may be set manually, and in this embodiment, N is set to 30.
6) In the continuous N frames of images, calculating the coordinates of key points corresponding to the left and right wrists or the left and right ankles in the nth frame of image (N is more than 1 and less than or equal to N) in real time according to the following formula to obtain epsilon, taking the key point of the left wrist as an example, and the coordinate of the key point of the left wrist in the 1 st frame of image is
Figure GDA0003254009020000131
The coordinates of the key point of the left wrist in the nth frame image are
Figure GDA0003254009020000132
And calculating according to a formula to obtain the epsilon corresponding to the frame. When epsilon is more than or equal to epsilon0Judging that the key point moves greatly; when epsilon < epsilon1When the key point is detected to be in the preset position, the key point is judged to have only small movement. E is changed from e within a preset second (preferably 0.5 second)1Up to epsilon0Then rapidly drops to epsilon1The limb associated with this keypoint is represented by a large amplitude back and forth movement of the infant. In this example ε0Is set to 0.3, ∈1Set to 0.1. μ in the formula1、μ2Can be set manually, and mu in the embodiment1Set to 0.6, mu2Set to 0.4.
Figure GDA0003254009020000133
7) If theta is greater than theta1～θ8Wherein the variation Delta theta of one angle theta is larger than the percentage K of the angle thetaθ(KθThe value is 0.3), namely when delta theta is larger than theta multiplied by 30%, the relevant limb part is considered to have larger swinging actions, such as swinging arms, pedaling feet and the like. For example, in the 1 st and 3 rd frame images, θ1Are respectively theta1、θ1', the corresponding angle change amount Delta theta1＝|θ1’-θ1When Δ e1When the angle is larger than theta multiplied by 30 percent, the swing action is considered to occur between the left wrist and the left arm;
8) if at a period of time t1Within seconds (t)1The value is 3-10), the moving times of hands or feet of the baby are continuously recorded to be more than N1If the moving frequency of the hands or the feet is high, the infant is judged to have abnormal behaviors, and the infant struggles possibly due to special conditions; t is t1A value of 3, N1The value is 5.
9) If at a period of time t2Within seconds (t)2The value is 3-10), and continuously recording that the number of times of large swing actions of the limb part of the baby is more than N2If the swing frequency of the limb part is higher, the infant is judged to have abnormal behavior, and the infant struggles possibly due to special conditions; t is t2A value of 3, N2The value is 5.
The method comprises the following steps in the aspect of infant face detection:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) and sequentially inputting each frame image into a trained face landmark predictor shape _ predictor 68 face _ maps for detecting faces, wherein the predictor is a trained face feature point library and can identify faces by 68 key points. The predictor can be used to obtain the positions and the confidence degrees of the key points of the face of the baby, in the embodiment, 6 key points are respectively selected around the left eye and the right eye, and 6 key points are selected around the mouth.
3) The present embodiment uses α to indicate the degree to which the eyes are open. 6 key points are selected around the left eye and the right eye, and alpha is calculated according to the following formula. Wherein P is1～P6Is a key point around the eye, P1And P6Is the left and right two end points of the eye, P2And P4One third and two thirds of the location point, P, of the upper half of the eye contour, respectively3And P5One third and two thirds of the location point of the lower half of the eye contour, respectively.
Figure GDA0003254009020000134
6) The baby's mouth region is also detected in this way. As shown in FIG. 9, since there are 20 feature points in the mouth, 6 of them are selected to represent the feature points, and the degree of mouth opening is represented by β, which is calculated as follows, wherein Q1～Q6Is a key point around the mouth, Q1And Q6Is the left and right ends of the mouth, Q2And Q4One third and two thirds of the location point, Q, of the upper half of the mouth contour, respectively3And Q5One third and two thirds of the location point of the lower half of the mouth contour, respectively.
Figure GDA0003254009020000141
4) When alpha is kept at a constant alpha larger than 00When the eyes of the baby are in an open state; when α remains at some constant α close to 01When the infant is in the eye-closing state, the infant is in the eye-closing state; when alpha is from alpha0Rapidly drops to alpha1Indicating that the infant has performed a blinking action. Wherein alpha is0And alpha1Can be respectively preset artificially according to the actual conditions of the infants, alpha0The value range is 0.2-0.3, alpha1The value range is 0-0.1.
5) Similarly, when β remains at some constant β greater than 00When the baby is in the open state, the baby mouth is in the open state; when beta remains at some constant beta close to 01When the baby is in a closed state, the baby stands for that the mouth of the baby is in a closed state; wherein beta is0And beta1Can be respectively preset artificially according to the actual conditions of the infants, beta0The value range is 0.2-0.5, beta1The value range is 0-0.1.
At a period of time t4Within seconds (t)4The value range is 0-5), if the baby is detected to be in a quiet state, for example, eyes and mouths are closed within 10 seconds or occasionally blinking motions (blinking ranges from 2 to 5 times), the baby state is normal; if eye closure is detected, the mouth is open at beta0And beta1The change is repeated, which indicates that the baby is possibly in a crying state and needs to be paid attention by medical staff in time; t is t4The value is 3;
the method comprises the following steps in terms of sound detection:
1) acquiring an audio stream through a microphone in a baby incubator;
2) selecting time length t from audio stream5Sound fraction of seconds, t5The value is 10;
3) inputting the sound segments into a trained sound classification model, and outputting sound classification and confidence;
4) if the output sound is classified as baby crying and the confidence coefficient is more than 80%, the main controller sends a related alarm to the computer;
5) if the output sound is classified as an incubator alarm sound and the confidence is greater than 80%, the main controller sends a relevant alarm to the computer;
6) if the output sound is classified as normal environmental sound, the sound in the incubator is considered to be abnormal;
7) further, the training process of the sound classification model specifically includes:
8) firstly, collecting a data set of 3000 infant incubator audios, wherein the data set comprises 3 categories including artificially marked infant crying, incubator alarm sound and sound under normal conditions;
9) then calling a neural network model CNN14, wherein the model comprises 12 convolutional layers and 2 full-connection layers; setting training parameters of the sound classification neural network model, wherein the training parameters comprise training times Epoch, Learning rate Learning _ rate and the like;
the Epoch value is 5000, and the Learning _ rate value is 0.2;
10) and importing the audio data set into the neural network model, finishing training and storing the trained model.
11) After the detection module is called, if the abnormity is found, the corresponding alarm content is sent to a computer end for displaying;
12) if no exception is found, the above flow continues from the beginning.
When the sound detection module uses the neural network model to identify the sound in the incubator, the sound detection module is interfered by the crying of the baby in other incubators in the same room, so that the sound detection module cannot be used for final judgment directly according to the output result of the neural network. The mouth detection module is combined with the baby mouth detection module to judge, and if the sound classification detection module detects crying of the baby, the mouth detection module is required to judge whether crying movement exists in the mouth of the baby. If the two modules simultaneously judge that the baby cries, the related abnormal conditions in the incubator can be confirmed; if the sound classification detection module detects crying, but the mouth detection module judges that the mouth of the baby is not opened, the baby is considered to be crying interference of babies in other incubators, and therefore errors of alarm content are avoided.

Claims (8)

1. An artificial intelligence-based infant incubator system comprises an infant incubator main body and a conventional sensor module, wherein the conventional sensor module comprises an air temperature sensor, a skin temperature sensor, a humidity sensor, an oxygen sensor, a weighing sensor and a noise sensor; the system is characterized by also comprising a main controller, a camera module, a microphone module and a computer;
the camera module is used for collecting video pictures in the infant incubator and transmitting the video pictures to the main controller in a video stream mode;
the microphone module is used for collecting sound in the infant incubator and transmitting the sound to the main controller;
the main controller comprises a baby limb action detection module, a baby face detection module, a sound detection module, a sensor data acquisition module and a transmission module;
the baby limb motion detection module: analyzing the video stream file, judging whether the baby in the incubator has abnormal limb movement, and if so, sending related alarm information to a computer;
the infant face detection module: analyzing the video stream file, detecting the facial expressions of the infant, including whether eyes and mouth are opened or not and the respective angles under the opening condition, and if the expressions are judged to be crying, sending related alarm information to a computer;
the sound detection module: analyzing the audio stream file by a deep learning method, and judging whether the sound in the incubator is normal or not, whether the sound of crying and crying of the baby exists or not, or whether the sound of alarming of the incubator is normal or not;
the sensor data acquisition module: collecting data of a conventional sensor of an infant incubator;
the transmission module is as follows: the judgment result and the collected data are sent to a computer,
the computer is used for simultaneously displaying video pictures of a plurality of infant incubators, alarm information analyzed through artificial intelligence and various data acquired through various sensors;
the baby limb motion detection module:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame of image into the trained neural network model in sequence, and detecting the positions and confidence degrees of key points of the limbs of the baby, wherein the key points of the limbs comprise left and right wrists, left and right elbows, left and right ankles, left and right knees, left and right shoulders and left and right hips;
3) by connecting specific limb key points into line segments, the left arm, the left elbow, the right elbow, the left elbow, the right elbow, the left elbow, the right elbow, the left elbow, the right elbow, the left elbow, the right elbow, the left elbow, the right elbow, the left elbow, the right,The right elbow and the left and right thighs and calves; the number of the key points of the limbs is more than or equal to 12, and when the number of the key points of the limbs is 12, (x)1,y1) Is the key point coordinate of the left wrist; (x)2,y2) Coordinates of key points of the left elbow; (x)3,y3) Is the key point coordinate of the right wrist; (x)4,y4) Coordinates of key points of the right elbow; (x)5,y5) Is the key point coordinate of the left ankle; (x)6,y6) Coordinates of key points of the left knee; (x)7,y7) Is the coordinate of the key point of the right ankle; (x)8,y8) Is the key point coordinate of the right knee; (x)9,y9) Is the key point coordinate of the left shoulder; (x)10,y10) Is the key point coordinate of the right shoulder; (x)11,y11) The coordinates of key points of the left hip; (x)12,y12) Is the coordinate of the key point of the right hip; when the number of the limb key points is more than 12, the coordinates of the limb key points can be randomly selected among the 12 limb key points;
4) the angle formed between the significant segments, including the angle θ between the left arm and the left elbow, is calculated and recorded1Angle theta of right arm and right elbow2Angle between left elbow and body θ3Angle between right elbow and body θ4Angle θ between left thigh and calf5Angle θ between right thigh and lower leg6Angle theta formed by the line segment connecting the left thigh and the left and right buttocks7The angle theta formed by the line segment connecting the right thigh and the right and left buttocks8；θ1～θ8The calculation method of (2) is as follows:
at theta1For example, the sum θ is calculated first1The lengths a, b and c of three line segments connected by three related key points, wherein a represents the length of the line segment connected by the key points of the left elbow and the left shoulder, b represents the length of the line segment connected by the key points of the left elbow and the left wrist, and c represents the length of the line segment connected by the key points of the left shoulder and the left wrist, and then theta is calculated according to the cosine theorem1；
Figure FDA0003538448700000021
Figure FDA0003538448700000022
Figure FDA0003538448700000023
Figure FDA0003538448700000024
5) Combining the key point position in the continuous N frames of images with theta1～θ8Recording, and carrying out correlation calculation and comparison by taking the first frame image as an initial position state; n is a preset value, the value of N is related to the frame rate and time, and when the frame rate is g, if the change of pictures for f seconds is required to be checked, the N is g multiplied by f;
6) in the continuous N frame images, the nth frame (1)<N is less than or equal to N) in the image, calculating the coordinates of the key points corresponding to the left wrist, the right ankle or the left ankle in real time according to the following formula to obtain a parameter epsilon, taking the key point of the left wrist as an example, and the coordinate of the key point of the left wrist in the 1 st frame of image as
Figure FDA0003538448700000025
The coordinates of the key point of the left wrist in the nth frame image are
Figure FDA0003538448700000026
Calculating to obtain an epsilon corresponding to the frame according to a formula; when epsilon is more than or equal to epsilon0Judging that the key point moves greatly; when epsilon<ε1Judging that the key point only has small movement; when epsilon is within preset second1Up to epsilon0Then rapidly drops to epsilon1Representing that the limb associated with the key point of the baby moves back and forth by a larger amplitude; epsilon0、ε1Is a preset value of epsilon0The value is 0.2-0.5 epsilon1The value is 0.01-0.1; μ in the formula1、μ2Is a preset value, mu1The value is 0.5-0.8 mu2The value is 0.2-0.5;
Figure FDA0003538448700000027
6) if theta is greater than theta1～θ8Wherein the variation Delta theta of one angle theta is larger than the percentage K of the angle thetaθI.e. Delta theta>KθAt x θ, KθIf the value is decimal between 0 and 1, the limb part related to the theta is considered to have larger action;
7) if the moving frequency of hands or feet of the baby is continuously recorded to be greater than the preset frequency within the preset second, namely the moving frequency of the hands or the feet is higher, the baby is judged to have abnormal behaviors;
9) if the fact that the number of times of large actions of the limb part of the baby is larger than the preset number of times is continuously recorded within the preset second, namely the swing frequency of the limb part is high, the baby is judged to have abnormal behaviors;
10) if no limb movement abnormality is detected.
2. The artificial intelligence based infant incubator system of claim 1, wherein the infant face detection module:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame image into a trained neural network model for detecting the face, detecting the position and confidence of key points of the face of the baby, and selecting S around the left eye and the right eye1A key point, S1Is an even number;
3) using the aspect ratio α of the eye to represent the degree to which the eye is open; when the value of S1 is n, n is more than or equal to 4, and alpha is calculated according to the following formula; p1～PnIs a key point around the eye, where P1And PnIs a left and a right end points of an eye, P2、P4、P6To Pn-2Is the key point of the upper half of the eye contour, P3、P5、P7、Pn-1Is the lower half of the eye contour, and P2And P3Is a key point of upper and lower symmetry, P4And P5Is a key point of upper and lower symmetry, P6And P7Is the key point of the upper and lower symmetry, … …, and so on, Pn-2And Pn-1Are key points which are symmetrical up and down;
Figure FDA0003538448700000031
4) when alpha remains a constant alpha greater than 00When the eyes of the baby are in an open state; when α remains a constant α close to 01When the infant is in the eye-closing state, the infant is in the eye-closing state; when alpha is within preset second from alpha0Rapidly drops to alpha1Indicating that the infant has performed a blinking action; wherein alpha is0And alpha1Respectively preset artificially according to the actual conditions of the infants, alpha0The value range is 0.2-0.3, alpha1The value range is 0-0.1.
3. The artificial intelligence based infant incubator system of claim 1, wherein the main controller further comprises an infant mouth detection module that:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame image into a trained neural network model for detecting the mouth, detecting the position and confidence of key points of the mouth of the baby, and selecting S around the mouth2A key point; selecting a plurality of key points to represent the opening degree of the mouth, and using beta to represent the opening degree of the mouth; when S is2When the value is n, n is more than or equal to 4 and is an even number, and beta is calculated by the following formula; q1～QnIs a key point around the mouth, where Q1And QnIs a mouth with two left and right end points, Q2、Q4、Q6To Qn-2Is the key point of the upper half of the mouth contour, Q3、Q5、Q7To Qn-1Is the lower half of the mouth contour, and Q2And Q3Is a key point of upper and lower symmetry, Q4And Q5Is a key point of upper and lower symmetry, Q6And Q7Is the key point of the upper and lower symmetry, … …, and so on, Qn-2And Qn-1Are key points which are symmetrical up and down;
Figure FDA0003538448700000032
3) likewise, when β remains a constant β greater than 00When the baby is in the open state, the baby mouth is in the open state; when beta remains a constant beta close to 01When the baby is in a closed state, the baby stands for that the mouth of the baby is in a closed state; wherein beta is0And beta1Can be respectively preset artificially according to the actual conditions of the infants, beta0The value range is 0.2-0.5, beta1The value range is 0-0.1;
4) when the mouth opening degree is beta0And beta1If the difference is repeated, it is abnormal.
4. The artificial intelligence based infant incubator system of claim 1, wherein the sound detection module:
1) acquiring an audio stream through a microphone in a baby incubator;
2) selecting sound segments with the time length within preset seconds from the audio stream;
3) inputting the sound segments into a trained sound classification model, and outputting sound classification and confidence;
4) when the output sound is classified as baby crying and the confidence is greater than X1When the alarm is received, the main controller sends a relevant alarm to the computer; x1The preset value is set between 80 percent and 90 percent;
5) when the output sound is classified as an incubator alarm sound and the confidence is greater than X2When the alarm is received, the main controller sends a relevant alarm to the computer; x2The preset value is set between 80 percent and 90 percent;
6) when the output sound is classified as normal environmental sound, the sound in the incubator is considered to be abnormal;
7) further, the training process of the sound classification model specifically includes:
8) firstly, collecting an audio data set related to a baby incubator, wherein the data set comprises 3 or more audio categories and sound clips including artificially marked baby crying, incubator alarm sound and other categories;
9) then establishing a sound classification neural network model, wherein the model comprises a convolution layer and a full-connection layer;
10) setting training parameters of the sound classification neural network model, wherein the training parameters comprise a training time Epoch and a Learning rate Learning _ rate;
11) and importing the audio data set into a sound classification neural network model, finishing training and storing the trained model.
5. An artificial intelligence based infant incubator method according to the system of claim 1, wherein: the method comprises the following steps of detecting the limb movement of the baby:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame of image into the trained neural network model in sequence, and detecting the positions and confidence degrees of key points of the limbs of the baby, wherein the key points of the limbs comprise left and right wrists, left and right elbows, left and right ankles, left and right knees, left and right shoulders and left and right hips;
3) marking the left arm, the left elbow, the right arm, the right elbow, the left thigh, the right thigh and the lower leg by connecting specific limb key points into line segments; the number of the key points of the limbs is more than or equal to 12, and when the number of the key points of the limbs is 12, (x)1,y1) Is the key point coordinate of the left wrist; (x)2,y2) Coordinates of key points of the left elbow; (x)3,y3) Is the key point coordinate of the right wrist;(x4,y4) Coordinates of key points of the right elbow; (x)5,y5) Is the key point coordinate of the left ankle; (x)6,y6) Coordinates of key points of the left knee; (x)7,y7) Is the coordinate of the key point of the right ankle; (x)8,y8) Is the key point coordinate of the right knee; (x)9,y9) Is the key point coordinate of the left shoulder; (x)10,y10) Is the key point coordinate of the right shoulder; (x)11,y11) The coordinates of key points of the left hip; (x)12,y12) Is the coordinate of the key point of the right hip; when the number of the limb key points is more than 12, the coordinates of the limb key points can be randomly selected among the 12 limb key points;
4) the angle formed between the significant segments, including the angle θ between the left arm and the left elbow, is calculated and recorded1Angle theta of right arm and right elbow2Angle between left elbow and body θ3Angle between right elbow and body θ4Angle θ between left thigh and calf5Angle θ between right thigh and lower leg6Angle theta formed by the line segment connecting the left thigh and the left and right buttocks7The angle theta formed by the line segment connecting the right thigh and the right and left buttocks8；θ1～θ8The calculation method of (2) is as follows:
at theta1For example, the sum θ is calculated first1The lengths a, b and c of three line segments connected by three related key points, wherein a represents the length of the line segment connected by the key points of the left elbow and the left shoulder, b represents the length of the line segment connected by the key points of the left elbow and the left wrist, and c represents the length of the line segment connected by the key points of the left shoulder and the left wrist, and then theta is calculated according to the cosine theorem1；
Figure FDA0003538448700000051
Figure FDA0003538448700000052
Figure FDA0003538448700000053
Figure FDA0003538448700000054
8) Combining the key point position in the continuous N frames of images with theta1～θ8Recording, and carrying out correlation calculation and comparison by taking the first frame image as an initial position state; n is a preset value, the value of N is related to the frame rate and time, and when the frame rate is g, if the change of pictures for f seconds is required to be checked, the N is g multiplied by f;
6) in the continuous N frame images, the nth frame (1)<N is less than or equal to N) in the image, calculating the coordinates of the key points corresponding to the left wrist, the right ankle or the left ankle in real time according to the following formula to obtain a parameter epsilon, taking the key point of the left wrist as an example, and the coordinate of the key point of the left wrist in the 1 st frame of image as
Figure FDA0003538448700000055
The coordinates of the key point of the left wrist in the nth frame image are
Figure FDA0003538448700000056
Calculating to obtain an epsilon corresponding to the frame according to a formula; when epsilon is more than or equal to epsilon0Judging that the key point moves greatly; when epsilon<ε1Judging that the key point only has small movement; when epsilon is within preset second1Up to epsilon0Then rapidly drops to epsilon1Representing that the limb associated with the key point of the baby moves back and forth by a larger amplitude; epsilon0、ε1Is a preset value of epsilon0The value is 0.2-0.5 epsilon1The value is 0.01-0.1; μ in the formula1、μ2Is a preset value, mu1The value is 0.5-0.8 mu2The value is 0.2-0.5;
Figure FDA0003538448700000057
7) if theta is greater than theta1～θ8Wherein the variation Delta theta of one angle theta is larger than the percentage K of the angle thetaθI.e. Delta theta>KθAt x θ, KθIf the value is decimal between 0 and 1, the limb part related to the theta is considered to have larger action;
8) if the moving frequency of hands or feet of the baby is continuously recorded to be greater than the preset frequency within the preset second, namely the moving frequency of the hands or the feet is higher, the baby is judged to have abnormal behaviors;
9) if the fact that the number of times of large actions of the limb part of the baby is larger than the preset number of times is continuously recorded within the preset second, namely the swing frequency of the limb part is high, the baby is judged to have abnormal behaviors;
10) if no limb movement abnormality is detected.
6. The artificial intelligence based infant incubator method of claim 5, wherein: the method also comprises the following steps of detecting the face of the baby:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame image into a trained neural network model for detecting the face, detecting the position and confidence of key points of the face of the baby, and selecting S around the left eye and the right eye1A key point, S1Is an even number;
3) using the aspect ratio α of the eye to represent the degree to which the eye is open; when the value of S1 is n, n is more than or equal to 4, and alpha is calculated according to the following formula; p1～PnIs a key point around the eye, where P1And PnIs a left and a right end points of an eye, P2、P4、P6To Pn-2Is the key point of the upper half of the eye contour, P3、P5、P7、Pn-1Is the lower half of the eye contour, and P2And P3Is a key point of upper and lower symmetry, P4And P5Is a key point of upper and lower symmetry, P6And P7Is the key point of the upper and lower symmetry, … …, and so on, Pn-2And Pn-1Are key points which are symmetrical up and down;
Figure FDA0003538448700000061
4) when alpha remains a constant alpha greater than 00When the eyes of the baby are in an open state; when α remains a constant α close to 01When the infant is in the eye-closing state, the infant is in the eye-closing state; when alpha is within preset second from alpha0Rapidly drops to alpha1Indicating that the infant has performed a blinking action; wherein alpha is0And alpha1Respectively preset artificially according to the actual conditions of the infants, alpha0The value range is 0.2-0.3, alpha1The value range is 0-0.1.
7. The artificial intelligence based infant incubator method of claim 5, wherein: the method also comprises baby mouth detection, and specifically comprises the following steps:
1) acquiring a video stream through a camera in the infant incubator, and reading each frame of image in the video stream;
2) inputting each frame image into a trained neural network model for detecting the mouth, detecting the position and confidence of key points of the mouth of the baby, and selecting S around the mouth2A key point; selecting a plurality of key points to represent the opening degree of the mouth, and using beta to represent the opening degree of the mouth; when S is2When the value is n, n is more than or equal to 4 and is an even number, and beta is calculated by the following formula; q1～QnIs a key point around the mouth, where Q1And QnIs a mouth with two left and right end points, Q2、Q4、Q6To Qn-2Is the key point of the upper half of the mouth contour, Q3、Q5、Q7、Qn-1Is the lower half of the mouth contour, and Q2And Q3Is a key point of upper and lower symmetry, Q4And Q5Is vertically symmetricalKey point, Q6And Q7Is the key point of the upper and lower symmetry, … …, and so on, Qn-2And Qn-1Are key points which are symmetrical up and down;
Figure FDA0003538448700000071
3) likewise, when β remains a constant β greater than 00When the baby is in the open state, the baby mouth is in the open state; when beta remains a constant beta close to 01When the baby is in a closed state, the baby stands for that the mouth of the baby is in a closed state; wherein beta is0And beta1Can be respectively preset artificially according to the actual conditions of the infants, beta0The value range is 0.2-0.5, beta1The value range is 0-0.1;
4) when the mouth opening degree is beta0And beta1If the difference is repeated, it is abnormal.
8. The artificial intelligence based infant incubator method of claim 5, wherein:
the method also comprises sound detection, and specifically comprises the following steps:
1) acquiring an audio stream through a microphone in a baby incubator;
2) selecting sound segments with the time length within preset seconds from the audio stream;
3) inputting the sound segments into a trained sound classification model, and outputting sound classification and confidence;
4) when the output sound is classified as baby crying and the confidence is greater than X1When the alarm is received, the main controller sends a relevant alarm to the computer; x1The preset value is set between 80 percent and 90 percent;
5) when the output sound is classified as an incubator alarm sound and the confidence is greater than X2When the alarm is received, the main controller sends a relevant alarm to the computer; x2The preset value is set between 80 percent and 90 percent;
6) when the output sound is classified as normal environmental sound, the sound in the incubator is considered to be abnormal;
7) further, the training process of the sound classification model specifically includes:
8) firstly, collecting an audio data set related to a baby incubator, wherein the data set comprises 3 or more audio categories and sound clips including artificially marked baby crying, incubator alarm sound and other categories;
9) then establishing a sound classification neural network model, wherein the model comprises a convolution layer and a full-connection layer;
10) setting training parameters of the sound classification neural network model, wherein the training parameters comprise a training time Epoch and a Learning rate Learning _ rate;
11) and importing the audio data set into a sound classification neural network model, finishing training and storing the trained model.
CN202110917475.7A 2021-08-11 2021-08-11 Artificial intelligence-based infant incubator system and method Active CN113762085B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110917475.7A CN113762085B (en) 2021-08-11 2021-08-11 Artificial intelligence-based infant incubator system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110917475.7A CN113762085B (en) 2021-08-11 2021-08-11 Artificial intelligence-based infant incubator system and method

Publications (2)

Publication Number Publication Date
CN113762085A CN113762085A (en) 2021-12-07
CN113762085B true CN113762085B (en) 2022-04-19

Family

ID=78788978

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110917475.7A Active CN113762085B (en) 2021-08-11 2021-08-11 Artificial intelligence-based infant incubator system and method

Country Status (1)

Country Link
CN (1) CN113762085B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114694269A (en) * 2022-02-28 2022-07-01 江西中业智能科技有限公司 Human behavior monitoring method, system and storage medium
CN114596279B (en) * 2022-03-08 2023-09-22 江苏省人民医院（南京医科大学第一附属医院） Non-contact respiration detection method based on computer vision
CN114821674B (en) * 2022-06-28 2022-11-18 合肥的卢深视科技有限公司 Sleep state monitoring method, electronic device and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN204256568U (en) * 2014-12-03 2015-04-08 南京金陵自动调温床有限公司 A kind of far and near range monitoring, there is the infant incubator of drawing and pulling type air duct board
CN204274880U (en) * 2014-12-03 2015-04-22 南京金陵自动调温床有限公司 A kind of far and near range monitoring, there is the infant incubator of built-in blue-ray light
CN106710599A (en) * 2016-12-02 2017-05-24 深圳撒哈拉数据科技有限公司 Particular sound source detection method and particular sound source detection system based on deep neural network
CN110368237A (en) * 2019-08-12 2019-10-25 宁波戴维医疗器械股份有限公司 A kind of infant incubator monitoring system
CN113158918A (en) * 2021-04-26 2021-07-23 深圳市商汤科技有限公司 Video processing method and device, electronic equipment and storage medium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110193964A1 (en) * 2010-02-07 2011-08-11 Mcleod Gregory F Method and System for Wireless Monitoring

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN204256568U (en) * 2014-12-03 2015-04-08 南京金陵自动调温床有限公司 A kind of far and near range monitoring, there is the infant incubator of drawing and pulling type air duct board
CN204274880U (en) * 2014-12-03 2015-04-22 南京金陵自动调温床有限公司 A kind of far and near range monitoring, there is the infant incubator of built-in blue-ray light
CN106710599A (en) * 2016-12-02 2017-05-24 深圳撒哈拉数据科技有限公司 Particular sound source detection method and particular sound source detection system based on deep neural network
CN110368237A (en) * 2019-08-12 2019-10-25 宁波戴维医疗器械股份有限公司 A kind of infant incubator monitoring system
CN113158918A (en) * 2021-04-26 2021-07-23 深圳市商汤科技有限公司 Video processing method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN113762085A (en) 2021-12-07

Similar Documents

Publication Publication Date Title
CN113762085B (en) Artificial intelligence-based infant incubator system and method
Chambers et al. Computer vision to automatically assess infant neuromotor risk
US20210290106A1 (en) Systems and methods of tracking patient movement
CN113257440A (en) ICU intelligent nursing system based on patient video identification
JP4189298B2 (en) Infant movement analysis system
TW200933538A (en) Nursing system
CN106553201A (en) Lifesaving robot and its implementation
CN109887238A (en) A kind of fall detection system and detection alarm method of view-based access control model and artificial intelligence
CN110853753A (en) Cognitive dysfunction old man rehabilitation and nursing system at home
CN114999646B (en) Newborn exercise development assessment system, method, device and storage medium
US20210225489A1 (en) Determining the likelihood of patient self-extubation
US20210287792A1 (en) Care system and automatic care method
CN109757923A (en) A kind of mattress with infantile state monitoring system
WO2018147252A1 (en) Bed monitoring system
CN113593693A (en) Remote health management platform
Wang et al. Vision analysis in detecting abnormal breathing activity in application to diagnosis of obstructive sleep apnoea
CN111652192A (en) Tumble detection system based on kinect sensor
Jung et al. Walking-in-place characteristics-based geriatric assessment using deep convolutional neural networks
Kitzig et al. MoveHN-A database to support the development of motion based biosignal processing systems
Mitas et al. Wearable system for activity monitoring of the elderly
CN110534209B (en) Choke rescue guidance system and method
Huang et al. Infant Contact-less Non-Nutritive Sucking Pattern Quantification via Facial Gesture Analysis.
Dusarlapudi et al. COVID-19 patient breath monitoring and assessment with MEMS accelerometer-based DAQ-a machine learning approach.
US11497418B2 (en) System and method for neuroactivity detection in infants
Jeon et al. SleePS: Sleep position tracking system for screening sleep quality by wristbands

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant