CN112535459A - Sleep abnormity detection method and device, electronic equipment and readable storage medium - Google Patents
Sleep abnormity detection method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN112535459A CN112535459A CN202011341969.7A CN202011341969A CN112535459A CN 112535459 A CN112535459 A CN 112535459A CN 202011341969 A CN202011341969 A CN 202011341969A CN 112535459 A CN112535459 A CN 112535459A
- Authority
- CN
- China
- Prior art keywords
- target object
- shot image
- image
- sleep
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/48—Other medical applications
- A61B5/4806—Sleep evaluation
- A61B5/4809—Sleep detection, i.e. determining whether a subject is asleep or not
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/48—Other medical applications
- A61B5/4806—Sleep evaluation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- Veterinary Medicine (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Heart & Thoracic Surgery (AREA)
- Biomedical Technology (AREA)
- Public Health (AREA)
- Medical Informatics (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Human Computer Interaction (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Anesthesiology (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The application provides a sleep anomaly detection method, a sleep anomaly detection device, an electronic device and a readable storage medium, wherein the method comprises the following steps: acquiring a current shot image; and determining whether the target object has a sleep abnormal state according to the position change condition of the target object in the current shot image and the historical shot image. Therefore, whether the abnormal sleep state occurs during the sleep of the human body can be effectively judged according to the position change conditions of the target object in the current shot image and the historical shot image, and the active identification of the abnormal sleep state occurring in the sleep process can be effectively realized. Furthermore, operations such as marking of the occurrence time of the abnormal sleep state in monitoring and abnormal active reminding can be carried out based on the recognized abnormal sleep state, so that a user can know the problems possibly existing in the sleep process of the target object conveniently, and the user can be helped to master the abnormal state possibly existing in the sleep process conveniently without depending on a close-fitting sensor device.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a sleep abnormality detection method and apparatus, an electronic device, and a readable storage medium.
Background
With the development of smart homes, people pay more and more attention to the living quality of people. At present, people often monitor sleep conditions by means of close-fitting sensor devices or by installing cameras and looking up monitoring records.
In the case of monitoring sleep conditions by means of a form-fitting sensor device, the sleep experience of the user is often impaired, since the sensor device still needs to be carried form-fitting during sleep. In the mode of installing the camera, the camera only realizes the function of data recording and cannot realize active identification of the abnormal state in the sleeping process, so that the functions of active marking, reminding and the like of whether the abnormal state occurs in the sleeping process cannot be realized, and the problem of poor practical application effect exists.
Disclosure of Invention
An object of the embodiments of the present application is to provide a sleep abnormality detection method, a sleep abnormality detection apparatus, an electronic device, and a readable storage medium, which are used to identify an abnormal state in a sleep process.
The embodiment of the application provides a sleep anomaly detection method for network equipment, which comprises the following steps: acquiring a current shot image; and determining whether the target object has a sleep abnormal state according to the position change condition of the target object in the current shot image and the historical shot image.
During normal human sleep, the human body does not usually undergo large movements, but usually does not move completely for a long time. Based on this, in the embodiment of the application, whether the abnormal sleep state occurs during the sleep of the human body can be effectively judged according to the position change conditions of the target object in the current shot image and the historical shot image, so that the active identification of the abnormal sleep state occurring in the sleep process can be effectively realized. Furthermore, operations such as marking of the occurrence time of the abnormal sleep state in monitoring and abnormal active reminding can be carried out based on the recognized abnormal sleep state, so that a user can know the problems possibly existing in the sleep process of the target object conveniently, and the user can be helped to master the abnormal state possibly existing in the sleep process conveniently without depending on a close-fitting sensor device.
Further, the method further comprises: detecting key points of a human body of a target object in the current shot image; acquiring a detection result of the human body key point of the target object in a historical shooting image; the determining whether the target object has the abnormal sleep state according to the position change condition of the target object in the current shot image and the historical shot image comprises the following steps: and determining whether the target object has an abnormal sleep state according to the position change condition of the human key points of the target object in the current shot image and the historical shot image.
In the implementation process, the target object is determined to exist in the image based on the human body key point detection technology, and meanwhile the abnormal sleep state detection based on the change of the human body key point position is achieved, so that the implementation is simple, and the detection result is reliable.
Further, the history shot image is a shot image which is the last shot image of the current shot image; determining whether the target object has a sleep abnormal state according to the position change condition of the human key point of the target object in the current shot image and the historical shot image, wherein the determining step comprises the following steps: and when the integral deviation between the human key point of the target object in the current shot image and the human key point of the target object detected in the last shot image is greater than a preset first deviation threshold value, determining that a sleep abnormal state occurs.
It is understood that the human body does not typically undergo large movements during normal human sleep. Therefore, in the implementation process, the overall deviation between the human body key point of the target object in the current captured image and the human body key point of the target object detected in the last captured image is compared with the preset first deviation threshold value, so as to determine whether the human body has moved greatly during the sleeping process, and further determine whether the abnormal sleeping state occurs according to the determination.
Further, determining whether the target object has an abnormal sleep state according to the position change condition of the key points of the human body of the target object in the current shot image and the historical shot image, and further comprising: calculating the position deviation between the key points of the human body corresponding to the same human body part in the current shot image and the last shot image; determining the integral deviation between the human body key point of the current shot image and the human body key point detected in the last shot image according to the position deviation corresponding to each human body key point; comparing the overall deviation with the preset first deviation threshold.
It should be understood that in practical applications, more than one human key point is often detected. In order to calculate the overall deviation between the human key points of the two images, the overall deviation can be determined by calculating the position deviation between the human key points corresponding to the same human body part in the current shot image and the previous shot image and further based on the position deviation corresponding to each human key point. The overall deviation obtained in the mode can well reflect the motion conditions of the human body in the two adjacent images, so that the reliability of the scheme of the embodiment of the application can be improved.
Further, the history shot image is a shot image which is the last shot image of the current shot image; determining whether the target object has a sleep abnormal state according to the position change condition of the human key point of the target object in the current shot image and the historical shot image, wherein the determining step comprises the following steps: and when the human body key point of the target object is detected in the last shot image and is positioned at the edge of the last shot image, determining that the abnormal sleep state occurs when the human body key point of the target object is not detected in the current shot image.
In real life, an abnormal situation such as a dream trip has a remarkable characteristic that a person can get up to walk and unconsciously do something, so that a dream trip usually leaves an imaging area when a pseudo-dream trip occurs. In the implementation process, the abnormal sleep state is determined based on whether the key points of the human body are detected in the two adjacent frames of images.
Further, the current shot image is an image shot in a preset sleep time period.
It should be understood that in practical application, the sleeping space (such as a bedroom) of the user is often used in the non-sleeping time periods, and the detection of the abnormal sleeping state in the time periods is not only meaningless, but also can cause a series of misjudgments due to the using behavior of the sleeping space by the user. Therefore, in the implementation mode, the current shot image is limited to be the image shot in the preset sleep time period, so that the condition of false detection caused by normal behaviors of the user under the non-sleep condition can be effectively reduced.
Further, the historical captured image is a continuous N-frame captured image before the current captured image; n is a preset positive integer; determining whether the target object has a sleep abnormal state according to the position change condition of the human key point of the target object in the current shot image and the historical shot image, wherein the determining step comprises the following steps: and determining that the abnormal sleep state occurs when the overall deviation between the human key points of the target object in all the two adjacent shot images in the current shot image and the historical shot image is less than a preset second deviation threshold.
It should be understood that although the human body does not perform a large movement during the normal sleep of the human, since the time for one sleep of the human is limited (generally about six to eight hours), if the human is in a state of being immobilized for a long time, a problem is likely to occur. Therefore, in the embodiment of the application, by judging that the overall deviation between the human key points of the target object in all the two adjacent frames of shot images is smaller than the preset second deviation threshold value in the continuous multi-frame shot images, whether the human body is in a stationary state for a long time is determined, and then the abnormality is accurately determined.
Further, the method further comprises: and when the target object is determined to have the abnormal sleep state, carrying out abnormal marking.
The abnormal marking is carried out when the target object is determined to have the abnormal sleep state, so that the user can look up the target object in a targeted mode according to the mark when looking up the monitoring, and the user can know and master the sleep state conveniently.
Further, the current shot image and the historical shot image are infrared imaging images; the determining whether the target object has the abnormal sleep state according to the position change condition of the target object in the current shot image and the historical shot image comprises the following steps: identifying human body infrared of the target object in the current shot image and the historical shot image; and determining whether the target object has an abnormal sleep state according to the position deviation of the human body infrared of the target object in the current shot image and the historical shot image.
In the implementation process, the motion condition of the human body in the sleeping process is detected in an infrared imaging mode, the implementation is simple, and the detection result is reliable.
The embodiment of the present application further provides a sleep abnormality detection apparatus, including: the device comprises an acquisition module, a detection module and a mark processing module; the acquisition module is used for acquiring a current shot image and acquiring a detection result of a human body key point of a target object in a historical shot image; the detection module is used for detecting key points of a human body of a target object in the current shot image; the mark processing module is used for determining whether the target object has a sleep abnormal state according to the position change condition of the human key points of the target object in the current shot image and the historical shot image.
An embodiment of the present application further provides an electronic device, including: a communication module, a processor and a memory; the communication module is used for being in communication connection with an image shooting device so as to acquire an image acquired by the image shooting device and process the image by the processor; the processor is configured to execute one or more programs stored in the memory to implement any of the above sleep anomaly detection methods for a network device.
The embodiment of the present application further provides a readable storage medium, where one or more programs are stored, and the one or more programs are executable by one or more processors to implement any of the above sleep anomaly detection methods for a network device.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a schematic basic flow chart of a sleep abnormality detection method according to an embodiment of the present application;
fig. 2 is a schematic diagram illustrating an image capturing apparatus according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a sleep abnormality detection apparatus according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
The first embodiment is as follows:
referring to fig. 1, a sleep abnormality detection method provided in the embodiment of the present application in fig. 1 includes:
s101: and acquiring the current shot image.
In the embodiment of the application, the image shooting device can be installed in sleeping spaces such as bedrooms, so that shooting and collecting of images of a user in sleeping are achieved through the image shooting device.
For example, in the embodiment of the present application, the image capturing device may be directed to a bed for a user to sleep. Meanwhile, the image shooting device can be in communication connection with the back-end processing equipment, so that shot images are sent to the back-end processing equipment, and the back-end processing equipment can process the shot images according to the scheme in the embodiment of the application.
It is to be understood that, in the embodiment of the present application, the image capturing apparatus may be a camera, an infrared imager, or other electronic devices having an imaging function.
Considering that a person mainly sleeps at night, in a possible implementation of the embodiment of the present application, a starlight level monitoring camera capable of monitoring in a dark light condition may be used for image acquisition.
S102: and determining whether the target object has an abnormal sleep state according to the position change conditions of the target object in the current shot image and the historical shot image.
In the embodiment of the present application, all human bodies located in the shooting area may be target objects.
In the embodiment of the application, the detection result of the human key points of the target object in the historical image can be simultaneously obtained by detecting the human key points of the target object in the current shot image, so that the determination of whether the target object has the abnormal sleep state or not can be realized based on the position change conditions of the human key points of the same target object in the current shot image and the historical shot image.
It should be understood that, in the embodiment of the present application, the detection of the human body key points of the target object from the image may be implemented, but not limited to, by using a key point detection model based on deep learning (such as an openposition model, a top-down method model, etc.).
It should be understood that, in the embodiment of the present application, when a plurality of target objects exist in an image, the distinction between each target object and its human key point may be achieved, but is not limited to, by a multi-person pose estimation algorithm or the like.
Since the human body does not usually perform large movements during normal sleep of the human body. Therefore, the motion amplitude of the human body is a good parameter for evaluating the sleep state of the user in the sleep process.
For this reason, in a possible implementation manner of the embodiment of the present application, the detection result of the human key point of the target object in the acquired historical captured image may be a detection result of a human key point of the target object in a captured image that is previous to the current captured image.
Therefore, the overall deviation of the key points of the human body of the target object in the last shot image and the current shot image can be realized, and the judgment on whether the target object has the abnormal sleep state currently is realized.
In this embodiment, a first deviation threshold may be preset, so that the overall deviation of the human key points of the target object in the previous captured image and the current captured image is compared with the first deviation threshold. If the overall deviation is greater than the first deviation threshold, it may be determined that the target object has moved greatly, and at this time, it may be determined that the target object has an abnormal sleep state. And if the overall deviation is less than or equal to the first deviation threshold value, the latest current shot image can be continuously obtained for continuous judgment.
In the embodiment of the present application, the first deviation threshold may be set by an engineer according to a large number of experiments or empirical values.
In the embodiment of the present application, the overall deviation of the target object between the key points of the human body in the current captured image and the historical captured image can be obtained by:
the position deviation between the human key points corresponding to the same human body part in the current shot image and the last shot image can be calculated, and then the overall deviation between the human key points of the current shot image and the human key points detected in the last shot image is determined according to the position deviation corresponding to each human key point.
It should be noted that, due to the motion of the target object, it may occur that a few key points of the human body in the current captured image do not exist in the previous captured image. In the embodiment of the present application, the human body key points can be ignored, and only the position deviation between the human body key points corresponding to the same human body part existing in the two images is calculated.
In the embodiment of the present application, the overall deviation between the human key point of the currently captured image and the human key point detected in the last captured image may be, but is not limited to, obtained by calculating an average of the position deviations of all human key points.
It is to be understood that in the above possible embodiments, activities such as kicking a quilt, turning over, dream walking, etc. may all be considered as abnormal sleep states. However, the sleep disorder state of the dream trip is different from the sleep disorder states such as quilt kicking and turning over, and the target object leaves the bed and does something unconsciously during the dream trip. Therefore, when a pseudo-dream occurs, the target object usually leaves the imaging area.
Based on the characteristics of the sleep abnormality of the dream trip, in a feasible implementation manner of the embodiment of the present application, it may be determined whether the target object leaves the image capturing area by determining whether a human body key point of the target object is detected in a previous captured image (the target object rises during the dream trip, and therefore the human body key point of the target object is necessarily detected), and whether the detected human body key point of the target object is located at an edge of the previous captured image, and determining whether the human body key point of the target object is detected in a current captured image.
If the human key point of the target object is detected in the last shot image, the human key point is located at the edge of the last shot image, and the human key point of the target object is not detected in the current shot image, the suspected dream trip condition can be considered to occur, so that the abnormal sleep state is determined to occur.
It should be understood that, in the embodiment of the present application, it may also be determined that a sleep abnormality occurs by detecting that the target object is away from the bed through an image recognition algorithm, and considering that a suspected dream-swimming situation occurs.
For example, the bed areas in the current captured image and the previous captured image may be calibrated through an image recognition algorithm, and then when the target object in the previous captured image is recognized through the image recognition algorithm not to be out of the bed area, and when the target object in the current captured image is out of the bed area, it may be considered that a suspected dream trip situation occurs, so as to determine that a sleep abnormality occurs.
It should be noted that, in practical applications, the occurrence of the dream trip condition is necessarily accompanied by a large movement of the target object. Therefore, in a possible implementation manner of the embodiment of the present application, the determination of whether or not the dream trip occurs may be performed in combination with the detection result of the position change with respect to the last captured image.
For example, when the last captured image is currently acquired, it may be determined whether there is a human body key point of the target object. If the difference exists, determining the overall deviation between the human body key point of the target object in the last shot image and the human body key point of the target object in the last shot image (hereinafter referred to as the last shot image), and then comparing the overall deviation with a preset first deviation threshold value to obtain a comparison result.
Then, when the current shot image is obtained, whether the human body key point of the target object exists is judged at first. If not, judging whether the last shot image has the human body key point of the target object. If yes, obtaining a comparison result corresponding to the last shot image. If the overall deviation is larger than a preset first deviation threshold value, the suspected dream trip situation can be considered to occur, and therefore the abnormal sleep state is determined to occur. If the comparison result is that the integral deviation is less than or equal to the preset first deviation threshold, the abnormal sleep state is not considered to occur, and the next frame of shot image is processed continuously according to the logic.
In addition, if the last captured image does not have the human key point of the target object, the dream trip process can be considered.
It should be noted that in the practical application process, the sleeping space (such as a bedroom) of the user is often used in the non-sleeping time periods, and the detection of the abnormal sleeping state in the time periods is not only meaningless, but also can cause a series of misjudgments due to the using behavior of the sleeping space by the user. For this reason, in the embodiment of the present application, for the above several possible embodiments, a sleep time period may be set in advance, so that only images captured in the sleep time period are processed in the above manner.
It should be understood that, although the human body does not move greatly during the normal sleep of the human, since the time for one sleep of the human is limited (generally about six to eight hours), if the human is in a state of being immobilized for a long time, an abnormal condition is likely to occur.
Therefore, in the embodiment of the present application, the detection result of the abnormal sleep state corresponding to the N consecutive frames of captured images before the current captured image may be obtained. And then determining that the abnormal sleep state occurs when the overall deviation between the human key points of the target object in all the two adjacent shot images is smaller than a preset second deviation threshold value in the current shot image and the historical shot image.
It should be noted that, in the embodiment of the present application, the two adjacent captured images for performing sleep abnormality detection are not limited to two consecutive video frames in the video captured by the image capturing device, and may also be video frames separated by a preset time length (for example, 1 second). Namely, one video frame can be extracted every preset time length to be used as a current shot image, and the processing is carried out according to the scheme provided by the embodiment of the application.
It should be noted that the value of N may be determined by an engineer according to the extraction time interval of the current shot image and according to different alarm time lengths of the human body obtained through experience or a large number of experimental tests.
It should be noted that the second deviation threshold may be set by an engineer based on experience or results of a number of experimental tests.
It should be understood that, in the embodiment of the present application, in addition to the detection of the abnormal sleep state by using a camera capable of clearly imaging as an image capturing device, and thus using key points of a human body or the like, the detection of the abnormal sleep state may also be realized by using an infrared imaging device such as an infrared camera.
At this time, the image captured by the image capturing device is an infrared imaging image. It should be understood that, since the temperature of the living body and the temperature of the peripheral object tend to have a large difference, in the embodiment of the present application, it may be determined whether the target object has the abnormal sleep state by identifying the human body infrared of the target object in the infrared imaging image and further according to the position deviation of the human body infrared of the target object in the current captured image and the historical captured image.
Similarly to the above, when the image captured by the image capturing device is an infrared imaging image, the historical captured image is a previous captured image of the current captured image, and then the position deviation of the human body infrared of the target object in the current captured image and the historical captured image can be calculated, so that the position deviation is compared with the preset third deviation threshold, and when the position deviation is greater than the preset third deviation threshold, the target object is determined to have the abnormal sleep state.
In the present embodiment, the positional deviation of the human infrared of the target object between the center positions in the currently captured image and the historically captured image may be, but is not limited to, taken as the positional deviation of the human infrared of the target object in the currently captured image and the historically captured image.
In addition, similar to the foregoing, in order to achieve detection of the dream state when the image captured by the image capturing device is an infrared imaging image, in a possible implementation manner of the embodiment of the present application, it may be determined that a sleep abnormality occurs when human infrared of the target object is detected in a previous captured image and the human infrared is located at an edge of the previous captured image, and human infrared of the target object is not detected in a current captured image.
In addition, similar to the foregoing, in order to achieve detection of the dream state when the image captured by the image capturing device is an infrared imaging image, in another possible implementation manner of the embodiment of the present application, when the human infrared of the target object is detected in the previous captured image, and the human infrared is located at an edge of the previous captured image, and a position deviation of the target object from the human infrared in the previous captured image is greater than a preset third deviation threshold, and when the human infrared of the target object is not detected in the current captured image, it is determined that the abnormal sleep state occurs.
It should be noted that in the practical application process, the sleeping space (such as a bedroom) of the user is often used in the non-sleeping time periods, and the detection of the abnormal sleeping state in the time periods is not only meaningless, but also can cause a series of misjudgments due to the using behavior of the sleeping space by the user. For this reason, in the above-described several possible embodiments when the image captured by the image capturing device is an infrared imaging image, a sleep time period may be set in advance so that only the image captured during the sleep time period is processed in accordance with the above-described possible embodiments.
As described above, for the detection of the abnormal condition that the target object is in the state of being kept still for a long time, in the embodiment of the present application, the detection result of the abnormal sleep state corresponding to the consecutive M frames of captured images before the current captured image may be acquired. And then determining that the abnormal sleep state occurs when the position deviation between the human body infrared of the target object in all the two adjacent shot images is smaller than a preset fourth deviation threshold value in the current shot image and the historical shot image.
It should be noted that the value of M may be determined by an engineer according to the extraction time interval of the current shot image and according to different alarm time intervals of the human body obtained through experience or a large number of experimental tests.
It should be noted that the fourth deviation threshold may be set by an engineer according to experience or a result of a large number of experimental tests.
It should be noted that, in the embodiment of the present application, when it is determined that the sleep abnormal state occurs, the abnormal flag may be performed. The labeling method includes but is not limited to: marking a current shooting picture or a last shooting picture of the current shooting picture for determining the abnormal sleep state in the video shot by the image shooting device, marking the time for determining the abnormal sleep state in the video shot by the image shooting device, and the like.
In addition, in the embodiment of the application, the user terminal can be combined, and after the abnormal sleep state is determined, corresponding abnormal information is sent to the user terminal. The exception information includes but is not limited to: the time when the abnormal sleep state is detected, and the type of the abnormal sleep state (such as suspected sleepwalking, no movement for a long time, etc.).
According to the sleep abnormity detection method provided by the embodiment of the application, whether the abnormal sleep state occurs in the target object during sleeping is actively detected according to the position change conditions of the target object in the current shot image and the historical shot image, so that the active identification of the abnormal sleep state occurring in the sleeping process can be effectively realized. Furthermore, operations such as marking of the occurrence time of the abnormal sleep state in monitoring and abnormal active reminding can be carried out based on the recognized abnormal sleep state, so that a user can know the problems possibly existing in the sleep process of the target object conveniently, and the user can be helped to master the abnormal state possibly existing in the sleep process conveniently without depending on a close-fitting sensor device.
Example two:
the present embodiment exemplifies a scheme of the present application by taking a specific abnormality detection process as an example on the basis of the first embodiment.
The image capture device is mounted in the orientation shown in figure 2,
and step 0, the image shooting device acquires the current shot image.
Step 1, a key point detection model based on deep learning is adopted to detect key points of a human body in a current shot image.
And 2, when no key point of the human body is detected, skipping to the step 0, and re-acquiring the current shot image. And when the key points of the human body are detected, the step 3 is carried out.
And 3, judging whether the overall deviation between the human body key points in the current shot image and the human body key points in the last shot image is larger than a preset first deviation threshold value or not. And if so, jumping to the step 4. If not, jumping to the step 0, and re-acquiring the current shot image.
It will be appreciated that there are often times when a person is sleeping that covers a quilt. The keypoint detection model can generally only detect parts that are not occluded. Under the condition that a person is asleep, the body usually keeps for a long time for a certain sleeping posture, so that key points of the human body obtained by each detection are basically kept unchanged. When abnormal sleep state occurs, people usually open the quilt and leak more key points, and correspondingly, the key points of the human body also move violently. The occurrence of a sleep exception condition may be identified at this time.
And 4, recording the starting time of the abnormal sleep state when the abnormal condition exists. And skipping to the step 0 to obtain the current shot image again.
Through the scheme of this embodiment, can realize the automatic recording to the abnormal state of sleeping, avoid wearing the inconvenience that the sensor brought. After the abnormal sleep state occurs, the monitoring of the corresponding event section can be searched more efficiently under the condition that the monitoring is not needed to be looked over all night, and whether the abnormal sleep state occurs or not and the occurrence frequency can be known more conveniently.
It should be understood that the abnormal sleep state in the present application does not only include unconscious behaviors such as kicking quilt, moving dream, etc., but also is considered as the abnormal sleep state in the present application for the autonomous behaviors such as actively getting out of bed to go to toilet, drinking water, etc. during the sleep process of the user. It should be understood that even if the user is actively getting out of bed to go to toilet, drinking water and other autonomous behaviors in the sleeping process, after the user is identified and recorded, the method has very important reference significance for the user, so that the user can conveniently know and count information such as the average time and duration of the autonomous behaviors such as actively getting out of bed to go to toilet, drinking water and the like, and the user can conveniently master the body health of the user.
Example three:
based on the same inventive concept, the embodiment of the application also provides a sleep abnormity detection device. Referring to fig. 3, fig. 3 shows a sleep abnormality detection apparatus 100 corresponding to the method of the first embodiment. It should be understood that the specific functions of the sleep abnormality detection apparatus 100 can be referred to the above description, and the detailed description is appropriately omitted here to avoid redundancy. The sleep abnormality detection apparatus 100 includes at least one software functional module that can be stored in a memory in the form of software or firmware or solidified in an operating system of the sleep abnormality detection apparatus 100. Specifically, the method comprises the following steps:
referring to fig. 3, the sleep abnormality detection apparatus 100 includes: an acquisition module 101 and a processing module 102. Wherein:
the acquiring module 101 is configured to acquire a current captured image;
the processing module 102 is configured to determine whether a sleep abnormality occurs in the target object according to a position change condition of the target object in the current captured image and the historical captured image.
In this embodiment of the present application, the obtaining module 101 is further configured to perform human body key point detection on a target object in the currently captured image; acquiring a detection result of the human body key point of the target object in a historical shooting image; the processing module 102 is specifically configured to determine whether the target object has an abnormal sleep state according to the position change condition of the key point of the human body of the target object in the current captured image and the historical captured image.
In a possible implementation manner of the embodiment of the present application, the historical captured image is a last captured image of the current captured image; the processing module 102 is specifically configured to determine that a sleep abnormality occurs when an overall deviation between a human body key point of a target object in the currently captured image and a human body key point of the target object detected in the last captured image is greater than a preset first deviation threshold.
In the possible embodiment, the processing module 102 is further specifically configured to calculate a position deviation between key points of a human body corresponding to the same human body part in the current captured image and the previous captured image; determining the integral deviation between the human body key point of the current shot image and the human body key point detected in the last shot image according to the position deviation corresponding to each human body key point; comparing the overall deviation with the preset first deviation threshold.
In yet another possible implementation manner of the embodiment of the present application, the historical captured image is a last captured image of the current captured image; the processing module 102 is specifically configured to determine that a sleep abnormal state occurs when a human body key point of the target object is detected in the last captured image, the human body key point is located at an edge of the last captured image, and the human body key point of the target object is not detected in the current captured image.
In the two possible implementation manners of the embodiment of the present application, the currently-captured image is an image captured within a preset sleep time period.
In yet another possible implementation manner of the embodiment of the present application, the historical captured image is a consecutive N frames of captured images before the current captured image; and N is a preset positive integer.
The processing module 102 is specifically configured to determine that a sleep abnormality occurs when the overall deviation between the human key points of the target object in all the two adjacent captured images in the current captured image and the historical captured image is smaller than a preset second deviation threshold.
In the embodiment of the application, the current shot image and the historical shot image are infrared imaging images; the processing module 102 is specifically configured to identify human infrared of the target object in the current captured image and the historical captured image; and determining whether the target object has an abnormal sleep state according to the position deviation of the human body infrared of the target object in the current shot image and the historical shot image.
It should be understood that, for the sake of brevity, the contents described in some embodiments are not repeated in this embodiment.
Example four:
the embodiment provides an electronic device, which can be seen in fig. 4, and includes a processor 401, a memory 402, and a communication module 403. Wherein:
the communication module 403 is configured to be in communication connection with the image capturing device to acquire an image captured by the image capturing device, and deliver the acquired image to the processor 401 for processing.
The processor 401 is configured to execute one or more programs stored in the memory 402 to implement the sleep anomaly detection method in the first embodiment or the second embodiment.
The image capturing apparatus in the embodiment of the present application may be an apparatus such as a camera that can implement an image capturing function.
It should be further noted that, in the embodiment of the present application, the image capturing apparatus may be integrated in an electronic device.
In the embodiment of the present application, when the image capturing apparatus is integrated in an electronic device, the communication module 403 may be implemented by a data bus. When the image capturing apparatus is not integrated in the electronic device, the communication module 403 may be implemented by a wireless communication module such as bluetooth or WiFi, or may be implemented by a wired communication module such as a USB interface.
It will be appreciated that the configuration shown in fig. 4 is merely illustrative and that the electronic device may also include more or fewer components than shown in fig. 4, or have a different configuration than shown in fig. 4, for example, may also have input/output interfaces, etc.
The present embodiment further provides a readable storage medium, such as a floppy disk, an optical disk, a hard disk, a flash Memory, a usb (Secure Digital Card), an MMC (Multimedia Card), etc., in which one or more programs for implementing the above steps are stored, and the one or more programs can be executed by one or more processors to implement the sleep abnormality detection method in the first embodiment or the second embodiment. And will not be described in detail herein.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
In addition, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
Furthermore, the functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
In this context, a plurality means two or more.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (11)
1. A sleep abnormality detection method characterized by comprising:
acquiring a current shot image;
and determining whether the target object has a sleep abnormal state according to the position change condition of the target object in the current shot image and the historical shot image.
2. The sleep abnormality detection method as set forth in claim 1, characterized in that the method further includes:
detecting key points of a human body of a target object in the current shot image;
acquiring a detection result of the human body key point of the target object in a historical shooting image;
the determining whether the target object has the abnormal sleep state according to the position change condition of the target object in the current shot image and the historical shot image comprises the following steps:
and determining whether the target object has an abnormal sleep state according to the position change condition of the human key points of the target object in the current shot image and the historical shot image.
3. The sleep abnormality detection method according to claim 2, characterized in that the history photographed image is a photographed image immediately previous to the current photographed image;
determining whether the target object has a sleep abnormal state according to the position change condition of the human key point of the target object in the current shot image and the historical shot image, wherein:
and when the integral deviation between the human key point of the target object in the current shot image and the human key point of the target object detected in the last shot image is greater than a preset first deviation threshold value, determining that a sleep abnormal state occurs.
4. The sleep abnormality detection method according to claim 3, wherein determining whether a sleep abnormality state occurs in the target subject based on a position change situation of a human body key point of the target subject in the current captured image and the history captured image, further comprises:
calculating the position deviation between the key points of the human body corresponding to the same human body part in the current shot image and the last shot image;
determining the integral deviation between the human body key point of the current shot image and the human body key point detected in the last shot image according to the position deviation corresponding to each human body key point;
comparing the overall deviation with the preset first deviation threshold.
5. The sleep abnormality detection method according to claim 2, characterized in that the history photographed image is a photographed image immediately previous to the current photographed image;
determining whether the target object has a sleep abnormal state according to the position change condition of the human key point of the target object in the current shot image and the historical shot image, wherein the determining step comprises the following steps:
and when the human body key point of the target object is detected in the last shot image and is positioned at the edge of the last shot image, determining that the abnormal sleep state occurs when the human body key point of the target object is not detected in the current shot image.
6. The sleep abnormality detection method according to any one of claims 1 to 5, characterized in that the currently shot image is an image shot within a preset sleep time period.
7. The sleep abnormality detection method according to claim 2, characterized in that the history photographed image is a consecutive N-frame photographed image before the current photographed image; n is a preset positive integer;
determining whether the target object has a sleep abnormal state according to the position change condition of the human key point of the target object in the current shot image and the historical shot image, wherein the determining step comprises the following steps:
and determining that the abnormal sleep state occurs when the overall deviation between the human key points of the target object in all the two adjacent shot images in the current shot image and the historical shot image is less than a preset second deviation threshold.
8. The sleep abnormality detection method according to claim 1, characterized in that the currently shot image and the historically shot image are infrared imaging images;
the determining whether the target object has the abnormal sleep state according to the position change condition of the target object in the current shot image and the historical shot image comprises the following steps:
identifying human body infrared of the target object in the current shot image and the historical shot image;
and determining whether the target object has an abnormal sleep state according to the position deviation of the human body infrared of the target object in the current shot image and the historical shot image.
9. A sleep abnormality detection apparatus characterized by comprising: the device comprises an acquisition module and a processing module;
the acquisition module is used for acquiring a current shot image;
the processing module is used for determining whether the target object has a sleep abnormal state according to the position change conditions of the target object in the current shot image and the historical shot image.
10. An electronic device, comprising: a communication module, a processor and a memory;
the communication module is used for being in communication connection with an image shooting device so as to acquire an image acquired by the image shooting device and process the image by the processor;
the processor is configured to execute one or more programs stored in the memory to implement the method of any of claims 1 to 8.
11. A readable storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011341969.7A CN112535459B (en) | 2020-11-24 | 2020-11-24 | Sleep abnormality detection method, sleep abnormality detection device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011341969.7A CN112535459B (en) | 2020-11-24 | 2020-11-24 | Sleep abnormality detection method, sleep abnormality detection device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112535459A true CN112535459A (en) | 2021-03-23 |
| CN112535459B CN112535459B (en) | 2023-08-18 |
Family
ID=75016351
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011341969.7A Active CN112535459B (en) | 2020-11-24 | 2020-11-24 | Sleep abnormality detection method, sleep abnormality detection device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112535459B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113221802A (en) * | 2021-05-24 | 2021-08-06 | 新疆爱华盈通信息技术有限公司 | Quilt kicking identification method and device and electronic equipment |
| CN116593151A (en) * | 2023-07-17 | 2023-08-15 | 创新奇智(青岛)科技有限公司 | Dental socket chest expander testing method and device, electronic equipment and readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2003299636A (en) * | 2002-04-12 | 2003-10-21 | Matsushita Electric Works Ltd | Detector for body motion upon sleeping |
| CN101937604A (en) * | 2010-09-08 | 2011-01-05 | 无锡中星微电子有限公司 | Sleep monitoring system based on human body detection and method thereof |
| CN107734617A (en) * | 2017-10-31 | 2018-02-23 | 广东欧珀移动通信有限公司 | Closing application program method, apparatus, storage medium and electronic equipment |
| JP2018164615A (en) * | 2017-03-28 | 2018-10-25 | 国立大学法人大阪大学 | Sleep depth determination system, sleep depth determination device and sleep depth determination method |
| US20200323485A1 (en) * | 2017-04-28 | 2020-10-15 | Optim Corporation | Sleep anomaly notification system, sleep anomaly notification method, and program |
-
2020
- 2020-11-24 CN CN202011341969.7A patent/CN112535459B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2003299636A (en) * | 2002-04-12 | 2003-10-21 | Matsushita Electric Works Ltd | Detector for body motion upon sleeping |
| CN101937604A (en) * | 2010-09-08 | 2011-01-05 | 无锡中星微电子有限公司 | Sleep monitoring system based on human body detection and method thereof |
| JP2018164615A (en) * | 2017-03-28 | 2018-10-25 | 国立大学法人大阪大学 | Sleep depth determination system, sleep depth determination device and sleep depth determination method |
| US20200323485A1 (en) * | 2017-04-28 | 2020-10-15 | Optim Corporation | Sleep anomaly notification system, sleep anomaly notification method, and program |
| CN107734617A (en) * | 2017-10-31 | 2018-02-23 | 广东欧珀移动通信有限公司 | Closing application program method, apparatus, storage medium and electronic equipment |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113221802A (en) * | 2021-05-24 | 2021-08-06 | 新疆爱华盈通信息技术有限公司 | Quilt kicking identification method and device and electronic equipment |
| CN116593151A (en) * | 2023-07-17 | 2023-08-15 | 创新奇智(青岛)科技有限公司 | Dental socket chest expander testing method and device, electronic equipment and readable storage medium |
| CN116593151B (en) * | 2023-07-17 | 2023-09-12 | 创新奇智(青岛)科技有限公司 | Dental socket chest expander testing method and device, electronic equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112535459B (en) | 2023-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9396400B1 (en) | Computer-vision based security system using a depth camera | |
| JP5143212B2 (en) | Method and apparatus for modeling behavior using probability distribution functions | |
| US8787617B2 (en) | Imaging apparatus, mobile body detecting method, mobile body detecting circuit and program | |
| CN112535459B (en) | Sleep abnormality detection method, sleep abnormality detection device, electronic equipment and readable storage medium | |
| JP6199791B2 (en) | Pet health examination apparatus, pet health examination method and program | |
| CN105554439A (en) | Modification of at least one parameter used by a video processing | |
| Khawandi et al. | Implementation of a monitoring system for fall detection in elderly healthcare | |
| CN112712020B (en) | Sleep monitoring method, device and system | |
| US10657783B2 (en) | Video surveillance method based on object detection and system thereof | |
| CN112001230A (en) | Sleeping behavior monitoring method and device, computer equipment and readable storage medium | |
| CN112949417A (en) | Tumble behavior identification method, equipment and system | |
| CN110781808A (en) | Fall detection method, device, equipment and storage medium | |
| US20190303656A1 (en) | Multi-level state detecting system and method | |
| CN110211316B (en) | Monitoring early warning method and system | |
| CN115346060A (en) | Picture abnormity identification method and device, electronic equipment and storage medium | |
| JP5805511B2 (en) | Image monitoring device | |
| JP2020194227A (en) | Face hiding determination device, face hiding determination method, face hiding determination program, and occupant monitoring system | |
| CN112381843A (en) | Industrial automatic scanning monitoring method, system, terminal equipment and storage medium | |
| KR20110102321A (en) | Image motion detection method and apparatus | |
| KR20100071222A (en) | Video saving method with variable frame rate according to the amount of human object motion of video and video authentication method in surveillance camera system | |
| JP2020149132A (en) | Image processing apparatus and image processing program | |
| CN111225178A (en) | Video monitoring method and system based on object detection | |
| KR20200028550A (en) | Device and method for determining an emergency situation through object detection | |
| CN117992128A (en) | Awakening method and device | |
| KR102351332B1 (en) | System and method for monitoring respiration during sleep |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |