CN104083160A - Sleep state monitoring method and device based on machine vision - Google Patents
Sleep state monitoring method and device based on machine vision Download PDFInfo
- Publication number
- CN104083160A CN104083160A CN201410310294.8A CN201410310294A CN104083160A CN 104083160 A CN104083160 A CN 104083160A CN 201410310294 A CN201410310294 A CN 201410310294A CN 104083160 A CN104083160 A CN 104083160A
- Authority
- CN
- China
- Prior art keywords
- testee
- video image
- frequency
- heart rate
- sleep state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Measuring And Recording Apparatus For Diagnosis (AREA)
- Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)
Abstract
The invention discloses a sleep state monitoring method and device based on machine vision. The method includes the steps of respectively detecting vision behavior characteristics and physiological parameter characteristics for representing tested persons through eye video images, lip video image and cheek video images, and carrying out fuse judgment on the parameters to obtain sleep state detection results of the tested persons. The device comprises an embedded system which is used for extracting the vision characteristics of the opening-closing frequency of the mouths, the positions of the heads, the maintaining time of the positions and the position change frequency of the tested persons and the physiological parameter characteristics, such as the pulse rates, of the tested persons, carrying out fusing processing and judgment alarm giving on the characteristic information, outputting alarm signals to an alarm apparatus when anomalies of the sleeping states of the tested persons are found, and meanwhile outputting abnormal signals to parents or related guardians. By means of the sleep state monitoring method and device, behaviors of the tested persons are not influenced, more information including the physiological parameters of the tested persons is obtained, and the reliability of monitoring on the sleeping states of the tested persons is greatly improved.
Description
Technical field
The present invention relates to field of machine vision, relate in particular to a kind of sleep state monitoring method and device based on machine vision.
Background technology
The sleep sufficient and degree of depth is able-bodied basic guarantee, but causes the factor that sleep quality is not high a lot.In order to diagnose the reason that sleep quality is not high, Many researchers has proposed various sleep monitoring devices.Then, there is many deficiencies in prior art: as the Chinese invention patent (publication number: CN1584529) " non-electrode sleeping and respiration disorder event sensing device " of Yu Mengsun proposition, utilize that the monitoring of air bag in mattress is aroused in interest, body moving and respiratory variations, reliability is low; The Chinese invention patent (publication number: CN1803089) sleep disorder measuring device and the method thereof of non-perceptibility " unconstrained, " that Tang Yiping proposes, pick up and microprocessor etc. are placed in medicated pillow, detected person is sent after falling asleep to the sound of snoring and record and judge, same poor reliability; The Chinese invention patent (publication number: CN1923132) " detecting method, device and the application thereof of respiration force in sleep " that Yang Fusheng proposes, by pressure fine motion sensing device obtain that monitored person's Sleeping Center is dirtyly beated, breathing, the moving fine motion signal of body, still there is the problem of poor reliability; The Chinese invention patent (publication number: CN101044981) " apparatus for observing sleep breathing status " that Zhou Changan proposes, is also detected person to be sent after falling asleep to the sound of snoring record and judge, same poor reliability; Open the Chinese invention patent (publication number: CN102188233A) " the apneic device of a kind of monitoring sleep " of inferior proposition, utilize the measurement human body breast abdominal respiration moving wave shape of amorphous wire giant stress impedance effect development, both need to wrap the health at infant, and only can monitor again the breathing of infant; The Chinese invention patent (publication number: CN102274022) " a kind of sleep state monitoring method based on EEG signals " that Meng Chun proposes, gathering brain electricity will affect the sleep of this human observer, and is difficult to life-time service; The Chinese invention patent (publication number: CN102415879A) " a kind of sleep monitoring device based on piezoelectric film sensor " that Du Yan proposes, utilize piezoelectric film sensor to measure that aerocyst pressure monitoring in mattress is aroused in interest, body moving and respiratory variations, reliability is low; Open the raw Chinese invention patent (publication number: CN102499656A) " a kind of wristlet type sleep monitoring device " proposing, pulse signal and the temperature signal of sensor acquisition sleeper by wrist strap, reliability is low and easily affect wearer's sleep; The Chinese invention patent (publication number: CN102648845A) " wireless automatic monitoring and early warning system are jumped, breathed to a kind of Sleeping Center " that Shen Jingpeng proposes, need to lay electrode to testee and measure electrocardio and breath signal, very large on testee's sleep impact.The Chinese invention patent (publication number: CN102835951A) " a kind of mobile wrist strap equipment and method of work thereof " that Liu Xin proposes, monitor and record user mobilizing exercises every day data and sleep information by the acceleration transducer in wrist strap and gyroscope, infrared inspection circuit and blood oxygen check circuit, its reliability is low, very large on testee's sleep impact.Jiao rises the Chinese invention patent (publication number: CN102973273A) " a kind of sleep-respiratory function monitoring system based on infrared radiation detection " of proposition, have advantages of non-cpntact measurement, but only can measure breath signal, and reliability is not high enough.M. raise the Chinese invention patent (publication number: CN103167849A) " diagnosis of sleep apnea and relevant dysfunction and/or the method for the treatment of and device " of this proposition, the inflatable utensil (vest) of the outer surface by being applied to chest and/or abdominal part is measured testee's respiration parameter, testee need to dress special vest, sleep impact on testee is very large, and only can measure breath signal, and reliability is not high enough.The Chinese invention patent (publication number: CN1751652) " a kind of wrist aroused in interest and breathing cycle detection method and device " that Lv Xiaodong proposes, is affected greatly by the action of detected person in sleeping.The Chinese invention patent (publication number: CN1860987) " monitoring and interfering system and the processing method thereof of sleep and dream " that Shen Zheng proposes, need to lay electrode to testee and measure EEG signals, very large on testee's sleep impact.The Chinese invention patent (publication number: CN103263260A) " using physio-parameter detection system and the Depth of sleep monitoring system of comb filter " that Song Jun etc. propose, essence is by air mattress transmission of signal, will have a strong impact on the reliability of signal detection in monitored person's sleep attitude.The Chinese invention patent (publication number: CN102648845A) " monitoring of sleeping respiration information abdomen formula and the early warning system of wireless transmission " that on woods, port proposes, adopt the contact type measurements such as abdomen formula sensing zone, respiration pickup, very large on testee's sleep impact.The Chinese invention patent (publication number: CN103006223A) " a kind of domestic noncontact sleep monitoring device and method " that Lu Guohua etc. propose, adopts microwave technology, among testee's radiation in microwave for a long time.The Chinese invention patent (publication number: CN103271741A) " a kind of sleeping posture monitor " that the lunar calendar proposes, only can monitor testee's sleeping position.The Chinese invention patent (publication number: CN103445777A) " monitoring method of sleep and fatigue monitoring class wrist-watch device and normalization dingus " that Hu Wendong etc. propose, adopt skin resistance sensor, temperature sensor, atmosphere pressure sensor, acceleration transducer and only can be placed on wrist portion, being difficult to measure the information of assessment sleep quality.
Generally speaking, existing monitoring testee's technology, or can not monitor testee's the important physiological parameter such as heart rate, breathing, or can only have the sensor of contact to measure, or the technology that adopts microwave to have radiation realizes.
Summary of the invention
The invention provides a kind of sleep state monitoring method and device based on machine vision, the present invention obtains the physiological parameter that comprises testee in interior more information, has increased substantially the reliability of the sleep state monitoring to testee, described below:
A sleep state monitoring method based on machine vision, said method comprising the steps of:
Obtain the video image of testee face by being arranged on two of the left and right tops photographic head at testee's medicated pillow position;
Video image is cleaned, obtain the video image after cleaning;
Video image after cleaning is cut apart, obtained eye video image, lip video image dough-making powder buccal video image;
By eye video image, lip video image dough-making powder buccal video image, detect respectively the visual behaviour feature and the physiological parameter feature that characterize testee:
Above-mentioned parameter is merged to judgement, draw testee's sleep state testing result.
Described by eye video image, lip video image dough-making powder buccal video image, detect respectively sign testee's visual behaviour feature and the step of physiological parameter feature and be specially:
1) frequency of the face opening and closing by lip video images detection testee, by the visual signature of eye video images detection testee's head position and the time maintaining thereof, transposition frequency;
2) obtain testee's heart rate, frequency of respiration and heart rate variability sequence thereof by buccal, opposite video image analysis.
In the time that two photographic head are all infrared camera, the described step that obtains testee's heart rate, frequency of respiration and heart rate variability sequence thereof by buccal, opposite video image analysis is specially:
(a) pixel of the R passage to cheek portion region is averaged, and has N data in R passage;
(b) to data in R passage be normalized successively, albefaction processing and independent component analysis;
(c) data after each independent component analysis are carried out to Fourier variation, obtain power spectrum, obtain the corresponding frequency of power spectrum maximum amplitude, i.e. testee's heart rate, separately obtain power spectrum time corresponding frequency of maximum amplitude, i.e. testee's frequency of respiration;
(d) by testee's heart rate and frequency of respiration calculating testee's heart rate variability value.
In the time that two photographic head are all colour imagery shot, the described step that obtains testee's heart rate, frequency of respiration and heart rate variability sequence thereof by buccal, opposite video image analysis is specially:
(a) the RGB passage in separation cheek portion region, averages to the pixel of each passage;
(b) data are preserved into 3 passages, each passage N data;
(c) to each channel data be normalized successively, albefaction processing and independent component analysis;
(d) data after each passage independent component analysis are carried out to Fourier variation, obtain power spectrum, obtain the corresponding frequency of power spectrum maximum amplitude, i.e. testee's heart rate, separately obtain power spectrum time corresponding frequency of maximum amplitude, i.e. testee's frequency of respiration;
(e) by testee's heart rate and frequency of respiration calculating testee's the heart rate variability value of single channel, and then by the calculating of multiple passages is obtained to heart rate variability sequence.
In the time that two photographic head are respectively infrared camera and colour imagery shot, the described step that obtains testee's heart rate, frequency of respiration and heart rate variability sequence thereof by buccal, opposite video image analysis is specially:
(a) buccal portion video image colour imagery shot being got, the RGB passage in separation cheek portion region, averages to the pixel of each passage;
(b) pixel of the R passage in the cheek portion region in buccal portion video image infrared camera being got is averaged;
(c) to data in four passages be normalized successively, albefaction processing and independent component analysis;
(d) data are preserved into 4 passages, each passage N data;
(e) to each channel data be normalized successively, albefaction processing and independent component analysis;
(f) data after each passage independent component analysis are carried out to Fourier variation, obtain power spectrum, obtain the corresponding frequency of power spectrum maximum amplitude, i.e. testee's heart rate, separately obtain power spectrum time corresponding frequency of maximum amplitude, i.e. testee's frequency of respiration;
(g) by testee's heart rate and frequency of respiration calculating testee's the heart rate variability value of single channel, and then by the calculating of multiple passages is obtained to heart rate variability sequence;
A sleep state monitoring device based on machine vision, described sleep state monitoring device comprises: 2 photographic head, encoder, display, embedded system, alarm,
Described photographic head is for shooting with video-corder the image of testee's head and face portion and be input to described encoder in real time;
Described encoder is sent into described embedded system for the image information of photographic head is encoded into data image signal;
Described embedded system is for extracting the visual signature of frequency of testee's frequency, the head position of face opening and closing and the time maintaining thereof, transposition, and the testee's such as pulse frequency physiological parameter feature, these characteristic informations are carried out to fusion treatment and judgement is reported to the police; In the time that testee's sleep state notes abnormalities, output alarm signal is to described alarm, and output abnormality signal is to the head of a family or relevant monitoring personnel simultaneously;
Described alarm is used for sending alarm sound;
Described display is for showing testee's physiological parameter.
Described photographic head is: infrared camera and/or colour imagery shot.
In order to get the video image of degree of precision, described colour imagery shot is: ANC Duo HD1080P, described infrared camera is Samsung SCO-2080RP.
Further, in order to improve processing speed, described embedded system is: the one in i.MX6 processor, OMAP processor and TigerSHARC processor.
The beneficial effect of technical scheme provided by the invention is: the present invention provides a kind of safety in order to ensure testee's safety, real-time monitoring method and device, the present invention had both detected testee's physiological parameter feature, detect again the visual signature of testee's head and face portion, and only adopt photographic head to obtain above-mentioned information, both the non-cpntact measurement behavior that does not affect testee possessing and the burden of wearing that increases testee had been retained, obtain again the physiological parameter that comprises testee in interior more information, increase substantially the reliability of the sleep state monitoring to testee.
Brief description of the drawings
Fig. 1 is the flow chart of a kind of sleep state monitoring method based on machine vision provided by the invention;
Fig. 2 is the flow chart of processing the testees' such as extraction pulse frequency and breathing physiological parameter feature based on image provided by the invention;
Fig. 3 is another flow chart of a kind of sleep state monitoring method based on machine vision provided by the invention;
Fig. 4 is another flow chart of processing the testees' such as extraction pulse frequency and breathing physiological parameter feature based on image provided by the invention;
Fig. 5 is another flow chart of a kind of sleep state monitoring method based on machine vision provided by the invention;
Fig. 6 is another flow chart of processing the testees' such as extraction pulse frequency and breathing physiological parameter feature based on image provided by the invention;
Fig. 7 is the structure chart of a kind of sleep state monitoring device based on machine vision provided by the invention.
In accompanying drawing, the list of parts of each label representative is as follows:
1: photographic head; 2: encoder;
3: display; 4: embedded system;
5: alarm.
Wherein, photographic head 1 is respectively the combination of infrared camera, colour imagery shot and infrared camera and colour imagery shot in three device embodiment.
Detailed description of the invention
For making the object, technical solutions and advantages of the present invention clearer, below embodiment of the present invention is described further in detail.
Embodiment 1
A sleep state monitoring method based on machine vision, adopts 2 infrared cameras while obtaining video image, referring to Fig. 1 and Fig. 2, the method comprises the following steps:
101: the video image that obtains testee face by being arranged on two of the left and right tops infrared camera at testee's medicated pillow position;
Wherein, in order to improve the precision of the video image getting, infrared camera used in this step adopts Samsung SCO-2080RP, when specific implementation, can also select according to the needs in practical application, and the embodiment of the present invention is not restricted this.
102: video image is cleaned, obtain the video image after cleaning;
Wherein, this cleaning method, according to the difference of the video image collecting, can look like to come the processing methods such as optimization data collection with image cutting-out screening, image gray processing and histogram equalization, image normalization, video frame matching, image albefaction processing, removal singular graph.For example: can and extract the pre-treatment step such as image characteristic point normalization and can remove the impact that environmental light intensity changes by histogram equalization.Being operating as in detail of this step is conventionally known to one of skill in the art, and the embodiment of the present invention does not repeat this.
103: the video image after cleaning is cut apart, obtained eye video image, lip video image dough-making powder buccal video image;
Wherein, the method that video image after cleaning is cut apart is: according to the video image of the people face detecting, " three five, front yards " criterion according to human face structure is divided the region of face, is partitioned into the video image of eye video image, lip video image dough-making powder buccal with this." three five, front yards " criterion is conventionally known to one of skill in the art, and the embodiment of the present invention does not repeat this.
104: by eye video image, lip video image dough-making powder buccal video image, detect respectively the visual behaviour feature and the physiological parameter feature that characterize testee:
1) frequency of the face opening and closing by lip video images detection testee, by the visual signature of eye video images detection testee's head position and the time maintaining thereof, transposition frequency;
For example: if people's head position conversion frequency is too high, infer according to some experiences and data standard, can judge to a certain extent as people and doze off.
2) obtain testee's heart rate, frequency of respiration and heart rate variability sequence (being shown in Fig. 2) thereof by buccal, opposite video image analysis.
This step 2) be specially:
(a) pixel of the R passage to cheek portion region is averaged, and has N data in R passage;
(b) to data in R passage be normalized successively, albefaction processing and independent component analysis;
Wherein, the normalization, albefaction processing and the independent component analysis that in the processing of step (b), adopt, be several technology known in those skilled in the art, and the embodiment of the present invention does not repeat this.
(c) data after each independent component analysis are carried out to Fourier variation, obtain power spectrum, obtain the corresponding frequency of power spectrum maximum amplitude, i.e. testee's heart rate, separately obtain power spectrum time corresponding frequency of maximum amplitude, i.e. testee's frequency of respiration;
(d) by testee's heart rate and frequency of respiration calculating testee's heart rate variability value.
Wherein, this step has adopted the Lomb periodogram principle in this area, and when specific implementation, the embodiment of the present invention does not repeat this.
105: above-mentioned parameter is merged to judgement, draw testee's sleep state testing result.
Wherein, the fusion described in this step judges and can be:
1) can directly draw testee's sleep detection result by the visual behaviour feature that characterizes testee; Or,
2) can directly draw testee's sleep state testing result by heart rate variability sequence; Or,
3) show by characterizing testee's visual behaviour feature and the weighted sum of heart rate variability sequence whether testee enters dormant testing result.
Be about to sign testee's visual behaviour feature and the default threshold value of weighted sum of heart rate variability sequence and compare, if be greater than threshold value, directly judgement show that testee enters dormant testing result.Wherein, weight coefficient and threshold value are set according to the needs in practical application, and the embodiment of the present invention does not limit this.
Embodiment 2
A sleep state monitoring method based on machine vision, adopts 2 colour imagery shots while obtaining video image, referring to Fig. 3 and Fig. 4, the method comprises the following steps:
201: the video image that obtains testee face by being arranged on two of the left and right tops colour imagery shot at testee's medicated pillow position;
Wherein, in order to improve the precision of the video image getting, colour shooting used in this step adopts ANC Duo, when specific implementation, can also select according to the needs in practical application, and the embodiment of the present invention is not restricted this.
202: video image is cleaned, obtain the video image after cleaning;
203: the video image after cleaning is cut apart, obtained eye video image, lip video image dough-making powder buccal video image;
Wherein, the operating procedure of step 202 and step 203 is identical with embodiment 1, and the embodiment of the present invention does not repeat this.
204: by eye video image, lip video image dough-making powder buccal video image, detect respectively the visual behaviour feature and the physiological parameter feature that characterize testee:
1) frequency of the face opening and closing by lip video images detection testee, by the visual signature of eye video images detection testee's head position and the time maintaining thereof, transposition frequency;
For example: if people's head position conversion frequency is too high, infer according to some experiences and data standard, can judge to a certain extent as people and doze off.
2) obtain heart rate, frequency of respiration and this step 2 of heart rate variability sequence (referring to Fig. 4) thereof of testee by buccal, opposite video image analysis) be specially:
(a) the RGB passage in separation cheek portion region, averages to the pixel of each passage;
(b) data are preserved into 3 passages, each passage N data;
(c) to each channel data be normalized successively, albefaction processing and independent component analysis;
Wherein, eliminate three dependencys between passage by albefaction processing, prevented interchannel interference.The normalization, albefaction processing and the independent component analysis that in the processing of step (c), adopt, be several technology known in those skilled in the art, and the embodiment of the present invention does not repeat this.
(d) data after each passage independent component analysis are carried out to Fourier variation, obtain power spectrum, obtain the corresponding frequency of power spectrum maximum amplitude, i.e. testee's heart rate, separately obtain power spectrum time corresponding frequency of maximum amplitude, i.e. testee's frequency of respiration;
(e) by testee's heart rate and frequency of respiration calculating testee's the heart rate variability value of single channel, and then by the calculating of multiple passages is obtained to heart rate variability sequence.
Wherein, this step has adopted the Lomb periodogram principle in this area, and when specific implementation, the embodiment of the present invention does not repeat this.
205: above-mentioned parameter is merged to judgement, draw testee's sleep state testing result.
Wherein, step 205 operating procedure identical with embodiment 1, the embodiment of the present invention does not repeat this.
Embodiment 3
A sleep state monitoring method based on machine vision, adopts 1 infrared camera and 1 colour imagery shot while obtaining video image, referring to Fig. 5 and Fig. 6, the method comprises the following steps:
301: the video image that obtains testee face by being arranged on the infrared camera of left and right top at testee's medicated pillow position and colour imagery shot;
Wherein, in order to improve the precision of the video image getting, infrared camera used in this step adopts Samsung SCO-2080RP, colored shooting adopts ANC Duo, when specific implementation, can also select according to the needs in practical application, the embodiment of the present invention is not restricted this.In this step, get one and also had the three-channel video image of RGB, and only comprised the video image of R passage.
302: 2 video images are cleaned respectively, obtain 2 video images after cleaning;
303: 2 video images after cleaning are cut apart, obtained respectively eye video image, lip video image dough-making powder buccal video image;
Wherein, the operating procedure of step 302 and step 303 is identical with embodiment 1, and the embodiment of the present invention does not repeat this.
304: by eye video image, lip video image dough-making powder buccal video image, detect respectively the visual behaviour feature and the physiological parameter feature that characterize testee:
1) frequency of the face opening and closing by lip video images detection testee, by the visual signature of eye video images detection testee's head position and the time maintaining thereof, transposition frequency;
For example: if people's head position conversion frequency is too high, infer according to some experiences and data standard, can judge to a certain extent as people and doze off.
2) obtain testee's heart rate, frequency of respiration and heart rate variability sequence thereof by buccal, opposite video image analysis,
Referring to Fig. 6, this step 2) be specially:
(a) buccal portion video image colour imagery shot being got, the RGB passage in separation cheek portion region, averages to the pixel of each passage;
(b) pixel of the R passage in the cheek portion region in buccal portion video image infrared camera being got is averaged;
(c) to four passages (2 R passages, 1 G passage and 1 B passage), interior data are normalized successively, albefaction processing and independent component analysis;
(d) data are preserved into 4 passages, each passage N data;
(e) to each channel data be normalized successively, albefaction processing and independent component analysis;
Wherein, eliminate four dependencys between passage by albefaction processing, prevented interchannel interference.The normalization, albefaction processing and the independent component analysis that in the processing of step (e), adopt, be several technology known in those skilled in the art, and the embodiment of the present invention does not repeat this.
(f) data after each passage independent component analysis are carried out to Fourier variation, obtain power spectrum, obtain the corresponding frequency of power spectrum maximum amplitude, i.e. testee's heart rate, separately obtain power spectrum time corresponding frequency of maximum amplitude, i.e. testee's frequency of respiration;
(g) by testee's heart rate and frequency of respiration calculating testee's the heart rate variability value of single channel, and then by the calculating of multiple passages is obtained to heart rate variability sequence.
Wherein, this step has adopted the Lomb periodogram principle in this area, and when specific implementation, the embodiment of the present invention does not repeat this.
305: above-mentioned parameter is merged to judgement, draw testee's sleep state testing result.
Wherein, step 305 operating procedure identical with embodiment 1, the embodiment of the present invention does not repeat this.
Embodiment 4:
A sleep state monitoring device based on machine vision, and method in embodiment 1 institute is corresponding, referring to Fig. 7, this monitoring device comprises: 2 infrared cameras 1, encoder 2, display 3, embedded system 4, alarm 5,
Shoot with video-corder in real time the image of testee's head and face portion and be input to encoder 2 in 2 infrared cameras 1 of the positive front upper place of testee, encoder 2 is encoded into data image signal by the image information of infrared camera 1 and sends into embedded system 4, in embedded system 4, extract the visual signature of the frequency of testee's frequency, the head position of face opening and closing and the time maintaining thereof, transposition, and the testee's such as pulse frequency physiological parameter feature, these characteristic informations are carried out to fusion treatment and judgement is reported to the police; In the time that testee's sleep state notes abnormalities, alarm 5 sends alarm sound, and 4 while of embedded system, output abnormality signal was to the head of a family or relevant monitoring personnel.
Wherein, this infrared camera 1 adopts Samsung SCO-2080RP; The OMAP that embedded system adopts TI company to produce, display 3 can be LCD display etc.
Embodiment 5
A sleep state monitoring device based on machine vision, and method in embodiment 2 institute is corresponding, referring to Fig. 7, this monitoring device comprises: 2 colour imagery shots 1, encoder 2, display 3, embedded system 4, alarm 5,
Shoot with video-corder in real time the image of testee's head and face portion and be input to encoder 2 in 2 colour imagery shots 1 of the positive front upper place of testee, encoder 2 is encoded into data image signal by the image information of colour imagery shot 1 and sends into embedded system 4, in embedded system 4, extract the visual signature of the frequency of testee's frequency, the head position of face opening and closing and the time maintaining thereof, transposition, and the testee's such as pulse frequency physiological parameter feature, these characteristic informations are carried out to fusion treatment and judgement is reported to the police; In the time that testee's sleep state notes abnormalities, alarm 5 sends alarm sound, and 4 while of embedded system, output abnormality signal was to the head of a family or relevant monitoring personnel.
Wherein, this colour imagery shot 1 adopts ANC Duo HD1080P, the OMAP that embedded system adopts TI company to produce, and display 3 can be LCD display etc.
Embodiment 6:
A sleep state monitoring device based on machine vision, and method in embodiment 3 institute is corresponding, referring to Fig. 7, this monitoring device comprises: infrared camera and colour imagery shot 1, encoder 2, display 3, embedded system 4, alarm 5,
Infrared camera in the positive front upper place of testee and colour imagery shot 1 are shot with video-corder in real time respectively the image of testee's head and face portion and are input to encoder 2, encoder 2 is encoded into data image signal by the image information of infrared camera and colour imagery shot 1 and sends into embedded system 4, in embedded system 4, extract the visual signature of the frequency of testee's frequency, the head position of face opening and closing and the time maintaining thereof, transposition, and the testee's such as pulse frequency physiological parameter feature, these characteristic informations are carried out to fusion treatment and judgement is reported to the police; In the time that testee's sleep state notes abnormalities, alarm 5 sends alarm sound, and 4 while of embedded system, output abnormality signal was to the head of a family or relevant monitoring personnel.
Wherein, this colour imagery shot adopts ANC Duo HD1080P, and infrared camera adopts Samsung SCO-2080RP; The TigerSHARC that embedded system adopts ADI company to produce, display 3 can be LCD display etc.
The embodiment of the present invention to the model of each device except do specified otherwise, the model of other devices does not limit, and all can as long as can complete the device of above-mentioned functions.
It will be appreciated by those skilled in the art that accompanying drawing is the schematic diagram of a preferred embodiment, the invention described above embodiment sequence number, just to describing, does not represent the quality of embodiment.
The foregoing is only preferred embodiment of the present invention, in order to limit the present invention, within the spirit and principles in the present invention not all, any amendment of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.
Claims (9)
1. the sleep state monitoring method based on machine vision, is characterized in that, said method comprising the steps of:
Obtain the video image of testee face by being arranged on two of the left and right tops photographic head at testee's medicated pillow position;
Video image is cleaned, obtain the video image after cleaning;
Video image after cleaning is cut apart, obtained eye video image, lip video image dough-making powder buccal video image;
By eye video image, lip video image dough-making powder buccal video image, detect respectively the visual behaviour feature and the physiological parameter feature that characterize testee:
Above-mentioned parameter is merged to judgement, draw testee's sleep state testing result.
2. a kind of sleep state monitoring method based on machine vision according to claim 1, it is characterized in that, described by eye video image, lip video image dough-making powder buccal video image, detect respectively sign testee's visual behaviour feature and the step of physiological parameter feature and be specially:
1) frequency of the face opening and closing by lip video images detection testee, by the visual signature of eye video images detection testee's head position and the time maintaining thereof, transposition frequency;
2) obtain testee's heart rate, frequency of respiration and heart rate variability sequence thereof by buccal, opposite video image analysis.
3. a kind of sleep state monitoring method based on machine vision according to claim 2, it is characterized in that, in the time that two photographic head are all infrared camera, the described step that obtains testee's heart rate, frequency of respiration and heart rate variability sequence thereof by buccal, opposite video image analysis is specially:
(a) pixel of the R passage to cheek portion region is averaged, and has N data in R passage;
(b) to data in R passage be normalized successively, albefaction processing and independent component analysis;
(c) data after each independent component analysis are carried out to Fourier variation, obtain power spectrum, obtain the corresponding frequency of power spectrum maximum amplitude, i.e. testee's heart rate, separately obtain power spectrum time corresponding frequency of maximum amplitude, i.e. testee's frequency of respiration;
(d) by testee's heart rate and frequency of respiration calculating testee's heart rate variability value.
4. a kind of sleep state monitoring method based on machine vision according to claim 2, it is characterized in that, in the time that two photographic head are all colour imagery shot, the described step that obtains testee's heart rate, frequency of respiration and heart rate variability sequence thereof by buccal, opposite video image analysis is specially:
(a) the RGB passage in separation cheek portion region, averages to the pixel of each passage;
(b) data are preserved into 3 passages, each passage N data;
(c) to each channel data be normalized successively, albefaction processing and independent component analysis;
(d) data after each passage independent component analysis are carried out to Fourier variation, obtain power spectrum, obtain the corresponding frequency of power spectrum maximum amplitude, i.e. testee's heart rate, separately obtain power spectrum time corresponding frequency of maximum amplitude, i.e. testee's frequency of respiration;
(e) by testee's heart rate and frequency of respiration calculating testee's the heart rate variability value of single channel, and then by the calculating of multiple passages is obtained to heart rate variability sequence.
5. a kind of sleep state monitoring method based on machine vision according to claim 2, it is characterized in that, in the time that two photographic head are respectively infrared camera and colour imagery shot, the described step that obtains testee's heart rate, frequency of respiration and heart rate variability sequence thereof by buccal, opposite video image analysis is specially:
(a) buccal portion video image colour imagery shot being got, the RGB passage in separation cheek portion region, averages to the pixel of each passage;
(b) pixel of the R passage in the cheek portion region in buccal portion video image infrared camera being got is averaged;
(c) to data in four passages be normalized successively, albefaction processing and independent component analysis;
(d) data are preserved into 4 passages, each passage N data;
(e) to each channel data be normalized successively, albefaction processing and independent component analysis;
(f) data after each passage independent component analysis are carried out to Fourier variation, obtain power spectrum, obtain the corresponding frequency of power spectrum maximum amplitude, i.e. testee's heart rate, separately obtain power spectrum time corresponding frequency of maximum amplitude, i.e. testee's frequency of respiration;
(g) by testee's heart rate and frequency of respiration calculating testee's the heart rate variability value of single channel, and then by the calculating of multiple passages is obtained to heart rate variability sequence.
6. the sleep state monitoring device based on machine vision, described sleep state monitoring device comprises: 2 photographic head, encoder, display, embedded system, alarm, it is characterized in that,
Described photographic head is for shooting with video-corder the image of testee's head and face portion and be input to described encoder in real time;
Described encoder is sent into described embedded system for the image information of photographic head is encoded into data image signal;
Described embedded system is for extracting the visual signature of frequency of testee's frequency, the head position of face opening and closing and the time maintaining thereof, transposition, and the testee's such as pulse frequency physiological parameter feature, these characteristic informations are carried out to fusion treatment and judgement is reported to the police; In the time that testee's sleep state notes abnormalities, output alarm signal is to described alarm, and output abnormality signal is to the head of a family or relevant monitoring personnel simultaneously;
Described alarm is used for sending alarm sound;
Described display is for showing testee's physiological parameter.
7. a kind of sleep state monitoring device based on machine vision according to claim 6, is characterized in that, described photographic head is:
Infrared camera and/or colour imagery shot.
8. a kind of sleep state monitoring device based on machine vision according to claim 7, is characterized in that,
Described colour imagery shot is: ANC Duo HD1080P,
Described infrared camera is Samsung SCO-2080RP.
9. a kind of sleep state monitoring device based on machine vision according to claim 6, is characterized in that, described embedded system is:
One in i.MX6 processor, OMAP processor and TigerSHARC processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410310294.8A CN104083160A (en) | 2014-06-30 | 2014-06-30 | Sleep state monitoring method and device based on machine vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410310294.8A CN104083160A (en) | 2014-06-30 | 2014-06-30 | Sleep state monitoring method and device based on machine vision |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104083160A true CN104083160A (en) | 2014-10-08 |
Family
ID=51630999
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410310294.8A Pending CN104083160A (en) | 2014-06-30 | 2014-06-30 | Sleep state monitoring method and device based on machine vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104083160A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104545950A (en) * | 2014-12-23 | 2015-04-29 | 上海博康智能信息技术有限公司 | Non-contact type lie detection method and lie detection system thereof |
| CN106313048A (en) * | 2016-09-30 | 2017-01-11 | 北京光年无限科技有限公司 | Method and device for monitoring wake-up state of user based on intelligent robot |
| CN106491117A (en) * | 2016-12-06 | 2017-03-15 | 上海斐讯数据通信技术有限公司 | A kind of signal processing method and device based on PPG heart rate measurement technology |
| CN109645957A (en) * | 2018-12-21 | 2019-04-19 | 南京理工大学 | A kind of sound of snoring origin classification method |
| CN110021140A (en) * | 2018-01-09 | 2019-07-16 | 财团法人交大思源基金会 | Monitoring system and monitoring method applied to infant |
| CN110928203A (en) * | 2019-11-21 | 2020-03-27 | 苏宁智能终端有限公司 | Wearable terminal and shutdown method thereof |
| CN111214769A (en) * | 2018-11-26 | 2020-06-02 | 巧连科技股份有限公司 | Intelligent forced air intake respirator |
| CN111281347A (en) * | 2020-03-06 | 2020-06-16 | 韩赛红 | Passive lateral lying device, control method and sleep assisting system |
| CN111880642A (en) * | 2020-06-29 | 2020-11-03 | 北京小米移动软件有限公司 | Method and device for controlling terminal and storage medium |
| CN112102946A (en) * | 2019-11-21 | 2020-12-18 | 北京健康有益科技有限公司 | Sleep quality detection method and device based on deep learning |
| CN112752541A (en) * | 2018-09-25 | 2021-05-04 | 皇家飞利浦有限公司 | Deriving information about a person's sleep state and awake state from a sequence of video frames |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5689241A (en) * | 1995-04-24 | 1997-11-18 | Clarke, Sr.; James Russell | Sleep detection and driver alert apparatus |
| CN1613425A (en) * | 2004-09-15 | 2005-05-11 | 南京大学 | Method and system for drivers' fatigue prealarming biological identification |
| US20080212828A1 (en) * | 2007-02-16 | 2008-09-04 | Denso Corporation | Device, program, and method for determining sleepiness |
| CN101375791A (en) * | 2007-08-31 | 2009-03-04 | 佛山普立华科技有限公司 | System and method for monitoring sleeping condition of baby |
| CN101876815A (en) * | 2009-04-28 | 2010-11-03 | 深圳富泰宏精密工业有限公司 | Infant sleep monitoring system and method |
| CN101937604A (en) * | 2010-09-08 | 2011-01-05 | 无锡中星微电子有限公司 | Sleep monitoring system based on human body detection and method thereof |
| CN102236781A (en) * | 2010-04-22 | 2011-11-09 | 鸿富锦精密工业（深圳）有限公司 | System and method for sensing infant sleep |
| CN202078318U (en) * | 2011-01-24 | 2011-12-21 | 沈阳工业大学 | Multi-modality fatigue detection equipment for driver |
| KR101159847B1 (en) * | 2005-01-20 | 2012-06-25 | 삼성전자주식회사 | Apparatus and method for formalizing the detection of a driver's face for sensing sleepy driving |
| CN102973273A (en) * | 2012-11-29 | 2013-03-20 | 中国人民解放军第四军医大学 | Sleep respiratory function monitoring system based on infrared radiation detection |
| CN103271743A (en) * | 2012-12-10 | 2013-09-04 | 中国人民解放军第一五二中心医院 | Non-contact oxyhemoglobin saturation measuring device based on imaging device |
| CN103400471A (en) * | 2013-08-12 | 2013-11-20 | 电子科技大学 | Detecting system and detecting method for fatigue driving of driver |
| CN103462597A (en) * | 2013-09-18 | 2013-12-25 | 温州医科大学 | Method and device for preventing obstructive sleep sudden death based on intelligent mobile phone control |
| US20140135598A1 (en) * | 2011-08-05 | 2014-05-15 | Daimler Ag | Method and Device to Monitor at Least One Vehicle Passenger and Method to Control at Least One Assistance Device |
-
2014
- 2014-06-30 CN CN201410310294.8A patent/CN104083160A/en active Pending
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5689241A (en) * | 1995-04-24 | 1997-11-18 | Clarke, Sr.; James Russell | Sleep detection and driver alert apparatus |
| CN1613425A (en) * | 2004-09-15 | 2005-05-11 | 南京大学 | Method and system for drivers' fatigue prealarming biological identification |
| KR101159847B1 (en) * | 2005-01-20 | 2012-06-25 | 삼성전자주식회사 | Apparatus and method for formalizing the detection of a driver's face for sensing sleepy driving |
| US20080212828A1 (en) * | 2007-02-16 | 2008-09-04 | Denso Corporation | Device, program, and method for determining sleepiness |
| CN101375791A (en) * | 2007-08-31 | 2009-03-04 | 佛山普立华科技有限公司 | System and method for monitoring sleeping condition of baby |
| CN101876815A (en) * | 2009-04-28 | 2010-11-03 | 深圳富泰宏精密工业有限公司 | Infant sleep monitoring system and method |
| CN102236781A (en) * | 2010-04-22 | 2011-11-09 | 鸿富锦精密工业（深圳）有限公司 | System and method for sensing infant sleep |
| CN101937604A (en) * | 2010-09-08 | 2011-01-05 | 无锡中星微电子有限公司 | Sleep monitoring system based on human body detection and method thereof |
| CN202078318U (en) * | 2011-01-24 | 2011-12-21 | 沈阳工业大学 | Multi-modality fatigue detection equipment for driver |
| US20140135598A1 (en) * | 2011-08-05 | 2014-05-15 | Daimler Ag | Method and Device to Monitor at Least One Vehicle Passenger and Method to Control at Least One Assistance Device |
| CN102973273A (en) * | 2012-11-29 | 2013-03-20 | 中国人民解放军第四军医大学 | Sleep respiratory function monitoring system based on infrared radiation detection |
| CN103271743A (en) * | 2012-12-10 | 2013-09-04 | 中国人民解放军第一五二中心医院 | Non-contact oxyhemoglobin saturation measuring device based on imaging device |
| CN103400471A (en) * | 2013-08-12 | 2013-11-20 | 电子科技大学 | Detecting system and detecting method for fatigue driving of driver |
| CN103462597A (en) * | 2013-09-18 | 2013-12-25 | 温州医科大学 | Method and device for preventing obstructive sleep sudden death based on intelligent mobile phone control |
Non-Patent Citations (1)
| Title |
|---|
| MING-ZHER POH ET AL.: "Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam", 《IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104545950A (en) * | 2014-12-23 | 2015-04-29 | 上海博康智能信息技术有限公司 | Non-contact type lie detection method and lie detection system thereof |
| CN106313048A (en) * | 2016-09-30 | 2017-01-11 | 北京光年无限科技有限公司 | Method and device for monitoring wake-up state of user based on intelligent robot |
| CN106313048B (en) * | 2016-09-30 | 2019-01-22 | 北京光年无限科技有限公司 | The method and device of monitoring user's awake state based on intelligent robot |
| CN106491117A (en) * | 2016-12-06 | 2017-03-15 | 上海斐讯数据通信技术有限公司 | A kind of signal processing method and device based on PPG heart rate measurement technology |
| CN110021140A (en) * | 2018-01-09 | 2019-07-16 | 财团法人交大思源基金会 | Monitoring system and monitoring method applied to infant |
| CN112752541A (en) * | 2018-09-25 | 2021-05-04 | 皇家飞利浦有限公司 | Deriving information about a person's sleep state and awake state from a sequence of video frames |
| CN111214769A (en) * | 2018-11-26 | 2020-06-02 | 巧连科技股份有限公司 | Intelligent forced air intake respirator |
| CN109645957A (en) * | 2018-12-21 | 2019-04-19 | 南京理工大学 | A kind of sound of snoring origin classification method |
| CN110928203A (en) * | 2019-11-21 | 2020-03-27 | 苏宁智能终端有限公司 | Wearable terminal and shutdown method thereof |
| CN112102946A (en) * | 2019-11-21 | 2020-12-18 | 北京健康有益科技有限公司 | Sleep quality detection method and device based on deep learning |
| CN110928203B (en) * | 2019-11-21 | 2021-08-06 | 苏宁智能终端有限公司 | Wearable terminal and shutdown method thereof |
| CN111281347A (en) * | 2020-03-06 | 2020-06-16 | 韩赛红 | Passive lateral lying device, control method and sleep assisting system |
| CN111880642A (en) * | 2020-06-29 | 2020-11-03 | 北京小米移动软件有限公司 | Method and device for controlling terminal and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104083160A (en) | Sleep state monitoring method and device based on machine vision | |
| CN109222961B (en) | Portable sleep monitoring system and related sleep monitoring method | |
| CN104224132B (en) | sleep monitoring device and monitoring method thereof | |
| CN112806966B (en) | Non-interference type early warning system for sleep apnea | |
| CN106846735A (en) | A kind of intelligent mattress warning system | |
| CN108135487A (en) | For obtaining the equipment, system and method for the vital sign information of object | |
| CN104257353A (en) | Sleep apnea syndrome detecting system | |
| CN104077881A (en) | Infant monitoring method and device based on robot vision | |
| CN103070683A (en) | Sleep breathing mode identification method and device based on bioelectrical impedance | |
| CN109567756B (en) | Sleep state detection method based on artificial intelligence mattress | |
| JP2013505811A (en) | System and method for obtaining applied kinesiology feedback | |
| US20180303366A1 (en) | Method for Detecting Seizures | |
| CN106175770B (en) | The judgement system of apnea during a kind of sleep | |
| CN104545818A (en) | Sleep apnea syndrome detection method based on pulse and blood oxygen signals | |
| CN104173051A (en) | Automatic noncontact respiration assessing system and assessing method | |
| KR102068330B1 (en) | Device to detect sleep apnea | |
| CN109770863A (en) | Suitable for personnel's safety supervision bracelet under high temperature and humidity place | |
| CN109044275B (en) | Non-invasive sensing sleep quality analysis system and method based on fuzzy inference system | |
| KR20150066563A (en) | A monitoring device for analysing a sleep condition | |
| Massaroni et al. | Comparison of two methods for estimating respiratory waveforms from videos without contact | |
| Karlen et al. | Improving actigraph sleep/wake classification with cardio-respiratory signals | |
| CN113907742B (en) | Sleep breathing data monitoring method and device | |
| US20200100727A1 (en) | System for determining a set of at least one cardio-respiratory descriptor of an individual during sleep | |
| CN200994790Y (en) | System for measuring mental state parameter | |
| CN210990210U (en) | Non-inductive sleep monitoring device and monitoring system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20141008 |