CN111429345A - Method for visually calculating heart rate and heart rate variability with ultra-low power consumption - Google Patents

Method for visually calculating heart rate and heart rate variability with ultra-low power consumption Download PDF

Info

Publication number
CN111429345A
CN111429345A CN202010139255.1A CN202010139255A CN111429345A CN 111429345 A CN111429345 A CN 111429345A CN 202010139255 A CN202010139255 A CN 202010139255A CN 111429345 A CN111429345 A CN 111429345A
Authority
CN
China
Prior art keywords
heart rate
image
ultra
low power
filtering
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010139255.1A
Other languages
Chinese (zh)
Inventor
李丹疆
李东
刘萍
黎平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guiyang Xiangshuling Technology Co ltd
Original Assignee
Guiyang Xiangshuling Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guiyang Xiangshuling Technology Co ltd filed Critical Guiyang Xiangshuling Technology Co ltd
Priority to CN202010139255.1A priority Critical patent/CN111429345A/en
Publication of CN111429345A publication Critical patent/CN111429345A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/40Scaling the whole image or part thereof
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/02Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
    • A61B5/024Detecting, measuring or recording pulse rate or heart rate
    • A61B5/02405Determining heart rate variability
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/02Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
    • A61B5/024Detecting, measuring or recording pulse rate or heart rate
    • A61B5/02416Detecting, measuring or recording pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161Detection; Localisation; Normalisation

Abstract

The invention discloses a method for calculating heart rate and heart rate variability through ultra-low power consumption vision, which relates to the technical field of medical treatment for extracting physiological information, and comprises the following steps of using a visible light image or a near-infrared image as an input source, obtaining an RAW original image through a CMOS (complementary metal oxide semiconductor) from the input source, optimizing an ISP (internet service provider) image, zooming and cutting an image, searching a PCN (personal computer) face, tracking a target, defining a region mask, projecting a color space, performing Gaussian filtering, Butterworth filtering, performing normalization processing after trend filtering, filtering by a median filter, performing pulse wave description after spectrum analysis, outputting a video, and drawing a heart rate index; the invention is based on ARMv7a, does not have DSP and neural network acceleration units, and forms a low-cost edge calculation unit; the collection of the heart rate and the heart rate variability of the whole age group can be met without calibrating the band-pass filtering parameters; the color filter has good environmental light and motion interference resistance and is suitable for visible light and infrared monochromatic images.

Description

Method for visually calculating heart rate and heart rate variability with ultra-low power consumption
Technical Field
The invention relates to the technical field of medical treatment for extracting physiological information, in particular to a method for visually calculating heart rate and heart rate variability with ultralow power consumption.
Background
Edge computing provides the computing power needed near the data source, i.e., the data generation side and the application edge side, in the internet of things. For example, the internet of things technology is commonly applied to existing unmanned automobiles and electric automobiles. Future automobiles will be networked and a variety of rich applications for driving as well as individuals will be provided on the automobiles. In the process, all the acquired data and the application needing to be judged in real time cannot be sent back to the cloud computing center for computing; and the processing result is returned to the equipment end, so that a long time delay is generated in the calculation process.
Therefore, a large amount of real-time data needs to be pre-processed at the edge. Especially in the network application scenario of the internet of things, the computing is required to be performed at the edge end, rather than returning to a centralized cloud computing center. The edge computing is widely applied to the field of the Internet of things, and is particularly suitable for application scenes with special service requirements such as low time delay, high bandwidth, high reliability, mass connection, heterogeneous convergence, local security and privacy protection and the like.
The embedded system technology is a complex technology integrating computer software and hardware, sensor technology, integrated circuit technology and electronic application technology. If a simple metaphor is made for the human body of the Internet of things, the sensors are equivalent to senses of eyes, nose, skin and the like of the human body, the network is a nervous system for transmitting information, and the embedded system is the brain of the human body and is classified after receiving the information. This example visually describes the location and role of the sensors and embedded systems in the internet of things.
By adopting the edge computing technology, the home video data can be stored on the local edge computing gateway equipment, so that the privacy of the user is ensured not to be leaked; the linkage among the intelligent single products can also carry out near real-time coordination through local edge calculation; the edge computing node can also realize the synchronous updating of control and equipment state information with cloud computing at regular intervals.
The process of internet of things is increasingly becoming more and more popular such as: the problems cannot be completely solved by only cloud computing, the challenges of transmission, bandwidth, safety, data processing, data analysis and the like can be effectively relieved and solved by edge computing, and the future trend is cloud integration and cloud cooperation.
In the real world, there are small changes that are difficult to detect every moment, and these movements are so slight that they are difficult to detect without observing them. Sometimes, however, we have the need to magnify and emphasize these slight movements, and this section describes a special technique that magnifies the slight movements in the original video to be noticeable.
In 2013, the EVM Euler video amplification technology was successfully developed by the American Massachusetts institute of technology and engineering (MIT) computer science and artificial intelligence laboratory. The essence of this technique is to change the pixel motion of a video segment frame by frame while simultaneously magnifying these colors by 100 times, for example changing pink to crimson, so that they are visible to the naked eye. Many seemingly static scenes contain subtle changes that are not visible to the naked eye. However, these small changes can be extracted from the video using the algorithms they developed. They provide a method of visualizing these subtle changes through magnification and propose algorithms that extract interesting signals from these videos, such as the human pulse, the sound of vibrating objects and the movement of hot air.
The patent publication No. CN109259749 discloses an invention patent of a non-contact heart rate measuring method based on a visual camera, and discloses the following scheme: determining a region of interest ROI, processing a pixel signal, and extracting heart rate information; the invention utilizes a camera to capture video images of a target. The acquired images are then color enhanced and the selected ROI is located for these images. Then, effective signals are extracted, and the heart rate is obtained through a band-pass filter and fast Fourier transform.
The patent publication No. CN107529646A discloses a non-contact heart rate measurement method and device based on Euler image amplification, and is used for solving the technical problems that the existing non-contact heart rate measurement method has the defects of large interference of same frequency band noise of heart rate, large influence of ambient temperature, lack of judgment of authenticity of separation signals and the like. The invention comprises the following steps: s1: amplifying small changes of radial artery pulsation in the video through an Euler image amplification technology; s2: performing brightness variance statistics on brightness channels of pixel points of a video frame after amplifying small radial artery pulsation changes in a time domain, and performing skin segmentation on the video frame after amplifying the small radial artery pulsation changes at the same time to extract a radial artery pulsation area; s3: and performing time-frequency analysis on the radial artery beating area to calculate the heart rate.
The publication number is CN110236511A, and the patent name is a non-invasive heart rate measuring method based on video, and the weak signals of the face of a human face are amplified by adopting an improved Euler image amplification method so as to be convenient for signal extraction; the MTCNN face detection algorithm with strong real-time performance and high detection rate is adopted to improve the face detection rate, the detection speed is high, and the real-time performance is stronger; the G signal is combined as an input signal for heart rate calculation, so that the accuracy of the heart rate signal is improved, accurate, rapid and non-inductive noninvasive measurement of the heart rate is finally realized, and a foundation is provided for the real-time measurement and medical diagnosis of the subsequent human blood pressure.
The methods of the three patents with publication numbers CN109259749, CN107529646A, and CN110236511A are all based on euler video amplification methods, and cannot process near-infrared monochromatic video images, and the corresponding passband parameters of the bandpass filter need to be individually adapted one by one to limit the applicability thereof, so the overall scheme is not suitable for the application of edge calculation.
The euler video amplification method needs to manually select corresponding passband parameters of the bandpass filter when realizing micro-motion and color amplification. In general, people cannot directly determine the precise motion frequency and color change frequency of a moving object in a video, so that the method is difficult to operate on unknown videos; the linear image amplification technology utilizes a Laplacian pyramid or a Gaussian pyramid to carry out multi-resolution decomposition calculation, and the calculation amount is difficult to operate in an embedded edge calculation chip with ultra-low power consumption, so that the trial range is limited on computer hardware with high cost and high power consumption; and finally, the method which needs to use a three-channel mode of color space is not suitable for near-infrared gray level images.
Disclosure of Invention
The invention aims to: in order to solve the problems that in the prior art, an Euler video amplification method cannot process near-infrared monochromatic video images, corresponding passband parameters of a band-pass filter need to be individually adapted one by one to limit the applicability of the band-pass filter, and the overall scheme is not suitable for being used as the application of edge calculation, the invention provides the method for visually calculating the heart rate and the heart rate variability with ultralow power consumption.
The invention specifically adopts the following technical scheme for realizing the purpose:
a heart rate and heart rate variability method for ultra-low power consumption visual computation is characterized by comprising the following steps:
step 1, using a visible light image or a near infrared image as an input source;
step 2, an input source obtains an RAW original image through a CMOS;
step 3, the RAW original image is subjected to ISP image optimization processing;
step 4, zooming and cutting the image processed in the step 3;
step 5, carrying out PCN face retrieval, target tracking and area mask definition on the image processed in the step 4;
step 6, performing color space projection on the image processed in the step 5;
step 7, carrying out normalization processing after Gaussian filtering, Butterworth filtering and trend filtering processing;
step 8, the data after normalization processing is further filtered by a median filter;
and 9, performing spectrum analysis on the pulse wave description, outputting a video, and drawing the obtained heart rate index.
Further, for H264/H265/Mjepg encoding, steps 4-8 are replaced.
Further, in step 1, the near-infrared image input source is near-infrared 850nm and 940nm images.
Furthermore, the heart rate and heart rate variability data acquisition capacity can reach 30Hz and above.
Further, in step 7, the face mask area is effectively calculated to have a pixel minimum of 2000 pixels.
The scheme is based on RAW video data RAW to calculate, the image acquisition rate is 30 frames per second, the acquisition precision requirement can be met, software operation is realized based on ARMCPU and a built-in neon floating point accelerator, third-party hardware DSP, FPGA and a neural network accelerator are not depended on, the method is suitable for all microprocessor architectures of ARMv7a 800MHz and above, the acquisition capacity of heart rate and heart rate variability data can reach 30Hz and above, the algorithm is within 100MB in full load operation consumption, and the heart rate variability data can be acquired in real time.
The invention has the following beneficial effects:
1. the invention has simple structure, is based on ARMv7a, does not have DSP and neural network acceleration units, and forms a low-cost edge calculation unit; the collection of the heart rate and the heart rate variability of the whole age group can be met without calibrating the band-pass filtering parameters; non-specific index collection does not limit the race and skin color, light makeup and partial facial occlusion;
2. the detection results of the large-angle face and the front face are consistent; the device has good resistance to ambient light and motion interference; various algorithms of a pyramid used for spatial filtering are not consumed in computing performance; both visible and infrared monochromatic images are suitable;
3. local computing, which does not depend on Internet and cloud computing; the video streaming calculation does not need to cache video image materials;
drawings
FIG. 1 is a flow chart of a method and apparatus for ultra-low power consumption visual heart rate and heart rate variability calculation in accordance with the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Furthermore, the terms "first," "second," and the like are used merely to distinguish one description from another, and are not to be construed as indicating or implying relative importance.
In the description of the embodiments of the present invention, it should be noted that the terms "inside", "outside", "upper", and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings or orientations or positional relationships conventionally arranged when products of the present invention are used, and are only used for convenience in describing the present invention and simplifying the description, but do not indicate or imply that the devices or elements indicated must have specific orientations, be constructed in specific orientations, and operated, and thus, cannot be construed as limiting the present invention.
Example 1
As shown in fig. 1, the present embodiment provides a method for visually calculating heart rate and heart rate variability with ultra-low power consumption, which includes the following steps:
step 1, using a visible light image or a near infrared image as an input source;
step 2, an input source obtains an RAW original image through a CMOS;
step 3, the RAW original image is subjected to ISP image optimization processing;
step 4, zooming and cutting the image processed in the step 3;
step 5, carrying out PCN face retrieval, target tracking and area mask definition on the image processed in the step 4;
step 6, performing color space projection on the image processed in the step 5;
step 7, carrying out normalization processing after Gaussian filtering, Butterworth filtering and trend filtering processing;
step 8, the data after normalization processing is further filtered by a median filter;
and 9, performing spectrum analysis on the pulse wave description, outputting a video, and drawing the obtained heart rate index.
Further, for H264/H265/Mjepg encoding, steps 4-8 are replaced.
Further, in step 1, the near-infrared image input source is near-infrared 850nm and 940nm images.
Furthermore, the heart rate and heart rate variability data acquisition capacity can reach 30Hz and above.
Further, in step 7, the face mask area is effectively calculated to have a pixel minimum of 2000 pixels.
The scheme is based on ARMv7a, and a low-cost edge computing unit is formed without a DSP and a neural network acceleration unit; the collection of the heart rate and the heart rate variability of the whole age group can be met without calibrating the band-pass filtering parameters; the acquisition of nonspecific indexes does not limit the race and skin color, light makeup is carried out, the local shielding of the face is realized, the image acquisition rate is 30 frames per second, the acquisition precision requirement can be met, the software operation is realized based on the ARMCPU and a built-in neon floating point accelerator, the software operation is not dependent on third-party hardware DSP, FPGA and a neural network accelerator, the method is suitable for all microprocessor architectures of ARMv7a 800MHz and above, the acquisition capacity of the heart rate and heart rate variability data can reach 30Hz and above, the algorithm is within 100MB in full load operation consumption, and the heart rate variability data can be acquired in real time.

Claims (5)

1. A heart rate and heart rate variability method for ultra-low power consumption visual computation is characterized by comprising the following steps:
step 1, using a visible light image or a near infrared image as an input source;
step 2, an input source obtains an RAW original image through a CMOS;
step 3, the RAW original image is subjected to ISP image optimization processing;
step 4, zooming and cutting the image processed in the step 3;
step 5, carrying out PCN face retrieval, target tracking and area mask definition on the image processed in the step 4;
step 6, performing color space projection on the image processed in the step 5;
step 7, carrying out normalization processing after Gaussian filtering, Butterworth filtering and trend filtering processing;
step 8, the data after normalization processing is further filtered by a median filter;
and 9, performing spectrum analysis on the pulse wave description, outputting a video, and drawing the obtained heart rate index.
2. The ultra-low power visual computation heart rate and heart rate variability method of claim 1, used for H264/H265/Mjepg encoding instead of steps 4-8.
3. The method for ultra-low power visual computation of heart rate and heart rate variability of claim 1, wherein in step 1, the near-infrared image input source is a near-infrared 850nm or 940nm image.
4. The method of ultra-low power visual computation of heart rate and heart rate variability of claim 1, wherein heart rate and heart rate variability data capture capability is up to 30Hz and above.
5. The method of ultra-low power visual computation of heart rate and heart rate variability of claim 1, wherein in step 7, the face mask area is effective to compute pixels up to 2000pixels minimum.
CN202010139255.1A 2020-03-03 2020-03-03 Method for visually calculating heart rate and heart rate variability with ultra-low power consumption Pending CN111429345A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010139255.1A CN111429345A (en) 2020-03-03 2020-03-03 Method for visually calculating heart rate and heart rate variability with ultra-low power consumption

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010139255.1A CN111429345A (en) 2020-03-03 2020-03-03 Method for visually calculating heart rate and heart rate variability with ultra-low power consumption

Publications (1)

Publication Number Publication Date
CN111429345A true CN111429345A (en) 2020-07-17

Family

ID=71546353

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010139255.1A Pending CN111429345A (en) 2020-03-03 2020-03-03 Method for visually calculating heart rate and heart rate variability with ultra-low power consumption

Country Status (1)

Country Link
CN (1) CN111429345A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112929622A (en) * 2021-02-05 2021-06-08 浙江大学 Euler video color amplification method based on deep learning
CN113160518A (en) * 2021-04-02 2021-07-23 Tcl通讯(宁波)有限公司 Early warning system and early warning method based on edge calculation
CN113657345A (en) * 2021-08-31 2021-11-16 天津理工大学 Non-contact heart rate variability feature extraction method based on reality application scene
EP4075776A1 (en) * 2021-04-12 2022-10-19 Nokia Technologies Oy Mapping pulse propagation

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101114339A (en) * 2007-07-17 2008-01-30 李东亚 Visual medium audiences information feedback system and method thereof
CN105943015A (en) * 2016-06-04 2016-09-21 浙江大学 Wearable heart rate variability (HRV) monitoring device with active noise reduction function
CN109820499A (en) * 2018-12-24 2019-05-31 杨爽 The anti-interference heart rate detection method of height, electronic equipment and storage medium based on video
CN110575332A (en) * 2019-08-29 2019-12-17 江苏大学 Nursing bed and method based on near-infrared active stereoscopic vision and brain wave technology

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101114339A (en) * 2007-07-17 2008-01-30 李东亚 Visual medium audiences information feedback system and method thereof
CN105943015A (en) * 2016-06-04 2016-09-21 浙江大学 Wearable heart rate variability (HRV) monitoring device with active noise reduction function
CN109820499A (en) * 2018-12-24 2019-05-31 杨爽 The anti-interference heart rate detection method of height, electronic equipment and storage medium based on video
CN110575332A (en) * 2019-08-29 2019-12-17 江苏大学 Nursing bed and method based on near-infrared active stereoscopic vision and brain wave technology

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112929622A (en) * 2021-02-05 2021-06-08 浙江大学 Euler video color amplification method based on deep learning
CN113160518A (en) * 2021-04-02 2021-07-23 Tcl通讯(宁波)有限公司 Early warning system and early warning method based on edge calculation
CN113160518B (en) * 2021-04-02 2023-06-20 Tcl通讯(宁波)有限公司 Early warning system and early warning method based on edge calculation
EP4075776A1 (en) * 2021-04-12 2022-10-19 Nokia Technologies Oy Mapping pulse propagation
US11825206B2 (en) 2021-04-12 2023-11-21 Nokia Technologies Oy Mapping pulse propagation
CN113657345A (en) * 2021-08-31 2021-11-16 天津理工大学 Non-contact heart rate variability feature extraction method based on reality application scene
CN113657345B (en) * 2021-08-31 2023-09-15 天津理工大学 Non-contact heart rate variability feature extraction method based on realistic application scene

Similar Documents

Publication Publication Date Title
CN111429345A (en) Method for visually calculating heart rate and heart rate variability with ultra-low power consumption
Chen et al. Video-based heart rate measurement: Recent advances and future prospects
EP3057487B1 (en) Device and method for obtaining a vital sign of a subject
US9642536B2 (en) Mental state analysis using heart rate collection based on video imagery
KR101738278B1 (en) Emotion recognition method based on image
Dosso et al. Eulerian magnification of multi-modal RGB-D video for heart rate estimation
US20140200460A1 (en) Real-time physiological characteristic detection based on reflected components of light
Tao et al. Home activity monitoring using low resolution infrared sensor
WO2014145204A1 (en) Mental state analysis using heart rate collection based video imagery
Yin et al. Heart rate estimation based on face video under unstable illumination
Alzahrani et al. Preprocessing realistic video for contactless heart rate monitoring using video magnification
Shahadi et al. Eulerian video magnification: a review
Przybyło A deep learning approach for remote heart rate estimation
Szankin et al. Long distance vital signs monitoring with person identification for smart home solutions
Alnaggar et al. Video-based real-time monitoring for heart rate and respiration rate
Wang et al. VitaSi: A real-time contactless vital signs estimation system
CN106096544B (en) Contactless blink and heart rate combined detection system and method based on second-order blind identification
CN112716468A (en) Non-contact heart rate measuring method and device based on three-dimensional convolution network
Slapnicar et al. Contact-free monitoring of physiological parameters in people with profound intellectual and multiple disabilities
CN110321781A (en) A kind of signal processing method and device for heed contacted measure
Karmuse et al. A robust rppg approach for continuous heart rate measurement based on face
CN113326781B (en) Non-contact anxiety recognition method and device based on face video
CN110321782A (en) A kind of system detecting characteristics of human body's signal
Peng et al. MVPD: A multimodal video physiology database for rPPG
Malacarne et al. Improved remote estimation of heart rate in face videos

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20200717

RJ01 Rejection of invention patent application after publication