CN112294282A - Self-calibration method of emotion detection device based on RPPG - Google Patents

Self-calibration method of emotion detection device based on RPPG Download PDF

Info

Publication number
CN112294282A
CN112294282A CN201910710796.2A CN201910710796A CN112294282A CN 112294282 A CN112294282 A CN 112294282A CN 201910710796 A CN201910710796 A CN 201910710796A CN 112294282 A CN112294282 A CN 112294282A
Authority
CN
China
Prior art keywords
heart rate
power spectrum
face
pulse
spectrum amplitude
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910710796.2A
Other languages
Chinese (zh)
Inventor
王慧泉
何森
田磊
赵喆
韩嘉文
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tianjin Polytechnic University
Original Assignee
Tianjin Polytechnic University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tianjin Polytechnic University filed Critical Tianjin Polytechnic University
Priority to CN201910710796.2A priority Critical patent/CN112294282A/en
Publication of CN112294282A publication Critical patent/CN112294282A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/02Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
    • A61B5/024Detecting, measuring or recording pulse rate or heart rate
    • A61B5/02416Detecting, measuring or recording pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/0033Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0077Devices for viewing the surface of the body, e.g. camera, magnifying lens
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/16Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/165Evaluating the state of mind, e.g. depression, anxiety
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16HHEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

The invention discloses a self-calibration method of a non-contact heart rate emotion monitoring device based on a face video, which comprises the following steps: (1) opening a camera to collect a face video; (2) carrying out frame conversion on the acquired video to determine an ROI (region of interest); (3) r, G, B separating the interested region; (4) carrying out normalization processing, independent component analysis and Fourier change on the separated data, extracting a maximum power spectrum value and corresponding frequency, extracting a pulse numerical value, and storing the pulse numerical value in a pulse array; (4) calculating the heart rate number HR (pulse (round ((i-450)/15) +1) × 60 according to a formula; (5) calculating a heart rate amplitude HRA and a heart rate variability HRV according to the extracted heart rate waveform; (6) and establishing a model among HR, HRA and HRV, and calculating the emotion index H which is a.HR + b.HRA + c.H.

Description

Self-calibration method of emotion detection device based on RPPG
Technical Field
The invention relates to a heart rate measuring method based on a face video, which judges the psychological health and emotion of a human body through measured heart rate information and belongs to the related field of biomedical engineering.
Background
The heart rate is the most basic information of pulse wave and is one of four vital signs of human body. The stability of the compound directly reflects the quality of the heart function and is an important physiological index of human health. Heart rate is an important parameter for monitoring cardiovascular disease. Meanwhile, the heart rate is also an important index of conductive exercise, and the kinematics research shows that the heart rate is one of parameters with sensitive physiological changes, and the heart rate has important application in criminal investigation means such as lie detection.
Most traditional heart rate detection is achieved by means of pulse feeling or auscultation by a doctor, and such heart rate measurement requires a doctor to have a rich relevant knowledge and a great deal of practical experience. With the development of science and technology, people invent heart rate monitors to measure the heart rate of people, but the heart rate monitors are high in cost, are usually only used for clinical monitoring in hospitals, and are difficult to enter the daily life of people. Subsequently, the appearance of finger-clipped heart rate oximeter and the like greatly reduces the equipment cost on the basis of high measurement accuracy, is more convenient to use, and can accurately acquire the heart rate of a user as long as the finger-clipped heart rate oximeter is clipped on a finger, but the finger-clipped heart rate oximeter is required to be directly contacted with the human body, and the long-time contact can cause discomfort of a testee, so that the finger-clipped heart rate oximeter is not suitable for long-time heart rate measurement.
In recent years, along with the popularization of computers, cameras and the like, the proposal of an image PPG (photoplethysmography) technology provides a feasible idea for realizing non-invasive and non-contact real-time heart rate measurement. Because the heart of the human body is continuously contracted and expanded, the filling degree of blood in the blood vessel of the human body can be continuously changed along with the heartbeat, and the absorption of light can present pulsatile change consistent with the heartbeat along with the change of the blood volume, so that the intensity of light reflected by the surface of the skin can also correspondingly and periodically change, and the change is represented as the change of the skin color in the collected image. The human face video based pulse wave signal extraction method based on the rppg technology realizes the extraction of the human body pulse wave signal based on the human face video, the HR, HRA and HRV values are extracted from the pulse wave signal, and the emotion influence index H is calculated according to the established emotion influence model, so that the psychological health is evaluated.
Disclosure of Invention
The invention aims to provide a self-calibration method of a non-contact heart rate emotion detection device based on a face video, which has the advantages of no wound, non-contact, high precision, low algorithm complexity and the like and has wide application prospects in the fields of patient heart rate monitoring, family heart rate measurement, body sensing games and criminal investigation lie detection.
The technical scheme of the invention is as follows:
(1) collecting a video of a section of face area by using a common camera, and reading the total frame number of the video;
(2) searching a face region of the RGB graph by combining an Adaboost algorithm and a pyramid graph, acquiring the length h and the width w of the face region, and determining the cheek region as an interested ROI region;
(3) separating RGB channels of the region of interest and taking a mean value;
(4) dividing and normalizing the acquired data according to the size of the window;
(5) analyzing independent components, changing Fourier, removing linear components, and extracting a maximum power spectrum value and corresponding frequency;
(6) and obtaining an emotion influence factor H ═ a × HR + b × HRA + c × HRV + d according to the collected new heart rate signals. (H: index of emotional influence, HR: number of heart rates, HRA: amplitude of heart rates, HRV: number of heart rate variability)
The invention has the advantages and positive effects that:
the self-calibration method of the emotion detection device based on the RPPG can realize non-contact measurement of the heart rate of a human body, and simultaneously, the relationship between the emotion influence index H and the heart rate is calculated according to the measured heart rate measurement characteristic points. The heart rate monitoring method is high in accuracy, has no direct contact with a tested person, and has no radiation hazard to a tested human body. The method disclosed by the invention is beneficial to health monitoring in daily life, provides a new technical means for clinical medical treatment, and can be applied to the fields of diagnosis of human mental health, judgment of lie in criminal investigation means and the like.
Drawings
Fig. 1 is a schematic diagram of a non-contact heart rate monitoring system.
Fig. 2 is a flow chart of the overall implementation of the system.
Detailed Description
The first implementation mode comprises the following steps:
the first step is as follows: selecting an environment with proper illumination, determining a position which can clearly and completely image a face area, and fixing a camera;
the second step is that: starting a camera to acquire a video of a scene of a face area, wherein the face is allowed to move and deflect within an imaging range in the acquisition process, and the acquired data is stored as an MP4 format video file;
the third step: selecting a face region of the acquired video file, determining an ROI (region of interest), separating RGB (red, green and blue) channels and taking an average value;
the fourth step: storing the obtained RGB numerical value, continuously processing next frame data until the total frame number of the video is completely processed, and drawing a mean value curve of RGB three channels by matlab software;
the fifth step: according to the drawn curve, the G channel can be seen to contain pulse information;
and a sixth step: setting the sizes of an adopted window and a sliding window, then carrying out normalization processing on the stored RGB channel numerical values, storing a matrix of the first ten maximum power spectrum amplitude values and frequency values of each channel of each window, and removing linear components from data to obtain a separated component matrix;
the seventh step: performing power spectrum analysis on the data, extracting the maximum power spectrum amplitude value in each channel, and finding out a pulse angle mark corresponding to the maximum frequency spectrum amplitude value, thereby finding out the most possible pulse value of the current window and storing the most possible pulse value in a plurality of groups of pulses;
eighth step: when i is larger than 450, calculating the number of heartbeats according to the formula pulse (round ((i-450)/15) +1) × 60;
the ninth step: and calculating H according to H & lta & gt HR + b & ltHRA + c & ltHRV + d, and estimating the relation between the emotion and the heart rate.
The second embodiment:
testing 10 adults according to the procedure of embodiment one;
the first step is as follows: take a video of 10 adults when they were answering questions normally;
the second step is that: shooting videos of 10 adults in a lie state;
the third step: analyzing the video information of the answer questions in a normal state, and calculating the value of an emotion influence index H;
the fourth step: analyzing the video information of the answer questions in the lie state, and calculating the value of the emotion influence index H;
the fifth step: by comparing the H values, the emotional influence index H value is higher in the lie state.

Claims (8)

1. A self-calibration method of an emotion detection device based on RPPG is characterized by comprising the following steps:
acquiring a face region video; the face area video consists of a plurality of frames of RGB face images;
carrying out frame conversion on the face region video to determine a face ROI image;
r, G, B three channels of human face ROI image are separated to obtain RGB value of each channel;
extracting the maximum power spectrum amplitude and the pulse angle mark in each channel according to the RGB numerical values; the pulse angle mark is a pulse angle mark corresponding to the maximum power spectrum amplitude value;
obtaining a pulse value according to the maximum power spectrum amplitude and the pulse angle scale;
calculating heart rate data according to the pulse value; the heart rate data comprises heart rate number per minute, heart rate amplitude and heart rate variability;
and calculating the emotion index according to the heart rate data to realize self-calibration.
2. The self-calibration method of the RPPG based emotion detection device as claimed in claim 1, wherein after said R, G, B three-channel separation of the ROI image of the human face to obtain RGB values of each channel, further comprising:
and carrying out normalization processing on the RGB values of all channels.
3. The self-calibration method of the emotion detection device based on RPPG as claimed in claim 1, wherein the performing frame transformation on the face region video to determine a face ROI image specifically comprises:
combining an Adaboost algorithm with a pyramid image, searching the RGB face image, and determining the length and the width of the RGB face image;
expanding the RGB face image, and determining a face ROI image according to the length and the width of the RGB face image; the face ROI image bboxPolygon ═ x, y, x + w, y, x + w, y + h, x, y + h ]; wherein x is the x-axis coordinate of the face ROI image coordinate, y is the y-axis coordinate of the face ROI image coordinate, h is the length of the RGB face image, w is the width of the RGB face image, x ═ bbox (1) + bbox (3) × 0.15, y ═ bbox (2) + bbox (4) × 0.05, w ═ bbox (3) × 0.7, h ═ bbox (4) × 0.85.
4. The self-calibration method of the emotion detection device based on RPPG as claimed in claim 1, wherein said extracting maximum power spectrum amplitude and pulse angle scale in each channel according to said RGB values specifically comprises:
calculating the power spectrum amplitude of each channel of each window by adopting an independent component analysis method and Fourier transform according to the RGB numerical value;
sequencing the power spectrum amplitude values from top to bottom to obtain a power spectrum amplitude value sequence corresponding to each channel;
determining a maximum power spectrum amplitude group and a corresponding frequency matrix corresponding to each channel according to the power spectrum amplitude sequence; the maximum power spectrum amplitude value group consists of the first ten power spectrum amplitude values in the power spectrum amplitude value sequence; the frequency matrix is a frequency matrix corresponding to the first ten power spectrum amplitude values;
removing linear components in the maximum power spectrum amplitude value set and the frequency matrix to obtain a processed maximum power spectrum amplitude value set and a processed frequency matrix;
and determining the maximum power spectrum amplitude and the pulse angle standard in each channel according to the processed maximum power spectrum amplitude group and the processed frequency matrix.
5. The self-calibration method of the RPPG based emotion detection device as recited in claim 1, wherein the calculation formula of the heart rate number is:
HR=pulse(round((i-450)/15)+1)*60;
HR represents the heart rate, i is larger than 450, pulse represents an array, and the pulse array stores pulse values.
6. The self-calibration method of the RPPG based emotion detection device as claimed in claim 1, wherein said calculating the emotion index according to the heart rate data to achieve self-calibration specifically comprises:
constructing an emotion influence index model according to the heart rate data; the model of emotional impact index is a correlated non-linear model of the heart rate number, the heart rate amplitude, and the heart rate variability; the emotion influence index model
H=a*HR+b*HRA+c*HRV+d;
Where HR denotes the number of heart rates, HRA denotes the amplitude of the heart rate, HRV denotes the variability of the heart rate, a denotes the coefficient of the HR correlation, b denotes the coefficient of the HRA correlation, c denotes the coefficient of the HRV correlation, and d is a compensation constant.
7. The self-calibration method of the RPPG based emotion detection device as claimed in claim 1, wherein the face region video is obtained by setting a timing initial value to 0 and performing video acquisition for 1 minute on a scene of the face region by using a camera.
8. The self-calibration method of an RPPG based emotion detection device as recited in claim 4, wherein said extracting maximum power spectral amplitude and pulse angle scale in each channel according to said RGB values further comprises:
the size of the initial sliding window is 30s, and the size of the sliding window and the last overlapping part is 29 s.
CN201910710796.2A 2019-08-01 2019-08-01 Self-calibration method of emotion detection device based on RPPG Pending CN112294282A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910710796.2A CN112294282A (en) 2019-08-01 2019-08-01 Self-calibration method of emotion detection device based on RPPG

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910710796.2A CN112294282A (en) 2019-08-01 2019-08-01 Self-calibration method of emotion detection device based on RPPG

Publications (1)

Publication Number Publication Date
CN112294282A true CN112294282A (en) 2021-02-02

Family

ID=74485472

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910710796.2A Pending CN112294282A (en) 2019-08-01 2019-08-01 Self-calibration method of emotion detection device based on RPPG

Country Status (1)

Country Link
CN (1) CN112294282A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113827240A (en) * 2021-09-22 2021-12-24 北京百度网讯科技有限公司 Emotion classification method and emotion classification model training method, device and equipment
CN113892952A (en) * 2021-06-09 2022-01-07 上海良相智能化工程有限公司 Intelligent studying and judging system
CN117281516A (en) * 2023-11-24 2023-12-26 吾征智能技术(北京)有限公司 Mental stress monitoring system, method, medium and device
WO2024056087A1 (en) * 2022-09-16 2024-03-21 Jing Wei Chin Method and device for camera-based heart rate variability monitoring

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103271734A (en) * 2012-12-10 2013-09-04 中国人民解放军第一五二中心医院 Heart rate measuring method based on low-end imaging device
CN104545864A (en) * 2014-12-25 2015-04-29 中国科学院深圳先进技术研究院 Mental regulation method and mental regulation device
CN105105732A (en) * 2015-08-14 2015-12-02 姚丽峰 System and method for implementing detection on pulse and heart rate and monitoring on emotion and fatigue degree
CN105193431A (en) * 2015-09-02 2015-12-30 杨静 Device for analyzing mental stress state of human body
CN105989357A (en) * 2016-01-18 2016-10-05 合肥工业大学 Human face video processing-based heart rate detection method
CN105996993A (en) * 2016-04-29 2016-10-12 南京理工大学 System and method for intelligent video monitoring of vital signs
CN107506716A (en) * 2017-08-17 2017-12-22 华东师范大学 A kind of contactless real-time method for measuring heart rate based on video image
WO2019055919A1 (en) * 2017-09-15 2019-03-21 University Of Maryland, College Park Heart rate measurement for fitness exercises using video
CN109523545A (en) * 2018-11-28 2019-03-26 荆门博谦信息科技有限公司 A kind of contactless heart rate detection method and system
US20190171893A1 (en) * 2017-12-05 2019-06-06 Tata Consultancy Services Limited Face video based heart rate monitoring using pulse signal modelling and tracking

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103271734A (en) * 2012-12-10 2013-09-04 中国人民解放军第一五二中心医院 Heart rate measuring method based on low-end imaging device
CN104545864A (en) * 2014-12-25 2015-04-29 中国科学院深圳先进技术研究院 Mental regulation method and mental regulation device
CN105105732A (en) * 2015-08-14 2015-12-02 姚丽峰 System and method for implementing detection on pulse and heart rate and monitoring on emotion and fatigue degree
CN105193431A (en) * 2015-09-02 2015-12-30 杨静 Device for analyzing mental stress state of human body
CN105989357A (en) * 2016-01-18 2016-10-05 合肥工业大学 Human face video processing-based heart rate detection method
CN105996993A (en) * 2016-04-29 2016-10-12 南京理工大学 System and method for intelligent video monitoring of vital signs
CN107506716A (en) * 2017-08-17 2017-12-22 华东师范大学 A kind of contactless real-time method for measuring heart rate based on video image
WO2019055919A1 (en) * 2017-09-15 2019-03-21 University Of Maryland, College Park Heart rate measurement for fitness exercises using video
US20190171893A1 (en) * 2017-12-05 2019-06-06 Tata Consultancy Services Limited Face video based heart rate monitoring using pulse signal modelling and tracking
CN109523545A (en) * 2018-11-28 2019-03-26 荆门博谦信息科技有限公司 A kind of contactless heart rate detection method and system

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113892952A (en) * 2021-06-09 2022-01-07 上海良相智能化工程有限公司 Intelligent studying and judging system
CN113827240A (en) * 2021-09-22 2021-12-24 北京百度网讯科技有限公司 Emotion classification method and emotion classification model training method, device and equipment
CN113827240B (en) * 2021-09-22 2024-03-22 北京百度网讯科技有限公司 Emotion classification method, training device and training equipment for emotion classification model
WO2024056087A1 (en) * 2022-09-16 2024-03-21 Jing Wei Chin Method and device for camera-based heart rate variability monitoring
CN117281516A (en) * 2023-11-24 2023-12-26 吾征智能技术(北京)有限公司 Mental stress monitoring system, method, medium and device

Similar Documents

Publication Publication Date Title
Wang et al. A comparative survey of methods for remote heart rate detection from frontal face videos
US10004410B2 (en) System and methods for measuring physiological parameters
CN103908236B (en) A kind of automatic blood pressure measurement system
CN112294282A (en) Self-calibration method of emotion detection device based on RPPG
CN110384491A (en) A kind of heart rate detection method based on common camera
CN107358220A (en) A kind of human heart rate and the contactless measurement of breathing
CN106580301A (en) Physiological parameter monitoring method, device and hand-held device
CN107506716A (en) A kind of contactless real-time method for measuring heart rate based on video image
CN111714144A (en) Mental stress analysis method based on video non-contact measurement
CN110647815A (en) Non-contact heart rate measurement method and system based on face video image
CN110084085B (en) RPPG high-precision heart rate detection method based on forming signal
CN111345803B (en) Heart rate variability measuring method based on mobile device camera
CN110236515A (en) A kind of contactless heart rate detection method based on near-infrared video
CN111387959A (en) Non-contact physiological parameter detection method based on IPPG
CN112890792A (en) Cloud computing cardiovascular health monitoring system and method based on network camera
CN111714088B (en) Human body characteristic index detection method and system based on traditional Chinese medicine principle
CN110251115A (en) PPG method for extracting signal, system, equipment and medium based on body surface video
CN110353700A (en) Contactless method for detecting blood oxygen saturation
CN113576475B (en) Deep learning-based contactless blood glucose measurement method
Wang et al. KLT algorithm for non-contact heart rate detection based on image photoplethysmography
Hu et al. Study on Real-Time Heart Rate Detection Based on Multi-People.
Khong et al. Analysis of heart beat rate through video imaging techniques
Cruz et al. Non-contact heart rate and respiratory rate monitoring system using histogram of oriented gradients
AV et al. Non-contact heart rate monitoring using machine learning
CN114209299B (en) IPPG technology-based human physiological parameter detection channel selection method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20210202