JP6817782B2 - Pulse detection device and pulse detection method - Google Patents

Description

The present invention relates to a technique for measuring a pulse or a degree of stress from an image acquired by a camera.

Conventionally, a method of estimating and calculating a person's heart rate and pulse from a non-contact image acquired by a camera has been known.

For example, after calculating the mean value of each RGB in the region of the face image and processing it by independent component analysis (ICA), the heart rate is estimated from the peak frequency obtained by the frequency analysis of one component waveform. The method is known (see, for example, Non-Patent Document 1).

Further, in order to track one component waveform after ICA processing in time series, a method of pairing with the component waveform of the current time and further tracking the peak frequency representing the heart rate in time series to estimate the heart rate. Is known (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2012-239661

M. Z. Poh, D. J. McDuff, and R. W. Picard “Non-contact, Automated Cardiac Pulse Measurements Using Video Imaging and Blind Source Separation” Optics Express, May 2010, (US), May 2010 18, No. 10, pp. 10762-10774

However, the methods disclosed in Patent Document 1 and Non-Patent Document 1 have a problem that the detection accuracy of heartbeat, pulse, etc. is lowered by the movement of a person.

In addition, depending on the application, in order to protect the privacy of people, an unclear image may be acquired intentionally, but even in this case, the face detection fails, so that the heartbeat, pulse, etc. are detected. There was a problem that the accuracy was lowered. In particular, when a person is stationary, the face region cannot be specified by using motion detection or the like, so that the detection accuracy is significantly reduced.

The present invention has been made in view of the above points, and an object of the present invention is to provide an apparatus capable of accurately detecting a pulse regardless of the situation of an imaging target or imaging conditions.

In order to achieve the above object, in the present invention, the face detection function from the captured image to the pulse detection, the noise reduction function, the pulse calculation function, etc., depending on the situation of the imaging area, the situation of the photographed person, and the like. Can be selected from different processing methods.

Specifically, the pulse detection device of the present invention is a pulse detection device that detects the pulse of a living body, and is an imaging unit that captures a predetermined imaging region and acquires an image signal, and a living body based on the image signal. From a face detector that detects the face area or skin area, a noise remover that removes noise from the detected image signal from the face area or skin area, and an image signal from the face area or skin area from which noise has been removed. It is characterized by including a pulse calculator for calculating a pulse and a diffuser for scattering and / or dimming incident light on an imaging unit to obscure a captured image .

According to this configuration, the pulse can be obtained directly from the image acquired by the imaging unit such as a camera. In addition, by making the captured image unclear, it is possible to obtain the pulse from the same image while protecting the privacy of the person to be captured.

The face detector, noise remover, and pulse calculator each support a plurality of processing methods, and one of the above processing methods can be selected from the plurality of processing methods according to the imaging region and the condition of the living body. , Preferably configured to perform the process.

According to this configuration, even in various cases where there is movement of a living body such as a human being, the face is hidden, or the face is not facing the direction of the imaging unit in the imaging region, imaging by a camera or the like The pulse can be obtained from the image acquired by the unit.

The face detector is preferably configured to detect a local brightness difference in the brightness value of the image signal in the imaging region and detect the face region by a majority logic based on the brightness difference.

According to this configuration, when a clear image is acquired, the face can be detected by a simple method.

The face detector may be configured to divide the image in the imaging region and detect the face region or the skin region based on the image signal from each of the divided regions.

According to this configuration, the face region or the skin region can be detected by a simple method even when a plurality of living organisms are photographed.

The face detector divides the image in the imaging region and compares the time-varying waveform of the image signal from each divided region with the model waveform represented by a sine wave, and based on the result obtained, the face region. Alternatively, it may be configured to detect the skin area.

According to this configuration, the data retention time for face detection can be shortened, and face detection can be performed in a short time. Further, even when the face of a living body such as a human being is hidden or the face area in the image is small, the skin area can be detected.

The noise remover is preferably configured to remove a specific frequency component of the power spectrum obtained by frequency analysis of the position information of the face region in the imaging region from the image signal from the face region.

According to this configuration, noise caused by facial movement, which is close to the pulse component in the frequency domain, can be appropriately removed.

The noise remover may be configured to remove at least a part of the image signal other than the wavelength component used for detecting the pulse component from the image signal from the face region or the skin region.

According to this configuration, noise other than the pulse component included in the image signal can be appropriately removed with a simple configuration.

The pulse calculator is preferably configured to calculate the pulse of the living body by frequency-analyzing the image signal from the face region or the skin region after the noise is removed.

The pulse calculator may be configured to calculate the pulse of the living body by comparing the time-varying waveform of the image signal from the face region or the skin region with the model waveform represented by a sine wave.

According to this configuration, the data holding time for pulse calculation can be shortened, and the pulse can be calculated in a short time.

The pulse detector further has a biodetector that detects the presence or absence of a living body based on the periodic fluctuation component of the image signal, and the face detector converts the image signal from the existing region of the living body obtained by the biodetector. It is preferably configured to detect the face area or skin area based on it.

According to this configuration, even when the face cannot be detected immediately, it is possible to easily detect the face region or the skin region by detecting a living body such as a human being, and to perform pulse detection.

The biodetector is compatible with a plurality of processing methods, and is configured to select one processing method from a plurality of processing methods and execute the processing according to the imaging region and the condition of the living body. preferable.

According to this configuration, the presence or absence of a living body is determined even in various cases where the living body such as a person is moving, the face is hidden, or the face is not facing the direction of the imaging unit in the imaging region. It can be detected reliably, and pulse detection becomes possible.

The biological detector may be configured to divide an image in an imaging region and detect the presence or absence of a living body based on an image signal from each divided region.

According to this configuration, the presence or absence of a living body can be detected by a simple method even when a plurality of living bodies are photographed.

The biodetector is configured to divide an image in the imaging region and compare the time-varying waveform of the image signal from each divided region with the model waveform represented by a sine wave to detect the presence or absence of a living body. You may be.

According to this configuration, the data retention time for biological detection can be shortened, and biological detection can be performed in a short time. Further, even when the face of a living body such as a human being is hidden or the face area in the image is small, the presence or absence of the living body can be detected.

It is preferable to further provide a stress measuring device for measuring the degree of stress in the living body based on the pulse interval obtained from the pulse calculator.

According to this configuration, the degree of stress can be obtained directly from an image acquired by an imaging unit such as a camera.

It is preferable to further provide an arousal level measuring device for measuring the arousal level of the living body based on the pulse interval obtained from the pulse rate calculator.

According to this configuration, the arousal level can be obtained directly from the image acquired by the imaging unit such as a camera.

The pulse detection method of the present invention is a pulse detection method for detecting a pulse of a living body, which is an imaging step of capturing a predetermined imaging region and obtaining an image signal, and a face region or a skin region of the living body based on the image signal. A face detection step to be detected, a noise removal step to remove noise from an image signal from the detected face area or skin area, and a pulse calculation to calculate a pulse from an image signal from the face area or skin area from which noise has been removed. It is characterized by comprising a step and a light-diffusing step of scattering and / or dimming the light from the imaging region and inputting the light to the imaging unit to make the captured image unclear .

According to this method, the pulse can be obtained directly from the image acquired by the imaging unit such as a camera. In addition, by making the captured image unclear, it is possible to obtain the pulse from the same image while protecting the privacy of the person to be captured.

The face detection step, the noise removal step, and the pulse calculation step each correspond to a plurality of processing methods, and one processing method is selected from the plurality of processing methods according to the imaging region and the condition of the living body. It is preferable to carry out the process.

According to this configuration, even in various cases where there is movement of a living body such as a human being, the face is hidden, or the face is not facing the direction of the imaging unit in the imaging region, imaging by a camera or the like The pulse can be obtained from the image acquired by the unit.

As described above, according to the pulse detection device of the present invention, in various cases where there is movement of a person, the face is hidden, or the face is not facing the direction of the imaging unit in the imaging region. However, the pulse, stress level, and arousal level of a person can be directly obtained from the image acquired by the imaging unit.

It is a block diagram which shows the functional structure of the pulse detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of the processing method in each functional block in a pulse detection apparatus. It is a block diagram which shows the flow of the signal in the pulse detection apparatus which concerns on Embodiment 1. FIG. This is a preferable example of the pulse detection flowchart according to the first embodiment. This is an example of a person's image acquired by the imaging unit according to the first embodiment. It is a figure which shows an example of the face detection method which concerns on Embodiment 1. FIG. It is a figure which shows an example of the area division of the image which concerns on Embodiment 1. FIG. It is a figure which shows an example of setting of the face search range in the front-rear frame. It is a figure which shows the time change of the luminance value of the image signal in the face region which concerns on Embodiment 1. FIG. This is an example of a sine wave model for a pulse component according to the first embodiment. This is another example of the sine wave model for the pulse component according to the first embodiment. It is still another example of the sine wave model for the pulse component according to the first embodiment. It is a figure which shows the time change of the luminance value of the image signal in the face region before and after the difference of the sine wave model which concerns on Embodiment 1. FIG. It is a power spectrum obtained by frequency analysis of the position coordinates of the face. It is a power spectrum obtained by differentiating the signal corresponding to the change in the face position from the luminance value of the image signal from the face region and further performing frequency analysis. It is a figure which shows the sine wave model corresponding to the frequency of the movement of the face obtained in FIG. It is a figure which shows the time change of the luminance value of the image signal from a face region. It is a figure which shows the time change of the luminance value of the image signal from a face region after noise reduction. It is a figure which shows an example of the skin area detection method which concerns on Embodiment 1. FIG. It is a figure which shows an example of the hardware configuration of the pulse detection system which concerns on Embodiment 1. FIG. It is a block diagram which shows the functional structure of the pulse detection system which concerns on Embodiment 2. FIG. It is a figure which shows the example of the processing function of a person detector. It is a block diagram which shows the flow of the signal in the pulse detection apparatus which concerns on Embodiment 2. FIG. This is a preferable example of the pulse detection flowchart according to the second embodiment. This is an example of a person's image acquired by the imaging unit according to the second embodiment. It is a figure which shows an example of the person detection method which concerns on Embodiment 2. FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The following description of preferred embodiments is merely exemplary and is not intended to limit the invention, its applications or its uses.

(Embodiment 1)
(Configuration of pulse detector and flow of image signal processing)
FIG. 1 is a block diagram showing a functional configuration of the pulse detection device according to the present embodiment.

FIG. 2 is a diagram showing an example of a processing function of each block of the pulse detection device.

FIG. 3 is a block diagram showing a signal flow in the pulse detection device.

The pulse detection device 10 includes an imaging unit 12, a face detector 13, a noise reducer 14, a pulse calculator 15, a stress level measuring device 16, and an arousal level measuring device 17 as functional blocks. There is.

The image pickup unit 12 has an image pickup element (not shown) such as a CCD or CMOS image sensor, and an optical system (not shown) including a lens or the like, and the image acquired by the image pickup unit 12 is an image. It is sent to the face detector 13 as a signal. The imaging unit 12 may be fixed or movable. The imaging region is determined according to the setting of the optical system and the movable region of the imaging unit 12.

The image sensor is preferably a color image sensor that outputs light incident from a peripheral region for each color component by using a color filter or the like, but is not particularly limited to this, and is an image sensor that outputs a black-and-white image. It may be. Further, although a CCD, a CMOS image sensor, or the like is generally used as the image sensor, a photodiode array may be used.

Further, as the image pickup device, an image pickup device that receives near-infrared light may be used, or a combination of this and another image pickup device may be used.

In the present embodiment, the image signal is an RGB color system signal represented by three primary colors of R (Red: red) component, G (Green: green) component, and B (Blue: blue) component. However, the present invention is not particularly limited to this, and may be an HSV color system signal or a YUV color system signal. It may also be a black-and-white image signal.

Further, the image signal is sent to the face detector 13 at a predetermined frame rate, for example, 30 frames / second, but may be another appropriate frame rate.

The face detector 13 is configured to detect a human face in the image pickup region photographed by the image pickup unit 12 based on the image signal sent from the image pickup unit 12.

The noise reducer 14 is configured to remove noise components from an image signal detected and identified by the face detector 13 from a region corresponding to a face in the imaging region (hereinafter referred to as a face region).

The pulse calculator 15 is configured to estimate a person's pulse rate and pulse interval based on an image signal from the face region, which has been noise-removed by the noise reducer 14.

The stress degree measuring device 16 is configured to measure the stress degree of a person to be measured based on the pulse interval obtained by the pulse calculator 15.

The arousal level measuring device 17 is configured to measure the arousal level of a person to be measured based on the pulse interval obtained by the pulse rate calculator 15.

As shown in FIG. 2, in the pulse detection device 10 according to the present embodiment, each of the face detector 13, the noise reducer 14, and the pulse calculator 15 can execute processing by a plurality of different methods.

Therefore, as shown in FIG. 3, the imaging unit 12 obtains the optimum combination according to the situation of the imaging area, the situation of people in the imaging area, the processing result in each functional block, and the like. It is possible to select the processing method in each functional block for the image signal.

The blocks such as face detection 1 and face detection 2 shown in FIG. 3 correspond to the processing methods 1 and 2 of the face detector shown in FIG.

(Preferable example of pulse detection method)
The pulse detection method according to the present embodiment will be described below.

FIG. 4 is a preferable example of the pulse detection flowchart according to the present embodiment.

(Step S10: Imaging step)
First, the imaging unit 12 photographs the surrounding area. The imaging region may be outdoors or indoors.

(Step S20: Face detection step)
The image signal acquired by the image capturing unit 12 is sent to the face detector 13, and based on this image signal, a human face in the imaging region is detected.

If the face detection fails, the process proceeds to step S42, which will be described later.

(Face detection step 1)
For example, when the distance between the image pickup unit 12 and the person to be imaged is short, the movement of the person is small, and a clear face image is obtained, a known method is used as the face detection method. Can be done.

As a known method, for example, a method of extracting a so-called Har-like feature amount and detecting a face by machine learning such as AdaBoost can be used.

In this case, the edges of the face are detected from a large number of local brightness differences in the image.

Using a large amount of face image data and non-face image data prepared in advance, a discriminator for discriminating whether the extracted Har-like feature amount corresponds to a face or not is generated.

After generating a plurality of classifiers, the data corresponding to the face is finally extracted from the image data by these majority logics, and the face area is output. An outline of such a technique is disclosed, for example, in the following papers (P. Viola, and M. Jones, "Rapid object detection using a boosted cascade of computer visions," Processing of Electrical and Pattern Engineers. and Pattern Recognition, p. 511. (2001)).

In addition, face image data and non-face image data can be obtained from a database such as OpenCV (http://opencv.org/).

The method is not limited to the above, and another known method may be used.

(Face detection step 2)
Another method can be used when the distance between the image pickup unit 12 and the person to be imaged is long, or when the movement of the person is large.

FIG. 5 is an example of a person's image acquired by the imaging unit according to the present embodiment.

FIG. 6 is a diagram showing an example of a face detection method according to the present embodiment.

FIG. 7 is a diagram showing an example of region division of an image according to the present embodiment.

FIG. 8 is a diagram showing an example of setting the face search range in the front and rear frames.

If the face is hidden or turned sideways, it may not be possible to detect the human face immediately using known methods.

In such a case, as shown in FIG. 6, the image is divided into regions, the average of the brightness values of the images is obtained in each of the divided regions, and the time change of these values is observed.

For example, since the light of the G component is greatly affected by the absorption of hemoglobin, if the image signal of the RGB color system is used, the periodic fluctuation related to the human pulse can be easily observed by using the brightness value of the G component. .. By observing this periodic fluctuation, it is possible to detect whether or not a human face is observed in the target area. It is also possible to grasp the outer shape of a human face by appropriately selecting the division size.

It is possible to observe periodic fluctuations related to a human pulse even in an image signal other than the RGB color system, and the luminance value used when detecting the face region is limited to the above wavelength component. is not.

Further, regarding the area division of the image, it is not always necessary to divide the image evenly as shown in FIG. 6, and the areas may overlap as shown in FIG. 7.

In this case, each of the divided regions has a different size, and the positions are also different and there are overlapping portions.

For example, if the regions are divided evenly, the two faces may enter the same region and it may not be possible to separate them. However, by dividing the regions randomly as shown in FIG. 7, these are separated. Accuracy is improved. However, if the area is set to allow duplication of areas in this way, there are many areas and it takes a lot of calculation time. Therefore, it is possible to shorten the calculation time by performing tracking processing and searching only around the result one frame before.

In FIG. 8, the frame shown by the solid line corresponds to the search range in the current frame, and the shaded range represents the face detection position and size one frame before.

As can be seen from FIG. 8, the search range in the current frame is often located near the face detection position one frame before, and the size of this range is often the same as that one frame before.

A known particle filter (particle filter) is used for the tracking process. There are three parameters, the position (x, y) of the search frame and the size L. As the likelihood of the particle filter, the magnitude of the largest peak when FFT (Fast Fourier Transform: Fast Fourier Transform) is performed is used as it is.

FFT is an example of a signal frequency analysis method.

By using this particle filter, the calculation time can be shortened and the tracking process can be performed without performing a full-screen search.

(Face detection step 3)
FIG. 9 is a diagram showing a time change of the brightness value of the image signal in the face region according to the present embodiment.

FIG. 10 is an example of a sine wave model for a pulse component according to the present embodiment.

FIG. 11 is another example of a sine wave model for a pulse component according to the present embodiment.

FIG. 12 is still another example of a sinusoidal model for pulse components according to this embodiment.

FIG. 13 is a diagram showing a time change of the brightness value of the image signal in the face region before and after the difference of the sine wave model according to the present embodiment.

As shown in FIG. 9, the image signal from the face region, in this case, the average luminance value of the G component, changes periodically with respect to the measurement time.

The face region can be detected by comparing this waveform with the sine wave model waveform shown below. This will be described in detail below.

First, as described above, the image is divided into regions, and the time change of the average luminance value is obtained in each of the divided regions.

Separately from this, a plurality of sine wave model waveforms represented by the equation (1) are prepared.

here,
Vm: maximum amplitude ω: angular frequency t: time θ: phase.

For example, a sine wave model waveform (FIG. 10) corresponding to a pulse waveform of 60 bpm (beats per minute) is used as a standard waveform, and a waveform shifted in phase as shown in FIG. 11 or an angular frequency is changed as shown in FIG. Create a waveform (corresponding to 90 bpm).

At this time, for example, a waveform in which the angular frequency is changed at intervals corresponding to 10 bpm and a waveform in which the phase is changed in units of 10 degrees are prepared in a table format.

As shown in FIG. 13, a sine wave model waveform prepared in advance is differentiated from the time-varying waveform of the brightness value actually obtained in each divided region.

For the maximum amplitude of the model waveform, the amplitude obtained from the periodic fluctuation of the actually obtained luminance value is used.

As shown in FIG. 13, the waveform shown by the dotted line is the waveform before the difference, and the waveform shown by the solid line is the waveform after the difference.

After the difference, it can be seen that the periodic fluctuation component seen in the original waveform is removed and the undulations are small. If the pulse represented by the sine wave model waveform and the pulse of the subject match, the waveform after the difference will have less undulations.

Since it is considered that the undulations are smaller if the standard deviation of the waveform after the difference is small, each difference is performed using the plurality of sinusoidal model waveforms described above, and the case where the standard deviation of the waveform after the difference is the lowest is the subject. It is assumed that it is a pulse of.

For example, if the calculated pulse rate is a value corresponding to a person's pulse, for example, 60 bpm to 120 bpm, the area is a face area, and if it is out of this range, the face does not exist.

The above range is just an example because the pulse of a person changes greatly due to exercise or the like.

In order to perform face detection from the periodic change of the brightness value of the image signal, it is necessary to retain a plurality of data for a dozen to several tens of seconds immediately before the detection. According to this method, the number of data to be retained can be reduced, and face detection can be performed in a shorter time.

Further, as described above, an appropriate face detection method differs depending on the situation of the captured image, particularly the situation of a person.

In the present embodiment, any one of face detection steps 1 to 3 can be arbitrarily and appropriately selected and used according to the imaging region or the situation of a person in the imaging region, and the detection result can be obtained in any of the steps. As, the face area is output.

(Step S30: Face position periodicity determination step / Step S40: Noise reduction step 1)
If the face detection is successful, then it is determined whether or not the face position is periodically moving (step S30). If there is no periodicity in the movement of the face position, the process proceeds to step SS41 described later.

When exercising such as jogging or exercising, the face moves, but since the movement component rides on the image signal, it becomes noise when calculating the pulse. Therefore, the change in the face position is frequency-analyzed, and the frequency is different from the observed value to reduce the noise (step S40).

In the present embodiment, the periodicity determination of the face position is performed by using the function of the face detector 13, but a functional block for determination may be provided separately.

FIG. 14 is a power spectrum obtained by frequency analysis of the position coordinates of the face.

FIG. 15 is a power spectrum obtained by differentiating the signal corresponding to the change in the face position from the luminance value of the image signal from the face region and further performing frequency analysis.

In the present embodiment, the position coordinates of the center point of the face, which is the position information of the face on the image, are subjected to FFT processing to obtain a power spectrum. In addition, since the position coordinates of the center point of the face may become unstable over time, frequency analysis is performed on the average position coordinates of the detected characteristic organs of the face, such as the eyes and nose, or those characteristic organ points. It is preferable to do so. Correspondingly, as shown in FIG. 1, the face detector 13 may have a face organ detection unit 13a.

The face detector 13 may have a face organ detection unit 13a regardless of whether or not the face position coordinates are frequency-analyzed.

It can be seen that the positions surrounded by the dotted lines in FIGS. 14 and 15 are the peaks corresponding to the movement of the face, and the face positions change with a certain periodicity. In this example, there is a peak near 1.14 Hz.

Further, in FIG. 15, the presence of peaks can be seen around 1.14 Hz and around 1.28 Hz. The former is a component corresponding to the movement of the face position, and the latter is a component corresponding to the pulse.

Actually, the range that can be taken as a pulse is set to be on the right side of the alternate long and short dash line in FIG. 15, that is, on the high frequency side, and the peak of the highest frequency within that range is estimated as the pulse.

As shown in FIG. 15, the peak caused by facial movement and the peak caused by pulse exist at close positions on the frequency. Therefore, if the frequency component due to the change in the face position is not calculated in advance, the former peak may be erroneously detected as a pulse.

According to the present embodiment, in order to prevent this, by obtaining the power spectrum obtained by frequency analysis of the face position coordinates in advance, it is possible to prevent erroneous measurement of the pulse due to the fluctuation of the face position due to exercise or the like. ..

Further, the power spectrum shown in FIG. 14 can be obtained by performing a difference using a sine wave model instead of directly different from the brightness value of the image signal from the detected face region.

FIG. 16 is a diagram showing a sine wave model corresponding to the frequency of facial movement obtained in FIG.

FIG. 17 is a diagram showing a time change of the brightness value of the image signal from the face region.

Here, the luminance value of the image signal from the face region is the average value of the luminance values from the region detected as the face region.

The noise component due to the movement of the face is removed by differentiating the sine wave waveform shown in FIG. 16 from the time-varying waveform of the brightness value shown in FIG.

(Step S50: Pulse calculation step 1)
Next, frequency analysis such as FFT is performed on the brightness value of the image signal from the face region where noise is reduced, and the pulse is estimated and calculated from the obtained signal. Furthermore, the stress level and alertness level of a person are calculated using the pulse interval.

FIG. 18 is a diagram showing a time change of the brightness value of the image signal from the face region after noise reduction.

As shown in FIG. 18, the pulse interval can be obtained from the time change of the brightness value of the image signal from the face region before the frequency analysis.

(Step S60: Stress degree and alertness measurement step)
Using the pulse interval obtained in step S50, the degree of stress and / or the degree of arousal of the person to be imaged is measured (step S60).

Known methods are used to measure stress and alertness.

From the time series data of the pulse interval, the power spectral density is calculated using, for example, an autoregressive model, the frequency component thereof is analyzed, and the stress degree is measured.

In the power spectral density obtained by the above method, the region corresponding to the frequency from 0.05 Hz to 0.15 Hz is defined as the LF (Low Frequency) component region, and the frequency is set to the frequency from 0.15 Hz to 0.40 Hz. The corresponding region is defined as the HF (High Frequency) component region, and the integrated intensity in each region is obtained.

In general, it is said that the signal in the LF component region corresponds to the blood pressure fluctuation, and the signal is generated in any case when the sympathetic nerve is tense or the parasympathetic nerve is tense.

On the other hand, it is said that the signal in the HF component region corresponds to the respiratory fluctuation, and the signal intensity increases when the parasympathetic nerve is dominant.

The stress level can be obtained by using (signal strength in the LF component region) / (signal strength in the HF component region) as a stress index (sympathetic nerve activity).

When relaxed, the parasympathetic nerves dominate and the signal strength in the HF component region increases and the stress index decreases, whereas when tense, the signal strength in the HF component region is relatively high. Decreases and the stress index rises.

Similarly, the arousal level can be evaluated from the activity of the sympathetic nerve by obtaining the signal strength in the LF component region and the signal strength in the HF component region.

Regarding the arousal level, it has been reported that whether or not the arousal state is reflected is reflected in the signal intensity of the HF component region. For example, the arousal level can be estimated by paying attention to this intensity.

(Step S41: Noise reduction step 2)
In step S30, when the movement of the face position does not show periodicity, the signal corresponding to the pulse component and the other signals are separated from the image signal from the face region, and the signal corresponding to the pulse component is used. Noise is reduced by differentiating other signals.

In the case of an RGB color-based image signal, the luminance value of the G component can be used as the pulse component, and the luminance values of the other components can be used as the noise component.

At this time, as the noise component, the brightness value of the R component, the brightness value of the B component, or the average value of the brightness values of the R component and the B component can also be used.

Further, in the case of an image signal of HSV color system or YUV color system, the brightness value of one primary color component is used as a pulse component, and the brightness value of another primary color component, for example, the brightness value of one component or two components. The average value of the brightness values of may be used as a noise component.

In any case, the luminance value of a component other than the primary color system, for example, a near infrared light component may be used as a noise component.

Further, when a black-and-white image signal or the like is used, noise can be reduced by a differential calculation with a moving average.

For example, if the time-series data of the luminance value of the image signal is A and the time-series data obtained by performing a moving average with respect to A is B, noise can be reduced by differentiating A from B. ..

(Step S51: Pulse calculation step 1 / Step S61: Stress degree and alertness measurement step)
When step S41 is completed, the process proceeds to step S51 and step S61 in sequence.

The pulse estimation and calculation methods in step S51 and the stress degree and alertness measurement methods in step S61 are the same as those shown in steps S50 and S60, respectively.

(Step S42: Skin area detection step)
FIG. 19 is a diagram showing an example of a skin region detection method according to the present embodiment.

If the face detection fails in step S20, the process proceeds to step S42.

Here, when the face detection fails, it is expected that the face is hidden or the face is not oriented toward the imaging unit 12. This also applies when the face area in the captured image is extremely small.

In such a case, as described in step S20, the image is divided into regions, the average value of the brightness values of the images is obtained in each of the divided regions, and the time change of this value is observed.

For example, it is easy to observe periodic changes related to the pulse from areas other than the face where the skin is exposed. By observing this periodic change, it is possible to detect whether or not the human skin is exposed in the target area.

In this case as well, the detection sensitivity can be improved by appropriately setting the division size of the image area.

Further, regarding the area division of the image, it is not always necessary to divide the image evenly as shown in FIG. 19, and the areas may overlap as shown in FIG. 7.

Further, as described with reference to FIG. 8, the calculation time can be shortened by performing the tracking process using the particle filter and searching only the periphery of the result one frame before.

By using this particle filter, it becomes possible to cope with changes in size due to hiding of the face or back and forth.

Further, the detection of the skin area in this step is performed by using the function of the face detector 13.

(Step S52: Pulse calculation step 2)
By the method used in the face detection step 3 in step S20, the pulse can be estimated and calculated from the image signal from the skin region.

In this case, preparing a plurality of sine wave model waveforms represented by the equation (1) is the same as in the case of face detection step 3, but the frequency and phase steps need to be made finer, for example, in 1 bpm steps. Prepare the waveform changed in step 1 and the waveform whose phase is changed in units of several degrees in a table format, select an appropriate waveform from the table created in this step, and perform in face detection step 3. Used for face detection.

As shown in FIG. 13, a sine wave model waveform prepared in advance is differentiated from the actually obtained luminance value.

For the maximum amplitude of the model waveform, a value close to the amplitude obtained from the periodic fluctuation of the actually obtained luminance value is used.

In FIG. 13, the waveform shown by the dotted line is the waveform before the difference, and the waveform shown by the solid line is the waveform after the difference.

After the difference, it can be seen that the periodic fluctuation component seen in the original waveform is removed.

If the pulse represented by the sine wave model waveform and the pulse of the subject match, the waveform after the difference will have less undulations.

Since it is considered that the undulations are smaller if the standard deviation of the waveform after the difference is small, each difference is performed using the plurality of sinusoidal model waveforms described above, and the case where the standard deviation of the waveform after the difference is the lowest is the subject. It is assumed that it is a pulse of.

According to this method, it is not necessary to hold multiple data for a certain period of time as compared with the case of obtaining the heart rate from the peak of the spectrum by frequency analysis, and the influence on the pulse calculation when the pulse change occurs during the period is suppressed. it can.

(Step S62: Stress degree and alertness measurement step)
When step S52 is completed, the process proceeds to step S62.

The method for measuring the stress level and the arousal level in step S62 is the same as that shown in step S60.

As described above, according to the present embodiment, in various cases where there is movement of a person, the face is hidden, or the face does not face the direction of the image pickup unit 12 in the image pickup region. However, in each functional block, by selecting an appropriate processing method and performing pulse detection processing, it is possible to directly obtain a person's pulse, stress level, and alertness level from the image acquired by the imaging unit 12. it can.

The pulse detection method in the present embodiment is not limited to the method shown in FIG. 4, and as shown in FIG. 3, the situation of the imaging region, the situation of a person in the imaging region, and each functional block. The pulse detection can be performed by combining various processing methods prepared in advance according to the processing result and the like.

In the present embodiment, the target of pulse detection is a human being, but the pulse detection according to the present invention can be applied to a living body other than a human such as an animal.

(Hardware configuration)
Function of the pulse detection device 1 0 shown in FIG. 1, except for part of the imaging unit 12 can be realized generally in software. In this case, for example, the pulse detection is performed by executing the program corresponding to the pulse detection flow shown in FIG. 4 on the hardware. The hardware configuration for realizing this function is shown in FIG.

FIG. 20 is a diagram showing an example of the hardware configuration of the pulse detection system according to the present embodiment.

The pulse detection system 100 includes a computer 110, a video camera 102, a display 111, an operation unit 112, and a system bus 101.

The computer 110, the video camera 102, the display 111, and the operation unit 112 are connected to the system bus 101, and control signals and data are exchanged via the system bus 101.

The video camera 102 photographs a predetermined boundary area around it, and inputs the captured moving image data as an image signal to the computer 110 via the system bus 101.

The video camera 102 corresponds to the image pickup unit 12 shown in FIG. 1, and has an image pickup element such as a CMOS image sensor and an optical system including a lens and the like, although not shown.

The image signal input to the computer 110 is subjected to various processing by the computer 110, and the obtained result is displayed on the display 111. There are various objects to be displayed, not only the pulse rate, pulse interval, stress level and alertness level, but also the detected face area, which is the result of the pulse detection process, and the real-time change waveform of the brightness value of the image signal. Can be displayed.

The display 111 may have an audio output function, if necessary.

The operation unit 112 is an input device represented by a keyboard, a mouse, a joystick, or the like. For example, when starting or stopping the computer 110, calling or executing a pulse detection program, selecting a processing result to be displayed on the display 111, or the like. Used.

For example, when the display 111 is a touch panel, or when the pulse detection system 100 separately includes a voice input unit (not shown) for inputting an operation command or the like, the operation unit 112 is particularly provided. It does not have to be.

The computer 110 has a CPU 103, a signal processing unit 104, a communication unit 105, a ROM 106, a RAM 107, and an HDD 108 as components, and these elements are connected to the system bus 101 to form a system. Control signals and data are exchanged via the bus 101.

For example, the program corresponding to the pulse detection flow shown in FIG. 4 is stored in the HDD 108 in advance, and the CPU 103 executes the program read instruction according to the operation by the operation unit 112. The program read from the HDD 108 is stored in the RAM 107 according to an instruction from the CPU 103. A table of sine wave model waveforms and the like are also read from the HDD 108 and stored in the RAM 107.

The program or table read from the HDD 108 may be stored in the ROM 106 depending on its size. Once the reading from the HDD 108 is performed, the program can be directly stored in the RAM 107 from the ROM 106 thereafter.

The program stored in the RAM 107 is executed according to the instruction from the CPU 103. The CPU 103 interprets the instructions from the program and performs various data processing and control.

The signal processing unit 104 according to an instruction from the CPU 103, performs processing such as extraction of the region division and the face region of the image.

The communication unit 105 exchanges data with the outside via a physical interface (not shown) or the like. For example, the database or the like used for face detection is stored in the HDD 108 or the ROM 106 via the communication unit 105. It is possible to communicate with the outside by wired communication or wireless communication using an optical fiber or the like, and the communication method is appropriately determined by the specifications of the system and the computer.

The program itself can be read from an external source via the communication unit 105, and the processing result of the program can also be stored in the external source via the communication unit 105. In this case, it is not particularly necessary to provide the HDD 108 only for storing the program and the processing result.

Further, instead of or separately from the HDD 108, a portable recording medium read / write unit may be provided. A program, a table of sine wave model waveforms, or the like may be stored in this recording medium, and the processing result may be written directly to the recording medium.

If the recording medium is an optical disk, an optical disk drive may be provided, if it is an IC card, an IC card reader / writer may be provided, and if it is a USB memory, a USB terminal connection port may be provided.

The above configuration is just an example, and can be changed as appropriate according to the imaging target and location, the content of the pulse detection program, and the like.

For example, when using a smartphone having a built-in camera or a notebook computer having a built-in camera, it is possible to realize the pulse detection system 100 with one device.

(Embodiment 2)
When attempting to detect a pulse using a surveillance camera or a camera installed in an electronic device, the following problems may occur.

First, in order to acquire an image, it is necessary to always point the camera in the direction of the person to be photographed, but the subject may feel uncomfortable because the feeling of being constantly monitored is strong. In addition, if the communication system of the camera is hacked, the daily life image of the person to be photographed may be unintentionally leaked, and privacy may be infringed.

Therefore, it is conceivable to make the lens of the camera a special type, for example, a lenticular type, or to arrange a diffuser member such as frosted glass in front of the lens to make the image unclear.

However, when the image is unclear, if the person is moving, it is possible to detect the person from the movement, but in the scene where the person is relaxing in the living room at home, the movement of the person is small, so the person cannot be detected. .. In situations where humans cannot be detected, it is difficult to detect the face and skin areas.

Therefore, in the present embodiment, a configuration is disclosed in which a person can be detected before face detection or skin area detection, and pulse detection can be performed reliably.

FIG. 21 is a block diagram showing a functional configuration of the pulse detection device according to the present embodiment.

FIG. 22 is a diagram showing an example of the processing function of the human detector.

FIG. 23 is a block diagram showing a signal flow in the pulse detection device.

The difference between the configuration shown in the present embodiment and the configuration shown in the first embodiment is that the diffuser portion 11 shown in FIG. 1 is provided and the image signal from the imaging unit 12 is sent to the human detector 20. The point is that after detecting the presence or absence of a person in the image, the pulse is detected using the configuration of the face detector 13 or later.

As described above, the light diffuser 11 scatters the light incident from the surrounding region by making the lens of the image pickup unit 12 a special type, for example, a lenticular type, or by arranging a light diffuser such as frosted glass in front of the lens. And / or has the function of dimming.

The diffuser portion 11 may be incorporated in the video camera 102 or may be separately provided in the hardware configuration shown in FIG.

In the pulse detection device 10 according to the present embodiment, as shown in FIG. 2, each of the face detector 13, the noise reducer 14, and the pulse calculator 15 can execute processing by a plurality of different methods. Further, as shown in FIG. 22, the human detector 20 can perform the process in a plurality of different ways.

Therefore, as shown in FIG. 23, the imaging unit 12 obtains the optimum combination according to the situation of the imaging area, the situation of people in the imaging area, the processing result in each functional block, and the like. It is possible to select the processing method in each functional block for the image signal.

It should be noted that the blocks such as face detection 1 and face detection 2 shown in FIG. 3 correspond to the processing methods 1 and 2 of the face detector shown in FIG. 2 as in the first embodiment. Further, the blocks of the person detection 1 and the person detection 2 shown in FIG. 23 correspond to the processing methods 1 and 2 of the person detector shown in FIG.

The pulse detection method in the present embodiment will be described with reference to FIG. 24. The steps described in the first embodiment are limited to the minimum necessary range.

FIG. 24 is a preferable example of the pulse detection flowchart according to the present embodiment.

(Step S100: Diffuse step / Imaging step)
Light from the surrounding region incident on the imaging unit 12 is scattered and / or dimmed by the diffuser 11. In this state, the imaging unit 12 photographs the surrounding area. The imaging region may be outdoors or indoors.

(Step S110: Person detection step)
The image signal acquired by the image capturing unit 12 is sent to the face detector 13, and based on this image signal, a human face in the imaging region is detected.

If the person detection fails, it is assumed that there is no person in the imaging region, and the process returns to step S100 to continue shooting.

(People detection step 1)
FIG. 25 is an example of a person's image acquired by the imaging unit according to the present embodiment.

FIG. 26 is a diagram showing an example of a person detection method according to the present embodiment.

In FIG. 25, the figure on the left is an example of an image when a lenticular type lens is used, and the figure on the right is an example of an image when an image is taken through frosted glass. In either case, the image is unclear, and as it is, for example, the Har-like feature amount cannot be extracted.

Therefore, in step S20 of the pulse detection flow in the first embodiment, as described in the face detection step 2, the captured image is divided into regions, and the average value of the brightness values of the images is calculated in each of the divided regions. Obtain and observe the time change of this value.

Even in an unclear image, it is easy to observe periodic changes related to the pulse from the area where the face and skin are exposed. By observing this periodic change, it is possible to detect whether or not the human face or skin is exposed in the target area.

Further, the detection sensitivity can be improved by appropriately setting the division size of the image area.

Regarding the area division of the image, it is not always necessary to divide the image evenly as shown in FIG. 19, and the areas may overlap as shown in FIG. 7.

Further, as described with reference to FIG. 8, the calculation time can be shortened by performing the tracking process using the particle filter and searching only the periphery of the result one frame before.

By using this particle filter, it becomes possible to cope with changes in size due to hiding of the face or back and forth.

(People detection step 2)
In step S20 of the pulse detection flow in the first embodiment, as described in the face detection step 3, the image is divided into regions, and the time-varying waveform and the sine wave model waveform of the average luminance value in each of the divided regions. By comparing with, the human area can be detected.

In order to perform face detection from the periodic change of the brightness value of the image signal, it is necessary to retain a plurality of data for a dozen to several tens of seconds immediately before the detection. According to this method, the number of data to be retained can be reduced, and people can be detected in a shorter time.

In addition, an appropriate person detection method differs depending on the situation of the captured image, particularly the situation of a person. For example, when a person is moving, it is preferable to be able to detect the person in a short time.

In the present embodiment, the person detection step 1 or 2 can be arbitrarily and appropriately selected and used depending on the situation of the person in the imaging power range.

The person detector 20 may be provided with a motion detection unit 20a. In this case, the area where a person is present is determined by integrating the information about the motion in the imaging region detected by the motion detection unit 20a and the information from the brightness value of the image.

In either case, the presence or absence of a person can be detected, and as the detection result, the area where the person exists, specifically, the number of people and the position in the imaging area is output.

(Step S120: From face detection step to step S160: Stress degree and alertness measurement step)
If the person is successfully detected in step S110, the process proceeds to face detection step S120. If the face detection is successful, it is determined whether or not the face position is moving periodically (step S130), and if the face position is moving periodically, the noise reduction step S140, the pulse calculation step S150, and the stress The process proceeds to step S160 for measuring the degree and arousal.

The face detection method in step S120, the face position periodicity determination method in step S130, the noise reduction method in step S140, the pulse estimation and calculation method in step S150, and the stress degree and alertness measurement method in step S160 are the first embodiment. It is the same as that shown in step S20, step S30, step S40, step S50 and step S60 shown in the pulse detection flow in.

(From step S141: noise reduction step to step S161: stress degree and alertness measurement step)
If no periodicity is found in the movement of the face position in step S130, the process proceeds sequentially to noise reduction step S141, pulse calculation step S151, stress degree, and alertness measurement step S161.

The noise reduction method in step S141, the pulse estimation and calculation method in step S151, and the stress degree and alertness measurement method in step S161 are shown in steps S41, S51 and S61 shown in the pulse detection flow in the first embodiment. It is the same as the one.

(Step S142: From noise reduction step to step S162: Stress degree and alertness measurement step)
If the face detection fails in step S120, the process proceeds sequentially to the skin area detection step S142, the pulse calculation step S152, the stress degree, and the arousal degree measurement step S162.

The skin region detection method in step S142, the pulse estimation and calculation method in step S152, and the stress degree and alertness measurement method in step S162 are described in steps S42, S52 and S62 shown in the pulse detection flow in the first embodiment. It is similar to the one shown.

As described above, according to the present embodiment, while protecting the privacy of the person to be photographed, there is movement of the person, the face is hidden, or the face is the image pickup unit 12 in the image pickup area. Even in various cases where the image is not oriented in the above direction, by selecting an appropriate processing method and performing pulse detection processing in each functional block, directly from the image acquired by the imaging unit 12, It is possible to obtain the pulse, stress, and alertness of a person.

The pulse detection method in the present embodiment is not limited to the method shown in FIG. 24, and as shown in FIG. 23, the situation of the imaging region, the situation of a person in the imaging region, and each functional block. The pulse detection can be performed by combining various processing methods prepared in advance according to the processing result and the like.

In the present embodiment, the target of pulse detection is a human being, but the pulse detection according to the present invention can be applied to a living body other than a human such as an animal.

(Other embodiments)
In addition to the above-described configuration, other configurations may be adopted. For example, in the first embodiment, a configuration in which the diffuser portion 11 is provided may be adopted.

By providing the light diffusing unit 11, the incident light on the imaging unit 12 is scattered or dimmed, so that the obtained image becomes an unclear image, but it is possible to avoid identifying a person directly from the image information. This applies to examinations such as pulse when it is clear that the target person is in front of the camera. For example, when it is necessary to particularly protect the privacy of a patient in a pulse or stress degree test in a hospital laboratory, the pulse detection device or pulse detection method of the present invention may be applied.

Further, in the second embodiment, a configuration in which the diffuser portion 11 is not provided can be adopted.

In this case, for example, in the image pickup unit 12, the lens is mechanically moved to shift the imaging focus to make the screen unclear, so that the diffuser unit 11 becomes unnecessary and the privacy of the person to be photographed. You can also protect.

However, in order to know whether or not the photographed object is a living body such as a human being, it is necessary to make a determination by the human detector 20.

In the first, second, and other embodiments, the light source at the time of photographing may be natural light or artificial light such as a fluorescent lamp or LED lighting. For example, the wavelength component used for pulse calculation and the wavelength component used for noise reduction can be appropriately changed according to the wavelength range of the light source.

Further, each functional block may be realized by wired logic except for a hardware configuration such as an image sensor and an optical system in the image pickup unit 12. It can also be realized by a dedicated LSI or a dedicated block in the LSI.

The pulse detection device of the present invention is acquired by the imaging unit even in various cases where there is a movement of a person, the face is hidden, or the face is not facing the direction of the imaging unit in the imaging region . It is extremely useful because the pulse, stress, and arousal of a person can be obtained directly from the image.

10 Pulse detector 11 Diffuse unit 12 Imaging unit 13 Face detector 13a Face organ detector 14 Noise reducer 15 Pulse calculator 16 Stress level measuring device 17 Arousal level measuring device 20 Human detector (biological detector)
20a Motion detection unit 100 Pulse detection system 101 System bus 102 Video camera 103 CPU
104 Signal processing unit 105 Communication unit 106 ROM
107 RAM
108 HDD
110 Computer 111 Display 112 Operation unit

Claims (21)

It is a pulse detection device that detects the pulse of a living body.
An imaging unit that captures a predetermined imaging region and acquires an image signal,
A face detector that detects the face region or skin region of the living body based on the image signal, and
A noise remover that removes noise from the detected image signal from the face area or the skin area,
A pulse calculator that calculates a pulse from the image signal from the face region or the skin region from which noise has been removed, and a pulse calculator.
A pulse detection device including a light-diffusing unit that scatters and / or dims the incident light to the image pickup unit to obscure the captured image . The face detector, the noise remover, and the pulse calculator each correspond to a plurality of processing methods, and one of the plurality of processing methods can be used depending on the imaging region and the condition of the living body. The pulse detection device according to claim 1, wherein the processing method is selected and the processing is executed. The face detector
The pulse detection apparatus according to claim 2, wherein a local brightness difference in the brightness value of the image signal in the imaging region is detected, and the face region is detected by a majority logic based on the brightness difference. .. The face detector
The pulse detection device according to claim 2, wherein an image in the imaging region is divided, and the face region or the skin region is detected based on the image signal from each of the divided regions. The face detector
Based on the result obtained by dividing the image in the imaging region and comparing the time-varying waveform of the image signal from each divided region with the model waveform represented by a sine wave, the face region or the said The pulse detection device according to claim 2, which is configured to detect a skin area. The noise remover
The pulse according to claim 2, wherein a specific frequency component of a power spectrum obtained by frequency-analyzing the position information of the face region in the imaging region is removed from an image signal from the face region. Detection device. The noise remover
The pulse detection according to claim 2, wherein at least a part of the image signal other than the wavelength component used for detecting the pulse component is removed from the image signal from the face region or the skin region. apparatus. The pulse calculator
The pulse detection device according to claim 2, wherein the image signal from the face region or the skin region after the noise is removed is frequency-analyzed to calculate the pulse of the living body. The pulse calculator
The pulse detection according to claim 2, wherein the pulse detection of the living body is configured to compare the time-varying waveform of the image signal from the face region or the skin region with the model waveform represented by a sine wave. apparatus. It further has a biological detector that detects the presence or absence of the living body based on the periodic fluctuation component of the image signal.
The pulse detection according to claim 1, wherein the face detector is configured to detect the face region or the skin region based on an image signal from the existing region of the living body obtained by the biodetector. apparatus. The biodetector is compatible with a plurality of processing methods, and one of the plurality of processing methods is selected from the plurality of processing methods according to the imaging region and the condition of the living body to execute the processing. The pulse detection device according to claim 10, which is configured. The biodetector
The pulse detection device according to claim 11, wherein an image in the imaging region is divided and the presence or absence of the living body is detected based on the image signal from each of the divided regions. The biodetector
The image in the imaging region is divided, and the time-varying waveform of the image signal from each divided region is compared with the model waveform represented by a sine wave to detect the presence or absence of the living body. The pulse detection device according to claim 11. The pulse detection device according to any one of claims 1 to 3 , further comprising a stress measuring device for measuring the stress degree of the living body based on the pulse interval obtained from the pulse calculator. Based on the pulse interval obtained from said pulse calculator, further comprising wakefulness measuring device for measuring the degree of awakening of the subject, the pulse detection device according to any one of claims 1 to 1 4. It is a pulse detection method that detects the pulse of a living body.
An imaging step of imaging a predetermined imaging region and obtaining an image signal,
A face detection step for detecting the face region or skin region of the living body based on the image signal, and
A noise removal step for removing noise from the detected image signal from the face area or the skin area,
A pulse calculation step of calculating a pulse from an image signal from the face region or the skin region from which noise has been removed, and a pulse calculation step.
A pulse detection method comprising a light-diffusing step of scattering and / or dimming light from the imaging region and inputting it into an imaging unit to blur a captured image . The face detection step, the noise removal step, and the pulse calculation step each correspond to a plurality of processing methods, and one of the plurality of processing methods is described according to the imaging region and the condition of the living body. The pulse detection method according to claim 16 , wherein the processing method is selected and the processing is executed. A biological detection step of detecting the presence or absence of the living body based on the periodic fluctuation component of the image signal is further provided.
In the face detection step
The pulse detection method according to claim 17 , wherein the face region or the skin region is detected based on an image signal from the existing region of the living body obtained by the biological detection step. The biological detection step corresponds to a plurality of processing methods, and one said processing method is selected from the plurality of processing methods according to the imaging region and the situation of the living body, and the processing is executed. Item 8. The pulse detection method according to Item 18 . The pulse detection method according to any one of claims 16 to 19 , further comprising a stress measurement step of measuring the stress degree of the living body based on the pulse interval obtained in the pulse calculation step. On the basis of the pulse interval obtained in the pulse calculating step, further comprising the alertness measurement step of measuring alertness of the living body, pulse detection method according to any one of claims 1 6 to 2 0.