JP2018068720A - Pulse detector and pulse detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a pulse detector capable of detecting a pulse according to a condition of a person in an image.SOLUTION: A pulse detector 10 comprises as function blocks, an imaging part 12, a face detector 13, a noise reduction tool 14, a pulse calculation tool 15, a stress degree measurement tool 16, and a vigilance measurement tool 17. In the face detector 13, the noise reduction tool 14, the pulse calculation tool 15, processing can be executed by selecting a processing method from plural different processing methods. According to a condition of a person in an imaging area or a processing result in each function block, a processing method in each function block is selected so as to achieve an optimal combination, for detecting pulse.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for measuring a pulse and a stress level from an image acquired by a camera.

  2. Description of the Related Art Conventionally, a method for estimating and calculating a person's heart rate and pulse from an image acquired by a camera in a non-contact manner is known.

  For example, after calculating each average value of RGB in the face image area and processing by independent component analysis (ICA), the heart rate is estimated from the peak frequency obtained by frequency analysis of one component waveform. The method is known (for example, refer nonpatent literature 1).

  In addition, in order to track one component waveform after ICA processing in time series, a method of performing pairing with the current time component waveform and further tracking the peak frequency representing the heartbeat in time series to estimate the heart rate Is known (see, for example, Patent Document 1).

JP 2012-239661 A

M.M. Z. Poh, D .; J. et al. McDuff, and R.M. W. Picard “Non-contact, Automated Cardiac Pulse Measurements Using Video Imaging and Blind Source Separation”, Optics Express, (US), May 7, 2010. 18, no. 10, pp. 10762-10774

  However, the methods disclosed in Patent Document 1 and Non-Patent Document 1 have a problem in that the detection accuracy of heartbeats, pulses, and the like decreases due to the movement of a person.

  Depending on the application, in order to protect people's privacy, an intentionally blurred image may be acquired. In this case, however, detection of heartbeats and pulses is detected by face detection failure. There was a problem that the accuracy decreased. In particular, when a person is stationary, the face area cannot be specified using motion detection or the like, and thus the detection accuracy is significantly reduced.

  The present invention has been made in view of this point, and an object of the present invention is to provide an apparatus capable of detecting a pulse with high accuracy regardless of a situation of an imaging target and imaging conditions.

  In order to achieve the above object, according to the present invention, a face detection function from a captured image to pulse detection, a noise reduction function, and a pulse calculation function depending on the situation of the imaging region and the situation of the photographed person. Can be selected from among different processing methods.

  Specifically, the pulse detection device of the present invention is a pulse detection device that detects a pulse of a living body, and captures a predetermined imaging region and acquires an image signal, and a living body based on the image signal. From a face detector that detects a face area or skin area, a noise remover that removes noise from an image signal from the detected face area or skin area, and an image signal from the face area or skin area from which noise has been removed And a pulse calculator for calculating a pulse.

  According to this configuration, a pulse can be directly obtained from an image acquired by an imaging unit such as a camera.

  Each of the face detector, noise remover, and pulse calculator corresponds to a plurality of processing methods, and selects one processing method from a plurality of processing methods according to the imaging region and the state of the living body. , Preferably configured to perform processing.

  According to this configuration, even in a case where there is a movement of a living body such as a person, a face is hidden, or a face is not facing the direction of the imaging unit in the imaging area, imaging such as a camera is performed. The pulse can be obtained from the image acquired by the section.

  The face detector is preferably configured to detect a local light / dark difference in the luminance value of the image signal in the imaging region and detect the face region by majority logic based on the light / dark difference.

  According to this configuration, a face can be detected by a simple method when a clear image is acquired.

  The face detector may be configured to divide an image in the imaging region and detect a face region or a skin region based on an image signal from each divided region.

  According to this configuration, even when a plurality of living bodies are photographed, the face area or the skin area can be detected by a simple method.

  The face detector divides the image in the imaging region, and based on the result obtained by comparing the time-varying waveform of the image signal from each divided region with the model waveform represented by the sine wave, Or you may be comprised so that the said skin area | region may be detected.

  According to this configuration, the data holding time for face detection can be shortened, and face detection can be performed in a short time. Further, even when the face of a living body such as a person is hidden or the face area in the image is small, the skin area can be detected.

  The noise remover is preferably configured to frequency-analyze position information of the face area in the imaging area and remove a specific frequency component of the obtained position information from the image signal from the face area.

  According to this configuration, it is possible to appropriately remove noise caused by the movement of the face that is close to the pulse component in the frequency domain.

  The noise remover may be configured to remove at least a part of the image signal other than the wavelength component used for detecting the pulse component from the image signal from the face region or the skin region.

  According to this configuration, noise other than the pulse component included in the image signal can be appropriately removed with a simple configuration.

  The pulse calculator is preferably configured to frequency-analyze the image signal from the face area or skin area after the noise is removed to calculate the pulse of the living body.

  The pulse calculator may be configured to calculate a pulse of a living body by comparing a time-change waveform of an image signal from a face region or a skin region with a model waveform represented by a sine wave.

  According to this configuration, the data holding time for pulse calculation can be shortened, and the pulse can be calculated in a short time.

  The pulse detection device further includes a living body detector that detects the presence or absence of a living body based on a periodic variation component of the image signal, and the face detector converts the image signal from the living body obtained by the living body detector into an image signal. It is preferably configured to detect a face region or a skin region based on it.

  According to this configuration, even when a face cannot be detected immediately, detection of a face region or a skin region can be facilitated by detecting a living body such as a person, and pulse detection can be performed.

  The living body detector corresponds to a plurality of processing methods, and is configured to select one processing method from the plurality of processing methods and execute the processing according to the imaging region and the state of the living body. preferable.

  According to this configuration, the presence / absence of a living body can be detected even in various cases where a living body such as a person moves, a face is hidden, or a face does not face the direction of the imaging unit in the imaging region. It can be detected reliably and the pulse can be detected.

  The living body detector may be configured to divide the image in the imaging region and detect the presence or absence of the living body based on the image signal from each divided region.

  According to this configuration, the presence or absence of a living body can be detected by a simple method even when a plurality of living bodies are photographed.

  The living body detector is configured to divide the image in the imaging region, and compare the time-varying waveform of the image signal from each divided region with the model waveform represented by a sine wave to detect the presence or absence of the living body. It may be.

  According to this configuration, the data holding time for the living body detection can be shortened, and the living body detection can be performed in a short time. Further, even when the face of a living body such as a person is hidden or the face area in the image is small, the presence or absence of the living body can be detected.

  It is preferable to further include a diffuser that scatters and / or attenuates light incident on the imaging unit.

  According to this configuration, by making the captured image unclear, it is possible to obtain a pulse from the same image while protecting the privacy of the person who is the imaging target.

  It is preferable to further include a stress measuring device for measuring the stress level of the living body based on the pulse interval obtained from the pulse calculator.

  According to this configuration, the degree of stress can be directly obtained from an image acquired by an imaging unit such as a camera.

  It is preferable to further include a wakefulness measuring device for measuring the wakefulness of the living body based on the pulse interval obtained from the pulse calculator.

  According to this configuration, the arousal level can be obtained directly from an image acquired by an imaging unit such as a camera.

  The pulse detection method of the present invention is a pulse detection method for detecting a pulse of a living body, imaging a predetermined imaging region and obtaining an image signal, and a facial region or skin region of the living body based on the image signal. A face detection step to detect, a noise removal step to remove noise from the image signal from the detected face area or skin area, and a pulse calculation to calculate a pulse from the image signal from the face area or skin area from which the noise has been removed And a step.

  According to this method, a pulse can be directly obtained from an image acquired by an imaging unit such as a camera.

  Each of the face detection step, the noise removal step, and the pulse calculation step corresponds to a plurality of processing methods, and selects one processing method from the plurality of processing methods according to the imaging region and the state of the living body, It is preferable to execute the processing.

  According to this configuration, even in a case where there is a movement of a living body such as a person, a face is hidden, or a face is not facing the direction of the imaging unit in the imaging area, imaging such as a camera is performed. The pulse can be obtained from the image acquired by the section.

  As described above, according to the pulse detection device of the present invention, for various cases where there is a movement of a person, a face is hidden, or a face is not facing the direction of the imaging unit in the imaging region. In addition, it is possible to directly determine a person's pulse, stress level, and arousal level from an image acquired by the imaging unit.

1 is a block diagram showing a functional configuration of a pulse detection device according to Embodiment 1. FIG. It is a figure which shows the example of the processing method in each functional block in a pulse detection apparatus. FIG. 3 is a block diagram illustrating a signal flow in the pulse detection device according to the first embodiment. 3 is a preferred example of a pulse detection flowchart according to the first embodiment. 3 is an example of an image of a person acquired by the imaging unit according to Embodiment 1. 6 is a diagram illustrating an example of a face detection method according to Embodiment 1. FIG. 6 is a diagram illustrating an example of region division of an image according to Embodiment 1. FIG. It is a figure which shows an example of the setting of the face search range in the front and back frames. FIG. 6 is a diagram illustrating temporal changes in luminance values of image signals in a face area according to the first embodiment. 3 is an example of a sine wave model for a pulse component according to the first embodiment. It is another example of the sine wave model with respect to the pulse component based on Embodiment 1. FIG. It is another example of the sine wave model with respect to the pulse component based on Embodiment 1. FIG. FIG. 6 is a diagram illustrating a temporal change in luminance value of an image signal in a face area before and after a sine wave model difference according to the first embodiment. It is a power spectrum obtained by frequency analysis of the face position coordinates. This is a power spectrum obtained by subtracting a signal corresponding to a change in the face position from the luminance value of the image signal from the face area and further performing frequency analysis. It is a figure which shows the sine wave model corresponding to the frequency of the movement of the face obtained in FIG. It is a figure which shows the time change of the luminance value of the image signal from a face area | region. It is a figure which shows the time change of the luminance value of the image signal from a face area after noise reduction. It is a figure which shows an example of the skin region detection method which concerns on Embodiment 1. FIG. 2 is a diagram illustrating an example of a hardware configuration of a pulse detection system according to Embodiment 1. FIG. It is a block diagram which shows the functional structure of the pulse detection system which concerns on Embodiment 2. FIG. It is a figure which shows the example of the processing function of a human detector. FIG. 10 is a block diagram showing a signal flow in the pulse detection device according to the second embodiment. 6 is a preferred example of a pulse detection flowchart according to Embodiment 2. 6 is an example of an image of a person acquired by an imaging unit according to Embodiment 2. It is a figure which shows an example of the person detection method which concerns on Embodiment 2. FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The following description of the preferred embodiments is merely exemplary in nature and is in no way intended to limit the invention, its application, or its application.

(Embodiment 1)
(Configuration of pulse detector and flow of image signal processing)
FIG. 1 is a block diagram showing a functional configuration of the pulse detection device according to the present embodiment.

  FIG. 2 is a diagram illustrating an example of a processing function of each block of the pulse detection device.

  FIG. 3 is a block diagram showing a signal flow in the pulse detection device.

  The pulse detection device 10 includes an imaging unit 12, a face detector 13, a noise reducer 14, a pulse calculator 15, a stress level measuring device 16, and a wakefulness level measuring device 17 as functional blocks. Yes.

  The imaging unit 12 includes an imaging element (not shown) such as a CCD or a CMOS image sensor, and an optical system (not shown) including a lens and the like, and an image acquired by the imaging unit 12 is an image. The signal is sent to the face detector 13 as a signal. Note that the imaging unit 12 may be fixed or movable. The imaging area is determined according to the setting of the optical system and the movable area of the imaging unit 12.

  Note that the image pickup device is preferably a color image pickup device that outputs light incident from a peripheral region for each color component using a color filter or the like, but is not particularly limited thereto, and is an image pickup device that outputs a monochrome image. It may be. In general, a CCD, a CMOS image sensor, or the like is used as the imaging device, but a photodiode array may be used.

  In addition, an image sensor that receives near-infrared light may be used as the image sensor, or a combination of this and other image sensors may be used.

  In this embodiment, the image signal is an RGB color system signal represented by three primary colors of an R (Red: red) component, a G (Green: green) component, and a B (Blue: blue) component. However, the present invention is not particularly limited to this, and may be an HSV color system signal or a YUV color system signal. Further, it may be a monochrome image signal.

  Further, the image signal is sent to the face detector 13 at a predetermined frame rate, for example, 30 frames / second, but may be another appropriate frame rate.

  The face detector 13 is configured to detect a human face in the imaging area captured by the imaging unit 12 based on the image signal sent from the imaging unit 12.

  The noise reducer 14 is configured to remove a noise component from an image signal from an area corresponding to a face in the imaging area (hereinafter referred to as a face area) detected and specified by the face detector 13.

  The pulse calculator 15 is configured to estimate a person's pulse rate and pulse interval based on the image signal from the face area from which noise has been removed by the noise reducer 14.

  The stress level measuring device 16 is configured to measure the stress level of the person to be measured based on the pulse interval obtained by the pulse measuring device 15.

  The wakefulness measuring device 17 is configured to measure the wakefulness of a person who is a measurement target based on the pulse interval obtained by the pulse measuring device 15.

  As shown in FIG. 2, in the pulse detection device 10 according to the present embodiment, each of the face detector 13, the noise reducer 14, and the pulse calculator 15 can execute processing by a plurality of different methods.

  Therefore, as shown in FIG. 3, the imaging unit 12 obtains an optimal combination according to the situation of the imaging area, the situation of the person in the imaging area, the processing result in each functional block, and the like. It is possible to select a processing method in each functional block for the received image signal.

  Note that blocks such as face detection 1 and face detection 2 shown in FIG. 3 correspond to processing methods 1 and 2 of the face detector shown in FIG.

(Preferred example of pulse detection method)
The pulse detection method according to the present embodiment will be described below.

  FIG. 4 is a preferred example of a pulse detection flowchart according to the present embodiment.

(Step S10: Imaging step)
First, the imaging unit 12 captures a surrounding area. The imaging area may be outdoors or indoors.

(Step S20: Face detection step)
The image signal acquired by the imaging unit 12 is sent to the face detector 13, and a human face in the imaging region is detected based on this image signal.

  If face detection fails, the process proceeds to step S42 described later.

(Face detection step 1)
For example, when a distance between the imaging unit 12 and the person who is the imaging target is close, there is little movement of the person, and a clear face image is obtained, a known method is used as the face detection method. Can do.

  As a known method, for example, a method of extracting a so-called Haar-like feature and detecting a face by machine learning such as Adaboost can be used.

  In this case, the face edge is detected from a number of local contrast differences in the image.

  Using a large amount of face image data and non-face image data prepared in advance, a discriminator for discriminating whether the extracted Haar-like feature quantity corresponds to a face or not a face is generated.

  After generating a plurality of discriminators, finally, data corresponding to the face is extracted from the image data by these majority logics, and the face area is output. An outline of such technology is disclosed in, for example, the following paper (P. Viola, and M. Jones, “Rapid object detection using a boosted cascade of simple features,” Proceedings of IE Conon and Pattern Recognition, p. 511. (2001)).

  Further, face image data and non-face image data can be obtained from a database such as OpenCV (http://opencv.org/).

  In addition, it is not limited to the above, You may use another known method.

(Face detection step 2)
Another method can be used when the distance between the imaging unit 12 and the person who is the object to be imaged is long, or when the movement of the person is large.

  FIG. 5 is an example of a human image acquired by the imaging unit according to the present embodiment.

  FIG. 6 is a diagram illustrating an example of a face detection method according to the present embodiment.

  FIG. 7 is a diagram showing an example of region division of an image according to the present embodiment.

  FIG. 8 is a diagram illustrating an example of setting the face search range in the previous and next frames.

  If the face is hidden or turned sideways, a human face may not be detected immediately using a known method.

  In such a case, as shown in FIG. 6, the image is divided into regions, the average of the luminance values of the images is obtained in each divided region, and the time change of this value is observed.

  For example, G component light is greatly affected by the absorption of hemoglobin. Therefore, if the luminance value of the G component is used for an RGB color system image signal, periodic fluctuations related to a human pulse can be easily observed. . By observing this periodic fluctuation, it is possible to detect whether or not a human face is observed in the target region. It is also possible to grasp the outline of a person's face by appropriately selecting the division size.

  Note that it is possible to observe periodic fluctuations related to the human pulse even in image signals other than the RGB color system, and the luminance value used when detecting the face area is limited to the above wavelength components. is not.

  Further, as shown in FIG. 6, it is not always necessary to divide the area of the image, and the areas may overlap as shown in FIG.

  In this case, the divided areas have different sizes, and there are portions where the positions are different and overlap.

  For example, if the area is divided evenly, two faces may enter the same area, and there is a possibility that they cannot be separated. However, by separating the areas into random areas as shown in FIG. Accuracy is improved. However, if the area setting that allows the overlapping of the areas is performed as described above, a large number of areas exist and a long calculation time is required. Therefore, it is possible to reduce the calculation time by performing a tracking process and searching only around the result of the previous frame.

  In FIG. 8, a frame indicated by a solid line corresponds to the search range in the current frame, and the shaded range represents the face detection position and size in the previous frame.

  As can be seen from FIG. 8, there are many search ranges in the current frame near the face detection position one frame before, and the size of this range is almost the same as that in the previous frame.

  A known particle filter (particle filter) is used for the tracking process. There are three parameters: the position (x, y) and the size L of the search frame. As the likelihood of the particle filter, the size of the largest peak when FFT (Fast Fourier Transform) is performed is used as it is.

  Note that FFT is an example of a signal frequency analysis method.

  By using this particle filter, it is possible to perform tracking processing while reducing the calculation time without searching the entire screen.

(Face detection step 3)
FIG. 9 is a diagram showing a temporal change in the luminance value of the image signal in the face area according to the present embodiment.

  FIG. 10 is an example of a sine wave model for the pulse component according to the present embodiment.

  FIG. 11 is another example of the sine wave model for the pulse component according to the present embodiment.

  FIG. 12 shows still another example of the sine wave model for the pulse component according to the present embodiment.

  FIG. 13 is a diagram showing a temporal change in the luminance value of the image signal in the face area before and after the difference of the sine wave model according to the present embodiment.

  As shown in FIG. 9, the image signal from the face area, in this case, the average luminance value of the G component, periodically changes with respect to the measurement time.

  By comparing this waveform with the following sine wave model waveform, the face area can be detected. This will be described in detail below.

  First, as described above, an image is divided into regions, and the temporal change in average luminance value is obtained in each of the divided regions.

  Separately, a plurality of sine wave model waveforms represented by Expression (1) are prepared.

here,
Vm: Maximum amplitude ω: Angular frequency t: Time θ: Phase

  For example, a sine wave model waveform (FIG. 10) corresponding to a pulse waveform of 60 bpm (beats per minute) is used as a standard waveform, and the phase is shifted as shown in FIG. 11 or the angular frequency is changed as shown in FIG. Create a waveform (corresponding to 90 bpm).

  At this time, for example, a waveform in which the angular frequency is changed at intervals corresponding to 10 bpm and a waveform in which the phase is changed in units of 10 degrees are prepared in a table format.

  As shown in FIG. 13, a sine wave model waveform prepared in advance is subtracted from the time-varying waveform of the luminance value actually obtained in each divided area.

  As the maximum amplitude of the model waveform, the amplitude obtained from the periodic fluctuation of the actually obtained luminance value is used.

  As shown in FIG. 13, the waveform indicated by the dotted line is the waveform before the difference, and the waveform indicated by the solid line is the waveform after the difference.

  After the difference, it can be seen that the periodic fluctuation component seen in the original waveform is removed, and there is little undulation. If the pulse expressed by the sine wave model waveform matches the pulse of the subject, the waveform after the difference has less undulation.

  If the standard deviation of the waveform after the difference is small, it is considered that there is less undulation, so each of the differences is performed using the multiple sine wave model waveforms described above, and the subject with the lowest standard deviation of the waveform after the difference is the subject It is assumed that the pulse is

  For example, if the calculated pulse rate is a human pulse, for example, a value corresponding to 60 bpm to 120 bpm, the region is a face region, and if it is outside this range, it is assumed that no face exists.

  In addition, since a person's pulse changes a lot by exercise etc., said range is an example to the last.

  In order to perform face detection from a periodic change in the luminance value of an image signal, it is necessary to hold a plurality of pieces of data from the tens of seconds to tens of seconds immediately before detection. According to this method, the number of data to be held can be reduced, and face detection can be performed in a shorter time.

  Further, as described above, an appropriate face detection method differs depending on the situation of the captured image, particularly the situation of a person.

  In the present embodiment, any one of the face detection steps 1 to 3 can be selected and used arbitrarily according to the imaging region or the situation of a person in the imaging region, and the detection result is obtained at any step. As shown in FIG.

(Step S30: Face position periodicity determination step / Step S40: Noise reduction step 1)
If the face detection is successful, it is next determined whether or not the face position is periodically moved (step S30). If there is no periodicity in the movement of the face position, the process proceeds to Step SS41 described later.

  When exercising such as jogging or exercising, the face moves, but since the movement component is on the image signal, it becomes noise when calculating the pulse. Thus, noise reduction is performed by analyzing the frequency of the change in the face position and subtracting the frequency from the observed value (step S40).

  In this embodiment, the periodicity determination of the face position is performed using the function of the face detector 13, but a functional block for determination may be provided separately.

  FIG. 14 is a power spectrum obtained by frequency analysis of the face position coordinates.

  FIG. 15 is a power spectrum obtained by subtracting a signal corresponding to a change in the face position from the luminance value of the image signal from the face area and further performing frequency analysis.

  In the present embodiment, the power spectrum is obtained by performing FFT processing on the position coordinates of the center point of the face, which is position information of the face on the image. In addition, since the position coordinates of the center point of the face may become unstable in time series, frequency analysis is performed on the detected facial characteristic organs, such as the eyes and nose, or the average positional coordinates of those characteristic organ points. It is preferable to do this. Corresponding to this, as shown in FIG. 1, the face detector 13 may include a face organ detection unit 13 a.

  Note that the face detector 13 may include a face organ detection unit 13a regardless of whether the face position coordinates are subjected to frequency analysis.

  14 and 15, the position surrounded by the dotted line is a peak corresponding to the movement of the face, and it can be seen that the face position changes with a certain periodicity. In this example, there is a peak near 1.14 Hz.

  Further, in FIG. 15, the presence of peaks is observed near 1.14 Hz and near 1.28 Hz. The former is a component corresponding to the movement of the face position, and the latter is a component corresponding to the pulse.

  Actually, the range that can be taken as a pulse is set to the right side of the one-dot chain line in FIG. 15, that is, the high frequency side, and the peak of the highest frequency within the range is estimated as the pulse.

  As shown in FIG. 15, the peak due to facial motion and the peak due to pulse are close to each other in frequency. Therefore, when the frequency component due to the change in the face position is not calculated in advance, the former peak may be erroneously detected as a pulse.

  According to the present embodiment, in order to prevent this, by pre-determining the power spectrum obtained by frequency analysis of the face position coordinates, it is possible to prevent erroneous pulse measurement due to face position fluctuations due to movement or the like. .

  In addition, the power spectrum shown in FIG. 14 can be obtained by performing a difference using a sine wave model, instead of directly obtaining a difference from the luminance value of the image signal from the detected face region.

  FIG. 16 is a diagram showing a sine wave model corresponding to the frequency of facial motion obtained in FIG.

  FIG. 17 is a diagram illustrating a temporal change in the luminance value of the image signal from the face area.

  Here, the luminance value of the image signal from the face area is an average value of the luminance values from the area detected as the face area.

  The noise component due to facial motion is removed by subtracting the sine wave waveform shown in FIG. 16 from the time-varying waveform of the luminance value shown in FIG.

(Step S50: Pulse calculation step 1)
Next, frequency analysis such as FFT is performed on the luminance value of the image signal from the face area with reduced noise, and the pulse is estimated and calculated from the obtained signal. Furthermore, a person's stress level and arousal level are calculated using the pulse interval.

  FIG. 18 is a diagram illustrating a temporal change in the luminance value of the image signal from the face area after noise reduction.

  As shown in FIG. 18, the pulse interval can be obtained from the temporal change of the luminance value of the image signal from the face area before frequency analysis.

(Step S60: Step of measuring stress level and arousal level)
Using the pulse interval obtained in step S50, the stress level and / or wakefulness level of the person who is the imaging target is measured (step S60).

  A known method is used for the measurement of the stress level and the arousal level.

  From the time series data of the pulse interval, for example, the power spectral density is calculated using an autoregressive model, and the frequency component is analyzed to measure the degree of stress.

  In the power spectral density obtained by the above method, an area corresponding to a frequency from 0.05 Hz to 0.15 Hz is an LF (Low Frequency) component area, and the frequency is from 0.15 Hz to 0.40 Hz. The corresponding region is set as an HF (High Frequency: high frequency) component region, and the integrated intensity in each region is obtained.

  In general, the signal in the LF component region corresponds to blood pressure fluctuation, and it is said that a signal is generated in any case when the sympathetic nerve is in tension or when the parasympathetic nerve is in tension.

  On the other hand, it is said that the signal in the HF component region corresponds to respiratory fluctuations, and that the signal intensity increases when the parasympathetic nerve is dominant.

  The stress level can be obtained by using (signal strength of LF component region) / (signal strength of HF component region) as a stress index (sympathetic nerve activity).

  The parasympathetic nerve is dominant when relaxed, and the signal intensity in the HF component region increases and the stress index decreases, while the signal strength in the HF component region is relatively high when nervous. Decreases and the stress index increases.

  Similarly, the arousal level can be evaluated from the sympathetic nerve activity by obtaining the signal strength of the LF component region and the signal strength of the HF component region.

  As for the degree of arousal, there is a report that whether or not it is an arousal state is reflected in the signal intensity of the HF component region. For example, the degree of arousal can be estimated by focusing on this intensity.

(Step S41: Noise reduction step 2)
In step S30, when no periodicity is seen in the movement of the face position, in the image signal from the face region, the signal corresponding to the pulse component and the other signal are separated, and from the signal corresponding to the pulse component, Noise is reduced by subtracting other signals.

  In the case of an RGB color system image signal, the luminance value of the G component can be a pulse component, and the luminance values of the other components can be noise components.

  At this time, the luminance value of the R component, the luminance value of the B component, or the average value of the luminance values of the R component and the B component can also be used as the noise component.

  If the image signal is an HSV color system or YUV color system, the luminance value of one primary color component is used as a pulse component, and the luminance value of another primary color component, for example, the luminance value of one component or two components The average value of the luminance values may be a noise component.

  In either case, a component other than the primary color system, for example, the luminance value of the near infrared light component may be used as the noise component.

  In addition, when a monochrome image signal or the like is used, noise can be reduced by calculating a difference from the moving average.

  For example, when time series data of luminance values of an image signal is A and time series data obtained by performing a moving average on A is B, noise can be reduced by subtracting B from A. .

(Step S51: Pulse calculation step 1 / Step S61: Stress level and arousal level measurement step)
When step S41 ends, the process proceeds sequentially to step S51 and step S61.

  The pulse estimation and calculation method in step S51 and the stress level and wakefulness level measurement method in step S61 are the same as those shown in step S50 and step S60, respectively.

(Step S42: Skin region detection step)
FIG. 19 is a diagram illustrating an example of a skin region detection method according to the present embodiment.

  If the face detection fails in step S20, the process proceeds to step S42.

  Here, as face detection failure, it is expected that the face is hidden or the face direction is not directed toward the imaging unit 12. The case where the face area in the photographed image is extremely small is also applicable to this case.

  In such a case, as described in step S20, the image is divided into regions, the average value of the luminance values of the image is obtained in each divided region, and the time change of this value is observed.

  For example, it is easy to observe a periodic change related to a pulse from an area other than the face where the skin is exposed. By observing this periodic change, it is possible to detect whether human skin is exposed in the target region.

  Also in this case, detection sensitivity can be improved by appropriately setting the division size of the image region.

  Further, as shown in FIG. 19, it is not always necessary to divide the image area as shown in FIG. 19, and the areas may overlap as shown in FIG.

  Further, as described with reference to FIG. 8, the tracking time using the particle filter is performed, and only the vicinity of the result of the previous frame can be searched to reduce the calculation time.

  By using this particle filter, it becomes possible to cope with changes in size due to the face being hidden or back and forth.

  In addition, the detection of the skin area in this step is performed using the function of the face detector 13.

(Step S52: Pulse calculation step 2)
By the method used in face detection step 3 in step S20, the pulse can be estimated and calculated from the image signal from the skin region.

  In this case, preparing a plurality of sine wave model waveforms represented by Expression (1) is the same as in the face detection step 3, but it is necessary to further refine the frequency and phase increments, for example, in increments of 1 bpm. The waveform changed in step 1 and the waveform whose phase has been changed in units of several degrees are prepared in a table format, and a waveform is appropriately selected from the table created in this step, and the face detection step 3 Used for face detection.

  As shown in FIG. 13, a sine wave model waveform prepared in advance is subtracted from the actually obtained luminance value.

  Note that the maximum amplitude of the model waveform is a value close to the amplitude obtained from the periodic fluctuation of the actually obtained luminance value.

  In FIG. 13, the waveform indicated by the dotted line is the waveform before the difference, and the waveform indicated by the solid line is the waveform after the difference.

  It can be seen that after the difference, the periodic fluctuation component seen in the original waveform is removed.

  If the pulse expressed by the sine wave model waveform matches the pulse of the subject, the waveform after the difference has less undulation.

  If the standard deviation of the waveform after the difference is small, it is considered that there is less undulation, so each of the differences is performed using the multiple sine wave model waveforms described above, and the subject with the lowest standard deviation of the waveform after the difference is the subject Let's say that the pulse.

  According to this method, it is not necessary to hold multiple data for a certain period, compared to the case where the heart rate is obtained from the spectrum peak by frequency analysis, and the influence on the pulse calculation when a pulse change occurs during the period is suppressed. it can.

(Step S62: Stress level and arousal level measurement step)
When step S52 ends, the process proceeds to step S62.

  The stress level and arousal level measuring method in step S62 is the same as that shown in step S60.

  As described above, according to the present embodiment, in various cases where there is a human movement, a face is hidden, or a face is not facing the direction of the imaging unit 12 according to the present embodiment. However, in each functional block, by selecting an appropriate processing method and performing pulse detection processing, a person's pulse, stress level, and arousal level can be obtained directly from the image acquired by the imaging unit 12. it can.

  Note that the pulse detection method in the present embodiment is not limited to the method shown in FIG. 4, and as shown in FIG. 3, the situation of the imaging region, the situation of people in the area, and each function block In accordance with the processing result or the like, pulse detection can be performed by combining various processing methods prepared in advance.

  In this embodiment, the target of pulse detection is a person, but the pulse detection according to the present invention can be applied to a living body other than a person such as an animal.

(Hardware configuration)
The functions of the pulse detection device 1 shown in FIG. 1 can be realized almost on software except for a part of the imaging unit 12. In this case, for example, the pulse detection is performed by executing a program corresponding to the pulse detection flow shown in FIG. 4 on hardware. A hardware configuration for realizing this function is shown in FIG.

  FIG. 20 is a diagram illustrating an example of a hardware configuration of the pulse detection system according to the present embodiment.

  The pulse detection system 100 includes a computer 110, a video camera 102, a display 111, an operation unit 112, and a system bus 101.

  The computer 110, the video camera 102, the display 111, and the operation unit 112 are connected to the system bus 101, and control signals and data are exchanged via the system bus 101.

  The video camera 102 captures a predetermined surrounding area and inputs the captured moving image data as an image signal to the computer 110 via the system bus 101.

  The video camera 102 corresponds to the imaging unit 12 illustrated in FIG. 1 and includes an imaging device such as a CMOS image sensor and an optical system including a lens and the like (not shown).

  The image signal input to the computer 110 is subjected to various processes by the computer 110, and the obtained result is displayed on the display 111. There are various objects to be displayed, including not only the pulse rate and pulse interval, stress level and arousal level, but also the detected face area and the real-time change waveform of the luminance value of the image signal as the result of the pulse detection process. Can be displayed.

  The display 111 may have a sound output function as necessary.

  The operation unit 112 is an input device represented by a keyboard, a mouse, a joystick, and the like. For example, when the computer 110 is started and ended, a pulse detection program is called and executed, a processing result to be displayed on the display 111 is selected, and the like. Used.

  For example, when the display 111 is a touch panel, or when the system 100 includes a voice input unit (not shown) for inputting an operation command or the like, the operation unit 112 is not particularly provided. Also good.

  The computer 110 includes, as components, a CPU 103, a signal processing unit 104, a communication unit 105, a ROM 106, a RAM 107, and an HDD 108. These components are connected to a system bus 101 and are connected to a system. Control signals and data are exchanged via the bus 101.

  For example, a program corresponding to the pulse detection flow shown in FIG. 4 is stored in the HDD 108 in advance, and a program read command is executed by the CPU 103 in accordance with an operation on the operation unit 112. A program read from the HDD 108 is stored in the RAM 107 in accordance with an instruction from the CPU 103. A table of sine wave model waveforms and the like are also read from the HDD 108 and stored in the RAM 107.

  Note that the programs and tables read from the HDD 108 may be stored in the ROM 106 depending on the size. Once reading from the HDD 108 is performed, the program can be directly stored in the RAM 107 from the ROM 106 thereafter.

  A program stored in the RAM 107 is executed in accordance with an instruction from the CPU 103. The CPU 103 interprets instructions from the program and performs various data processing and control.

  The signal processing unit 105 performs processing such as image region division and face region extraction in accordance with instructions from the CPU 103.

  The communication unit 105 exchanges data with the outside via a physical interface (not shown) or the like. For example, a database or the like used for face detection is stored in the HDD 108 or the ROM 106 via the communication unit 105. Communication with the outside can be performed by wired communication or wireless communication using an optical fiber or the like, and the communication method is appropriately determined depending on the specifications of the system and the computer.

  Note that the program itself can be read from an external source via the communication unit 105, and the processing result of the program can also be stored in the external source via the communication unit 105. In this case, it is not particularly necessary to provide the HDD 108 only for storing programs and processing results.

  Further, a portable recording medium read / write unit may be provided instead of or separately from the HDD 108. A program, a table of sine wave model waveforms, etc. may be stored in this recording medium, and the processing result may be directly written in the recording medium.

  If the recording medium is an optical disk, an optical disk drive may be provided, if it is an IC card, an IC card reader / writer may be provided, and if it is a USB memory, a USB terminal connection port may be provided.

  Note that the above-described configuration is merely an example, and can be changed as appropriate in accordance with the subject to be imaged, the location, the contents of the pulse detection program, and the like.

  For example, when using a smartphone with a built-in camera or a notebook computer with a built-in camera, the pulse detection system 100 can be realized with a single device.

(Embodiment 2)
When trying to detect a pulse using a surveillance camera or a camera installed in an electronic device, the following problems may occur.

  First, in order to acquire an image, it is necessary to point the camera in the direction where the person to be photographed is always present. In addition, when a camera communication system is hacked, a normal life image of a person to be photographed may be unintentionally leaked and privacy may be infringed.

  In view of this, it is conceivable that the camera lens is made a special type, for example, a lenticular type, or a light diffusing member such as frosted glass is arranged in front of the lens to take a picture with a blurred image.

  However, if the image is unclear, if the person is moving, the person can be detected from the movement, but the person cannot be detected because the movement of the person is small in a relaxing scene in the living room. . In situations where human detection is not possible, it is also difficult to detect faces and skin areas.

  Therefore, in this embodiment, a configuration is disclosed in which human detection is performed before face detection or skin area detection, and pulse detection can be reliably performed.

  FIG. 21 is a block diagram showing a functional configuration of the pulse detection device according to the present embodiment.

  FIG. 22 is a diagram illustrating an example of the processing function of the human detector.

  FIG. 23 is a block diagram showing a signal flow in the pulse detection device.

  The difference between the configuration shown in the present embodiment and the configuration shown in the first embodiment is that the diffuser 11 shown in FIG. 1 is provided and the image signal from the imaging unit 12 is sent to the human detector 20. Thus, after detecting the presence or absence of a person in the image, the pulse detection is performed using the configuration after the face detector 13.

  As described above, the light diffusing unit 11 scatters light incident from the surrounding area by making the lens of the imaging unit 12 a special type, for example, a lenticular type, or by arranging a light diffusing member such as frosted glass in front of the lens. And / or has the function of dimming.

  The light diffuser 11 may be incorporated in the video camera 102 in the hardware configuration shown in FIG. 20 or may be provided separately.

  In the pulse detection device 10 according to the present embodiment, as shown in FIG. 2, each of the face detector 13, the noise reducer 14, and the pulse calculator 15 can execute processing by a plurality of different methods. Further, as shown in FIG. 22, the human detector 20 can execute processing by a plurality of different methods.

  Therefore, as shown in FIG. 23, the imaging unit 12 obtains an optimal combination according to the situation of the imaging area, the situation of people in the imaging area, the processing result in each functional block, and the like. It is possible to select a processing method in each functional block for the received image signal.

  The blocks such as face detection 1 and face detection 2 shown in FIG. 3 correspond to the processing methods 1 and 2 of the face detector shown in FIG. The human detection 1 and human detection 2 blocks shown in FIG. 23 correspond to the processing methods 1 and 2 of the human detector shown in FIG.

  A pulse detection method in the present embodiment will be described with reference to FIG. Note that the steps described in Embodiment 1 are limited to the minimum necessary extent.

  FIG. 24 is a preferred example of a pulse detection flowchart according to the present embodiment.

(Step S100: Diffuse Step / Imaging Step)
Light from a surrounding region incident on the imaging unit 12 is scattered and / or attenuated by the light scattering unit 11. In this state, the imaging unit 12 captures the surrounding area. The imaging area may be outdoors or indoors.

(Step S110: Person detection step)
The image signal acquired by the imaging unit 12 is sent to the human detector 13, and based on this image signal, a human face in the imaging region is detected.

  If the person detection fails, it is assumed that there is no person in the imaging area, and the process returns to step S100 to continue shooting.

(People detection step 1)
FIG. 25 is an example of a human image acquired by the imaging unit according to the present embodiment.

  FIG. 26 is a diagram showing an example of a person detection method according to the present embodiment.

  In FIG. 25, the left figure is an example of an image when a lenticular type lens is used, and the right figure is an example of an image when photographing is carried out through ground glass. In either case, the image is unclear, and for example, Haar-like feature values cannot be extracted.

  Therefore, in step S20 of the pulse detection flow in the first embodiment, as described in face detection step 2, the captured image is divided into regions, and the average value of the luminance value of the image is calculated in each divided region. Obtain and observe the time change of this value.

  Even if the image is unclear, it is easy to observe a periodic change related to the pulse from the area where the face and skin are exposed. By observing this periodic change, it is possible to detect whether a human face or skin is exposed in the target region.

  Further, detection sensitivity can be improved by appropriately setting the division size of the image area.

  As shown in FIG. 19, it is not always necessary to divide the area of the image as shown in FIG. 19, and the areas may overlap as shown in FIG.

  Further, as described with reference to FIG. 8, the tracking time using the particle filter is performed, and only the vicinity of the result of the previous frame can be searched to reduce the calculation time.

  By using this particle filter, it becomes possible to cope with changes in size due to the face being hidden or moving back and forth.

(People detection step 2)
As described in face detection step 3 in step S20 of the pulse detection flow in the first embodiment, the image is divided into regions, and the time-varying waveform of the average luminance value and the sine wave model waveform in each of the divided regions. The human region can be detected by comparing with.

  In order to perform face detection from a periodic change in the luminance value of an image signal, it is necessary to hold a plurality of pieces of data from the tens of seconds to tens of seconds immediately before detection. According to this method, the number of data to be held can be reduced, and human detection can be performed in a shorter time.

  In addition, an appropriate person detection method varies depending on the situation of the captured image, particularly the situation of a person. For example, when a person is moving, it is preferable that the person can be detected in a short time.

  In the present embodiment, the human detection step 1 or 2 can be arbitrarily and appropriately selected and used depending on the situation of the person within the imaging power range.

  The human detector 20 may be provided with a motion detector 20a. In this case, the area where the person is present is determined by integrating the information on the motion in the imaging area detected by the motion detection unit 20a and the information from the luminance value of the image.

  In any case, the presence / absence of a person can be detected, and the presence area of the person, specifically, the number and position of the person in the imaging area is output as the detection result.

(From step S120: face detection step to step S160: stress level and arousal level measurement step)
If person detection is successful in step S110, the process proceeds to face detection step S120. If the face detection is successful, it is determined whether or not the face position is periodically moved (step S130). If the face position is periodically moved, the noise reduction step S140, the pulse calculation step S150, the stress Then, the process proceeds to step S160.

  The face detection method in step S120, the face position periodicity determination method in step S130, the noise reduction method in step S140, the pulse estimation and calculation method in step S150, and the stress level and arousal level measurement method in step S160 are described in the first embodiment. These are the same as those shown in Step S20, Step S30, Step S40, Step S50 and Step S60 shown in the pulse detection flow in FIG.

(From step S141: noise reduction step to step S161: stress level and arousal level measurement step)
If no periodicity is found in the movement of the face position in step S130, the process proceeds in sequence to noise reduction step S141, pulse calculation step S151, stress level, and arousal level measurement step S161.

  The noise reduction method in step S141, the pulse estimation and calculation method in step S151, and the stress level and wakefulness measurement method in step S161 are shown in steps S41, S51, and S61 shown in the pulse detection flow in the first embodiment. It is the same as each.

(From Step S142: Noise Reduction Step to Step S162: Stress Level and Arousal Level Measurement Step)
In step S120, if face detection fails, the process proceeds to skin region detection step S142, pulse calculation step S152, stress level, and arousal level measurement step S162.

  The skin region detection method in step S142, the pulse estimation and calculation method in step S152, and the stress level and arousal level measurement method in step S162 are the same as those in steps S42, S52, and S62 shown in the pulse detection flow in the first embodiment. Each is the same as shown.

  As described above, according to the present embodiment, while protecting the privacy of the person to be imaged, the person is moving, the face is hidden, the face is the imaging unit 12 in the imaging area. Even in various cases in which the direction is not directed, in each functional block, by selecting an appropriate processing method and performing pulse detection processing, directly from the image acquired by the imaging unit 12, A person's pulse rate, stress level, and arousal level can be obtained.

  The pulse detection method in the present embodiment is not limited to the method shown in FIG. 24. As shown in FIG. 23, the situation of the imaging region, the situation of the person in the situation, and each function block In accordance with the processing result or the like, pulse detection can be performed by combining various processing methods prepared in advance.

  In this embodiment, the target of pulse detection is a person, but the pulse detection according to the present invention can be applied to a living body other than a person such as an animal.

(Other embodiments)
In addition to the configuration described above, other configurations can be adopted. For example, in Embodiment 1, the structure which provides the diffuser part 11 can also be taken.

  By providing the light diffusing unit 11, the incident light to the imaging unit 12 is scattered or dimmed, so that the obtained image becomes a blurred image, but it is possible to avoid specifying a person directly from the image information. This is the case when the subject person is clearly in front of the camera, such as a pulse test. For example, the pulse detection apparatus and the pulse detection method of the present invention may be applied to the case where it is particularly necessary to protect the privacy of a patient in a pulse or stress level test in a hospital laboratory.

  Moreover, in Embodiment 2, the structure which does not provide the diffuser part 11 can also be taken.

  In this case, for example, by moving the lens mechanically in the imaging unit 12 to shift the imaging focus and blur the screen, the diffuser unit 11 becomes unnecessary, and the privacy of the person who is photographed Can also be protected.

  However, it is necessary to make a determination by the human detector 20 in order to know whether or not the photographed object is a living body such as a person.

  In the first and second embodiments and other embodiments, the light source at the time of photographing may be natural light or artificial light such as a fluorescent lamp or LED illumination. Depending on the wavelength range of the light source, for example, the wavelength component used for pulse calculation and the wavelength component used for noise reduction can be appropriately changed.

  In addition, each functional block may be realized by wired logic, except for hardware configurations such as an imaging element and an optical system in the imaging unit 12. Realization with a dedicated LSI or a dedicated block in the LSI is also possible.

  The pulse detection device of the present invention is acquired by the imaging unit 12 even in various cases where there is a person's movement in the imaging region, the face is hidden, or the face is not facing the direction of the imaging unit. It is extremely useful because the pulse, stress level, and arousal level of a person can be obtained directly from the captured image.

DESCRIPTION OF SYMBOLS 10 Pulse detection apparatus 11 Diffuse part 12 Imaging part 13 Face detector 13a Face organ detection part 14 Noise reducer 15 Pulse calculator 16 Stress degree measuring instrument 17 Arousal degree measuring instrument 20 Human detector (biological detector)
20a Motion detection unit 100 Pulse detection system 101 System bus 102 Video camera 103 CPU
104 Signal processing unit 105 Communication unit 106 ROM
107 RAM
108 HDD
110 Computer 111 Display 112 Operation unit

Claims (23)

A pulse detection device for detecting a pulse of a living body,
An imaging unit for imaging a predetermined imaging area and acquiring an image signal;
A face detector for detecting a face area or a skin area of the living body based on the image signal;
A noise remover that removes noise from the detected image signal from the face area or the skin area;
And a pulse calculator for calculating a pulse from the image signal from the face area or the skin area from which noise has been removed.   The face detector, the noise remover, and the pulse calculator each correspond to a plurality of processing methods, and one of the plurality of processing methods is selected according to the imaging region and the state of the living body. The pulse detection device according to claim 1, configured to select a processing method and execute the processing. The face detector is
The pulse detection device according to claim 2, configured to detect a local light / dark difference in a luminance value of the image signal in the imaging region and detect the face region by majority logic based on the light / dark difference. . The face detector is
The pulse detection device according to claim 2, wherein the pulse detection device is configured to divide an image in the imaging region and detect the face region or the skin region based on the image signal from each of the divided regions. The face detector is
Based on the result obtained by dividing the image in the imaging region and comparing the time-varying waveform of the image signal from each divided region with the model waveform represented by a sine wave, The pulse detection device according to claim 2, configured to detect a skin region. The noise remover is
The position information of the face area in the imaging area is frequency-analyzed, and the specific frequency component of the obtained position information is removed from the image signal from the face area. Pulse detector. The noise remover is
The pulse detection according to claim 2, wherein at least a part of a component other than a wavelength component used for detection of a pulse component in the image signal is removed from the image signal from the face region or the skin region. apparatus. The pulse calculator is
The pulse detection device according to claim 2, configured to calculate a pulse of the living body by performing frequency analysis on an image signal from the face area or the skin area after noise is removed. The pulse calculator is
The pulse detection according to claim 2, configured to calculate a pulse of the living body by comparing a time-varying waveform of an image signal from the face region or the skin region with a model waveform represented by a sine wave. apparatus. Further comprising a living body detector for detecting the presence or absence of the living body based on a periodic fluctuation component of the image signal;
2. The pulse detection according to claim 1, wherein the face detector is configured to detect the face region or the skin region based on an image signal from the presence region of the living body obtained by the living body detector. apparatus.   The living body detector corresponds to a plurality of processing methods, and selects one processing method from the plurality of processing methods according to the imaging region and the state of the living body, and executes the processing. The pulse detection device according to claim 10, which is configured. The biological detector is
The pulse detection device according to claim 11, configured to divide an image in the imaging region and detect presence or absence of the living body based on the image signal from each of the divided regions. The biological detector is
The image in the imaging region is divided, and the time-varying waveform of the image signal from each divided region is compared with a model waveform represented by a sine wave, and configured to detect the presence or absence of the living body. The pulse detection device according to claim 11.   The pulse detection device according to any one of claims 1 to 13, further comprising a diffuser that scatters and / or attenuates light incident on the imaging unit.   The pulse detection device according to any one of claims 1 to 14, further comprising a stress measurement device that measures a stress level of the living body based on a pulse interval obtained from the pulse calculator.   The pulse detection device according to any one of claims 1 to 15, further comprising a wakefulness measuring device that measures the wakefulness of the living body based on a pulse interval obtained from the pulse calculator. A pulse detection method for detecting a pulse of a living body,
An imaging step of imaging a predetermined imaging area and obtaining an image signal;
A face detection step of detecting a face area or a skin area of the living body based on the image signal;
A noise removal step of removing noise from the detected image signal from the face area or the skin area;
And a pulse calculating step of calculating a pulse from the image signal from the face area or the skin area from which noise has been removed.   The face detection step, the noise removal step, and the pulse calculation step each correspond to a plurality of processing methods, and one of the plurality of processing methods is selected according to the imaging region and the state of the living body. The pulse detection method according to claim 17, wherein the processing is executed by selecting a processing method. A living body detecting step of detecting the presence or absence of the living body based on a periodic variation component of the image signal,
In the face detection step,
The pulse detection method according to claim 18, wherein the face area or the skin area is detected based on an image signal obtained from the living body presence area obtained by the living body detection step.   The living body detection step corresponds to a plurality of processing methods, and selects one processing method from the plurality of processing methods according to the imaging region and the state of the living body, and executes the processing. Item 20. The pulse detection method according to Item 19.   21. The pulse detection method according to claim 17, further comprising a scattering step of scattering and / or dimming light from the imaging region and inputting the light to the imaging unit.   The pulse detection method according to any one of claims 17 to 21, further comprising a stress measurement step of measuring a stress level of the living body based on a pulse interval obtained in the pulse calculation step.   The pulse detection method according to any one of claims 17 to 22, further comprising a wakefulness measurement step of measuring the wakefulness of the living body based on the pulse interval obtained in the pulse calculation step.