WO2022137555A1

WO2022137555A1 - Pulse detection device and pulse detection method

Info

Publication number: WO2022137555A1
Application number: PCT/JP2020/048929
Authority: WO
Inventors: 彰彦菅原
Original assignee: 株式会社ソニー・インタラクティブエンタテインメント
Priority date: 2020-12-25
Filing date: 2020-12-25
Publication date: 2022-06-30

Abstract

A frame storage unit 20 stores therein a prescribed number of frames of an image captured of a region including a user's skin. A pixel value acquisition unit 30 divides the captured image into a plurality of blocks and acquires a pixel value of each block. A time-series pixel array acquisition unit 40 acquires a time-series pixel array which has arranged therein the pixel values of the respective blocks in the prescribed number of frames. A pulse data array acquisition unit 50 calculates an average value of respective elements in the time-series pixel array as a direct-current component, acquires a pulse component by subtracting the direct-current component from a prescribed element in the time-series pixel array, and acquires a pulse data array in which pulse components in a prescribed period is arranged. A pulse detection unit 60 detects a pulse rate in the form of a frequency obtained by calculating an autocorrelation of the pulse data array.

Description

Pulse detection device and pulse detection method

The present invention relates to a pulse detection technique.

Some game consoles in the home are equipped with a camera, which captures the image of the user playing the game, detects the user's facial expression from the captured image of the user, and grasps the mental state of the user during game play to play the game. It can be reflected in the development of. Further, in order to detect the degree of tension of the user during game play, a sensor for detecting the pulse is attached to the user, and the measurement result of the pulse output by the sensor is input to the game machine and reflected in the game. ing.

Patent Document 1 describes a method of measuring vital signs from a temporal change in the density of a captured image of a subject.

Japanese Unexamined Patent Publication No. 2005-218507

There was a problem that it would be a burden on the user to attach the sensor to the user to detect the pulse. In addition, with the conventional pulse detection method using captured images, it is difficult to detect fine fluctuations in the pulse, and since body movements such as respiration affect it, it is difficult to accurately measure the pulse. there were.

The present invention has been made in view of these problems, and an object of the present invention is to provide a pulse detection technique capable of detecting a pulse from a captured image of a user with high accuracy.

In order to solve the above problems, the pulse detection device of an embodiment of the present invention divides the captured image into a plurality of blocks and a storage unit for storing a predetermined number of frames of the captured image in the region including the user's skin. A pixel value acquisition unit that acquires the pixel value of each block, a time-series pixel array acquisition unit that acquires a time-series pixel array in which the pixel values of each block of a predetermined number of frames are arranged, and each element of the time-series pixel array. By calculating the average value of the DC component as a DC component and subtracting the DC component from a predetermined element of the time-series pixel array, the pulse component is acquired, and the pulse data array in which the pulse components are arranged for a predetermined period is acquired. It includes a data sequence acquisition unit and a pulse detection unit that detects a frequency obtained by obtaining an autocorrelation of the pulse data sequence as a pulse number.

Another aspect of the present invention is a pulse detection method. In this method, a pixel value acquisition step of dividing an image captured by a predetermined number of frames in an area including a user's skin into a plurality of blocks and acquiring a pixel value of each block, and the pixel value of each block of a predetermined number of frames are combined. The time-series pixel array acquisition step for acquiring the arranged time-series pixel array and the average value of each element of the time-series pixel array are calculated as DC components, and the DC component is subtracted from a predetermined element of the time-series pixel array. As a result, the pulse data sequence acquisition step of acquiring the pulse component and acquiring the pulse data array in which the pulse components are arranged for a predetermined period, and the frequency obtained by obtaining the autocorrelation of the pulse data array are detected as the pulse count. Includes a pulse detection step to be performed.

It should be noted that any combination of the above components and the conversion of the expression of the present invention between methods, devices, systems, computer programs, data structures, recording media, etc. are also effective as aspects of the present invention.

According to the present invention, the pulse can be detected with high accuracy from the captured image of the user.

It is a block diagram of the pulse detection apparatus which concerns on 1st Embodiment. It is a figure explaining the data structure of the signal used in the pulse detection apparatus of FIG. It is a flowchart explaining the pulse wave detection procedure by the pulse detection apparatus of FIG. FIG. 4A shows an autocorrelation graph of the detected pulse wave, and FIG. 4B is an enlarged view of 200 near the peak of the autocorrelation graph of FIG. 4A. It is a block diagram of the pulse detection apparatus which concerns on 2nd Embodiment. FIG. 6A is a diagram showing a detected pulse wave and its autocorrelation graph, and FIG. 6B is a diagram showing a reference pulse wave and its autocorrelation graph. FIG. 7 (a) is a diagram showing a detected pulse wave, and FIG. 7 (b) is a diagram showing an amplitude reliability. It is a block diagram of the pulse detection apparatus which concerns on 3rd Embodiment. 9 (a) to 9 (d) are diagrams illustrating a plurality of areas set in the captured image. 10 (a) and 10 (b) are diagrams showing the time variation of the pulse rate and the reliability detected in each area. It is a block diagram of the pulse detection apparatus which concerns on 4th Embodiment. It is a figure explaining the pixel exclusion by the pixel exclusion part of FIG.

(First Embodiment)
FIG. 1 is a block diagram of a pulse detection device 100 according to the first embodiment. The pulse detection device 100 includes an imaging unit 10, a frame storage unit 20, a pixel value acquisition unit 30, a time-series pixel array acquisition unit 40, a pulse data array acquisition unit 50, a pulse detection unit 60, and an autocorrelation graph storage unit 70. ..

The image pickup unit 10 captures an image of a body portion where the skin is exposed, such as the user's face, and stores it in the frame storage unit 20. A predetermined number of captured images are stored in the frame storage unit 20.

There are many capillaries on the skin of the face, which is suitable for taking a pulse. Other than the face, the palms and soles of the feet also have less melanin, making it easier to photograph capillaries. When you beat the pulse, the capillaries also pulsate. When the capillaries of the skin are photographed, the reflected light also changes due to the pulsation of the capillaries, so that the pulse can be detected from the time change of the pixel value of the photographed image.

The pixel value acquisition unit 30 divides the captured image stored in the frame storage unit 20 into a plurality of blocks and acquires the pixel value of each block. The area suitable for pulse wave detection is the area of the face where the skin is exposed and there is little movement. For example, the face area of the captured image is divided into blocks such as the forehead, right cheek, left cheek, and nose, and the pixel value of each block is acquired. The number of pixels in each block may be different.

It is preferable that the pixel value acquisition unit 30 acquires a green value as a pixel value when the color of the pixel of each block is represented by RGB.

The skin has a dermis under the epidermis, and if the epidermis is transparent and thin, visible light can penetrate into the dermis and photograph capillaries. The transmission power of visible light depends on the wavelength, and the longer the wavelength, the deeper the skin penetrates. Red light has the longest wavelength and penetrates deep into the skin, but the light penetrates too far and shoots extra things other than capillaries. Blue light has the shortest wavelength and does not penetrate deep into the skin, making it unsuitable for imaging capillaries. Green light penetrates to the dermis below the epidermis and is easily absorbed by red blood cells, making it suitable for photographing capillaries.

The pixel value acquisition unit 30 acquires spatially smoothed pixel values by adding a predetermined number of pixel values of each block. All the pixel values in the block may be added, or some pixel values in the block may be added. In the case of 8-bit pixels, the pulse wave signal detected from one pixel is small to the extent that there is no change in the least significant bit, so the pulse wave signal is amplified by adding the pixel values in the block. can do. Further, adding the pixel values in the block acts as a spatial low-pass filter (LPF) that spatially smoothes the pixel values in the block.

The time-series pixel array acquisition unit 40 acquires a time-series pixel array in which the pixel values of each block acquired by the pixel value acquisition unit 30 are arranged by a predetermined number of frames. For example, when the frame rate of the moving image is 30 frames / second, a time-series pixel array in which 30 pixel values for 1 second, that is, 30 frames are arranged in chronological order is generated for each block.

The time-series pixel array acquisition unit 40 may acquire a time-series pixel array smoothed in time by applying a low-pass filter to the time-series pixel array. The image sensor used for imaging generally has noise in the detected value. Applying a low-pass filter to a time-series pixel array acts as a temporal low-pass filter that removes noise in the time direction of the detection value of the sensor.

The pulse data array acquisition unit 50 calculates the average value of each element of the time-series pixel array as a direct current (DC) component, extracts the central element of the time-series pixel array as an alternating current (AC) component, and extracts the direct current component from the alternating current component. The pulse component is obtained by subtracting. Subtracting a direct current component from the alternating current component of the time-series pixel array has the effect of applying a high-pass filter (HPF) to the time-series pixel array.

The pulse data array acquisition unit 50 advances the acquisition of the pulse component for each frame, and acquires the pulse data array in which the pulse components for a predetermined period are arranged. The predetermined period is the time of at least two cycles of a standard pulse (eg, 2.5 seconds). By using a short-time pulse wave signal of about two cycles, it becomes possible to detect a momentary pulse. If a long-term pulse wave signal is used, the pulse will be averaged, and it will not be possible to capture momentary changes in the pulse.

The pulse data array acquisition unit 50 may acquire a time-smoothed pulse data array by applying a low-pass filter to the pulse data array. The pulse wave signal obtained here may have an amplitude fluctuation in the time direction depending on the calculation method of the AC component. Applying a low-pass filter to the pulse data array acts as a temporal low-pass filter that eliminates the temporal fluctuation of the amplitude of the pulse wave.

Here, it is preferable that the filter strength of the low-pass filter applied to the time-series pixel array is stronger than the filter strength of the low-pass filter applied to the pulse data array. This is because the temporal noise contained in the raw data detected by the image sensor is relatively large, and the distortion of the pulse wave signal waveform due to the fluctuation of the amplitude is not larger than that.

The pulse detection unit 60 obtains the autocorrelation of the pulse data array and stores the autocorrelation graph in the autocorrelation graph storage unit 70. The pulse detection unit 60 detects the point where the autocorrelation graph of the pulse wave signal has the second maximum, acquires the time on the horizontal axis of the second maximum point as the period of the pulse wave signal, and obtains the reciprocal of the pulse wave. Obtained as the frequency of the signal. The pulse detection unit 60 outputs the frequency of the pulse wave signal detected by autocorrelation as the pulse rate.

FIG. 2 is a diagram illustrating a signal data structure used in the pulse detection device 100.

The pixel value acquisition unit 30 acquires pixel values from each block of the forehead 14a, the right cheek 14b, and the left cheek 14c of the user's face image 12 captured by the image pickup unit 10, and adds the pixel values in the blocks. It is stored in the time-series pixel array R [], which is a raw data array. For example, by storing the pixel values for 30 frames per second in the array, the time-series pixel array R [] having 30 elements is acquired. As the frame of the captured image advances, each element of the time-series pixel array R [] is shifted to the right, the i-th element is copied to the (i + 1) th element, and the first element of the time-series pixel array R [] is copied. The pixel value obtained from the new frame is stored in.

The pixel value acquisition unit 30 applies a low-pass filter having a filter intensity of N_raw_data to the time-series pixel array R []. Specifically, as an example, the moving average of the following equation is repeated as many times as the number of times according to the filter intensity N_raw_data. For example, the moving average is repeated 10 times with the filter intensity N_raw_data as 10.
R'[i] = 0.25 * R [i-1] +0.5 * R [i] +0.25 * R [i + 1]

Let LPR [] be the time-series pixel array obtained by applying a low-pass filter having a filter strength of N_raw_data to the time-series pixel array R [].

The pulse data array acquisition unit 50 calculates the average value of each element of the time-series pixel array LPR [] as a DC component, and subtracts the DC component from the central element of the time-series pixel array LPR [] to obtain the first pulse. Acquire component P [0].

Next, the pulse data sequence acquisition unit 50 copies the pulse component P [0] to P [1] and acquires the pulse component P [0] of the next frame. Further, the pulse data sequence acquisition unit 50 copies the pulse component P [1] to P [2], copies P [0] to P [1], and further copies the pulse component P [0] of the next frame. get. By repeating this for a predetermined period, for example, 2.5 seconds, a pulse data array P [] in which pulse components are arranged is acquired. When the frame rate is 30 frames / sec, the number of elements of the pulse data array P [] is 30 * 2.5 = 75.

As the frame of the captured image advances in this way, each element of the pulse data array P [] is shifted to the right, the i-th element is copied to the (i + 1) element, and the first element of the pulse data array P [] is copied. The element stores the pulse component of the new frame.

The pulse data array acquisition unit 50 applies a low-pass filter having a filter intensity of N_pulse_data to the pulse data array P []. Specifically, as an example, the moving average of the following equation is repeated as many times as the number of times according to the filter intensity N_pulse_data. For example, the moving average is repeated three times with the filter intensity N_pulse_data set to 2.
P'[i] = 0.25 * P [i-1] +0.5 * P [i] +0.25 * P [i + 1]

Let LPP [] be the pulse data sequence obtained by applying a low-pass filter with a filter intensity of N_pulse_data to the pulse data sequence P [].

The pulse detection unit 60 obtains the autocorrelation AC [t] with respect to the time delay t of the pulse data array LPP [] by the following equation.
AC [t] = SUM (LPP [n] * LPP [n + t])

The pulse detection unit 60 obtains the autocorrelation AC [t] from t = 0 to t = N (N is the number of elements of LPP []), and the time T at which AC [T] becomes the second maximum point is the pulse wave. It is acquired as a cycle and the frequency that is the reciprocal of it is output as the pulse rate.

Generally, there is a method of performing a fast Fourier transform on a pulse wave signal to detect the frequency of the pulse wave signal. At least 512 data are required for the fast Fourier transform, and in the case of 30 frames / sec, the pulse wave signal of 10-odd seconds is analyzed, and the average value of the pulse of 10-odd seconds can be obtained, but it is instantaneous. No pulse can be obtained. The pulse detection method of the present embodiment uses autocorrelation, and when the pulse wave signals are overlapped at a position deviated by one cycle, the autocorrelation is maximized and the cycle of the pulse wave signal can be detected. Therefore, a momentary pulse can be obtained from the autocorrelation of the pulse wave signal for a short period of about two cycles. The pulse detection method of the present embodiment is suitable for detecting fine fluctuations in the pulse.

FIG. 3 is a flowchart illustrating a pulse wave detection procedure by the pulse detection device 100.

The pixel value acquisition unit 30 acquires spatially leveled pixel values from a predetermined number of pixel values in each block of the captured image (S10).

The time-series pixel array acquisition unit 40 acquires a time-series pixel array in which the pixel values of blocks having a predetermined number of frames are arranged (S20). The time-series pixel array acquisition unit 40 applies a low-pass filter to the time-series pixel array (S30).

The pulse data array acquisition unit 50 calculates the average value of each element of the time-series pixel array as a DC component, and acquires the pulse component by subtracting the DC component from the central element of the time-series pixel array (S40). The central element was selected as the AC component, and the DC component was subtracted from the AC component to give it the function of a high-pass filter. The reason for selecting the central element as the AC component is that there are about the same number of elements before and after the central element when applying a low-pass filter to the time series pixel array, and the same degree of weighting is applied before and after the time. This is because a low-pass filter is applied.

The pulse data array acquisition unit 50 acquires a pulse data array in which pulse components for a predetermined period are arranged (S50). The pulse data array acquisition unit 50 applies a low-pass filter to the pulse data array (S60).

The pulse detection unit 60 obtains the period of the pulse wave signal from the autocorrelation of the pulse data array, and detects the frequency which is the reciprocal of the cycle as the pulse rate (S70).

In the above description, the frequency is obtained from the maximum point of the autocorrelation graph of the detected pulse wave signal, but the pulse detection unit 60 applies a quadratic curve to the vicinity of the peak of the autocorrelation of the detected pulse wave to obtain a peak. By detecting the frequency corresponding to the pulse rate as the pulse rate, the detection accuracy of the pulse rate can be improved. When the frame rate of the captured image is 30 fps, when the frequency is detected from the maximum point of the autocorrelation graph, the time resolution becomes 1/30 second, and the frequency detection accuracy is not high. Therefore, a quadratic curve is applied near the peak to interpolate the autocorrelation graph, and the frequency is detected at the maximum point of the fitted quadratic curve to improve the detection accuracy.

FIG. 4A shows an autocorrelation graph of the detected pulse wave, and FIG. 4B is an enlarged view of 200 near the peak of the autocorrelation graph of FIG. 4A. At the time resolution of 1/30 second, the position indicated by reference numeral 210b is the maximum point of the autocorrelation graph, but when a quadratic curve 230 that passes through the

values

210a, 210b, 210c of the autocorrelation graph near the peak is applied, The position where the quadratic curve 230 is maximized is given by reference numeral 220. The detection accuracy can be improved by obtaining the frequency of the detected pulse wave from the maximum point 220 of the quadratic curve 230.

(Second embodiment)
FIG. 5 is a block diagram of the pulse detection device 100 according to the second embodiment. The pulse detection device 100 according to the second embodiment further includes a reliability calculation unit 80 and a reference graph storage unit 90 in addition to each configuration of the pulse detection device 100 of FIG. The description of the configuration and operation common to the pulse detection device 100 of FIG. 1 will be omitted.

The reference graph storage unit 90 stores an autocorrelation graph of a pulse data array in which noise-free pulse components for a predetermined period are arranged as a reference graph. In order to model a frequent pulse wave as a reference pulse wave, for example, a noise-free reference pulse wave autocorrelation graph such as when the pulse rate is stable, when the pulse rate rises, and when the pulse rate falls is used as a reference graph. Register in the storage unit 90.

In order to compare the autocorrelation graph of the actual detected pulse wave with the reference graph of the reference pulse wave, it is necessary to align the fundamental frequencies. The fundamental frequency obtained from the autocorrelation graph of the detected pulse wave changes according to the pulse rate, but the fundamental frequency of the reference graph of the reference pulse wave is fixed to, for example, 60 bpm (beats per minute). In order to evaluate the similarity of waveforms by cross-correlation between the autocorrelation graph of the detected pulse wave and the reference graph of the reference pulse wave, it is necessary to make the fundamental frequencies of both the same. Therefore, "frequency standardization" is performed to expand and contract the data of the pulse data array of the detected pulse wave in the time axis direction so that the fundamental frequency of the autocorrelation graph of the detected pulse wave becomes 60 bpm, which is the same as the reference graph of the detected pulse wave.

FIG. 6 (a) shows the detected pulse wave and its autocorrelation graph, and FIG. 6 (b) shows the reference pulse wave and its autocorrelation graph. The value on the horizontal axis of the second maximum point of the autocorrelation fluff is the fundamental period, and its reciprocal is the fundamental frequency. Frequency standardization is performed to correct the fundamental frequency of the autocorrelation of the detected pulse wave so as to match the fundamental frequency of the autocorrelation of the reference pulse wave.

The reliability calculation unit 80 calculates the cross-correlation between the auto-correlation graph of the detected pulse wave stored in the auto-correlation graph storage unit 70 and the reference graph of the reference pulse wave stored in the reference graph storage unit 90, and cross-correlates. Obtain the waveform reliability based on the value. The cross-correlation CC is given by the sum of the products of the autocorrelation graph AC of the detected pulse wave and the reference graph REF of the reference pulse wave as shown in the following equation.
CC = SUM (AC [n] * REF [n])

The higher the cross-correlation value, the higher the similarity between the waveforms of the autocorrelation graph of the detected pulse wave and the reference graph of the reference pulse wave. The reliability calculation unit 80 obtains a cross-correlation value with the auto-correlation graph of the detected pulse wave for all the reference graphs of the reference pulse wave registered in the reference graph storage unit 90, and determines the maximum value as the final waveform reliability. And. The waveform reliability is an index for determining the pulse wave-likeness of the waveform of the detected pulse wave.

Further, the reliability calculation unit 80 obtains the amplitude reliability by comparing the amplitude of the waveform of the detected pulse wave signal with the predetermined upper limit value and lower limit value.

FIG. 7 (a) shows the detected pulse wave. The horizontal axis is time, the vertical axis is the green pixel value, and the time change of the pixel value is shown in a graph. The amplitude of the detected pulse wave is given by the difference between the maximum value and the minimum value of the detected pulse wave.

FIG. 7 (b) shows the amplitude reliability. The horizontal axis is the amplitude of the detected pulse wave, and the vertical axis is the amplitude reliability. When the amplitude of the detected pulse wave is the lower limit value th1 or more and the upper limit value th2 or less, the amplitude reliability is 1.0. If the amplitude of the detected pulse wave is in the range between the lower limit value th1 and the upper limit value th2, it is an amplitude like a pulse wave, and the amplitude reliability is 1.0.

When the amplitude of the detected pulse wave is smaller than the lower limit value th1, the amplitude is smaller than the assumed pulse wave, so the amplitude reliability is gradually lowered from 1.0 as the amplitude becomes smaller than the lower limit value th1. If the amplitude of the detected pulse wave is smaller than the lower limit, it may be a minute signal or noise other than the pulse wave.

When the amplitude of the detected pulse wave is larger than the upper limit value th2, the amplitude is larger than the assumed pulse wave, so the amplitude reliability is gradually lowered from 1.0 as the amplitude increases from the upper limit value th2. If the amplitude of the detected pulse wave is larger than the upper limit, it may be noise, probably because the user's body is moving.

The reliability calculation unit 80 calculates and outputs the final pulse wave reliability of the detected pulse rate based on the waveform reliability and the amplitude reliability. For example, the product of waveform reliability and amplitude reliability is calculated as the final pulse wave reliability of the pulse rate.

(Third embodiment)
FIG. 8 is a block diagram of the pulse detection device 100 according to the third embodiment. The pulse detection device 100 according to the third embodiment further includes an area reliability storage unit 92 in addition to each configuration of the pulse detection device 100 of FIG. The description of the configuration and operation common to the pulse detection device 100 of FIG. 5 will be omitted.

In the pulse detection device 100 according to the first and second embodiments, the captured image stored in the frame storage unit 20 is divided into a plurality of blocks, and the pulse component is acquired from the pixel value of each block. In the pulse detection device 100 according to the embodiment, the captured image is divided into a plurality of areas, and the pulse component is acquired from the pixel value of each area. For example, in addition to a narrow area such as the forehead of the face, the right cheek, and the left cheek, there is a wide area including the eyes and nose, and multiple areas can be specified by allowing duplication. The number of pixels in each area may be different.

The pixel value acquisition unit 30 acquires the pixel values of each area and adds the predetermined number of pixel values of each area to acquire the spatially smoothed pixel values.

The time-series pixel array acquisition unit 40 acquires a time-series pixel array in which the pixel values of each area acquired by the pixel value acquisition unit 30 are arranged by a predetermined number of frames, and applies a low-pass filter to the time-series pixel array. ..

The pulse data array acquisition unit 50 acquires a pulse component by subtracting the average value of each element of the time-series pixel array from the central element of the time-series pixel array in each area, and arranges the pulse components for a predetermined period. Generate a data array and apply a low-pass filter to the pulse data array.

The pulse detection unit 60 obtains the autocorrelation of the pulse data array of each area, stores the autocorrelation graph of each area in the autocorrelation graph storage unit 70, and the frequency of the pulse wave signal detected from the autocorrelation graph of each area. Is output as the pulse rate of each area.

The reliability calculation unit 80 calculates the cross-correlation between the auto-correlation graph of the detected pulse wave stored in the auto-correlation graph storage unit 70 and the reference graph of the reference pulse wave stored in the reference graph storage unit 90. , Obtain the waveform reliability based on the cross-correlation value. Further, the reliability calculation unit 80 obtains the amplitude reliability by comparing the amplitude of the waveform of the detected pulse wave signal in each area with a predetermined upper limit value and lower limit value. The reliability calculation unit 80 calculates the final pulse wave reliability of the detected pulse rate for each area based on the waveform reliability and the amplitude reliability, and the pulse reliability for each area is the area reliability storage unit. Store in 92.

The pulse detection unit 60 may select the pulse with the highest reliability stored in the area reliability storage unit 92 from the pulses detected from each area and output it as the final pulse. The pulse detection unit 60 may obtain the final pulse by weighting and combining the pulse detected from each area with a pulse having a reliability of a predetermined threshold value or more based on the reliability.

9 (a) to 9 (d) are diagrams illustrating a plurality of areas set in the captured image. As an example, the forehead 16a of the user's face image 12, the right cheek 16b, the left cheek 16c, and the wide area 16d including the eyes and nose are set as a plurality of areas. Here, the wide area 16d partially overlaps with the forehead 16a, the right cheek 16b, and the left cheek 16c.

In FIG. 9A, since the entire face of the user is exposed, the pulse rate is detected with high reliability in the four areas 16a to 16d.

In FIG. 9B, since the forehead is covered with hair, the pulse rate detected from the forehead 16a is unreliable. Further, since the forehead portion of the wide area 16d is blocked by the hair, it is difficult to detect the pulse component from the pixel value of the forehead portion. Reliability is reduced.

In FIG. 9C, since the user wears glasses, a part of the right cheek 16b and the left cheek 16c is blocked by the glasses, so that the case is compared with the right cheek 16b and the left cheek 16c in FIG. 9A. , The reliability of the detected pulse rate is reduced. Further, it becomes difficult to detect the pulse component from the pixels around the eyes in the wide area 16d, and the reliability of the detected pulse rate is lowered as compared with the wide area 16d in FIG. 9A.

In FIG. 9D, since the user puts his / her hand on the mouth, the right cheek 16b, the left cheek 16c, and a part of the wide area 16d are blocked by the hand, so that the right cheek 16b and the left cheek in FIG. 9A are shown. Compared to 16c and 16d over a wide area, the reliability of the detected pulse rate is lower.

In addition to this, pulse wave signals cannot be detected in some areas due to wearing obstacles such as hats, head-mounted displays, masks, makeup, dark melanin pigments on the skin, the effects of reflected light and shadows, and facial movements. Or, even if it can be detected, the signal may be weakened.

The degree of decrease in the reliability of the pulse rate differs depending on the size of the region where the pulse wave signal is weakened or cannot be detected due to an obstacle or the like. By comparing the pulse rate detected in each area and the reliability at each time and adopting the pulse rate in the area with high reliability at each time, it is possible to maintain high pulse detection accuracy at each time.

FIGS. 10 (a) and 10 (b) are diagrams showing changes in the pulse rate and reliability detected in each area over time.

FIG. 10A is a graph of the time change of the pulse rate detected in each area of the forehead 16a, the right cheek 16b, the left cheek 16c, and the wide area 16d. Depending on the time of day, part of the area may not be visible due to facial movements, or part of the area may be blocked by obstacles such as hands, which weakens the pulse wave signal in that area and increases the detected pulse rate. It has become unstable.

FIG. 10B is a graph of the time change of the reliability of the detected pulse wave in each area of the forehead 16a, the right cheek 16b, the left cheek 16c, and the wide area 16d. It can be seen that the reliability of a specific area is extremely reduced depending on the time of day. The pulse rate detected from the area is not adopted in the time zone when the reliability of the detected pulse wave in each area is lower than the predetermined threshold value. In the time zone when the reliability of the detected pulse wave in each area is equal to or higher than a predetermined threshold, the pulse rate detected from that area is adopted, and the weighted average is taken by the reliability to calculate the final pulse rate. The final pulse rate is the pulse rate detected from the area with the highest reliability.

(Fourth Embodiment)
FIG. 11 is a block diagram of the pulse detection device 100 according to the fourth embodiment. The pulse detection device 100 according to the fourth embodiment further includes a pixel exclusion unit 32 and a brightness change compensation unit 34 in addition to each configuration of the pulse detection device 100 of FIG. The description of the configuration and operation common to the pulse detection device 100 of FIG. 1 will be omitted.

The pixel value acquisition unit 30 divides the captured image into a plurality of blocks, acquires the pixel value of each block, and supplies the captured image to the pixel exclusion unit 32.

The pixel exclusion unit 32 narrows down the target pixels of each block by excluding the pixels unsuitable for pulse wave detection from the pixel values of each block.

Specifically, the pixel exclusion unit 32 narrows down the target pixels of each block by excluding the pixels whose green value is equal to or less than a predetermined threshold when the color of the pixels of each block is represented by RGB. Since the pulse wave signal is small in dark pixels, it becomes noise. Therefore, dark pixels whose green value of the pixel is equal to or less than a predetermined threshold value are excluded. In the case of 8-bit pixels having a maximum value of 255, for example, pixels having a green value of 20 or less are excluded.

The pixel exclusion unit 32 may narrow down the target pixels of each block by excluding the pixels whose red value is equal to or greater than a predetermined threshold value when the color of the pixels of each block is represented by RGB. No pulse wave signal is detected in the pixels whose brightness is saturated. Since the bright reflected light from the skin is saturated in red before that in green, whether or not the brightness is saturated can be determined by the value of red. Therefore, pixels whose red value is equal to or greater than a predetermined threshold value are excluded. In the case of 8-bit pixels having a maximum value of 255, for example, pixels having a red value of 254 or more are excluded. For example, even if the green value is 150, if the red value is 255, the pixel is excluded.

It is preferable to exclude the red pixel whose brightness is saturated from the target pixel because it is used for the brightness change compensation by the brightness change compensation unit 34 described later.

The pixel exclusion unit 32 may narrow down the target pixels of each block by further excluding pixels that are far from a predetermined threshold value from the skin color when the color of the pixels of each block is represented by hue. When the hue is 0.0 on the red side and 1.0 on the purple side, for example, pixels having a hue in the range of 0.3 to 0.85 are excluded.

The pixel exclusion unit 32 supplies the target pixels of each block narrowed down in this way to the brightness change compensation unit 34.

Brightness changes due to changes in shadows due to facial movements and changes in lighting for the face, making pulse wave detection difficult. Therefore, it is necessary to compensate for the change in brightness. On the same skin, there is a positive correlation between changes in green and changes in red and blue. Therefore, the value of the green pixel having a strong pulse wave signal is compensated by using the value of the red pixel or the blue pixel having a weak pulse wave signal to make the brightness constant.

The brightness change compensation unit 34 compensates for the change in brightness by dividing the average value of the green values when the color of the target pixel of each block is represented by RGB by the average value of the red or blue values.

The brightness change compensation unit 34 divides the average value of the green values when the color of the target pixel of each block is represented by RGB by the sum of the average value of the red values and the average value of the blue values to obtain brightness. You may compensate for the change in color.

The brightness change compensation unit 34 supplies the green value of the target pixel whose brightness change has been compensated in this way to the time-series pixel array acquisition unit 40.

In the above description, after the pixels are eliminated by the pixel exclusion unit 32, the brightness change compensation unit 34 compensates for the brightness change, but the brightness change compensation unit 34 does not compensate for the brightness change. The target pixels narrowed down by the pixel exclusion unit 32 by eliminating the pixels may be supplied to the time-series pixel array acquisition unit 40. Further, the brightness change compensating unit 34 compensates for the brightness change for the pixel value of each block supplied from the pixel value acquiring unit 30 without eliminating the pixels by the pixel eliminating unit 32, and the time-series pixels. It may be supplied to the sequence acquisition unit 40.

FIG. 12 is a diagram illustrating pixel exclusion by the pixel exclusion unit 32 of FIG. The pixel exclusion unit 32 excludes pixels whose green value is equal to or less than a predetermined threshold value with respect to the pixels of the block 18a. Next, the pixel exclusion unit 32 excludes pixels whose red value is equal to or greater than a predetermined threshold value with respect to the pixels of the block 18a. Finally, the pixel exclusion unit 32 converts the color space of the pixels of the block 18a from RGB to HSV composed of three components of hue, saturation, and lightness, and from a predetermined range assumed as the skin color in hue. Eliminate out-of-order pixels. As a result, the target pixels are narrowed down as shown by the block 18b. The pixels excluded here are shown by hatching.

As described above, according to the pulse detection device 100 of the present embodiment, fine fluctuations in the pulse can be accurately measured by obtaining the autocorrelation of the pulse wave signal.

The present invention has been described above based on the embodiments. It is understood by those skilled in the art that the embodiments are exemplary and that various modifications are possible for each of these components and combinations of processing processes, and that such modifications are also within the scope of the present invention. ..

This invention can be used for pulse detection technology.

10 image pickup unit, 20 frame storage unit, 30 pixel value acquisition unit, 32 pixel exclusion unit, 34 brightness change compensation unit, 40 time series pixel array acquisition unit, 50 pulse data array acquisition unit, 60 pulse detection unit, 70 autocorrelation Graph storage unit, 80 reliability calculation unit, 90 reference graph storage unit, 92 area reliability storage unit, 100 pulse detection device.

Claims

A storage unit that stores a predetermined number of captured images in an area including the user's skin, and a storage unit.
A pixel value acquisition unit that divides the captured image into a plurality of blocks and acquires the pixel value of each block.
A time-series pixel array acquisition unit that acquires a time-series pixel array in which the pixel values of each block having a predetermined number of frames are arranged, and a time-series pixel array acquisition unit.
The average value of each element of the time-series pixel array is calculated as a DC component, and the pulse component is obtained by subtracting the DC component from a predetermined element of the time-series pixel array, and the pulse components for a predetermined period are arranged. The pulse data array acquisition unit that acquires the pulse data array,
A pulse detection device including a pulse detection unit that detects a frequency obtained by obtaining an autocorrelation of the pulse data array as a pulse rate.
The pulse detection device according to claim 1, wherein the pixel value acquisition unit acquires a green value as a pixel value when the color of the pixel of each block is represented by RGB.
The pulse detection device according to claim 1 or 2, wherein the predetermined period is a time of at least two cycles of a standard pulse.
The pulse according to any one of claims 1 to 3, wherein the pixel value acquisition unit acquires spatially smoothed pixel values by adding a predetermined number of pixel values of each block. Detection device.
Any of claims 1 to 4, wherein the time-series pixel array acquisition unit acquires a time-series pixel array smoothed in time by applying a low-pass filter to the time-series pixel array. The pulse detection device described in Crab.
The pulse detection device according to claim 5, wherein the pulse data sequence acquisition unit acquires a time-smoothed pulse data sequence by applying a low-pass filter to the pulse data sequence.
The pulse detection device according to claim 6, wherein the filter strength of the low-pass filter for the time-series pixel array is stronger than the filter strength of the low-pass filter for the pulse data array.
The auto-correlation graph of the pulse data array in which the noise-free pulse components of a predetermined period are arranged and the pulse count is P is registered as a reference graph, and the auto-correlation graph of the pulse data array acquired by the pulse data array acquisition unit is pulsed. Claims 1 to 7 further include a reliability calculation unit that calculates the cross-correlation with the reference graph after correcting the number to P, and obtains the waveform reliability based on the cross-correlation value. The pulse detection device according to any one of.
The reliability calculation unit is characterized in that the amplitude reliability is obtained by comparing the amplitude of the waveform of the pulse data array acquired by the pulse data array acquisition unit with a predetermined upper limit value and lower limit value. 8. The pulse detection device according to 8.
The pulse detection unit according to any one of claims 1 to 9, wherein the pulse detection unit detects a frequency corresponding to a peak obtained by applying a quadratic curve to the vicinity of the peak of autocorrelation of the pulse data array as a pulse rate. The described pulse detector.
A pixel value acquisition step of dividing an image captured by a predetermined number of frames in an area including the user's skin into a plurality of blocks and acquiring the pixel value of each block, and
A time-series pixel array acquisition step for acquiring a time-series pixel array in which the pixel values of each block having a predetermined number of frames are arranged, and
The average value of each element of the time-series pixel array is calculated as a DC component, and the pulse component is obtained by subtracting the DC component from a predetermined element of the time-series pixel array, and the pulse components for a predetermined period are arranged. The pulse data array acquisition step to acquire the pulse data array,
A pulse detection method comprising a pulse detection step of detecting a frequency obtained by obtaining an autocorrelation of the pulse data array as a pulse rate.
A pixel value acquisition function that divides an image captured by a predetermined number of frames in an area including the user's skin into multiple blocks and acquires the pixel value of each block.
A time-series pixel array acquisition function that acquires a time-series pixel array in which the pixel values of each block having a predetermined number of frames are arranged, and
The average value of each element of the time-series pixel array is calculated as a DC component, and the pulse component is obtained by subtracting the DC component from a predetermined element of the time-series pixel array, and the pulse components for a predetermined period are arranged. The pulse data array acquisition function to acquire the pulse data array,
A program characterized in that a computer realizes a pulse detection function that detects a frequency obtained by obtaining an autocorrelation of the pulse data array as a pulse rate.