JP2022031001A - Heartbeat measurement system, heartbeat measurement method and heartbeat measurement program - Google Patents

Abstract

To more accurately measure the heartbeat on the basis of an image.SOLUTION: A heartbeat measurement system S comprises an image data acquisition unit 211, a pulse wave signal detection unit 213, and a heartbeat measurement unit 215. The image data acquisition unit 211 acquires a plurality of images generated by continuously imaging a measurement object person as a subject. The pulse wave signal detection unit 213 detects a pulse wave signal being a signal indicating pulse wave components of the measurement object person on the basis of the change of a value expressing a color between the plurality of images. The heartbeat measurement unit 215 detects the timing of the heartbeat in the pulse wave signal by analyzing the cross correlation between the pulse wave signal and a model signal being a model of the waveform indicating the heartbeat, and measures the heartbeat of the measurement object person on the basis of the detected timing of the heartbeat.SELECTED DRAWING: Figure 3

Description

The present invention relates to a heart rate measuring system, a heart rate measuring method, and a heart rate measuring program.

Conventionally, it has been widely practiced to measure heartbeat for the purpose of prevention of heart disease and the like. It is desirable that this heart rate measurement be performed as easily as possible. To meet this demand, for example, a small wearable device capable of measuring heart rate is used. However, even if it is small, wearing a wearable device may be a burden to the measurer.
Therefore, there is a technique for measuring the heartbeat without contacting the person to be measured. For example, in the technique disclosed in Patent Document 1, an image of a person to be measured is taken as a subject. Then, the heartbeat is measured based on the change in brightness in this image.

Japanese Unexamined Patent Publication No. 2020-902817

However, it is considered that there is room for improvement in the general technique for measuring the heartbeat based on an image, as disclosed in the above-mentioned Patent Document 1 and the like, in order to improve the accuracy of the measurement.

The present invention has been made in view of such a situation. An object of the present invention is to measure the heartbeat based on an image with higher accuracy.

In order to solve the above problems, the heart rate measurement system according to the embodiment of the present invention is
An image acquisition means for acquiring a plurality of images generated by continuously shooting a measurement target person as a subject, and an image acquisition means.
A pulse wave signal detecting means for detecting a pulse wave signal which is a signal indicating a pulse wave component of the measurement target person based on a change in a value expressing a color between the plurality of images.
By analyzing the cross-correlation between the pulse wave signal and the model signal which is a model of the waveform showing the heartbeat, the timing of the heartbeat in the pulse wave signal is detected, and the measurement target is based on the detected timing of the heartbeat. A measuring means for measuring a person's heartbeat and
It is characterized by having.

According to the present invention, it is possible to measure the heartbeat based on an image with higher accuracy.

It is a figure which shows the system structure of the heart rate measurement system which concerns on one Embodiment of this invention. It is a block diagram which shows an example of the structure of the photographing apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows an example of the structure of the heart rate measuring apparatus which concerns on one Embodiment of this invention. It is a graph which shows the correction of the waveform of a pulse wave signal. It is a graph which shows an example of a model signal used for cross-correlation analysis. It is a graph which shows an example of the distribution of the number of detections for demonstrating the measurement of the heartbeat timing. It is a graph which shows an example of the measurement result of the measurement with the cross-correlation analysis in this embodiment. It is a flowchart which shows the flow of the imaging process executed by the imaging apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the flow of the heart rate measurement processing executed by the heart rate measuring apparatus which concerns on one Embodiment of this invention.

Hereinafter, an example of an embodiment of the present invention will be described with reference to the accompanying drawings.

[System configuration]
FIG. 1 is a diagram showing a system configuration of the heart rate measurement system S according to the present embodiment. As shown in FIG. 1, the heart rate measuring system S includes a photographing device 10 and a heart rate measuring device 20. Further, FIG. 1 also shows a measurement target person who is a target person whose heart rate is measured by the heart rate measurement system S.

The photographing device 10 and the heart rate measuring device 20 are connected to each other so as to be able to communicate with each other. This communication may be performed in accordance with any communication method, and the communication method is not particularly limited. Further, the connection in this communication may be a wired connection or a wireless connection. Further, this communication may be performed directly between the devices or may be performed via a network including a relay device. In this case, the network is realized by, for example, a LAN (Local Area Network), a network such as the Internet or a mobile phone network, or a network in which these are combined.

The heart rate measurement system S is a system in which an image of a person to be measured is taken by the photographing device 10 and the heart rate is measured based on the image by the heart rate measuring device 20 with higher accuracy.

The photographing device 10 generates a plurality of images by continuously photographing the person to be measured as a subject. Then, the photographing device 10 transmits the image data corresponding to the generated plurality of images to the heart rate measuring device 20. The photographing device 10 can be realized by, for example, an information processing device having a photographing function such as a smartphone, a digital camera, or a video camera.

The heart rate measuring device 20 acquires the image data transmitted by the photographing device 10. Then, the heart rate measuring device 20 measures the heart rate of the measurement target person based on the acquired image data. The heart rate measuring device 20 can be realized by, for example, an information processing device having an arithmetic processing capability such as a personal computer or a server device.
When measuring the heartbeat, the heartbeat measuring device 20 detects a pulse wave signal which is a signal indicating a pulse wave component of a measurement target person based on a change in a value expressing a color between a plurality of images. Further, the heart rate measuring device 20 detects the timing of the heartbeat in the pulse wave signal by analyzing the cross-correlation between the pulse wave signal and the model signal which is a model of the waveform showing the heartbeat, and also detects the timing of the heartbeat in the pulse wave signal. The heartbeat of the person to be measured is measured based on.

In this way, in the heart rate measurement system S, the pulse wave signals are detected from the image of the measurement target person by the cooperation of each device. Then, the heartbeat measurement system S detects the timing of the heartbeat by analyzing the cross-correlation between the pulse wave signal and the model signal, and measures the heartbeat based on this.
Thereby, according to the heart rate measurement system S, the measurement regarding the heart rate based on the image can be performed with higher accuracy. Therefore, according to the heart rate measurement system S, for example, it affects the measurement of the change in the orientation of the measurement target person who is the subject (for example, the change in the face orientation) and the change in brightness due to the surrounding environment. Even if there is such an event, it is possible to measure the heartbeat of the person to be measured. Further, according to the heart rate measurement system S, it is possible to measure the heart rate without contacting the measurement target person, and the measurement target person does not need to wear a wearable device or the like, so that the measurement related to the heart rate can be realized more easily. It becomes possible.

Next, the configuration and functions of the photographing device 10 and the heart rate measuring device 20 for realizing such processing will be described in more detail.

[Configuration of shooting device]
Next, the configuration of the photographing apparatus 10 will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the configuration of the photographing apparatus 10.
As shown in FIG. 2, the photographing apparatus 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a communication unit 14, and a storage unit 15. A unit 16, an output unit 17, and an imaging unit 18 are provided. Each of these parts is connected by a signal line and sends and receives signals to and from each other.

The CPU 11 executes various processes (for example, a shooting process described later) according to a program recorded in the ROM 12 or a program loaded from the storage unit 15 into the RAM 13.
Data and the like necessary for the CPU 11 to execute various processes are also appropriately stored in the RAM 13.

The communication unit 14 controls communication for the CPU 11 to communicate with another device (for example, a heart rate measuring device 20).
The storage unit 15 is composed of a semiconductor memory such as a DRAM (Dynamic Random Access Memory) and stores various data.

The input unit 16 is composed of various buttons and the like, and inputs various information according to a user's instruction operation. The output unit 17 is composed of a display, a speaker, or the like, and outputs images and sounds. When the photographing device 10 is realized by, for example, a smartphone or a digital camera, the input unit 16 is configured by a touch sensor and is arranged on the display of the output unit 17 so as to have a touch panel. Is also possible.
The image pickup unit 18 is composed of an image pickup device including a lens, an image pickup element, and the like, and captures a digital image of a subject.

In the photographing apparatus 10, the "imaging process" is performed by the cooperation of each of these parts. Here, the photographing process is a series of processes in which a plurality of images are generated by continuously photographing the measurement target person as a subject, and image data corresponding to the generated plurality of images is transmitted to the heart rate measuring device 20. Is.

When the shooting process is executed, as shown in FIG. 2, the shooting control unit 111 and the image data transmission unit 112 function in the CPU 11.
Further, an image data storage unit 251 is provided in one area of the storage unit 15.
Data necessary for realizing processing is appropriately transmitted and received between these functional blocks, even if not specifically mentioned below.

The shooting control unit 111 is instructed or communicated by the photographer (the person to be measured may be the photographer or another person may be the photographer) accepted by the input unit 16. Based on the instruction information from the photographer received through the unit 14, the image pickup unit 18 controls continuous shooting with the measurement target person as the subject. For example, the shooting control unit 111 outputs a live view screen and a predetermined user interface for assisting the shooting by the photographer from the output unit 17, and controls such as reflecting the operation instruction received by the input unit 16. .. Here, in the present embodiment, the imaging control unit 111 assumes that the face of the measurement target is taken as a subject for continuous imaging, but this is only an example for explanation. The imaging device 10 may capture images of a portion other than the face of the person to be measured (for example, a portion of the upper body near the neck, hands, or heart).

Further, the shooting control unit 111 generates a plurality of images by continuous shooting by the imaging unit 18. Then, the photographing control unit 111 stores the image data corresponding to the generated plurality of images in the image data storage unit 251. That is, the image data storage unit 251 functions as a storage unit for storing image data. The image data corresponding to the plurality of images may be in a general-purpose moving image format. For example, it may be a moving image format of 30 FPS (Frames Per Second) expressed by RGB.

The image data transmission unit 112 transmits the image data stored in the image data storage unit 151 by the shooting control unit 111 to the heart rate measuring device 20. For transmission, the generated image data may be stored in the image data storage unit 151, and the image data stored in the image data storage unit 151 may be collectively transmitted at once after the shooting is completed. , Real-time transmission may be performed with the generation of image data by the photographing control unit 111.

[Structure of heart rate measuring device]
Next, the configuration of the heart rate measuring device 20 will be described with reference to FIG. FIG. 3 is a block diagram showing an example of the configuration of the heart rate measuring device 20. As shown in FIG. 3, the heart rate measuring device 20 includes a CPU 21, a ROM 22, a RAM 23, a communication unit 24, a storage unit 25, an input unit 26, an output unit 27, and a drive 28. .. Each of these parts is connected by a signal line and sends and receives signals to and from each other.

The CPU 21 executes various processes (for example, a heartbeat determination process described later) according to a program recorded in the ROM 22 or a program loaded from the storage unit 25 into the RAM 23.
Data and the like necessary for the CPU 21 to execute various processes are also appropriately stored in the RAM 23.

The communication unit 24 controls communication for the CPU 21 to communicate with another device (for example, a photographing device 10).
The storage unit 25 is composed of a semiconductor memory such as a DRAM and stores various data.

The input unit 26 is composed of various buttons, a touch panel, or an external input device such as a mouse or a keyboard, and inputs various information according to a user's instruction operation.
The output unit 27 is composed of a display, a speaker, or the like, and outputs images and sounds.

A removable medium (not shown) made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 28. The programs and various data read from the removable media by the drive 28 are installed in the storage unit 25 as needed.

In the heart rate measuring device 20, the "heart rate measuring process" is performed by the cooperation of each of these parts.
Here, the heart rate measurement process is a series of processes for acquiring image data transmitted by the photographing device 10 and measuring the heart rate of the measurement target person based on the image data.

When this heart rate measurement process is executed, as shown in FIG. 3, in the CPU 21, the image data acquisition unit 211, the area extraction unit 212, the pulse wave signal detection unit 213, the waveform correction unit 214, and the heart rate measurement unit 215 and, work.
Further, in one area of the storage unit 25, an image data storage unit 251, a pulse wave signal storage unit 252, and a model signal storage unit 253 are provided.
Data necessary for realizing processing is appropriately transmitted and received between these functional blocks, even if not specifically mentioned below.

The image data acquisition unit 211 acquires image data corresponding to a plurality of images with the measurement target person as a subject transmitted from the photographing device 10. Then, the image data acquisition unit 211 stores the acquired image data in the image data storage unit 251. That is, the image data storage unit 251 functions as a storage unit for storing image data.

The photographing device 10 may store the image data in the removable media. Then, the image data acquisition unit 211 may acquire the image data from the removable media inserted in the drive 28 instead of acquiring the image data by communication.

In the region extraction unit 212, the pulse wave signal detection unit 213, which will be described later, detects the pulse wave signal, which is a signal indicating the pulse wave component of the measurement target person, from the image data from each of the plurality of images corresponding to the image data. , Performs a process of extracting a predetermined area. Specifically, the region extraction unit 212 is a portion other than the background, clothes, and face that is unnecessary when detecting the pulse wave component from each of the plurality of images corresponding to the image data stored in the image data storage unit 251. Extracts the face area, which is the area excluding the area where is photographed. Further, the pulse wave signal detection unit 213 extracts from each of the extracted facial regions a skin region, which is a region excluding the region where the hair, eyeball, and oral cavity, which are unnecessary when detecting the pulse wave component, are photographed. do. Then, the pulse wave signal detection unit 213 outputs the image after extracting this skin region to 313. This skin area becomes a target area for detecting the pulse wave signal by the pulse wave signal detection unit 213 described later.

Here, the extraction of the face region and the skin region by the region extraction unit 212 can be realized by any method. For example, the extraction of the face region by the region extraction unit 212 can be realized by a method called the Viola-Jones method, which is widely used as a method for extracting the face region. The Viola-Jones method is based on Haar Like features that show features in the facial area (eg, the area around the eyes is darker than the cheeks, or the areas on either side of the nose are darker). Generate a Haar Cascade classifier. Then, based on this Har Cascade classifier, the face region can be extracted from the image with high accuracy.

Next, the region extraction unit 212 further extracts a skin region from each of the extracted face regions. For example, the extraction of the skin region by the region extraction unit 212 can be realized by using a classifier such as OC-SVM (One Class Support Vector Machine). In this case, the region extraction unit 212 first converts the image data represented by RGB so as to be represented by the luminance signal Y and YCrCb represented by the two color difference signals Cr and Cb. Then, the region extraction unit 212 separates the luminance signal Y and extracts the skin region based only on the two color difference signals Cr and Cb. As described above, the region extraction unit 212 can extract the skin region by reducing the influence of the shadow of the skin by once separating the lightness component indicated by the luminance signal Y.

Then, the region extraction unit 212 detects outliers by OC-SVM. In this case, for example, the pulse wave signal detection unit 213 considers that the center of the face region is the skin, and uses the Cr and Cb values of the pixels in this center region as normal data to perform learning for one class. , Determine the identification boundaries for outlier detection. Then, the region extraction unit 212 detects a region (corresponding to a region other than the skin) corresponding to the outliers based on the obtained identification boundary. This allows the skin area and the non-skin area to be separated. Then, the region extraction unit 212 reconverts the skin region into the original image represented by RGB again. On the other hand, the region extraction unit 212 replaces each RGB value with zero for the non-skin region. The region extraction unit 212 can extract the skin region from the image in this way.

By such a process of extracting the face region and extracting the skin region, the region extraction unit 212 extracts the skin region from each of the plurality of images corresponding to the image data (that is, from each frame in the moving image). (Hereinafter referred to as "skin area image") can be generated. Then, the region extraction unit 212 outputs a plurality of skin region images corresponding to each of the plurality of images to the pulse wave signal detection unit 213.

The pulse wave signal detection unit 213 is a signal indicating the pulse wave component of the measurement target person based on the change in the value expressing the color between the plurality of skin region images input from the region extraction unit 212. Detect the signal. For example, the detection of the pulse wave signal by the region extraction unit 212 can be realized by a method called rPPG (Remote Photo-PlethysmoGraphy), which is widely used as a method for detecting a pulse wave signal. In rPPG, since the blood component (for example, hemoglobin) absorbs light more than the surrounding tissue, attention is paid to the fact that the pulse wave component, which is a component associated with blood flow, is superimposed on the diffuse reflection of the skin.

Specifically, rPPG detects a pulse wave signal, which is a signal indicating the pulse wave component of the measurement target, based on the change in the RGB value in the region of interest in the image (here, the skin region in the face region). .. In this case, if the image includes regions such as the hair, the eyeball, and the oral cavity, these movements can be disturbing. However, in the present embodiment, since these non-skin regions are excluded and the skin regions are extracted by the region extraction unit 212, the influence of such disturbance can be reduced.

Further, in this case, the pulse wave signal detection unit 213 performs a predetermined pretreatment on the skin region image in consideration of the individual difference in the skin color of each measurement target person and the color of the light source, and then rPPG. The pulse wave signal may be detected by. For example, from each of the plurality of skin region images (that is, from each frame of the moving image consisting of the skin region image), C (t), which is a spatial average RGB value for each frame, is calculated. Then, C _n (t) for each frame is calculated by dividing each C (t) of each frame by the average value of the values C (t) of all the frames. Then, the pulse wave signal is detected by _rPPG for Cn (t) after this pretreatment. As a result, it is possible to suppress individual differences in the skin color of each measurement target person and the influence of the color of the light source.

Further, the pulse wave signal detection unit 213 may further apply a method called POS (Plane-Orthogonal-to-Skin) in the detection of the pulse wave signal by rPPG. POS is a method that focuses on the fact that a pulse wave component and a component derived from a change in brightness are mixed in the change in RGB value, and the pulse is directed to an axis orthogonal to the direction of the vector in which the change in brightness occurs in RGB space. C (t) (or C _n (t) after the pretreatment described above), which is a spatial average RGB value, is projected for wave component extraction. As a result, the pulse wave signal detection unit 213 can detect the pulse wave signal with higher accuracy. The details of POS are disclosed in, for example, Non-Patent Document 1 below.

<Non-Patent Document 1>
W. Wang, et al., "Algorithmic Principles of Remote PPG", [online], IEEE Transactions on Biomedical Engineering, Vol. 64, No. 7, 2017, [Search on July 13, 2nd year of Reiwa], Internet < URL: https://pure.tue.nl/ws/portalfiles/portal/31563684/TBME_00467_2016_R1_preprint.pdf>

The pulse wave signal detection unit 213 detects the pulse wave signal in this way, and outputs the detected pulse wave signal to the waveform correction unit 214.

The waveform correction unit 214 corrects the pulse wave signal input from the pulse wave signal detection unit 213 for the measurement of the heartbeat by the heart rate measurement unit 215. FIG. 4 is a graph showing the correction of the waveform of the pulse wave signal by the pulse wave signal detection unit 213. As shown in FIG. 4A, as a premise, the pulse wave signal has a fluctuation of the baseline due to a change in lighting at the time of photographing or the like. Therefore, the waveform correction unit 214 performs trend removal using a moving average filter. For example, the waveform correction unit 214 converts the pulse wave signal into a Z-score by converting the pulse wave signal so that the average is 0 and the standard deviation is 1. Next, the waveform correction unit 214 extracts a moving average line having a time window of a predetermined length (for example, a width of 1 second) as a baseline signal from the pulse wave signal. Then, the waveform correction unit 214 subtracts this baseline signal from the pulse wave signal. Then, the pulse wave signal shown in FIG. 4A becomes a trend-removed pulse wave signal as shown in FIG. 4B. Further, the waveform correction unit 214 further processes the pulse wave signal to be transmitted to a band-pass filter (for example, a band-pass filter of 0.5 [Hz] -5 [Hz]) in consideration of the frequency characteristics of the pulse wave. You may do.

The waveform correction unit 214 corrects the pulse wave signal in this way. Then, the waveform correction unit 214 stores the corrected pulse wave signal in the pulse wave signal storage unit 252. That is, the pulse wave signal storage unit 252 functions as a storage unit for storing the pulse wave signal.

The heart rate measuring unit 215 detects the timing of the heartbeat in the pulse wave signal by analyzing the cross-correlation between the pulse wave signal and the model signal which is a model of the waveform showing the heartbeat, and is based on the detected heartbeat timing. To measure the heartbeat of the person to be measured.

The model signal is a signal corresponding to a model of a waveform showing an ideal heartbeat in rPPG. As a waveform showing an ideal heartbeat, for example, a waveform corresponding to the heartbeat in rPPG, which is a typical heartbeat waveform generally seen in a measurement target person, or a waveform of a plurality of heartbeats is averaged. Heartbeat waveforms can be used.

FIG. 5 is a graph showing an example of a model signal used for cross-correlation analysis by the heart rate measuring unit 215. The model signal having such a waveform is created in advance by a user who uses the heart rate measurement system S and is stored in the model signal storage unit 253. That is, the model signal storage unit 253 functions as a storage unit for storing the model signal. It should be noted that this user can appropriately modify this model signal or the like so that the measurement can be performed with higher accuracy by referring to the measurement result or the like regarding the heartbeat using this model signal.

When performing cross-correlation analysis with this model signal, the heart rate measuring unit 215 first reads the pulse wave signal from the pulse wave signal storage unit 252, and a time window of a predetermined length to be analyzed (for example, a width of 2 seconds). ). In addition, the heart rate measurement unit 215 converts the cut out pulse wave signal to be analyzed into a Z-score, and calculates the cross-correlation coefficient by performing cross-correlation analysis with the model signal stored in the model signal storage unit 253. do. Then, the heartbeat measuring unit 215 detects a portion having a high calculated correlation coefficient as the timing of the heartbeat. However, at this point, the heartbeat measuring unit 215 only counts the detected heartbeat timing as a candidate for the heartbeat timing.

Then, the heartbeat measuring unit 215 newly cuts out the analysis target from the pulse wave signal, detects the heartbeat timing in this way, and repeats counting it as a candidate for the heartbeat timing. Here, the cutout of the analysis target overlaps with the portion previously analyzed (that is, partially overlaps the portion previously analyzed) and has a time window of a predetermined length (for example, 0.1 second). Width) This is done by shifting. That is, the analysis target is shifted by a time window of a predetermined length (for example, 0.1 second width), and the cross-correlation analysis with the model signal is repeated. Then, when such a shift is repeated and all the pulse wave signals stored in the heart rate measuring unit 215 are cut out as analysis targets and the cross-correlation analysis is performed, the repetition of the cross-correlation analysis ends.

Next, the heart rate measuring unit 215 measures the heart rate of the person to be measured based on the number of times the heart rate timing is detected in the repetition of this cross-correlation analysis (that is, the number of times counted as a candidate for the heart rate timing). do. FIG. 6 is a graph showing an example of the distribution of the number of detections for explaining the measurement of the heartbeat timing by the heartbeat measuring unit 215. As an example, as shown in FIG. 6A, the distribution of the number of times the heartbeat timing is detected by the heartbeat measuring unit 215 in the repetition of the cross-correlation analysis (that is, the number of times counted as a candidate for the heartbeat timing) is Suppose it was created. In this case, the heart rate measuring unit 215 searches for a peak (for example, a local maximum value) by performing a peak search for a waveform showing such a distribution.

In FIG. 6B, the peaks searched in this way are indicated by “dashed line circles”. The heart rate measuring unit 215 selects the peak thus searched as the timing of the measured heart rate. On the other hand, even if the timing is counted as a candidate for the heartbeat timing several times, the timing with a small number of counts that is not searched for as a peak is not selected as the measured heartbeat timing and is discarded. .. As a result, it is possible to select a timing with a higher likelihood and measure the timing of the heartbeat of the measurement target person with higher accuracy.

Further, the heart rate measuring unit 215 outputs the measurement result. For example, the heart rate measuring unit 215 outputs the measured heart rate timing itself of the measurement target person as a measurement result. Alternatively, the heart rate measuring unit 215 outputs the interval of the heartbeat timing of the measured person to be measured as a measurement result. This output is, for example, a display on a display included in the output unit 27, an audio output from a speaker included in the output unit 27, or a paper medium from a printing device (not shown) via a communication unit 24. It may be printing to or transmitting to another device (not shown) via the communication unit 24. A user who uses the heart rate measurement system S can utilize the output measurement result as useful information for, for example, prevention of heart disease.

FIG. 7 is a graph showing an example of the measurement result of the measurement accompanied by the cross-correlation analysis in the present embodiment. For comparison, FIG. 7A shows, as the correct value, the heartbeat interval measured by an electrocardiograph that measures the heartbeat by bringing the electrode into contact with the body of the person to be measured. Further, FIG. 7B shows, as a conventional example, the heart rate interval measured when a peak search is simply performed on the pulse wave signal measured by rPPG. On the other hand, FIG. 7C shows the measurement when various correction processes, cross-correlation analysis with the model signal, and selection based on the detection frequency distribution are performed as described above as the present embodiment. Indicates the heart rate interval. It should be noted that the measurement with the electrocardiograph corresponding to FIG. 7 (A) and the imaging for obtaining the image data corresponding to FIGS. 7 (B) and (C) are simultaneously performed for the same measurement target person. Was done. Therefore, ideally, the measured heart rate intervals should be the same.

However, when FIG. 7 (A) and FIG. 7 (B) are compared, a very large error occurs. That is, it can be seen that a very large measurement error occurs when a peak search is simply performed on the pulse wave signal measured by rPPG as compared with an electrocardiograph that can measure with high accuracy. On the other hand, when FIG. 7 (A) and FIG. 7 (C) are compared, the results are very close and almost no error occurs. That is, it can be seen that, according to the present embodiment, the measurement can be performed with the same high accuracy as compared with the electrocardiograph which can measure with the high accuracy.
As described above, according to the present embodiment, the measurement regarding the heartbeat based on the image can be performed with higher accuracy as in the electrocardiograph. In addition, according to the present embodiment, it is possible to measure the heartbeat without contacting the measurement target person, and the measurement target person does not need to wear an electrocardiograph, so that the measurement related to the heartbeat can be realized more easily. It becomes possible.

[Shooting process]
Next, with reference to FIG. 8, the flow of the photographing process executed by the photographing apparatus 10 will be described. FIG. 8 is a flowchart illustrating a flow of a photographing process executed by the photographing apparatus 10. The shooting process is executed in accordance with an instruction operation for starting shooting from the photographer (the person to be measured may be the photographer or another person may be the photographer).

In step S11, the imaging control unit 111 starts generating a plurality of images by controlling continuous imaging with the measurement target person as the subject by the imaging unit 18.
In step S12, the photographing control unit 111 stores the image data corresponding to the plurality of images generated in step S11 in the image data storage unit 251.

In step S13, the shooting control unit 111 determines whether or not to end the shooting by the imaging unit 18. Here, the shooting is terminated on condition that the shooting in step S11 has elapsed for a predetermined time and that the photographer has instructed the shooting to end. When the shooting is finished, it is determined as Yes in step S13, and the process proceeds to step S14. On the other hand, if the shooting is not completed, it is determined as No in step S13, the process returns to step S11, and the process is repeated.

In step S14, the image data transmission unit 112 transmits the image data stored in the image data storage unit 151 by the shooting control unit 111 to the heart rate measuring device 20. This ends this process. In the figure, it is assumed that the generated image data is stored in the image data storage unit 151, and the image data stored in the image data storage unit 151 is collectively transmitted at once after the shooting is completed. However, as described above, the image data may be transmitted in real time as the image data is generated by the photographing control unit 111.

By the photographing process described above, the photographing apparatus 10 generates a plurality of images by continuously photographing the measurement target person as a subject, and transfers the image data corresponding to the generated plurality of images to the heart rate measuring apparatus 20. Can be sent.

[Measurement model construction process]
Next, with reference to FIG. 9, the flow of the heart rate measurement process executed by the heart rate measuring device 20 will be described. FIG. 9 is a flowchart illustrating a flow of a heart rate measurement process executed by the heart rate measuring device 20. The heart rate measurement process is executed in accordance with an instruction operation for starting the heart rate measurement process from the user of the heart rate measurement system S.

In step S21, the image data acquisition unit 211 acquires image data corresponding to a plurality of images of the measurement target person transmitted from the photographing device 10 as a subject.

In step S22, the image data acquisition unit 211 stores the image data acquired in step S21 in the image data storage unit 251.
In step S23, the area extraction unit 212 extracts a face region from the image data stored in the image data storage unit 251 in step S22.

In step S24, the region extraction unit 212 extracts a skin region from the face region extracted in step S23 and generates a plurality of skin region images.
In step S25, the pulse wave signal detection unit 213 is a signal indicating the pulse wave component of the measurement target person based on the change in the value expressing the color among the plurality of skin region images generated in step S24. Detects a certain pulse wave signal.

In step S26, the waveform correction unit 214 corrects the pulse wave signal detected in step S25 for the measurement related to the heartbeat.
In step S27, the waveform correction unit 214 stores the pulse wave signal corrected in step S26 in the pulse wave signal storage unit 252.

In step S28, the heart rate measuring unit 215 analyzes the cross-correlation between the pulse wave signal stored in the pulse wave signal storage unit 252 in step S27 and the model signal which is a model of the waveform indicating the heartbeat. Detects the timing of the heartbeat in the wave signal.

In step S29, the heart rate measuring unit 215 measures the heart rate of the person to be measured based on the number of times the heart rate timing is detected in step S28 (that is, the number of times counted as a candidate for the heart rate timing).

In step S30, the heart rate measuring unit 215 outputs the measurement result measured in step S29. This ends this process.

According to each process described above, as described with reference to FIG. 7, the measurement regarding the heartbeat based on the image can be performed with higher accuracy as in the electrocardiograph. In addition, according to each of the processes described above, it is possible to measure the heart rate without contacting the measurement target person, and the measurement target person does not need to wear an electrocardiograph, so that the measurement related to the heart rate can be performed more easily. It will be possible to realize it.

[Modification example]
Although the embodiment of the present invention has been described above, this embodiment is merely an example and does not limit the technical scope of the present invention. The present invention can take various other embodiments without departing from the gist of the present invention, and can be modified in various ways such as omission and substitution. In this case, these embodiments and variations thereof are included in the scope and gist of the invention described in the present specification and the like, and are included in the scope of the invention described in the claims and the equivalent scope thereof.

As an example, the device configuration of the heart rate measurement system S in the above-described embodiment is only an example, and can be appropriately changed. For example, in the above-described embodiment, the photographing device 10 and the heart rate measuring device 20 are realized as separate devices, but the photographing device 10 and the heart rate measuring device 20 may be realized as an integrated device. .. In addition, for example, the heart rate measuring device 20 may be distributed and realized by a plurality of computers, for example, such as a cloud system.

[Configuration example]
As described above, the heart rate measurement system S according to the present embodiment includes an image data acquisition unit 211, a pulse wave signal detection unit 213, and a heart rate measurement unit 215.
The image data acquisition unit 211 acquires a plurality of images generated by continuously shooting the measurement target person as a subject.
The pulse wave signal detection unit 213 detects a pulse wave signal which is a signal indicating a pulse wave component of a measurement target person based on a change in a value expressing a color between a plurality of images.
The heart rate measuring unit 215 detects the timing of the heartbeat in the pulse wave signal by analyzing the cross-correlation between the pulse wave signal and the model signal which is a model of the waveform showing the heartbeat, and is based on the detected heartbeat timing. To measure the heartbeat of the person to be measured.
In this way, the heart rate measurement system S detects the pulse wave signal from the image of the person to be measured. Then, the heartbeat measurement system S detects the timing of the heartbeat by analyzing the cross-correlation between the pulse wave signal and the model signal, and measures the heartbeat based on this.

Thereby, according to the heart rate measurement system S, the measurement regarding the heart rate based on the image can be performed with higher accuracy. Therefore, according to the heart rate measurement system S, for example, it affects the measurement of the change in the orientation of the measurement target person who is the subject (for example, the change in the face orientation) and the change in brightness due to the surrounding environment. Even if there is such an event, it is possible to measure the heartbeat of the person to be measured. Further, according to the heart rate measurement system S, the heart rate can be measured without contacting the measurement target person, and the measurement target person does not need to wear a wearable device or the like, so that the measurement related to the heart rate can be realized more easily. It becomes possible.

The heart rate measurement unit 215 detects the timing of the heartbeat by shifting the part to be analyzed for the cross-correlation of the pulse wave signal by superimposing a part on the part to be analyzed before and then analyzing the cross-correlation. Repeat that.
The heart rate measuring unit 215 measures the heart rate of the person to be measured based on the number of times the heart rate timing is detected in the repetition.
This makes it possible to perform a plurality of analyzes on the same portion of the pulse wave signal. Therefore, the likelihood of measurement is improved, and the measurement regarding the heartbeat of the person to be measured can be performed with higher accuracy.

The heart rate measuring unit 215
By obtaining a distribution along the time series of the number of times the heartbeat timing is detected and performing a peak search for the obtained distribution, the timing of the heartbeat with a small number of detections is the timing of the heartbeat used for measurement. Exclude from.
As a result, the timing of the heartbeat with a small number of detections can be discarded. Therefore, the likelihood of measurement is improved, and the measurement regarding the heartbeat of the person to be measured can be performed with higher accuracy.

The plurality of images are images that include the face portion of the measurement target as the subject.
The heart rate measuring unit 215 measures the interval between the heartbeats of the measurement target as a measurement related to the heartbeat.
This makes it possible to measure the heartbeat interval of the person based on the facial image of the person who is the subject.

[Realization of functions by hardware and software]
The function of executing a series of processes according to the above-described embodiment can be realized by hardware, software, or a combination thereof. In other words, it suffices if the function of executing the above-mentioned series of processes is realized in any one of the heart rate measurement systems S, and the mode in which this function is realized is not particularly limited.

For example, when the function of executing the above-mentioned series of processes is realized by a processor that executes the arithmetic processing, the processor that executes the arithmetic processing is composed of various processing units such as a single processor, a multiprocessor, and a multicore processor. In addition to the above, the present invention includes a combination of these various processing units and a processing circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

Further, for example, when the function of executing the above-mentioned series of processes is realized by software, the programs constituting the software are installed in the computer via a network or a recording medium. In this case, the computer may be a computer having dedicated hardware built-in, or a general-purpose computer capable of performing a predetermined function by installing a program (for example, a general-purpose personal computer or the like). It may be an electronic device in general). Further, the step for describing the program may include only the processes performed in time series in the order thereof, but may include the processes executed in parallel or individually. Further, the steps for describing the program may be executed in any order within a range that does not deviate from the gist of the present invention.

The recording medium on which such a program is recorded may be provided to the user by being distributed separately from the computer main body, or may be provided to the user in a state of being incorporated in the computer main body in advance. In this case, the storage medium distributed separately from the computer itself is composed of a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disc is composed of, for example, a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versaille Disc), a Blu-ray (registered trademark) Disc (Blu-ray disc), or the like. The magneto-optical disk is composed of, for example, an MD (MiniDisc) or the like. These storage media are attached to, for example, the drive 28 of FIG. 3 and incorporated into the computer body. Further, the recording medium provided to the user in a state of being incorporated in the computer body in advance is, for example, the ROM 12 of FIG. 2 in which the program is recorded, the ROM 22 of FIG. 3, the storage unit 15 of FIG. 2, or the storage of FIG. It is composed of a hard disk and the like included in the unit 25.

10 Imaging device, 20 Heart rate measuring device, 11,21 CPU, 12,22 ROM, 13,23 RAM, 14,24 communication unit, 15,25 storage unit, 16,26 input unit, 17,27 output unit, 18 image pickup Unit, 28 drive, 111 shooting control unit, 112 image data transmission unit, 151,251 image data storage unit, 211 image data acquisition unit, 212 area extraction unit, 213 pulse wave signal detection unit, 214 waveform correction unit, 215 heart rate measurement Unit, 216 measurement unit, 252 pulse wave signal storage unit, 253 model signal storage unit, S heart rate measurement system

Claims (6)

An image acquisition means for acquiring a plurality of images generated by continuously shooting a measurement target person as a subject, and an image acquisition means.
A pulse wave signal detecting means for detecting a pulse wave signal which is a signal indicating a pulse wave component of the measurement target person based on a change in a value expressing a color between the plurality of images.
By analyzing the cross-correlation between the pulse wave signal and the model signal which is a model of the waveform showing the heartbeat, the timing of the heartbeat in the pulse wave signal is detected, and the measurement target is based on the detected timing of the heartbeat. A measuring means for measuring a person's heartbeat and
A heart rate measurement system characterized by being equipped with. The measuring means is
The heartbeat timing is repeatedly detected by shifting the part to be analyzed for the cross-correlation of the pulse wave signal by overlapping the part to be analyzed before and then analyzing the cross-correlation.
Based on the number of times the heartbeat timing is detected in the repetition, the measurement regarding the heartbeat of the measurement target person is performed.
The heart rate measuring system according to claim 1. The measuring means is
By obtaining a distribution along the time series of the number of times the heartbeat timing is detected and performing a peak search for the obtained distribution, the heartbeat used when performing the measurement is obtained for the timing of the heartbeat with a small number of detections. Exclude from the timing of
The heart rate measuring system according to claim 2. The plurality of images are images including the face portion of the measurement target as a subject.
The measuring means measures the interval between the heartbeats of the person to be measured as a measurement relating to the heartbeat.
The heart rate measuring system according to any one of claims 1 to 3. An image acquisition step of acquiring a plurality of images generated by continuously shooting a measurement target person as a subject, and
A pulse wave signal detection step for detecting a pulse wave signal, which is a signal indicating a pulse wave component of the measurement target, based on a change in a value expressing a color between the plurality of images.
By analyzing the cross-correlation between the pulse wave signal and the model signal which is a model of the waveform showing the heartbeat, the timing of the heartbeat in the pulse wave signal is detected, and the measurement target is based on the detected timing of the heartbeat. Measurement steps to measure a person's heartbeat and
A heart rate measuring method comprising. An image acquisition function that acquires multiple images generated by continuously shooting with the person to be measured as the subject, and
A pulse wave signal detection function that detects a pulse wave signal, which is a signal indicating a pulse wave component of the measurement target, based on a change in a value expressing a color between the plurality of images.
By analyzing the cross-correlation between the pulse wave signal and the model signal which is a model of the waveform showing the heartbeat, the timing of the heartbeat in the pulse wave signal is detected, and the measurement target is based on the detected timing of the heartbeat. A measurement function that measures a person's heartbeat and
A heart rate measurement program characterized by the realization of a computer.

Images