JP2018033931A - Pulse wave measurement device and pulse wave measurement method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a pulse wave measurement device capable of achieving further improvement.SOLUTION: A pulse wave measurement device 10 includes a processor and a memory. The processor acquires a visible light image acquired by imaging a user at which visible light is radiated by a visible light source in a visible light region, acquires an infrared light image acquired by imaging the user at which infrared light is radiated by an infrared light source in an infrared light region, extracts a visible light wave form which is the wave form showing a pulse wave of the user from the acquired visible light image, extracts an infrared light wave form which is the wave form showing the pulse wave of the user from the acquired infrared light image, calculates the degree of correlation between the extracted visible light wave form and the extracted infrared light wave form, outputs a control signal for controlling a light amount of the infrared light emitted by the infrared light source to the infrared light source according to the calculated degree of correlation, and by using at least one of the visible light wave form and the infrared light wave form, calculates biological information and outputs the calculated biological information.SELECTED DRAWING: Figure 7

Description

  The present disclosure relates to a pulse wave measurement device and a pulse wave measurement method for measuring a person's pulse wave in a non-contact manner.

  Patent Document 1 discloses a technique for measuring heartbeat and sleep depth in a non-contact state using millimeter waves, visible light, infrared light, and the like.

  Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique for satisfactorily switching from an infrared photographing mode in which infrared light is irradiated to a subject to a normal photographing mode in a photographing apparatus.

JP2013-192620A JP 2004-146873 A JP 2007-130182 A

  However, the techniques disclosed in Patent Document 1 and Patent Document 2 require further improvement.

  A pulse wave measurement device according to an aspect of the present disclosure is a pulse wave measurement device including a processor and a memory, and the processor captures an image of a user irradiated with visible light from a visible light source in a visible light region. Obtaining the first visible light image obtained, obtaining the first infrared light image obtained by imaging the user irradiated with infrared light from the infrared light source in the infrared light region, and obtaining the first visible light image A first visible light waveform, which is a waveform indicating the user's pulse wave, is extracted from the first visible light image, and a waveform indicating the user's pulse wave is obtained from the acquired first infrared light image. A first infrared light waveform is extracted, a first correlation between the extracted first visible light waveform and the extracted first infrared light waveform is calculated, and the calculated first correlation is the second A first determination is made to determine whether or not the threshold is greater than or equal to As a result of the first determination, when it is determined that the calculated first correlation is equal to or greater than the second threshold, a first control signal for reducing the amount of visible light in the visible light source is supplied to the visible light source. And outputting a second control signal for increasing the amount of infrared light in the infrared light source to the infrared light source, and the visible light source is irradiated with visible light based on the first control signal. A second visible light image obtained by imaging the user in the visible light region is acquired, and the user irradiated with infrared light based on the second control signal by the infrared light source is infrared light. A second infrared light image obtained by imaging in a region is acquired, and a second visible light waveform that is a waveform indicating the pulse wave of the user is extracted and acquired from the acquired second visible light image. From the second infrared light image, The second infrared light waveform, which is a waveform indicating the pulse wave of the user, is extracted, and biological information is calculated using at least one of the extracted second visible light waveform and the second infrared light waveform. The biometric information is output.

  These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. The system, method, integrated circuit, computer program And any combination of recording media.

  According to the said aspect, the further improvement is realizable.

FIG. 1 is a schematic diagram showing a state in which the pulse wave measuring device according to the present embodiment is used by a user. FIG. 2 is a plan view of the pulse wave measurement device when the pulse wave measurement device is viewed from below. FIG. 3 is a block diagram illustrating an example of a hardware configuration of the pulse wave measurement device. FIG. 4 is a block diagram illustrating an example of a hardware configuration of the mobile terminal according to the first embodiment. FIG. 5 is a diagram for explaining an example of use of the pulse wave measuring apparatus. FIG. 6 is a diagram for explaining an example of use of the pulse wave measurement device. FIG. 7 is a block diagram illustrating an example of a functional configuration of the pulse wave measurement device according to the present embodiment. FIG. 8 is a graph showing an example of the luminance change of the visible light image and the infrared light image in the present embodiment. FIG. 9 is a graph showing an example of calculation of pulse wave timing in the present embodiment. FIG. 10 is a graph showing an example of heartbeat interval times acquired in time series. FIG. 11 is a graph for explaining a method of extracting an inflection point from a pulse wave. FIG. 12 is a graph showing a visible light waveform for explaining a method of calculating the slope between the vertex and the bottom point in the visible light waveform. FIG. 13 is a graph showing an infrared light waveform when a human skin image is acquired by an infrared light camera for each level of the light amount of the infrared light source. FIG. 14 is a graph showing the first heartbeat interval time and the second heartbeat interval time plotted with data in time series order. FIG. 15 is a diagram for describing a specific example of determining whether or not the heartbeat interval time is appropriate. FIG. 16 is a diagram for explaining an example in the case where excessive acquisition of peak points is performed in the visible light waveform and no excessive acquisition of peak points is performed in the corresponding infrared light waveform. FIG. 17 is a diagram for explaining a case where the degree of correlation is calculated using inflection points. FIG. 18 is a diagram for explaining an example that does not apply to the condition that the number of peak points in the first predetermined period exceeds the first threshold even though the number of peak points is excessive. FIG. 19 is a diagram illustrating an example for explaining that the peak point acquired during the adjustment of the light amount of the light source is not used for calculating the degree of correlation between the visible light waveform and the infrared light waveform. FIG. 20 is a diagram showing an example of the simplest steps for decreasing the light amount of the visible light source to zero and increasing the light amount of the infrared light source to an appropriate light amount using the pulse wave measurement device. is there. FIG. 21 illustrates that the light source control is waited until two or more predetermined feature points continuous from the waveform are extracted within the second predetermined period in each of the visible light waveform and the infrared light waveform. FIG. FIG. 22 is a diagram illustrating a display example on the presentation device. FIG. 23 is a flowchart showing the flow of processing of the pulse wave measuring apparatus in the present embodiment. FIG. 24 is a flowchart showing details of peak point excess acquisition determination processing in the present embodiment. FIG. 25 is a flowchart showing details of correlation degree calculation processing in the present embodiment. FIG. 26 is a flowchart showing details of light amount adjustment processing in the present embodiment. FIG. 27 is a diagram for explaining the characteristics of the pulse wave measurement device according to the modification.

(Knowledge that became the basis of the present invention)
The inventor has found that the following problems occur with respect to the technique described in the “Background Art” column.

  Patent Document 1 does not disclose adjustment of the light amount of an infrared light source when acquiring a pulse wave in a dark room, so that there is a problem that it is difficult to measure heartbeats and pulse waves in a dark room without contact.

  Further, in Patent Document 2, mode switching is used by using the ratio between the luminance of visible light and the luminance of infrared light. However, when the mode switching is applied to pulse wave measurement in a dark room, the ratio of luminance is used. There is a problem that the pulse wave cannot be easily measured by the switching by.

  Therefore, the present disclosure provides a pulse wave measuring device and the like that can accurately measure a pulse wave in a dark room.

  A pulse wave measurement device according to an aspect of the present disclosure is a pulse wave measurement device including a processor and a memory, and the processor captures an image of a user irradiated with visible light from a visible light source in a visible light region. Obtaining the first visible light image obtained, obtaining the first infrared light image obtained by imaging the user irradiated with infrared light from the infrared light source in the infrared light region, and obtaining the first visible light image A first visible light waveform, which is a waveform indicating the user's pulse wave, is extracted from the first visible light image, and a waveform indicating the user's pulse wave is obtained from the acquired first infrared light image. A first infrared light waveform is extracted, a first correlation between the extracted first visible light waveform and the extracted first infrared light waveform is calculated, and the calculated first correlation is the second A first determination is made to determine whether or not the threshold is greater than or equal to As a result of the first determination, when it is determined that the calculated first correlation is equal to or greater than the second threshold, a first control signal for reducing the amount of visible light in the visible light source is supplied to the visible light source. And outputting a second control signal for increasing the amount of infrared light in the infrared light source to the infrared light source, and the visible light source is irradiated with visible light based on the first control signal. A second visible light image obtained by imaging the user in the visible light region is acquired, and the user irradiated with infrared light based on the second control signal by the infrared light source is infrared light. A second infrared light image obtained by imaging in a region is acquired, and a second visible light waveform that is a waveform indicating the pulse wave of the user is extracted and acquired from the acquired second visible light image. From the second infrared light image, The second infrared light waveform, which is a waveform indicating the pulse wave of the user, is extracted, and biological information is calculated using at least one of the extracted second visible light waveform and the second infrared light waveform. The biometric information is output.

  According to this, the first red light obtained from the first visible light waveform obtained from the first visible light image obtained by imaging the pulse wave of the user and the first red light image obtained by imaging the same pulse wave. The first correlation with the external light waveform is calculated, and the amount of infrared light emitted from the infrared light source is controlled according to the first correlation. For this reason, the amount of infrared light can be adjusted appropriately, and biological information can be calculated accurately.

  Further, for example, in the calculation of the first correlation, the processor divides the first visible light waveform into a plurality of first unit waveforms in a pulse wave cycle unit that is a cycle of the pulse wave, and For each 1-unit waveform, a first peak point that is one of the first vertex that is the maximum value in the first unit waveform and the first base point that is the minimum value in the first unit waveform is extracted. The plurality of first peak points are extracted from the first visible light waveform, the first infrared light waveform is divided into a plurality of second unit waveforms in the pulse wave period unit, and the plurality of second unit waveforms are obtained. For each of the second peak, which is one of the second vertex that is the maximum value in the second unit waveform and the second base point that is the minimum value in the second unit waveform, A plurality of the second peaks from one infrared light waveform For each of the plurality of extracted first peak points, a first time at the first peak point and a first time at another first peak point adjacent to the first peak point in time series are extracted. By calculating a first heartbeat interval time that is a time interval between two times, a plurality of the first heartbeat interval times are calculated, and the second peak is extracted for each of the extracted second peak points. By calculating a second heartbeat interval time that is a time interval between the third time at the point and the fourth time at the other second peak point adjacent to the second peak point in time series, The second heart beat interval time of the plurality of first heart beat interval times and the plurality of second heart beat interval times corresponding to each other in time series using the following (formula 1): The first correlation coefficient may be calculated as the first correlation degree.

  According to this, the first correlation coefficient is calculated by comparing the first heartbeat interval time calculated from the first visible light waveform with the second heartbeat interval time calculated from the first infrared light waveform. Calculate as degrees. Therefore, the first correlation between the first visible light waveform and the first infrared light waveform can be easily calculated.

  Further, for example, the processor further determines whether the first standard deviation exceeds a fourth threshold and whether the second standard deviation exceeds the fourth threshold. And when it is determined as a result of the second determination that the first standard deviation exceeds the fourth threshold and the second standard deviation exceeds the fourth threshold, One second heart beat interval corresponding to the first heart beat interval time in time series among the first heart beat interval time of the first heart beat interval times and the second heart beat interval time. Third determination as to whether or not the first time difference with time is less than a fifth threshold, and fourth determination as to whether or not the first time difference is greater than a sixth threshold greater than the fifth threshold As a result of the third determination and the fourth determination, the first time difference is The second control signal is output to the infrared light source, and when the first time difference is determined to be greater than or equal to the fifth threshold and less than or equal to the sixth threshold, A third control signal that increases the amount of visible light in the light source is output to the visible light source, and a fourth control signal that decreases the amount of infrared light in the infrared light source is output to the infrared light source. It may be output.

  In addition, for example, if the processor determines that the first time difference is smaller than the fifth threshold as a result of the third determination and the fourth determination, the second standard deviation is A fifth determination is made to determine whether or not the threshold value is less than or equal to a threshold value. If the second standard deviation is determined to be less than or equal to the fourth threshold value as a result of the fifth determination, the first control signal is determined to be the visible And outputting the second control signal to the infrared light source and determining that the second standard deviation is greater than the fourth threshold value, the third control signal is output to the visible light source. The fourth control signal may be output to the infrared light source.

  Thereby, the light quantity of a visible light source and an infrared light source can be adjusted appropriately.

  Further, for example, the processor further includes a first vertex of one of the plurality of first vertices and one immediately after the first vertex of the plurality of first bottom points in the time series. An inclination of a first straight line connecting the first base point is stored in the memory as a first inclination, the second visible light waveform is divided into a plurality of third unit waveforms in the pulse wave period unit, For each of the third unit waveforms, a third peak that is one of the third vertex that is the maximum value in the third unit waveform and the third base point that is the minimum value in the third unit waveform is extracted. Thus, a plurality of the third peak points are extracted from the second visible light waveform, the second infrared light waveform is divided into a plurality of fourth unit waveforms in the pulse wave period unit, For each unit waveform, it is the maximum value in the fourth unit waveform A plurality of fourth peak points are extracted from the second infrared light waveform by extracting four peak points and a fourth peak point that is one of the fourth bottom points, which is the minimum value in the fourth unit waveform. Then, for each of the extracted third peak points, a fifth time at the third peak point and a sixth time at another third peak point adjacent to the third peak point in time series, By calculating a third heart beat interval time that is a time interval between the plurality of third heart beat interval times, each of the extracted fourth peak points is calculated at the fourth peak point. By calculating a fourth heart beat interval time which is a time interval between the seventh time and the eighth time at another fourth peak point adjacent to the fourth peak point in time series, Calculate 4 heartbeat intervals and use the following (Equation 2) A second correlation coefficient between the plurality of third heart beat interval times and the plurality of fourth heart beat interval times corresponding to each other is calculated, the second visible light image is obtained, and the second visible light waveform is obtained. Extraction of the second infrared light image, extraction of the second infrared light waveform, and calculation of the second correlation coefficient are repeated, and calculation of the second correlation coefficient is repeatedly performed. Furthermore, in the second infrared light waveform obtained by repeatedly performing, the fourth of the fourth vertices of the fourth vertices and the fourth of the fourth bottom points. The second slope, which is the slope of the second straight line connecting the fourth base point immediately after the vertex in the time series, is compared with the first slope stored in the memory, and the second slope The second control signal may be output to the infrared light source until the inclination becomes the first inclination. .

  According to this, in order to compare the second inclination in the second infrared light waveform after adjusting the light quantity of the infrared light source and the first inclination stored in the memory, the light quantity of the infrared light source. It is possible to effectively determine whether or not the light amount is appropriate.

  Further, for example, the processor further determines whether the third standard deviation exceeds the fourth threshold and whether the fourth standard deviation exceeds the fourth threshold. Performing the determination, and as a result of the sixth determination, if it is determined that the third standard deviation exceeds the fourth threshold and the fourth standard deviation exceeds the fourth threshold, One fourth heart beat corresponding to one third heart beat interval time in a time series among the third heart beat interval time of the plurality of third heart beat interval times and the fourth heart beat interval time. Performing a seventh determination as to whether a second time difference from the interval time is less than the fifth threshold and an eighth determination as to whether the second time difference is greater than the sixth threshold; As a result of the seventh determination and the eighth determination, the second time difference is larger than the sixth threshold value. If it is determined that the second control signal is output to the infrared light source, and if it is determined that the second time difference is not less than the fifth threshold and not more than the sixth threshold, the third control signal is The fourth light may be output to the visible light source and the fourth control signal may be output to the infrared light source.

  In addition, for example, if the processor determines that the second time difference is smaller than the fifth threshold as a result of the seventh determination and the eighth determination, the fourth standard deviation is the fourth A ninth determination is made to determine whether or not the threshold value is less than or equal to a threshold value, and when the fourth standard deviation is determined to be less than or equal to the fourth threshold value as a result of the ninth determination, the first control signal is And outputting the second control signal to the infrared light source and determining that the fourth standard deviation is greater than the fourth threshold value, the third control signal is output to the visible light source. The fourth control signal may be output to the infrared light source.

  In addition, for example, in the calculation of the second correlation coefficient that is repeatedly performed, the processor determines whether the number of the third peak points or the number of the fourth peak points exceeds a first threshold value in a first predetermined period. A tenth determination is made to determine whether or not, and as a result of the tenth determination, it is determined that the number of the third peak points or the number of the fourth peak points exceeds the first threshold in the first predetermined period. In this case, for each of the plurality of third vertices, an inflection between the third vertex and the third bottom point immediately after the third vertex in the time series of the third vertex. By extracting the first inflection point that is a point, a plurality of the first inflection points are extracted, and for each of the plurality of fourth vertices, the fourth vertex and the plurality of fourth bottom points Of the fourth base point immediately after the time series of the fourth vertex. A plurality of second inflection points are extracted by extracting a second inflection point, which is an inflection point, and for each of the extracted first inflection points, the first inflection point The time interval between the fifth time and the sixth time at the other first inflection point adjacent to the first inflection point is calculated as the third heartbeat interval time, and the plurality of extracted second times For each of the two inflection points, the time interval between the seventh time at the second inflection point and the eighth time at another second inflection point adjacent to the second inflection point is The plurality of third heartbeat interval times calculated using the first inflection point and the plurality of third heartbeat interval times calculated using the first inflection point, which are calculated as the fourth heartbeat interval time and correspond to each other in time series The correlation coefficient with the fourth heartbeat interval time may be calculated as the second correlation coefficient by using (Equation 2).

  For this reason, when many peak points are acquired, each heartbeat interval time can be calculated by using the inflection points.

  In addition, for example, the processor further includes the third heart beat interval time and the fourth heart beat interval time corresponding to each other in time series among the plurality of third heart beat interval times and the plurality of fourth heart beat interval times. When the absolute error during the period exceeds the third threshold, the number of the plurality of third peak points is compared with the number of the plurality of fourth peak points, and exceeds the third threshold. Among the third heartbeat interval time and the fourth heartbeat interval time determined to be, the heartbeat interval time calculated by the peak point that is determined to be larger as a result of the comparison is specified, and the specified heartbeat You may exclude the peak point used as the reference | standard of calculation of interval time from the calculation object of the said heartbeat interval time.

  For this reason, an excessively acquired peak point can be deleted, and an appropriate value of the third heart beat interval time or the fourth heart beat interval time can be obtained.

  For example, the processor further compares the number of the plurality of third peak points with the number of the plurality of fourth peak points, and the plurality of third heartbeat interval times and the plurality of fourth heartbeats. When the heartbeat interval time calculated based on the peak point that is determined to be smaller as a result of the comparison among the interval times is specified, and the standard deviation of the specified heartbeat interval time is equal to or less than the fourth threshold value For each of the plurality of third vertices, an inflection point between the third vertex and the third bottom point immediately after the third vertex in the time series of the third vertex. By extracting a first inflection point, a plurality of the first inflection points are extracted, and for each of the plurality of fourth vertices, the fourth vertex and the plurality of fourth bottom points The inflection point between the fourth base point immediately after the fourth vertex in the time series A plurality of second inflection points are extracted, and for each of the extracted first inflection points, a ninth time at the first inflection point is extracted. The time interval from the tenth time at the other first inflection point adjacent to the first inflection point is calculated as the third heartbeat interval time, and the extracted second inflection points are calculated. For each, the time interval between the eleventh time at the second inflection point and the twelfth time at another second inflection point adjacent to the second inflection point is the fourth heartbeat interval time. The plurality of third heart beat interval times calculated using the first inflection point and the plurality of fourth heart beat intervals calculated using the second inflection point, which correspond to each other in time series. The correlation coefficient with time may be calculated as the second correlation coefficient by using (Expression 2).

  Thereby, the light quantity of a visible light source and an infrared light source can be adjusted appropriately.

  In addition, for example, in the extraction of the plurality of first peak points, the processor acquires the first visible light waveform acquired in a period excluding a period in which the light amount of the visible light source is controlled by the first control signal. The plurality of first peak points are extracted from, and in the extraction of the plurality of second peak points, acquired in a period excluding a period in which the light amount of the infrared light source is controlled by the second control signal A period excluding a period in which the plurality of second peak points are extracted from the first infrared light waveform and the amount of the visible light source is controlled by the third control signal in the extraction of the plurality of third peak points. The plurality of third peak points are extracted from the second visible light waveform acquired in step S4, and the amount of the infrared light source is controlled by the fourth control signal in the extraction of the plurality of fourth peak points. Have During it may extract the plurality of fourth peak point from the second infrared light waveform acquired in the period except.

  For this reason, the first to fourth peak points can be appropriately extracted, and the biological information can be calculated with high accuracy.

  Further, for example, in the output of the first control signal or the third control signal, the processor extracts two or more first peak points that are continuous from the first visible light waveform within a second predetermined period. Or waiting for the output of the first control signal or the third control signal until two or more consecutive third peak points from the second visible light waveform are extracted within a second predetermined period. And, in the output of the second control signal or the fourth control signal, until two or more second peak points continuous from the first infrared light waveform are extracted within the second predetermined period, or And waiting for the output of the second control signal or the fourth control signal until two or more fourth peak points continuous from the second infrared light waveform are extracted within the second predetermined period. Also good.

  For this reason, the first to fourth peak points can be appropriately extracted, and the biological information can be calculated with high accuracy.

  The pulse wave measurement method according to an aspect of the present disclosure is a pulse wave measurement method, and includes a plurality of first visible lights in a visible light region of a user irradiated with first visible light emitted from a visible light source. An image is acquired, a plurality of second infrared light images in the infrared light region of the user irradiated with the first infrared light emitted from the infrared light source are acquired, and the plurality of first visible light images are acquired. The first visible light waveform is extracted, the first infrared light waveform is extracted from the plurality of first infrared light images, and the correlation between the first visible light waveform and the first infrared light waveform is extracted. When the degree of correlation is equal to or greater than a threshold value, the visible light source emits second visible light, the infrared light source emits second infrared light, and the second visible light Is smaller than the amount of the first visible light per unit time, and the amount of the second infrared light per unit time is the first visible light. A plurality of second visible light images in the visible light region of the user irradiated with the second visible light that are larger than the amount of external light per unit time are acquired, and the user irradiated with the second infrared light is acquired. Acquiring a plurality of second infrared light images in the infrared light region, extracting a second visible light waveform from the plurality of second visible light images, from the plurality of second infrared light images; Extracting a second infrared light waveform, calculating biological information using at least one of the second visible light waveform and the second infrared light waveform, outputting the biological information, and calculating the correlation (1) A plurality of first peak points at a plurality of first times included in a plurality of first unit waveforms are extracted, and the plurality of first peak points are included in the plurality of first unit waveforms. A plurality of first maximum points or a plurality of first maximum points included in the plurality of first unit waveforms. The first visible light waveform includes the plurality of first unit waveforms, the plurality of first maximum value points correspond to the plurality of first unit waveforms, respectively, and the plurality of first unit waveforms. The minimum value point and the plurality of first unit waveforms respectively correspond to each other, the plurality of first unit waveforms and the plurality of first times correspond to each other, and (2) a plurality of second unit waveforms included in the plurality of second unit waveforms. A plurality of second peak points in 2 hours are extracted, and the plurality of second peak points are included in the plurality of second maximum value points included in the plurality of second unit waveforms or the plurality of second unit waveforms. A plurality of second minimum value points, wherein the second visible light waveform includes the plurality of second unit waveforms, and the plurality of second maximum value points and the plurality of second unit waveforms respectively correspond to each other. , The plurality of first minimum value points correspond to the plurality of first unit waveforms, respectively, The plurality of second unit waveforms and the plurality of second times correspond to each other, and (3) a plurality of first heart beat interval times are calculated based on the plurality of first times, and the plurality of first heart beat interval times are calculated. Is a time between a first time and a second time, the plurality of first times includes the first time and the second time, and the time included in the first time is the first time and the second time (4) calculating a plurality of second heartbeat intervals based on the plurality of second times, wherein the plurality of second heartbeat intervals are the third time and the fourth time. The plurality of second times includes the third time and the fourth time, and the time included in the second time does not exist between the third time and the fourth time. (5) A pulse wave measuring method for calculating the degree of correlation using the following (Expression 1).

  These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. The system, method, integrated circuit, computer program And any combination of recording media.

(Embodiment)
In the present embodiment, a pulse wave measurement that acquires a user's pulse wave from each of the user's visible light image and infrared light image and controls the light source based on the degree of correlation between the acquired two pulse wave feature quantities The apparatus will be described.

[1-1. Constitution]
[1-1-1. Pulse wave measuring device]
The configuration of the pulse wave measurement device according to the present embodiment will be described.

  FIG. 1 is a schematic diagram illustrating a state in which the pulse wave measuring device 10 according to the present embodiment is used by a user U. FIG. 2 is a plan view of the pulse wave measuring device 10 when the pulse wave measuring device 10 is viewed from below. FIG. 3 is a block diagram illustrating an example of a hardware configuration of the pulse wave measuring device 10.

  FIG. 2 is a diagram illustrating an example of a UI for operating the pulse wave measurement device 10 to be displayed on the mobile terminal 30.

  The pulse wave measurement device 10 includes a visible light LED 21, a visible light camera 22, an infrared light LED 23, an infrared light camera 24, and a pulse wave calculation device 100. The pulse wave measuring device 10 may be configured to include only the pulse wave computing device 100.

  As shown in FIG. 3, the pulse wave measuring apparatus 10 includes a housing 20, and each component shown in FIG. 3 is arranged on a surface (for example, a lower surface) on the light irradiation side of the housing 20. . Specifically, in the pulse wave measurement device 10, a visible light LED (Light Emitting Diode) 21, a visible light camera 22, an infrared light LED 23, and an infrared light camera 24 are disposed on the lower surface of the housing 20. . The pulse wave measuring device 10 acquires a user's pulse wave using images captured by the visible light camera 22 and the infrared light camera 24, and is visible based on the degree of correlation between the two acquired pulse waves. A pulse wave arithmetic device 100 that controls the light amounts of the light LED 21 and the infrared light LED 23 is provided.

  The visible light LED 21 is a light source that emits visible light, and is, for example, a white LED. Visible light is light in the wavelength region of the visible light region (for example, 400 to 800 nm). The visible light LED 21 is arranged in an annular shape on the lower surface of the housing 20, for example. The visible light LED 21 may be configured by a plurality of bullet-type LEDs, may be configured by a plurality of surface mount device (SMD) LEDs, or may be a COB (Chip On Board) type. The LED may be configured. Further, the visible light LED 21 may not be arranged in an annular shape.

  The visible light camera 22 is a camera that captures visible light. The visible light camera 22 is disposed near the center of the visible light LED 21 disposed in an annular shape. That is, the visible light camera 22 is disposed at a position surrounded by the visible light LED 21. The visible light camera 22 is a camera including an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor Image Sensor). The visible light camera 22 applies RGB color filters to the image sensor, so that visible light, that is, light existing in a wavelength region of 400 to 800 nm is applied to the image sensor by RGB (Red, Green, Blue). These three types of signals are acquired.

  The infrared light LED 23 is a light source that emits infrared light. Infrared light is light in the wavelength region of the infrared light region (for example, 800 to 2500 nm). The infrared LED 23 is arranged inside the visible LED 21 and is arranged in an annular shape. The infrared LED 23 may be composed of a plurality of bullet-type LEDs, a plurality of surface-mount type (SMD) LEDs, or a COB (Chip On Board) type. You may comprise by LED. In addition, the infrared LED 23 may not be arranged in an annular shape. In addition, although the infrared light LED 23 is disposed inside the visible light LED 21, the infrared light LED 23 may be disposed outside the visible light LED 21.

  The infrared light camera 24 is a camera that captures infrared light. The infrared light camera 24 may be a camera that captures electromagnetic waves in a wavelength region (for example, 700 nm to 900 nm) including a part of the visible light region. The infrared light camera 24 is disposed near the center of the infrared light LED 23 disposed in an annular shape. That is, the infrared camera 24 is disposed at a position surrounded by the infrared LED 23. The infrared light camera 24 includes a filter different from that of the visible light camera 22 so that infrared light, that is, light existing in a wavelength region of 800 nm or more is acquired as only one type of monochrome signal in the image sensor. Let

  The pulse wave calculation device 100 is disposed inside the housing 20. The pulse wave computing device 100 includes a CPU (Central Processing Unit) 101, a main memory 102, a storage 103, and a communication IF (Interface) 104.

  The CPU 101 is a processor that executes a control program stored in the storage 103 or the like.

  The main memory 102 is a volatile storage area (main storage device) used as a work area used when the CPU 101 executes a control program.

  The storage 103 is a non-volatile storage area (auxiliary storage device) that holds control programs, various data, and the like.

  The communication IF 104 is a communication interface that transmits and receives data to and from other devices via a network. Specifically, the communication IF 104 outputs a control signal for controlling these devices to the visible light LED 21, the visible light camera 22, the infrared light LED 23, and the infrared light camera 24. Further, the communication IF 104 acquires image data captured by each of the visible light camera 22 and the infrared light camera 24.

  The communication IF 104 may be a communication interface that can be connected to the mobile terminal 30 for communication. Specifically, the communication IF 104 may be a wireless LAN (Local Area Network) interface that conforms to the IEEE 802.11a, b, g, and n standards, or a wireless communication interface that conforms to the Bluetooth (registered trademark) standards. There may be.

[1-1-2. Mobile device]
The hardware configuration of the portable terminal 30 will be described with reference to FIG.

  FIG. 4 is a block diagram illustrating an example of a hardware configuration of the mobile terminal according to the first embodiment.

  As illustrated in FIG. 4, the mobile terminal 30 includes a CPU 201, a main memory 202, a storage 203, a display 204, a communication IF 205, and an input IF 206. The portable terminal 30 is a communicable information terminal such as a smartphone or a tablet terminal.

  The CPU 201 is a processor that executes a control program stored in the storage 203 or the like.

  The main memory 202 is a volatile storage area (main storage device) used as a work area used when the CPU 201 executes a control program.

  The storage 203 is a non-volatile storage area (auxiliary storage device) that holds control programs, various data, and the like.

  A display 204 is a display device that displays a processing result in the CPU 201. The display 204 is, for example, a liquid crystal display or an organic EL display.

  The communication IF 205 is a communication interface that communicates with the pulse wave measurement device 10. The communication IF 205 may be, for example, a wireless LAN (Local Area Network) interface that conforms to the IEEE 802.11a, b, g, or n standard, or a wireless communication interface that conforms to the Bluetooth (registered trademark) standard. Good. In addition, the communication IF 205 is a radio that conforms to a communication standard used in a mobile communication system such as the third generation mobile communication system (3G), the fourth generation mobile communication system (4G), or LTE (registered trademark). It may be a communication interface.

  The input IF 206 is, for example, a touch panel that is arranged on the surface of the display 204 and receives input from the user to a UI (User Interface) displayed on the display 204. The input IF 206 may be an input device such as a numeric keypad or a keyboard.

  5 and 6 are diagrams for explaining an example of use of the pulse wave measuring device 10.

  For example, as illustrated in FIG. 5, the mobile terminal 30 may display a UI for operating the pulse wave measurement device 10 on the display 204. Further, the mobile terminal 30 may transmit a control signal to the pulse wave measuring device 10 in accordance with an input to the UI.

  In the pulse wave measuring device 10, the mobile terminal 30 can be used as a means for the user to switch on / off the visible light LED 21 and the infrared light LED 23. For example, the mobile terminal 30 can be used as a remote controller for the pulse wave measuring device 10 by starting a remote control application for controlling the pulse wave measuring device 10 on the mobile terminal 30. As shown in FIG. 5, the user can turn on the visible light LED 21 by selecting “illumination ON” regardless of whether the infrared light LED 23 is on or off.

  FIG. 6A is an image example when the visible light LED 21 is ON. When the user selects “infrared ON”, the infrared light source can be turned on regardless of whether the visible light LED 21 is on or off. For example, FIG. 6B shows a state in which the visible light LED 21 is OFF but the infrared light LED is ON. In the case of only infrared light illumination, since the user is not dazzled, the user can sleep as usual. Furthermore, when the user selects “OFF”, both the visible light LED 21 and the infrared light LED 23 are turned off, and no light is irradiated to the user.

  When the user selects the “sleep mode”, the visible light LED 21 is turned off and the infrared light LED 21 is turned off from the state in which the visible light LED 21 is turned on and the infrared light LED is turned off. By turning ON and gradually increasing the light amount, the optimum light amount of the infrared LED 23 is determined, and the pulse wave can be acquired even when the user is sleeping.

[1-2. Functional configuration]
Next, the functional configuration of the pulse wave measuring device 10 will be described with reference to FIG.

  FIG. 7 is a block diagram illustrating an example of a functional configuration of the pulse wave measurement device according to the present embodiment.

  As shown in FIG. 7, the pulse wave measurement device 10 includes a visible light source 121, a visible light imaging unit 122, an infrared light source 123, an infrared light imaging unit 124, and a pulse wave calculation device 100. .

  The visible light source 121 irradiates the user with visible light, and the light amount to be irradiated is adjusted by the light source control unit 114. The visible light source 121 is realized by, for example, the visible light LED 21. The visible light source 121 may be realized by a fluorescent lamp.

  In addition, although control of the light quantity of the visible light source 121 was performed by the light source control part 114, it is not restricted to this. The visible light source 121 may be manually controlled by the user himself / herself using a controller, for example. Thereby, for example, when the user can sleep under visible light of a predetermined light amount or less, the user sleeps while acquiring the user's pulse wave in a state where the visible light of the light amount or less is irradiated by the visible light source 121. Is possible.

  Further, the visible light source 121 may be set to be activated every time a user enters a room. Thereby, whenever a user enters a room, a user's pulse wave information can be acquired and the pulse wave information before sleep and after sleep can be acquired not only at the time of sleep.

  The visible light imaging unit 122 images an irradiation target irradiated with visible light from the visible light source 121 in the visible light region. Specifically, the visible light imaging unit 122 captures a visible light image obtained by imaging a user's skin as an irradiation target in a visible light region (for example, color), and a visible light waveform computing unit of the pulse wave computing device 100. To 111. The visible light imaging unit 122 outputs, for example, a plurality of visible light images captured at a plurality of different timings to the visible light waveform calculation unit 111. The visible light imaging unit 122 outputs, for example, a skin image obtained by imaging skin including a human face or hand as a visible light image. The skin image is an image obtained by capturing the same portion of the skin including a human face or hand at a plurality of timings that are temporally continuous, and includes, for example, a moving image or a plurality of still images. The visible light imaging unit 122 is realized by the visible light camera 22, for example.

  The infrared light source 123 irradiates the user with infrared light, and the amount of light to be irradiated is adjusted by the light source control unit 114. The infrared light source 123 is realized by, for example, an infrared LED 23.

  The infrared light imaging unit 124 images an irradiation target irradiated with infrared light from the infrared light source 123 in the infrared light region. Specifically, the infrared light imaging unit 124 captures an infrared light image obtained by imaging the user's skin as an irradiation target in an infrared light region (for example, monochrome), in the infrared of the pulse wave calculation device 100. The result is output to the optical waveform calculator 112. For example, the infrared light imaging unit 124 outputs a plurality of infrared light images captured at a plurality of different timings to the infrared light waveform calculation unit 112. The infrared light imaging unit 124 images the same part as the part imaged by the visible light imaging unit 122. For example, the infrared light imaging unit 124 outputs a skin image obtained by imaging skin including a human face or hand as an infrared light image. This is because the infrared light imaging unit 124 can acquire the same pulse wave in the visible light region and the infrared light region by capturing the same region as the region captured by the visible light imaging unit 122. This is because it is easy to compare feature quantities.

  Note that, as an imaging method of the same part, a region of interest (ROI) having the same size is set in the visible light imaging unit 122 and the infrared light imaging unit 124. And it is judged whether the same site | part is imaged by comparing using the pattern recognition about the image in the said ROI imaged with the visible light imaging part 122 and the infrared light imaging part 124, for example. May be. Further, face recognition is performed on each of the visible light image obtained by the visible light imaging unit 122 and the infrared light image obtained by the infrared light imaging unit 124, and the coordinates of the feature points in the eyes, nose, mouth, and the like are performed. The same part is identified by calculating the coordinates (relative position) from the feature points of the eyes, nose, mouth, etc., considering the size ratio of the eyes, nose, mouth, etc. May be.

  Similar to the skin image obtained by the visible light imaging unit 122, the skin image obtained by the infrared light imaging unit 124 has a plurality of timings at which the same portion of the skin including a human face or hand is temporally continuous. For example, it is composed of a moving image or a plurality of still images. The infrared light imaging unit 124 is realized by the infrared light camera 24, for example.

  The pulse wave calculation device 100 includes a visible light waveform calculation unit 111, an infrared light waveform calculation unit 112, a correlation degree calculation unit 113, a light source control unit 114, and a biological information calculation unit 115. Hereinafter, each component of the pulse wave calculation device 100 will be described in order.

(Visible light waveform calculation unit)
The visible light waveform calculation unit 111 acquires a visible light image from the visible light imaging unit 122, and extracts a visible light waveform that is a waveform indicating a user's pulse wave from the acquired visible light image. The visible light waveform calculation unit 111 extracts the first visible light waveform from the first visible light image acquired before the light amount of the visible light source 121 is controlled. Further, the visible light waveform calculation unit 111 extracts a second visible light waveform from the second visible light image acquired after the light amount of the visible light source 121 is controlled. The control of the light amount of the visible light source 121 means that the first control signal for reducing the light amount of visible light in the visible light source 121 or the visible light in the visible light source 121 by the light source control unit 114 described later. This means that a third control signal for increasing the amount of light is output to the visible light source 121. As described above, the plurality of visible light images acquired from the visible light imaging unit 122 include the first visible light image acquired before the control of the light amount of the visible light source 121 and the light amount of the visible light source 121. And the second visible light image acquired after the above control is performed. The visible light waveforms extracted from the plurality of visible light images include a first visible light waveform extracted from the first visible light image and a second visible light waveform extracted from the second visible light image. It is.

  The visible light waveform calculation unit 111 may extract a plurality of first feature points that are predetermined feature points in the extracted first visible light waveform. Specifically, when the visible light waveform calculation unit 111 divides the first visible light waveform into a plurality of first unit waveforms in a pulse wave cycle unit, which is a pulse wave cycle, for each of the plurality of first unit waveforms. The first visible light waveform is extracted by extracting the first peak point which is one of the first vertex that is the maximum value in the first unit waveform and the first bottom point that is the minimum value in the first unit waveform. To extract a plurality of first peak points. The first peak point is an example of a first feature point.

  The visible light waveform calculation unit 111 acquires a pulse wave timing as a feature point of the visible light waveform, and calculates a heartbeat interval time from the adjacent pulse wave timings. That is, the visible light waveform calculation unit 111 calculates the time between each of the extracted first feature points and another first feature point adjacent to the first feature point as the first heartbeat interval time. . For example, the visible light waveform calculation unit 111 may, for each of the plurality of extracted first peak points, the first time at the first peak point and another first peak adjacent to the first peak point in time series. A plurality of the first heart beat interval times are calculated by calculating a first heart beat interval time which is a time interval between the second time at the point.

  Specifically, the visible light waveform calculation unit 111 extracts a visible light waveform based on temporal changes in luminance extracted from a plurality of visible light images each associated with an imaged timing. In other words, each of the plurality of visible light images acquired from the visible light imaging unit 122 is associated with the time (time point) when the visible light image is captured by the visible light imaging unit 122. The visible light waveform calculation unit 111 acquires the pulse wave timing of the user (hereinafter also referred to as pulse wave timing) by acquiring the interval between predetermined feature points of the visible light waveform. Then, the visible light waveform calculation unit 111 calculates, for each of the obtained plurality of pulse wave timings, an interval between the pulse wave timing and the next pulse wave timing as a heartbeat interval time.

  In addition, the visible light waveform calculation unit 111 may extract a plurality of third feature points that are predetermined feature points in the extracted second visible light waveform. Specifically, when the visible light waveform calculation unit 111 divides the second visible light waveform into a plurality of third unit waveforms in units of pulse wave periods, the third unit waveform for each of the plurality of third unit waveforms. Are extracted from the second visible light waveform by extracting a third peak point that is one of the third vertex and the third bottom point that is the minimum value in the third unit waveform. A peak point may be extracted. The third peak point is an example of a third feature point.

  For each of the extracted third peak points, the visible light waveform calculation unit 111 uses the fifth time at the third peak point and another third peak point adjacent to the third peak point in time series. A plurality of the third heart beat interval times may be calculated by calculating a third heart beat interval time that is a time interval from the sixth time.

  For example, the visible light waveform calculation unit 111 uses the extracted visible light waveform to specify the timing with the largest luminance change, and specifies the specified timing as the pulse wave timing. Alternatively, the visible light waveform calculation unit 111 identifies the positions of the faces or hands in the plurality of visible light images using the face or hand pattern held in advance, and changes the luminance with time at the identified positions. To identify the visible light waveform. The visible light waveform calculation unit 111 calculates the pulse wave timing using the identified visible light waveform. Here, the pulse wave timing is a time at a predetermined feature point in the time waveform of the luminance, that is, the time waveform of the pulse wave. The predetermined feature point is, for example, the peak position (the time at the apex or bottom point) in the luminance time waveform. The peak position can be specified by using a known local search method including, for example, a hill-climbing method, an autocorrelation method, and a method using a differential function. The visible light waveform calculation unit 111 is realized by, for example, the CPU 101, the main memory 102, the storage 103, and the like.

  In general, a pulse wave is a change in blood pressure or volume in the peripheral vasculature as the heart beats. That is, the pulse wave is a change in the volume of the blood vessel when blood is pumped out from the heart due to contraction of the heart and reaches the face or hand. Thus, when the volume of the blood vessel in the face or hand changes, the amount of blood passing through the blood vessel changes, and the skin color changes depending on the amount of components in the blood such as hemoglobin. For this reason, the brightness of the face or hand in the captured image changes according to the pulse wave. That is, information on blood movement can be acquired by using temporal changes in luminance of the face or hand obtained from images obtained by capturing the face or hand at a plurality of timings. As described above, the visible light waveform calculation unit 111 acquires pulse wave timing by calculating information related to blood movement from a plurality of images captured in time series.

  In obtaining the pulse wave timing in the visible light region, it is desirable to use an image in which the luminance in the green wavelength region in the visible light image is captured. This is because, in an image captured in the visible light region, a change due to the pulse wave appears greatly in the luminance in the wavelength region near the green. In a visible light image including a plurality of pixels, the luminance in the green wavelength region of a pixel corresponding to a face or hand in a state where a lot of blood is flowing is equivalent to a face or hand in a state where a small amount of blood is flowing It is smaller than the luminance in the green wavelength region of the pixel to be used.

  FIG. 8A is a graph showing an example of a change in luminance of a visible light image, particularly in green, in the present embodiment. Specifically, FIG. 8A shows the luminance change of the green component (G) in the cheek region of the user in the visible light image captured by the visible light imaging unit 122. In the graph of FIG. 8A, the horizontal axis represents time, and the vertical axis represents the luminance of the green component (G). It can be seen that the luminance change shown in FIG. 8A periodically changes in luminance due to the pulse wave.

  When skin is imaged in a daily environment, that is, in a visible light region, a visible light image includes noise due to scattered light from lighting or various factors. Therefore, it is desirable that the visible light waveform calculation unit 111 performs signal processing using a filter or the like on the visible light image acquired from the visible light imaging unit 122, and obtains a visible light image including many skin brightness changes caused by pulse waves. . An example of a filter used for signal processing is a low-pass filter. That is, in this embodiment, the visible light waveform calculation unit 111 performs a visible light waveform extraction process using the luminance change of the green component (G) that has passed through the low-pass filter.

  (A) of FIG. 9 is a graph which shows an example of calculation of the pulse wave timing in this Embodiment. In the graph of FIG. 9A, the horizontal axis indicates time and the vertical axis indicates luminance. In the time waveform of the graph of FIG. 9A, each point at times t1 to t5 is an inflection point or a vertex. Each point in the time waveform of the graph includes an inflection point and a peak point (vertex and bottom point) as feature points. The vertex is a point at the maximum value that is convex upward in the time waveform, and the base point is the point at the minimum value that is convex downward in the time waveform. At each of the above points included in the time waveform, the time at the point (vertex) where the luminance is higher than any of the previous or subsequent time points, or the point (bottom point) where the luminance is lower than any of the previous or subsequent time points. Time is pulse wave timing.

  A method for specifying the position of the vertex, that is, a peak searching method will be described using the luminance time waveform of the graph shown in FIG. The visible light waveform calculation unit 111 sets the current reference point as the point at time t2 in the luminance time waveform. The visible light waveform calculation unit 111 compares the point at time t2 with the point at the previous time t1, and compares the point at time t2 with the point at the next time t3. The visible light waveform calculation unit 111 determines that the reference point is positive when the luminance of the reference point is higher than the luminance of the point at the previous time and the point at the next time. That is, in this case, the visible light waveform calculation unit 111 determines that the reference point is a peak point (vertex) and the time at the reference point is the pulse wave timing.

  On the other hand, the visible light waveform calculation unit 111 determines NO when the luminance of the reference point is smaller than the luminance of at least one of the point at the previous time and the point at the next time. That is, in this case, the visible light waveform calculation unit 111 determines that the reference point is not a peak point (vertex) and that the time at the reference point is not a pulse wave timing.

  In FIG. 9A, the luminance at the point at time t2 is larger than the luminance at the point at time t1, but the luminance at the point at time t2 is smaller than the luminance at the point at time t3. The point at time t2 is determined as NO. Next, the visible light waveform calculation unit 111 increments the reference point by one and sets the next point at time t3 as the reference point. Since the luminance at the point at time t3 is higher than the luminance at the point at time t2 immediately before time t3 and the point at time t4 immediately after time t3, the visible light waveform calculation unit 111 performs the calculation at time t3. The point is determined to be positive. The visible light waveform calculator 111 outputs the time at the point determined to be positive to the correlation calculator 113 as the pulse wave timing. Thereby, as shown in (b) of Drawing 9, the time of a white circle mark is specified as pulse wave timing.

  In addition, the visible light waveform calculation unit 111 considers that the heartbeat interval time is between 333 ms and 1000 ms, for example, based on knowledge of a general heart rate (eg, 60 bpm to 180 bpm) in specifying the pulse wave timing. Thus, the pulse wave timing may be specified. The visible light waveform calculation unit 111 does not need to perform the above-described luminance comparison at all points by considering a general heartbeat interval time, and is appropriate if the luminance comparison is performed only at some points. The pulse wave timing can be specified. That is, the above-described luminance comparison may be performed using each point in the range from 333 ms to 1000 ms from the recently acquired pulse wave timing as a reference point. In this case, the next pulse wave timing can be specified without performing luminance comparison using points before that range as reference points. Therefore, it is possible to acquire a pulse wave timing that is robust in a daily environment.

  The visible light waveform calculation unit 111 further calculates a heartbeat interval time by calculating a time difference between the obtained adjacent pulse wave timings. The heartbeat interval time varies in time series. For this reason, it can be used to calculate the degree of correlation at a predetermined feature point between the visible light waveform and the infrared light waveform by comparing with the heartbeat interval time of the pulse wave specified from the infrared light waveform acquired in the same period. Can do.

  FIG. 10 is a graph showing an example of heartbeat interval times acquired in time series. In the graph of FIG. 10, the horizontal axis indicates the data number associated with the heartbeat interval time acquired in time series, and the vertical axis indicates the heartbeat interval time. As shown in FIG. 10, it can be seen that the heartbeat interval time varies with time. The data number indicates the order in which data (here, heartbeat interval time) is stored in the memory. That is, the data number corresponding to the heartbeat interval time recorded in the nth (n is a natural number) is “n”.

  The visible light waveform calculation unit 111 may further extract the time of the inflection point immediately after the pulse wave timing in the visible light waveform. Specifically, the visible light waveform calculation unit 111 obtains the minimum point of the visible light differential luminance by calculating the first derivative of the luminance value of the visible light waveform, and sets the time that becomes the minimum point as the time of the inflection point. (Hereinafter referred to as inflection point timing) is calculated. That is, the visible light waveform calculation unit 111 may extract a plurality of inflection points from the top to the bottom as predetermined feature points.

  The visible light waveform calculation unit 111 also calculates the inflection point timing in consideration of the fact that the heartbeat interval time is between 333 ms and 1000 ms, for example, based on general heart rate knowledge. Point timing may be calculated. As a result, even if the visible light waveform includes an inflection point that is completely unrelated to the heartbeat, the inflection point is not specified, so that the inflection point timing can be calculated more accurately. .

  FIG. 11 is a graph for explaining a method of extracting an inflection point from a pulse wave. Specifically, (a) of FIG. 11 is a graph showing a visible light waveform obtained from a visible light image, and (b) of FIG. 11 plots the first derivative values of (a) of FIG. It is a graph. In FIG. 11A, a circle represents a peak among peak points, and an X represents an inflection point. In (b) of FIG. 11, a circle indicates a point corresponding to the vertex in (a) of FIG. 11, and an X indicates a point corresponding to the inflection point in (a) of FIG. In the graph of FIG. 11A, the horizontal axis represents time, and the vertical axis represents the luminance value. In the graph of FIG. 11B, the horizontal axis indicates time, and the vertical axis indicates the derivative of the luminance value.

  In the extraction of the visible light waveform, a visible light image in which green light is captured as described above is used. The principle of this visible light waveform extraction will be described. When the blood volume in blood vessels such as the face or hand increases or decreases according to the pulse wave, the amount of hemoglobin in the blood increases or decreases according to the blood volume. That is, the amount of hemoglobin that absorbs light in the green wavelength region increases or decreases according to the increase or decrease of the blood volume in the blood vessel. For this reason, in the visible light image captured by the visible light imaging unit 122, the color of the skin near the blood vessel changes according to the increase or decrease of the blood volume, and the luminance value of the green component in particular of the visible light varies. . Specifically, since hemoglobin absorbs green light, the luminance value in the visible light image decreases by the amount absorbed by hemoglobin.

  Further, the visible light waveform has a characteristic that the gradient from the top point to the next bottom point is steeper than the gradient from the bottom point to the top point. Therefore, it is relatively susceptible to noise between the bottom point and the apex. On the other hand, since the slope is steep from the top to the next bottom point, it is not easily affected by noise. For this reason, the inflection point timing existing between the vertex and the bottom point is also less susceptible to noise and has a feature that it can be obtained relatively stably. From the above, the visible light waveform calculation unit 111 may calculate the time difference between the inflection points existing from the top to the bottom as the heartbeat interval time.

  Moreover, the peak point of the visible light waveform described above is a portion where the differential coefficient becomes 0 immediately before the inflection point. Specifically, as shown in FIG. 11 (b), the time at which the differential coefficient immediately before the mark X, which is the inflection point, becomes 0, is the time indicated by the circle indicating the apex in FIG. 11 (a). You can see that Using this feature, the visible light waveform calculation unit 111 may limit the vertex acquired from the visible light waveform to only the vertex immediately before the inflection point.

  The visible light waveform calculation unit 111 further calculates the inclination from the top to the bottom of the visible light waveform. The visible light waveform calculation unit 111 includes one first vertex of one of the plurality of first vertices and one first bottom immediately after the one first vertex among the plurality of first bottom points. The first slope of the first straight line connecting the points is calculated. The inclination in the visible light waveform should be as large as possible by adjusting the luminance of the visible light source 121. This is because the greater the inclination, the greater the kurtosis of the vertex in the visible light waveform, and the smaller the time lag of the pulse wave timing due to filter processing or the like.

  FIG. 12 is a graph showing a visible light waveform for explaining a method of calculating the slope of the visible light waveform. In the graph of FIG. 12, the horizontal axis indicates time, the vertical axis indicates the luminance value, the circle indicates a vertex, and the triangle indicates a bottom point. The visible light waveform calculation unit 111 connects the vertex (circle) and the next base point (triangle) with a straight line, and calculates the slope of the straight line. The inclination calculated here differs depending on the amount of light emitted by the light source in the visible light source 121, the skin area of the user acquired by the visible light imaging unit 122, and the like. Therefore, the pulse wave can be acquired clearly, for example, the ROI corresponding to the light amount of the visible light source 121 and the user's site in the visible light imaging unit 122 so that the heartbeat interval time can be continuously acquired from 333 ms to 1000 ms. Can be set, inclination information can be recorded, and compared with inclination information in the pulse wave of infrared light. Further, the visible light waveform calculation unit 111 is in the initial state, that is, after the visible light source 121 is turned on, the light source control unit 114 causes the visible light amount of the visible light source 121 or the infrared light source 123 to be infrared. The inclination from the top to the bottom of the visible light waveform in the state until the amount of light is changed is recorded in the memory (for example, the storage 103) as the first inclination A. The pulse wave measuring device 10 gradually decreases the light amount of the visible light source 121 while comparing the feature points between the visible light waveform and the infrared light waveform, and increases the light amount of the infrared light source 123. The feature is to follow. In this way, in order to gradually reduce the amount of visible light, it is in the initial state that the gradient from the top to the bottom of the visible light waveform becomes the largest.

(Infrared light waveform calculation unit)
The infrared light waveform calculation unit 112 acquires an infrared light image from the infrared light imaging unit 124, and extracts an infrared light waveform that is a waveform indicating a user's pulse wave from the acquired infrared light image. The infrared light waveform calculation unit 112 extracts the first infrared light waveform from the first infrared light image acquired before the light amount of the infrared light source 123 is controlled. In addition, the infrared light waveform calculation unit 112 extracts a second infrared light waveform from the second infrared light image acquired after the light amount of the infrared light source 123 is controlled. The control of the light amount of the infrared light source 123 means that a light source control unit 114 (to be described later) increases the amount of infrared light in the infrared light source 123, or an infrared light source. That is, the fourth control signal that causes the amount of infrared light in 123 to be output is output to the infrared light source 123. Thus, the plurality of infrared light images acquired from the infrared light imaging unit 124 include the first infrared light image acquired before the control of the light amount of the infrared light source 123 and the infrared light. 2nd infrared light image acquired after control of the light quantity of the light source 123 is contained.

  The infrared light waveform calculation unit 112 may extract a plurality of second feature points that are predetermined feature points in the extracted first infrared light waveform. Specifically, when the infrared light waveform calculation unit 112 divides the first infrared light waveform into a plurality of second unit waveforms in units of pulse wave periods, the second waveform of each of the plurality of second unit waveforms is the second. By extracting a second peak point that is one of the second vertex that is the maximum value in the unit waveform and the second bottom point that is the minimum value in the second unit waveform, a plurality of peak values are extracted from the first infrared light waveform. A second peak point is extracted. The second peak point is an example of a second feature point.

  Similar to the visible light waveform calculation unit 111, the infrared light waveform calculation unit 112 acquires the pulse wave timing as a feature point of the infrared light waveform, and calculates the heartbeat interval time from the adjacent pulse wave timing. In other words, the infrared light waveform calculation unit 112 calculates the time between each of the extracted second feature points and the other second feature points adjacent to the second feature point as the second heartbeat interval time. To do. Specifically, the infrared light waveform calculation unit 112 extracts an infrared light waveform based on a temporal change in luminance extracted from a plurality of infrared light images. That is, each of the plurality of infrared light images acquired from the infrared light imaging unit 124 is associated with the time (time point) when the infrared light image is captured by the infrared light imaging unit 124. For example, the infrared light waveform calculation unit 112, for each of the plurality of extracted second peak points, the third time at the second peak point and another second adjacent to the second peak point in time series. A plurality of second heart beat interval times are calculated by calculating a second heart beat interval time that is a time interval from the fourth time at the peak point.

  The infrared light waveform calculation unit 112 may extract a plurality of fourth feature points that are predetermined feature points in the extracted second infrared light waveform. Specifically, when the infrared light waveform calculation unit 112 divides the second infrared light waveform into a plurality of fourth unit waveforms in units of pulse wave periods, the fourth waveform of each of the plurality of fourth unit waveforms is calculated. By extracting the fourth peak that is the maximum value in the unit waveform and the fourth peak point that is the minimum value in the fourth unit waveform, a plurality of peak values are extracted from the second infrared light waveform. A fourth peak point may be extracted. The fourth peak point is an example of a fourth feature point.

  For each of the plurality of extracted fourth peak points, the infrared light waveform calculation unit 112 has a seventh time at the fourth peak point and another fourth peak point adjacent to the fourth peak point in time series. A plurality of fourth heart beat interval times may be calculated by calculating the fourth heart beat interval time, which is the time interval between the eighth time and

  Here, similarly to the visible light waveform calculation unit 111, the infrared light waveform calculation unit 112 calculates a peak position as a predetermined feature point of the infrared light waveform, for example, a hill climbing method, an autocorrelation method, and a differential function. It can be identified using a known local search method including the method used. The infrared light waveform calculation unit 112 is realized by, for example, the CPU 101, the main memory 102, the storage 103, and the like, similarly to the visible light waveform calculation unit 111.

  In general, in an infrared light image, as in a visible light image, the luminance of a skin region, for example, a face or a hand in the image changes depending on the amount of a component in blood such as hemoglobin. That is, information on blood movement can be acquired by using temporal changes in luminance of the face or hand obtained from images obtained by capturing the face or hand at a plurality of timings. In this manner, the infrared light waveform calculation unit 112 acquires pulse wave timing by calculating information related to blood movement from a plurality of images taken in time series.

  In acquiring the pulse wave timing in the infrared light region, it is desirable to use an image in which the luminance in the wavelength region of 800 nm or more in the infrared light image is captured. This is because, in an image captured in the infrared light region, a change due to the pulse wave appears greatly in the luminance in the wavelength region near 800 to 950 nm.

  FIG. 8B is a graph showing an example of the luminance change of the infrared light image in the present embodiment. Specifically, FIG. 8B shows a change in luminance of the cheek region of the user in the infrared light image captured by the infrared light imaging unit 124. In the graph of FIG. 8B, the horizontal axis indicates time, and the vertical axis indicates luminance. It can be seen that the luminance change shown in (b) of FIG. 8 periodically changes in luminance due to the pulse wave.

  However, when skin is imaged in the infrared light region, the amount of absorption of infrared light by hemoglobin is less than when skin is imaged in the visible light region. That is, the infrared light image captured in the infrared light region is likely to contain noise due to various factors such as body movement. Therefore, by applying signal processing to the captured infrared light image using a filter or the like and irradiating the user's skin area with an appropriate amount of infrared light, the infrared light contains many skin luminance changes caused by pulse waves. It is desirable to obtain a light image. An example of a filter used for signal processing is a low-pass filter. In other words, in this embodiment, the infrared light waveform calculation unit 112 performs infrared light waveform extraction processing using the luminance change of the infrared light that has passed through the ρ bus filter. The method for determining the amount of infrared light by the infrared light source 123 is described in the correlation calculation unit 113 or the light source control unit 114.

  Next, a peak search method in the infrared light waveform calculation unit 112 will be described. The peak search in the infrared light waveform can use the same method as the peak search in the visible light waveform.

  As with the visible light waveform calculation unit 111, the infrared light waveform calculation unit 112 specifies the pulse wave timing based on knowledge of a general heart rate (for example, 60 bpm to 180 bpm), for example, 333 ms to 1000 ms. The pulse wave timing may be specified in consideration of the period up to. The infrared light waveform calculation unit 112 does not need to perform the above-described luminance comparison at all points by considering a general heartbeat interval time, and is appropriate if the luminance comparison is performed only at some points. It is possible to specify the correct pulse wave timing. That is, the above-described luminance comparison may be performed using each point in the range from 333 ms to 1000 ms from the recently acquired pulse wave timing as a reference point. In this case, the next pulse wave timing can be specified without performing luminance comparison using points before that range as reference points.

  Similar to the visible light waveform calculation unit 111, the infrared light waveform calculation unit 112 calculates a heartbeat interval time by calculating a time difference between the obtained adjacent pulse wave timings. Further, the infrared light waveform calculation unit 112 may further extract the time of the inflection point immediately after the pulse wave timing in the infrared light waveform. Then, for example, the infrared light waveform calculation unit 112 obtains the minimum point of the infrared light differential luminance by calculating the first derivative of the luminance value of the infrared light waveform, and sets the time that becomes the minimum point as the inflection point. Is calculated (inflection point timing). That is, the infrared light waveform calculation unit 112 may extract a plurality of inflection points from the top to the bottom as predetermined feature points.

  Similarly to the visible light waveform calculation unit 111, the infrared light waveform calculation unit 112 calculates a tilt from the top to the bottom of the infrared light waveform. In other words, the infrared light waveform calculation unit 112 has the fourth vertex of one of the plurality of fourth vertices and the plurality of fourth bottom points in the second infrared light waveform. A second slope that is the slope of the second straight line connecting the first base point immediately after the first time series is calculated.

  As described above, the infrared light waveform calculation unit 112 performs a process similar to that of the visible light waveform calculation unit 111 to extract a plurality of predetermined feature points as second feature points. However, the infrared light waveform varies greatly depending on the amount of infrared light emitted from the light source as compared with the visible light waveform. That is, the infrared light waveform is more susceptible to the light amount of the light source than the visible light waveform.

  FIG. 13 is a graph showing an infrared light waveform when a human skin image is acquired by an infrared light camera for each level of the light amount of the infrared light source. In FIG. 13, the light amount level in the infrared light source is increased in order from (a) to (d). That is, the light source level indicates that the light source level 1 has the smallest light amount, the light amount increases as the light source level increases, and the light source level 4 has the largest light amount. The light source level indicates that the control voltage of the light source increases by about 0.5 V every time the level increases by 1. Further, a circle in each graph of FIG. 13 indicates the peak position (vertex) of the pulse wave. As shown in FIG. 13A, when the amount of light in the light source is small, the noise becomes larger than the infrared light from the infrared light source, and it is difficult to specify the pulse wave timing. On the other hand, as shown in FIGS. 13C and 13D, if the amount of light in the light source is large, a change in skin brightness corresponding to the pulse wave is buried in the light amount of the light source, and the shape of the pulse wave becomes small. It is difficult to specify the pulse wave timing.

  By the way, when acquiring a pulse wave using an image that is irradiated with visible light and captured in the visible light region, even if the visible light is irradiated with a light amount that is not too strong for the user's eyes, the pulse wave is sufficiently emitted with the irradiation amount. You can get it. However, when acquiring a pulse wave using an image that is irradiated with infrared light and captured in the infrared light region, even if the light amount of the infrared light is controlled, as described above, it contains noise or the infrared light There is too much light. For this reason, it is difficult to acquire a pulse wave only within a considerably narrow light amount range. In addition, even if the amount of light from the infrared light source is determined in advance to a predetermined value, it varies depending on the area of the skin to be acquired, the skin quality of the user, the color of the skin, and the like. Is difficult. Therefore, it is necessary to control the amount of infrared light to an appropriate value while reducing the amount of visible light so that the visible light waveform matches the infrared light waveform by the correlation degree calculation unit 113 described below. There is.

(Correlation degree calculation part)
The correlation degree calculation unit 113 calculates the degree of correlation between the visible light waveform obtained from the visible light waveform calculation unit 111 and the infrared light waveform obtained from the infrared light waveform calculation unit 112. Correlation degree calculation unit 113 then determines a command to adjust each light quantity in visible light source 121 and infrared light source 123 according to the calculated correlation level, and sends the determined command to light source control unit 114.

  Correlation degree calculation unit 113 converts a plurality of first heartbeat interval times calculated from the first visible light waveform and a plurality of second heartbeat interval times calculated from the first infrared light waveform into visible light waveform calculation unit 111 and Obtained from the infrared light waveform calculation unit 112, respectively. Then, the correlation degree calculation unit 113 calculates a first correlation degree between the plurality of first heart beat interval times and the plurality of second heart beat interval times that correspond to each other in time series.

  Further, the correlation degree calculation unit 113 converts the plurality of third heart beat interval times calculated from the second visible light waveform and the plurality of fourth heart beat interval times calculated from the second infrared light waveform into a visible light waveform calculation unit. 111 and the infrared light waveform calculation unit 112, respectively. Then, the correlation degree calculation unit 113 may calculate the second correlation degree between the plurality of third heart beat interval times and the plurality of fourth heart beat interval times, which correspond to each other in time series.

  FIG. 14 is a graph showing the first heartbeat interval time and the second heartbeat interval time plotted with data in time series order. In the graph of FIG. 14, the horizontal axis indicates the data numbers in time series, and the vertical axis indicates the heartbeat interval time corresponding to each data number. Here, the data number indicates the order in which data of each heartbeat interval time is stored in the memory. That is, in the first heartbeat interval time, the data number corresponding to the heartbeat interval time recorded nth (n is a natural number) is “n”. In the second heartbeat interval time, the data number corresponding to the heartbeat interval time recorded nth (n is a natural number) is “n”. Furthermore, since the first heartbeat interval time and the second heartbeat interval time are the results of measuring the pulse wave at the same timing, as a general rule, as long as there is no measurement error, if the data numbers are the same, the timing is almost the same. It can be said that it is the result of measuring the pulse wave. That is, the plurality of first heart beat interval times and the plurality of second heart beat interval times include a pair of first heart beat interval times and second heart beat interval times corresponding to each other in time series.

  The correlation degree calculation unit 113 calculates the degree of correlation between the plurality of first heartbeat intervals and the plurality of second heartbeat intervals using the correlation method. Specifically, the correlation degree calculation unit 113 uses the following (Equation 1) to calculate the first time between a plurality of first heartbeat intervals and a plurality of second heartbeat intervals corresponding to each other in time series. One correlation coefficient is calculated as the first degree of correlation.

  Further, the correlation degree calculation unit 113 uses the following (Equation 2) to calculate the second phase relationship between the plurality of third heart beat interval times and the plurality of fourth heart beat interval times corresponding to each other in time series. The number is calculated as the second correlation degree.

  For example, if the first correlation coefficient is a second threshold value, for example, 0.8 or more, the correlation degree calculation unit 113 substantially matches the plurality of first heart beat interval times with the plurality of second heart beat interval times. For example, a signal of “TRUE” is transmitted to the light source control unit 114 as a signal indicating that they substantially match. On the other hand, if the correlation coefficient is a value smaller than a second threshold, for example, 0.8, the correlation degree calculation unit 113 has a plurality of first heartbeat intervals and a plurality of second heartbeat intervals. For example, a signal “FALSE” is transmitted to the light source control unit 114 as a signal indicating that they do not match. Correlation degree calculation unit 113 performs the above-described processing on the second correlation coefficient as well as the first correlation coefficient.

  Further, the correlation degree calculation unit 113 determines whether each heartbeat interval time is appropriate as well as the correlation between the first heartbeat interval time and the second heartbeat interval time, and transmits the determination result to the light source control unit 114. May be. Specifically, the correlation calculation unit 113 calculates the time interval between the first heartbeat interval time and the second heartbeat interval time that correspond to each other in time series among the plurality of first heartbeat interval times and the plurality of second heartbeat interval times. It is determined whether or not the absolute error between the two exceeds a third threshold (for example, 200 ms). For example, the correlation calculation unit 113 calculates the absolute error of the first heartbeat interval time and the second heartbeat interval time having the same data number, and determines whether or not the absolute error exceeds the third threshold value. Then, for example, when it is determined that the absolute error exceeds the third threshold value, the correlation calculation unit 113 determines that the number of peak points of either the visible light waveform or the infrared light waveform is excessive. To do. Then, the correlation degree calculation unit transmits the waveform (visible light waveform or infrared light waveform) having an excessive number of peak points to the light source control unit 114. The absolute error calculation is obtained by the following equation 3.

e = RRI _RGB -RRI _IR (Formula 3)

In Equation 1, e indicates the absolute error between the corresponding first heartbeat interval time and the second heartbeat interval time, RRI _RGB indicates the first heartbeat interval time, and RRI _IR indicates the second heartbeat interval time.

  Further, if e is smaller than (−1) × third threshold (for example, −200 ms), correlation degree calculation unit 113 determines that the number of peak points in visible light is excessive, and e is the third If it is larger than the threshold (for example, 200 ms), it is determined that the number of peak points in the infrared light is excessive. Then, correlation degree calculation section 113 transmits information indicating whether the waveform having the excessive number of peak points is the visible light waveform or the infrared light waveform to light source control section 114 as the determination result. In this way, it is possible to specify that the peak point is excessively acquired in either waveform or the acquisition of the peak point is unsuccessful from the shift in the corresponding heartbeat interval time between the two waveforms.

  Correlation degree calculation unit 113, for example, has a corresponding absolute error of the first heartbeat interval time and the second heartbeat interval time that exceeds the third threshold, and the peak point is excessively acquired in the visible light waveform. If it is determined that the signal is present, a “False, RGB” signal indicating the result of the determination is transmitted to the light source control unit 114. If the correlation degree calculation unit 113 determines that the absolute error exceeds the third threshold and that the peak point is excessively acquired in the infrared light waveform, the correlation calculation unit 113 determines that the light source control unit 114 performs the determination. A “False, IR” signal indicating the result is transmitted.

  FIG. 15 is a diagram for describing a specific example of determining whether or not the heartbeat interval time is appropriate. FIG. 15A is a graph showing a case where a plurality of acquired heartbeat intervals are not appropriate. FIG. 15B is a graph showing an example of a visible light waveform or an infrared light waveform corresponding to FIG. In the graph of FIG. 15A, the horizontal axis indicates the data numbers in time series, and the vertical axis indicates the heartbeat interval time corresponding to each data number. In the graph of FIG. 15B, the horizontal axis indicates time, and the vertical axis indicates luminance in the image.

  In FIG. 15A, the two heartbeat intervals surrounded by a dotted line are not appropriate. The heartbeat interval time generally fluctuates and fluctuates, but the value hardly fluctuates abruptly. For example, as shown in FIG. 15A, in the region other than the portion surrounded by the dotted line, the average value is about 950 ms, and the standard deviation is about 50 ms. However, the value of the heartbeat interval time at the two points surrounded by the dotted line changes abruptly to about 600 to 700 ms. This occurs because the portion with the broken line in FIG. 15B is acquired as the peak point. That is, it occurs when the peak points are acquired excessively in the visible light waveform calculation unit 111 or the infrared light waveform calculation unit 112.

  When only the visible light waveform calculation unit 111 or the infrared light waveform calculation unit 112 obtains a result as shown in FIG. 15, a plurality of first heart beat interval times and a plurality of second heart beat interval times are obtained. When the number of data is compared, the number of data does not match.

  This is shown in FIG. FIG. 16 is a diagram for explaining an example in the case where excessive acquisition of peak points is performed in the visible light waveform and no excessive acquisition of peak points is performed in the corresponding infrared light waveform.

  A plurality of data of the first or second heart beat interval time is stored in the storage 103 in the form of (data No, heart beat interval time), for example. Data indicating a plurality of first heartbeat intervals acquired in the visible light waveform is, for example, (x, t20-t11), (x + 1, t12-t20), (x + 2, t13-t12). Moreover, the data which shows the some 2nd heartbeat interval time acquired in an infrared-light waveform will be (x, t12-t11), (x + 1, t13-t12), for example. Thereby, when the data acquired in each of the visible light waveform and the infrared light waveform are compared, the number of data is shifted although it is the data acquired in the same time interval t11 to t13. As a result, the data correspondence between the subsequent first heartbeat interval time and the second heartbeat interval time is all shifted, and the degree of correlation of the time fluctuation of the heartbeat interval time is shifted.

  Therefore, the correlation degree calculation unit 113 has an absolute error of the heartbeat interval time in each data number of the third or fourth heartbeat interval time obtained by the visible light waveform calculation unit 111 and the infrared light waveform calculation unit 112. When the third threshold value is, for example, 200 ms or more, one pulse wave peak having the larger number of peak points is deleted. Then, the correlation degree calculation unit 113 performs a process of reducing the subsequent data numbers one by one from the data number corresponding to the deleted peak.

When it is determined that the peak points (that is, the predetermined feature points) are acquired excessively as described above, the correlation degree calculation unit 113 has a waveform having a larger number of the predetermined feature points (visible light waveform or infrared light). A predetermined feature point that is a reference for calculating the heartbeat interval time in the optical waveform may be excluded from the calculation target of the heartbeat interval time. In other words, if e is smaller than (−1) × the third threshold, the correlation degree calculation unit 113 sets the peak point that is the basis of the RRI _RGB calculation used to calculate the e as the first heartbeat interval time. Excluded from the operation target. If e is larger than the third threshold value, correlation degree calculation unit 113 excludes the peak point that is the basis for the RRI _IR calculation used to calculate e from the calculation target of the second heartbeat interval time.

  That is, the correlation calculation unit 113 calculates an absolute error between the third heart beat interval time and the fourth heart beat interval time corresponding to each other in time series among the plurality of third heart beat interval times and the plurality of fourth heart beat interval times. It is determined whether or not exceeds a third threshold. Then, when it is determined that the absolute error exceeds the third threshold, the correlation degree calculation unit 113 compares the number of the plurality of third peak points with the number of the plurality of fourth peak points. Correlation degree calculation unit 113 calculates the peak point of the third heartbeat interval time and the fourth heartbeat interval time determined to exceed the third threshold as a result of comparison and determined to be larger. Identify the beat interval time. The correlation degree calculation unit 113 excludes the peak point that is the reference for calculating the specified heartbeat interval time from the calculation target of the heartbeat interval time.

  Further, excessive acquisition of peak points occurs due to a lot of noise in the acquired waveform (visible light waveform or infrared light waveform). For this reason, it is ascertained whether the excessively acquired waveform is a visible light waveform or an infrared light waveform, and for example, a signal such as “FALSE, RGB” is generated as described above, and the generated signal Is transmitted to the light source control unit 114. That is, if the light source control unit 114 receives the “FALSE, RGB” signal, the heartbeat interval times between the visible light waveform and the infrared light waveform are not matched, and the cause is not matched. It can be grasped that it is a visible light waveform. As described above, since the data shift in the acquisition of the peak point between the visible light waveform and the infrared light waveform can be grasped and information indicating the grasped result can be transmitted to the light source control unit 114, the visible light waveform and the infrared light waveform can be transmitted. It becomes possible to acquire a user's pulse wave more accurately.

  In the correlation degree calculation unit 113, the determination of the degree of correlation between the first heartbeat interval time and the second heartbeat interval time is made with the second threshold being 0.8, but the present invention is not limited to this. Specifically, the second threshold value may be changed according to the accuracy of the biological information that the user wants to measure. For example, if the user wants to more accurately acquire biological information during sleep, for example, information such as heartbeat and blood pressure, by accurately extracting pulse waves with infrared light during sleep, the second criterion is used. May be increased to a value such as 0.9, for example.

  Further, when the second threshold value of the correlation coefficient as a reference is adjusted, the reliability may be displayed on the presentation device 40 as the reliability of the acquired data according to the adjusted second threshold value. For example, if the feature amount between the visible light waveform and the infrared light waveform does not match easily and the amount of light from the visible light source cannot be reduced during sleep or the like, the second correlation coefficient as the reference is used. The threshold value may be changed to a value smaller than 0.8, such as 0.6. At this time, since the accuracy related to the degree of correlation becomes small, it may be displayed on the presentation device 40 that the degree of reliability has become small.

  Correlation degree calculation unit 113, when the correlation coefficient of the first and second heartbeat interval times acquired in time series from the visible light waveform and the infrared light waveform is smaller than the second threshold, or visible light waveform calculation unit 111 When the peak point of the first predetermined period is excessively acquired in the infrared light waveform calculation unit 112, the visible light waveform and the infrared light waveform are obtained using the inflection points of the visible light waveform and the infrared light waveform, respectively. The degree of correlation may be determined. That is, a phase between a plurality of third heart beat interval times calculated using the first inflection points and a plurality of fourth heart beat interval times calculated using the second inflection points, which correspond to each other in time series. The number of relationships may be calculated as the second correlation coefficient by using (Equation 2).

  Specifically, as described above, when the correlation coefficient of the first and second heartbeat interval times in the visible light waveform and the infrared light waveform is smaller than a second threshold, for example, 0.8, or visible The number of peak points acquired by the optical waveform calculation unit 111 and the infrared light waveform calculation unit 112 do not match in the first predetermined section (for example, 5 seconds), and the number of peak points in at least one waveform is the first. When the threshold value (for example, 10) is exceeded, the inflection points in both the visible light waveform and the infrared light waveform are used, and the correlation of the time interval information between the inflection points is determined in each waveform. May be.

  That is, the correlation calculation unit 113 performs the tenth determination for determining whether the number of the plurality of third peak points or the number of the plurality of fourth peak points exceeds the first threshold in the first predetermined period. . If the degree of correlation calculation unit 113 determines that the number of third peak points or the number of fourth peak points exceeds the first threshold value in the first predetermined period, the correlation degree calculation unit 113 may perform the following processing.

  In other words, the correlation degree calculation unit 113 causes the visible light waveform calculation unit 111 to make a time series of the third vertex and the third vertex of the third vertex for each of the plurality of third vertices. A plurality of the first inflection points are extracted by extracting a first inflection point that is an inflection point between the immediately following third base point. In addition, the correlation calculation unit 113 causes the infrared light waveform calculation unit 112 to time-series the fourth vertex of the fourth vertex and the fourth bottom point for each of the fourth vertex. A plurality of second inflection points are extracted by extracting a second inflection point, which is an inflection point between the fourth base point immediately after in step. In addition, the correlation calculation unit 113 causes the visible light waveform calculation unit 111 to calculate the ninth time at the first inflection point and the first inflection point for each of the extracted first inflection points. A time interval from the tenth time at another adjacent first inflection point is calculated as a third heartbeat interval time. In addition, the correlation degree calculation unit 113 sends the infrared light waveform calculation unit 112 the seventh time at the second inflection point and the second inflection point for each of the extracted second inflection points. A time interval from the eighth time at another second inflection point adjacent to is calculated as the fourth heartbeat interval time. Then, the correlation degree calculation unit 113 corresponds to each other in the time series, and a plurality of third heart beat interval times calculated using the first inflection points and a plurality of fourth heart beats calculated using the second inflection points. The second correlation coefficient between the interval times is calculated as the second correlation degree using (Equation 2).

  Further, regardless of the result of the tenth determination, the correlation degree calculation unit 113, as described above, uses the plurality of third heartbeat interval times calculated using the first inflection point and the second variable as described above. The second correlation coefficient between the plurality of fourth heartbeat interval times calculated using the inflection points may be calculated as the second correlation degree using (Equation 2). In this case, the standard deviation of the heartbeat interval time calculated from the peak point that is determined to be smaller as a result of the comparison is less than or equal to the fourth threshold value.

  FIG. 17 is a diagram for explaining a case where the degree of correlation is calculated using inflection points. FIG. 17A is a graph showing peak points (vertices) acquired in the visible light waveform, and FIG. 17B is a graph showing peak points (vertices) acquired in the infrared light waveform. It is. In both (a) and (b) of FIG. 17, the horizontal axis indicates time, the vertical axis indicates luminance, the black circle indicates the acquired vertex, and the white circle indicates the acquired inflection point.

  In (a) of FIG. 17, the peak point is excessively acquired in the visible light waveform, and the peak point is equal to or higher than the first threshold value or exceeds the first threshold value in the first predetermined period (5 seconds). It can be seen that there are 10 or 11. On the other hand, in FIG. 17B, the peak point is acquired at a constant heartbeat interval in the infrared light waveform, and the standard deviation is 100 ms or less. At this time, the time-series data numbers indicating the first and second heartbeat interval times in the visible light waveform and the infrared light waveform are shifted.

  Therefore, the correlation degree calculation unit 113 uses the inflection points existing between the top and bottom points of each pulse wave, which are obtained from the visible light waveform calculation unit 111 and the infrared light waveform calculation unit 112, to display visible light. The degree of correlation between the waveform and the infrared light waveform may be calculated. For example, the correlation calculation unit 113 causes the visible light waveform calculation unit 111 and the infrared light waveform calculation unit 112 to calculate the first heartbeat interval time and the second heartbeat interval time calculated using the inflection point, and The degree of correlation between the first and second heartbeat intervals is calculated. As a specific calculation method, the evaluation is performed based on the correlation or absolute error of the heartbeat interval time between the inflection points of the visible light waveform and the infrared light waveform.

  Note that in the correlation calculation unit 113, when the correlation coefficient of the heartbeat interval time in the visible light waveform or the infrared light waveform is smaller than the second threshold, or the number of peak points in the visible light waveform or the infrared light waveform is In the first predetermined period, when the number of peak points in at least one waveform is greater than the first threshold, the correlation degree between the visible light waveform and the infrared light waveform using the heartbeat interval time between the inflection points. However, the present invention is not limited to this. For example, the correlation degree calculation unit 113 may calculate the degree of correlation between the visible light waveform and the infrared light waveform using the heartbeat interval time between the inflection points from the beginning without using the peak point. As a result, even if peak points cannot be accurately obtained from the visible light waveform or infrared light waveform, the time similar to the heartbeat interval time is calculated by calculating the heartbeat interval time between the inflection points. it can. However, the heartbeat interval time between the inflection points is less susceptible to noise than the heartbeat interval time that can be acquired from the peak point, but the inflection point is likely to fluctuate between the vertex and the bottom point. That is, the peak-to-vertex heartbeat interval time is stable, for example, the standard deviation tends to be within 100 ms, and the time error tends to be smaller than the heartbeat interval time between the inflection points. Therefore, in the present disclosure, the heartbeat interval time calculated from the peak point is preferentially used unless otherwise specified.

  In addition to the above, when the following condition is satisfied, the correlation degree calculation unit 113 is used for calculating the correlation degree instead of the heartbeat interval time for calculating the heartbeat interval time between the inflection points from the peak point. Also good. The condition is, for example, the standard deviation of the heartbeat interval time corresponding to the waveform having the smaller number of peak points of the visible light waveform and the infrared light waveform among the plurality of heartbeat intervals and the plurality of heartbeat intervals. Is less than or equal to a fourth threshold value (for example, 100 ms). This is because when determining whether or not an excessive number of peak points has been acquired based on only the number of peak points in the first predetermined period, the peak points in the first predetermined period are actually excessive. This does not apply to the condition that the number of s exceeds the first threshold, and there is a possibility that an excessively acquired peak point may be overlooked.

  For example, FIG. 18 is a diagram for describing an example in which the condition that the number of peak points in the first predetermined period exceeds the first threshold value although the number of peak points is excessive is not applicable. In both (a) and (b) of FIG. 18, the horizontal axis indicates time, the vertical axis indicates luminance, the black circle indicates the acquired vertex, and the white circle indicates the acquired inflection point.

  As shown in FIG. 18A, when the number of peak points acquired in 5 seconds in the visible light waveform is 8, the number of peak points in the first predetermined period exceeds the first threshold. However, the number of peak points different from the number of peak points acquired in the infrared light waveform shown in FIG. 18B is acquired. At this time, as described above, if even one peak point is acquired excessively, there is a problem that the data numbers in the first heartbeat interval time and the second heartbeat interval time are shifted one by one. Alternatively, if it can be shown that the heartbeat interval time of either one of the infrared light waveforms is substantially constant, it can be adjusted (deleted) according to the number of peak points of the waveform. Details of the adjustment of the peak point are as described with reference to FIG.

  It should be noted that when the standard deviation of the heartbeat interval time in the first predetermined period exceeds the fourth threshold value in both the visible light waveform and the infrared light waveform, the correlation degree calculation unit 113 selects an appropriate value from both waveforms. It is determined that the pulse wave timing cannot be acquired, and a “False, Both” signal indicating that appropriate pulse wave timing cannot be acquired from both waveforms is transmitted to the light source control unit 114.

  Correlation degree calculation unit 113, when the use of pulse wave measuring device 10 is started, and when peak points can be appropriately acquired in first predetermined period by visible light waveform calculation unit 111 (that is, the standard of heartbeat interval time) When the deviation is smaller than the fourth threshold), the visible light waveform calculation unit 111 is stored in the memory as a result of calculating the slope between the vertex and the bottom of the visible light waveform as the first slope A. Then, every time the light amount in the visible light source 121 or the infrared light source 123 is changed by the light source control unit 114, the correlation degree calculation unit 113 changes the second slope between the vertex and the bottom of the infrared light waveform. A command is sent to the light source control unit 114 so that the inclination A is 1. Further, the correlation calculation unit 113 does not need to use the peak point acquired during the adjustment of the light amount of the light source in the light source control unit 114 for calculation of the correlation between the visible light waveform and the infrared light waveform. Good.

  FIG. 19 is a diagram illustrating an example for explaining that the peak point acquired during the adjustment of the light amount of the light source is not used for calculating the degree of correlation between the visible light waveform and the infrared light waveform. In the graph of FIG. 19, the horizontal axis indicates time, the vertical axis indicates luminance, and the state in which the light amount of the light source is adjusted in the shaded area. White circles and black circles indicate the acquired peak points.

  As shown in FIG. 19, by adjusting the light amount of the light source, the luminance gain of the visible light waveform or infrared light waveform changes, and the kurtosis of the peak point also changes accordingly. When the peak point after the kurtosis is changed is filtered by the visible light waveform calculation unit 111 or the infrared light waveform calculation unit 112, the peak point is determined by the kurtosis of the peak of the raw waveform before the filter is applied. The position changes back and forth on the time axis. This error is not a problem as long as the heart rate is calculated as biometric information, but in the case of calculating blood pressure from the pulse wave propagation time, the influence of this error is large. Therefore, in the pulse wave measuring device 10 of the present disclosure, the visible light waveform acquired while the light amount of the visible light source 121 or the infrared light source 123 is controlled by the first to fourth control signals (described later) or It is not necessary to extract a predetermined feature point (that is, peak point) from the infrared light waveform.

  That is, the visible light waveform calculation unit 111 extracts a plurality of first peak points from the first visible light waveform acquired in the period excluding the period in which the light amount of the visible light source 121 is controlled by the first control signal. A plurality of first peak points are extracted. Further, the visible light waveform calculation unit 111 extracts a plurality of third peak points from the second visible light waveform acquired in the period excluding the period in which the light amount of the visible light source 121 is controlled by the third control signal. A plurality of third peak points are extracted.

  In addition, the infrared light waveform calculation unit 112 obtains the first infrared acquired in a period excluding the period in which the light amount of the infrared light source 123 is controlled by the second control signal in the extraction of the plurality of second peak points. A plurality of second peak points are extracted from the optical waveform. In addition, the infrared light waveform calculation unit 112 extracts the second infrared light acquired in a period excluding the period in which the light amount of the infrared light source 123 is controlled by the fourth control signal in the extraction of the plurality of fourth peak points. A plurality of fourth peak points are extracted from the optical waveform.

  It should be noted that when the correlation coefficient of the heartbeat interval time in the visible light waveform and the infrared light waveform is smaller than the second threshold, the correlation degree calculation unit 113 has an excessive number of peak points in one or both waveforms. Assuming that the heartbeat interval time error and the standard deviation of each heartbeat interval time are calculated and the predetermined condition is satisfied, the heartbeat interval time between the inflection points between the top and bottom of the waveform is used. However, it is not limited to this. For example, even if the correlation coefficient between the first heartbeat interval time and the second heartbeat interval time is smaller than the second threshold, the correlation degree calculation unit 113 can appropriately acquire the peak points in both waveforms (for example, both If the standard deviations in the heartbeat interval time of the waveform are both equal to or smaller than the fourth threshold value), only the “False” signal is transmitted to the light source control unit 114.

  In this way, the correlation degree calculation unit 113 outputs signals (for example, “True”, “False”, and the like corresponding to the calculated correlation degree and the extraction result of the predetermined feature points from the visible light waveform and the infrared light waveform. "False, RGB", "False, IR", or "False, Both") is transmitted to the light source control unit 114.

  As described above, the correlation degree calculation unit 113 performs the following determination based on the first heartbeat interval time and the second heartbeat interval time.

  That is, the correlation calculation unit 113 performs a second determination that determines whether the first standard deviation exceeds the fourth threshold and whether the second standard deviation exceeds the fourth threshold. . Further, when the correlation degree calculation unit 113 determines that the first standard deviation exceeds the fourth threshold and the second standard deviation exceeds the fourth threshold as a result of the second determination, One second heart beat interval time corresponding to the first heart beat interval time in time series among the first heart beat interval time of the first heart beat interval times and the plurality of second heart beat interval times. And a fourth determination as to whether or not the first time difference is greater than a sixth threshold value that is greater than the fifth threshold value.

  On the other hand, if the correlation calculation unit 113 determines that the first time difference is smaller than the fifth threshold as a result of the third determination and the fourth determination, whether or not the second standard deviation is less than or equal to the fourth threshold. A fifth determination is made to determine

  Further, the correlation calculation unit 113 may perform the following determination based on the third heartbeat interval time and the fourth heartbeat interval time.

  That is, the correlation calculation unit 113 performs a sixth determination that determines whether the third standard deviation exceeds the fourth threshold and whether the fourth standard deviation exceeds the fourth threshold. Further, as a result of the sixth determination, the correlation degree calculation unit 113 determines that the third standard deviation exceeds the fourth threshold and that the fourth standard deviation exceeds the fourth threshold. One fourth heart beat interval corresponding to the one third heart beat interval time among the plurality of third heart beat interval times and the one third heart beat interval time in time series. A seventh determination is made as to whether the second time difference from the time is less than a fifth threshold, and an eighth determination is made as to whether the second time difference is greater than a sixth threshold.

  On the other hand, when the correlation calculation unit 113 determines that the second time difference is smaller than the fifth threshold as a result of the seventh determination and the eighth determination, whether or not the fourth standard deviation is equal to or smaller than the fourth threshold. Ninth determination is performed.

(Light source controller)
The light source control unit 114 receives the amount of visible light from the visible light source 121 and the infrared light from the infrared light source 123 according to the correlation degree and the signal corresponding to the extraction result received from the correlation degree calculation unit 113. At least one of the light amounts is determined to be increased, decreased, or maintained, and first to fourth control signals corresponding to the determination result are output to the visible light source 121 and the infrared light source 123. .

  When the light source control unit 114 receives the “False” signal, the correlation coefficient of the first and second heartbeat interval times in the visible light waveform and the infrared light waveform is smaller than the second threshold, but the heartbeat of each waveform It can be determined that the interval time has been acquired appropriately. At this time, the light source control unit 114 is in a state where the signal of the infrared light waveform is weak with respect to the visible light waveform and a predetermined feature point in each waveform can be acquired. However, for example, the kurtosis of the peak point is small. It can be determined that the peak position is shifted each time by the filter processing or the like. Therefore, in this case, the light source control unit 114 determines the amount of light in the infrared light source 123 until the second inclination from the top to the bottom of the infrared light waveform becomes the first inclination A stored in the memory. increase.

  Further, when the light source control unit 114 receives the “TRUE” signal, the light source control unit 114 can determine that the predetermined feature points in the visible light waveform and the infrared light waveform match. For this reason, the light source control unit 114 reduces the amount of visible light in the visible light source 121 and changes the amount of infrared light in the infrared light source 123 from the top to the bottom of the infrared light waveform. The inclination is increased until the first inclination A stored in the memory is reached. That is, the light source control unit 114 decreases the amount of visible light in the visible light source and increases the amount of infrared light in the infrared light source when the correlation degree is equal to or greater than the second threshold. Further, when the amount of infrared light is increased, the amount of infrared light is increased until the second slope in the infrared light waveform becomes the first slope A stored in the memory (storage 103).

  The acquisition of the second visible light image, the extraction of the second visible light waveform, the acquisition of the second infrared light image, the extraction of the second infrared light waveform, and the calculation of the second correlation coefficient are performed as a pulse wave calculation. It is repeatedly performed in each processing unit of the apparatus 100. In the calculation of the second correlation coefficient that is repeatedly performed, the light source control unit 114 compares the second inclination with the first inclination stored in the memory, and the second inclination becomes the first inclination. Until it becomes, the second control signal is output to the infrared light source 123.

  For example, when the light source control unit 114 receives a “False, IR” signal, the light source control unit 114 can determine that the infrared light waveform calculation unit 112 has not properly acquired a predetermined feature point in the infrared light waveform. That is, for example, the “False, IR” signal indicates that there is a lot of noise in the infrared light waveform. For this reason, the light quantity in the visible light source 121 is not adjusted, and the light quantity in the infrared light source 123 is increased.

  That is, the light source control unit 114 performs the second control when it is determined that the absolute error e is greater than the sixth threshold value (200 [ms]) as a result of the third determination and the fourth determination. The signal is output to the infrared light source 123. When the light source control unit 114 determines that the absolute error e is greater than the sixth threshold value (200 [ms]) as a result of the seventh determination and the eighth determination, the second control is performed. The signal is output to the infrared light source 123. The light source control unit 114 increases the amount of light in the infrared light source 123 by outputting the second control signal to the infrared light source 123.

  Further, when the light source control unit 114 receives the “False, RGB” signal, the light source control unit 114 can determine that the visible light waveform calculation unit 111 has not properly acquired a predetermined feature point in the visible light waveform. In this case, the light source control unit 114 cannot determine whether or not the infrared light waveform calculation unit 112 can appropriately acquire a predetermined feature point in the infrared light waveform. Therefore, for example, if the standard deviation of the heartbeat interval time in the first predetermined period is equal to or smaller than the fourth threshold in the infrared light waveform, the light source control unit 114 decreases the light amount of the light source in the visible light source 121 and The light quantity of the light source in the external light source 123 is increased until the slope between the top and bottom of the infrared light waveform becomes A. Further, if the standard deviation in the infrared light waveform exceeds the fourth threshold value, the light source control unit 114 determines that both signals cannot be acquired, and changes the signal to “False, Both”.

  That is, when the light source control unit 114 determines that the second standard deviation is equal to or smaller than the fourth threshold as a result of the fifth determination, the light source control unit 114 outputs the first control signal to the visible light source 121 and the second control signal. Is output to the infrared light source 123. When it is determined that the second standard deviation is greater than the fourth threshold, the light source control unit 114 outputs the third control signal to the visible light source 121 and outputs the fourth control signal to the infrared light source. As described above, the fifth determination is performed when it is determined that the first time difference is smaller than the fifth threshold as a result of the third determination and the fourth determination, and the second standard deviation is the fourth threshold. It is a determination of whether or not:

  When the light source control unit 114 determines that the fourth standard deviation is equal to or smaller than the fourth threshold as a result of the ninth determination, the light source control unit 114 outputs the first control signal to the visible light source 121 and the second control signal. Is output to the infrared light source 123. If the light source control unit 114 determines that the fourth standard deviation is greater than the fourth threshold value as a result of the ninth determination, the light source control unit 114 outputs the third control signal to the visible light source 121 and the fourth control signal to the infrared light source. Output to the light source 123. As described above, the ninth determination is performed when it is determined that the second time difference is smaller than the fifth threshold as a result of the seventh determination and the eighth determination, and the fourth standard deviation is the fourth threshold. It is a determination of whether or not:

  Further, when the light source control unit 114 receives the “FALSE, Both” signal, the light source control unit 114 can determine that a predetermined feature point has not been acquired in both the visible light waveform and the infrared light waveform. In this case, the light source control unit 114 increases the light amount of the visible light source 121 until the inclination from the top to the bottom of the visible light waveform reaches the first inclination A. The light source control unit 114 may increase the light amount of the visible light source 121 until the initial light amount of the visible light waveform is stored in the memory. Further, the light source control unit 114 reduces the light amount of the infrared light source 123 to zero. That is, the light source control unit 114 can obtain the light quantity of the visible light source 121 and the infrared light source 123 in a state where the predetermined feature points cannot be obtained most reliably in both the visible light waveform and the infrared light waveform. The light quantity is adjusted to the initial state and the light quantity is adjusted again.

  That is, the light source controller 114 determines that the third control signal is output when the absolute error e, which is the first time difference, is determined to be not less than the fifth threshold and not more than the sixth threshold as a result of the third determination and the fourth determination. The light is output to the visible light source 121 and the fourth control signal is output to the infrared light source 123. In addition, when the light source control unit 114 determines that the absolute error e that is the second time difference is not less than the fifth threshold and not more than the sixth threshold as a result of the seventh determination and the eighth determination, the light source control unit 114 outputs the third control signal. The light is output to the visible light source 121 and the fourth control signal is output to the infrared light source 123. The light source control unit 114 outputs the third control signal to the visible light source 121 to increase the amount of light of the visible light source 121 and outputs the fourth control signal to the infrared light source 123, so that infrared light is emitted. The light quantity of the light source 123 is decreased.

  In other words, the light source control unit 114 has a case where the standard deviation of the plurality of first heartbeat interval times exceeds the fourth threshold value and the standard deviation of the plurality of second heartbeat interval times exceeds the fourth threshold value. When the difference between the first heartbeat interval time and the second heartbeat interval time corresponding to each other in the time series is smaller than the fifth threshold value ((−1) × third threshold value), the visible light in the visible light source 121 When the amount of infrared light is increased and the amount of infrared light in the infrared light source 123 is increased and the amount of infrared light is increased, the second slope in the infrared light waveform is stored in the memory. The amount of infrared light is increased until an inclination A of 1 is reached.

  In addition, the light source control unit 114 has a case where the standard deviation of the plurality of first heartbeat interval times exceeds the fourth threshold value, and the standard deviation of the plurality of second heartbeat interval times exceeds the fourth threshold value. When the difference between the first heartbeat interval time and the second heartbeat interval time corresponding to each other in the time series is larger than the sixth threshold (that is, the third threshold), the amount of infrared light in the infrared light source 123 When the amount of infrared light increases, the amount of infrared light is increased until the second slope in the infrared light waveform becomes the first slope A stored in the memory.

  In addition, the light source control unit 114 has a case where the standard deviation of the plurality of first heartbeat interval times exceeds the fourth threshold value, and the standard deviation of the plurality of second heartbeat interval times exceeds the fourth threshold value. When the difference between the first heartbeat interval time and the second heartbeat interval time corresponding to each other in the time series is a value between the fifth threshold value and the sixth threshold value, the visible light source 121 The amount of light is increased and the amount of infrared light in the infrared light source 123 is decreased.

  Note that the light source control unit 114 reduces the light amount of the infrared light source 123 to red unless a predetermined feature point cannot be acquired in both the visible light waveform and the infrared light waveform, such as “False, Both”. Although the second inclination of the external light waveform is increased until it reaches the first inclination A, the present invention is not limited to this. For example, when the average luminance value in the ROI exceeds a seventh threshold value, for example, 240, the light source control unit 114 embeds an image captured from the user's skin in noise information because the light amount of the light source is too strong. End up. The average luminance “240” is “240” from 0 to 255 indicating the luminance, and the larger the value is, the higher the luminance is. For this reason, in this case, the light source control unit 114 is considered that the second inclination of the infrared light waveform exceeds the first inclination A, so that the second inclination becomes the first inclination A. The amount of infrared light may be reduced.

  FIG. 20 is a diagram showing an example of the simplest steps for decreasing the light amount of the visible light source to zero and increasing the light amount of the infrared light source to an appropriate light amount using the pulse wave measurement device. is there. In all graphs (a) to (d) in FIG. 20, the horizontal axis represents time, and the vertical axis represents luminance. In FIG. 20, the visible light waveform is represented as RGB, and the infrared light waveform is represented as IR.

  (A) of FIG. 20 is a figure which shows the visible light waveform and infrared light waveform which were acquired in the initial state which the user turned ON the visible light source 121 with the pulse wave measuring device 10. FIG. The visible light waveform in (a) of FIG. 20 is a waveform having the largest inclination from the top to the bottom among the visible light waveforms in (a) to (d) of FIG. Therefore, the inclination from the top to the bottom of the visible light waveform at this time is stored in the memory as the first inclination A.

  At this time, the infrared light source 123 is OFF. For this reason, an infrared light waveform is hardly acquired. In this state, the correlation calculation unit 113 transmits, for example, a signal “False, IR” to the light source control unit 114. Therefore, the light source control unit 114 increases the light amount of the infrared light source 123 in the infrared light source 123. At this time, as the light amount of the infrared light source 123 is increased, the infrared light waveform calculation unit 112 can acquire a predetermined feature point of the infrared light waveform, and can acquire the second heartbeat interval time. In addition, the acquired standard deviation of the second heartbeat interval time falls within the fourth threshold. Then, as shown in FIG. 20 (b), the second deviation between the apex-bottom points of the infrared light waveform is maintained while maintaining the standard deviation of the second heartbeat interval time within the fourth threshold. The light amount of the infrared light source 123 is increased until the inclination becomes the first inclination A. When the second inclination becomes the first inclination A, the correlation degree calculation unit 113 transmits, for example, a signal of “TRUE, AMP = A” to the light source control unit 114. For this reason, the light source control unit 114 once stops the adjustment of the light source when receiving the signal “TRUE, AMP = A”.

  Next, from the state of FIG. 20B, the light source control unit 114 decreases the light amount of the visible light source in the visible light source 121. FIG. 20C shows a state in which the standard deviation of the heartbeat interval time is equal to or smaller than the fourth threshold in the infrared light waveform calculation unit 112 and the light source in the visible light source 121 is OFF. 20D shows a state where the light source in the visible light source 121 is OFF and the second slope in the infrared light waveform is the first slope A, that is, This is the final goal.

  In the process of changing from the state of FIG. 20B to the state of FIG. 20C, the amount of visible light is decreased by a constant interval, for example, by 1W. Then, every time the amount of visible light is decreased, the infrared light waveform calculation unit 112 and the correlation degree calculation unit 113 confirm whether or not a predetermined feature point can be appropriately acquired in the infrared light waveform. Further, if the infrared light waveform calculation unit 112 and the correlation degree calculation unit 113 can confirm that predetermined feature points can be appropriately acquired in the infrared light waveform, as shown in FIG. The amount of light in the light source of the external light source 123 is increased until the second slope in the infrared light waveform becomes the first slope A.

  Therefore, in the process of changing from the state of FIG. 20B to the state of FIG. 20C, the correlation calculation unit 113 sends a signal of “True” or “False, The light source control unit 114 adjusts the light amount of the infrared light source 123 until it becomes “True” every time it receives the “False, IR” signal. When the light source control unit 114 receives “False, RGB” from the correlation degree calculation unit 113 by decreasing the light amount of the visible light source 121, the process is terminated.

  Alternatively, in the process of changing from the state of FIG. 20C to the state of FIG. 20D, the correlation degree calculation unit 113 transmits a “False, RGB” signal to the light source control unit 114, and the light source The control unit 114 continues to increase the light amount of the light source in the infrared light source 123 until the second slope in the infrared light waveform reaches the first slope A. For example, the visible light waveform cannot be obtained, and the first When the signal of “False, RGB, AMP = A” indicating that the inclination of 2 has become the first inclination A is received from the correlation calculation unit 113, the light source control by the light source control unit 114 is finished. .

  Further, the light source control unit 114 acquires, in the visible light waveform calculation unit 111 or the infrared light waveform calculation unit 112, two or more predetermined feature points that are continuous from the visible light waveform or the infrared light waveform, respectively. After completion, the light source is controlled. That is, the light source control unit 114 outputs the first control signal or the third control signal until two or more first peak points continuous from the first visible light waveform are extracted within the second predetermined period, or Until the two or more third peak points continuous from the second visible light waveform are extracted within the second predetermined period, the output of the first control signal or the third control signal is waited. Further, the light source control unit 114 outputs, in the output of the second control signal or the fourth control signal, until two or more second peak points continuous from the first infrared light waveform are extracted within the second predetermined period. Or it waits for the output of a 2nd control signal or a 4th control signal until two or more 4th peak points which are continuous from a 2nd infrared-light waveform are extracted within a 2nd predetermined period.

  FIG. 21 illustrates that the light source control is waited until two or more predetermined feature points continuous from the waveform are extracted within the second predetermined period in each of the visible light waveform and the infrared light waveform. FIG. The graph in FIG. 21 shows a visible light waveform or an infrared light waveform. In the graph of FIG. 21, the horizontal axis represents time, and the vertical axis represents luminance.

  When the light source controller 114 changes the light amount of the visible light source 121 or the infrared light source 123, the luminance gain of the visible light waveform or the infrared light waveform changes. When the luminance gain changes, the position of the pulse wave timing shifts, so that a large error occurs in the calculation of timing such as the heartbeat interval time. In the present disclosure, the heartbeat interval time is mainly used as a material for determining the degree of correlation between the visible light waveform and the infrared light waveform, and two consecutive peak points are necessary to calculate the heartbeat interval time. It is. Therefore, as shown in FIG. 21, the light source control unit 114 adjusts the amount of light source after confirming that two or more peak points are continuously taken in the visible light waveform or the infrared light waveform.

(Biometric information calculation unit)
The biological information calculation unit 115 uses either one of the feature quantities of the visible light waveform acquired by the visible light waveform calculation unit 111 or the infrared light waveform acquired by the infrared light waveform calculation unit 112, and The biometric information is calculated. Specifically, when the visible light source 121 is ON and the visible light waveform calculation unit 111 can acquire a visible light waveform, the biological information calculation unit 115 receives the first heartbeat interval time from the visible light waveform calculation unit 111. To get. Then, the biological information calculation unit 115 calculates biological information such as a heart rate and a stress index using the first heartbeat interval time.

  On the other hand, the biological information calculation unit 115 is a case where the visible light source 121 is OFF or the visible light waveform calculation unit 111 cannot acquire a visible light waveform, and the infrared light waveform calculation unit 112 receives infrared light. When the waveform can be acquired, the second heartbeat interval time is acquired from the infrared light waveform calculation unit 112. Then, the biological information calculation unit 115 similarly calculates biological information such as a heart rate and a stress index using the second heartbeat interval time.

  The biological information calculation unit 115 can extract the feature amount (heartbeat interval time) of each waveform (visible light waveform and infrared light waveform) in both the visible light waveform calculation unit 111 and the infrared light waveform calculation unit 112. If it is, the biological information is calculated using the first heartbeat interval time from the visible light waveform calculation unit 111. This is because visible light is more robust to noise such as body movement and more reliable than infrared light.

  The biometric information calculation unit 115 may calculate biometric information using the acquired feature value of the visible light waveform, or calculate biometric information using the acquired feature value of the infrared light waveform. Also good. In addition, the biological information calculation unit 115 may calculate the user's biological information using the feature amount of the second visible light waveform acquired after the second control information is output from the light source control unit 114, or the light source The biometric information of the user may be calculated using the feature quantity of the first visible light waveform acquired before the second control information is output from the control unit 114. Similarly, the biological information calculation unit 115 may calculate the biological information of the user using the feature amount of the second infrared light waveform acquired after the first control signal is output from the light source control unit 114. The biometric information of the user may be calculated using the feature amount of the first infrared light waveform acquired before the first control signal is output from the light source control unit 114.

  The biometric information to be calculated is a heart rate or a stress index, but is not limited to this. For example, an acceleration pulse wave may be calculated from the obtained pulse wave to calculate an arteriosclerosis index. Alternatively, the pulse wave timing may be accurately acquired from two different user sites, and the blood pressure may be estimated from the time difference (pulse wave propagation time). Alternatively, the sleep depth may be calculated by calculating the superiority of the sympathetic nerve and the parasympathetic nerve from the fluctuation of the heartbeat interval time.

  The biological information calculation unit 115 may output information indicating “high stress”, “low stress”, or the like as the stress index according to the numerical value of LF / HF.

  In addition, the biological information calculation unit 115 can obtain the sleep depth as shown in Patent Document 3. Specifically, the sleep depth can be determined based on LF, HF, and the presence or absence of body movement. The sleep depth is an index indicating the degree of activity of the subject's brain. For example, it may be determined whether the sleep depth corresponds to non-REM sleep or REM sleep. Further, in non-REM sleep, it may be determined whether it corresponds to shallow sleep or deep sleep.

  In addition, the biometric information calculation unit 115 may output the numerical value as the sleep depth by assigning a numerical value corresponding to the determined sleep depth for each step.

  Note that LF (Low Frequency) and HF (Hi Frequency) can be obtained by performing processing as shown in Patent Document 3. That is, the pulse interval data (heart interval time) is converted into a frequency spectrum distribution by, for example, FFT (Fast Fourier Transform). Next, LF and HF are obtained from the obtained frequency spectrum distribution. Specifically, LF and HF are obtained by taking the arithmetic average of the total value of three points including the peak value of a plurality of power spectra and one point at regular intervals around the peak value. As other examples of the FFT method as the frequency analysis method, an AR model, a maximum entropy method, a wavelet method, or the like can be used.

(Presentation device)
The presentation device 40 is a device that presents biological information received from the biological information calculation unit 115. Specifically, the presentation device 40 is a device that presents biological information such as a heart rate, a stress index, and a sleep depth obtained from the biological information calculation unit 115. The presentation device 40 may be realized by, for example, the mobile terminal 30 and may display a graphic indicating biological information on the display 204 of the mobile terminal 30 or output a sound indicating biological information from a speaker (not shown) of the mobile terminal 30. May be.

  The presentation device 40 may be realized by the display when the pulse wave measurement device 10 has a built-in display, or may be realized by the speaker when the pulse wave measurement device 10 has a built-in speaker. May be.

  Although the presentation device 40 presents the biological information obtained from the biological information calculation unit 115, the present invention is not limited to this. The presentation device 40 may always present the light amount of the light source in the visible light source 121 and the light amount of the light source in the infrared light source 123, for example. Further, the presentation device 40 may present the degree of coincidence at the current time point from the correlation degree calculation unit 113, for example, in% display as the reliability. Specifically, the presentation device 40 may present a correlation coefficient between the visible light waveform and the infrared light waveform.

  FIG. 22 is a diagram illustrating a display example on the presentation device. As shown in FIG. 22, the presentation device 40 is a graphic showing the heart rate, the stress index, the sleep depth, and the current reliability (that is, the correlation coefficient between the heartbeat interval times of the visible light waveform and the infrared light waveform). Is displayed. The presentation device 40 may also display the ratio of the light amount of the visible light antigen and the infrared light source at the present time. In addition, the presentation device 40 refers to a table in which the sleep state is associated with the numerical values of the heart rate, the stress index, and the sleep depth based on these parameters as to what the user's sleep state is. The sleep state may be determined and the determined sleep state may be displayed. For example, the presentation device 40 displays “GOOD” when the heart rate is 65 or less, the stress index is 40 or less, and the sleep depth is 70 or more. Note that the presentation device 40 may not display the presentation content such as the biological information immediately after the calculation. That is, since the user basically sleeps, instead of immediately presenting the presentation contents such as biological information obtained by calculation, for example, recording (accumulating) For example, the presentation content may be presented when it gets up the next morning. Thereby, the user can confirm whether the good sleep was taken immediately after waking up.

[1-3. Operation]
Next, the operation of the pulse wave measuring apparatus 10 according to the present embodiment will be described. FIG. 23 is a flowchart showing a process flow of the pulse wave measuring apparatus 10 according to the present embodiment.

  First, the visible light source 121 is activated when the user enters the room or the user himself / herself controls the control. The visible light waveform calculation unit 111 acquires a visible light image obtained by imaging a user irradiated with visible light by the visible light source 121 in the visible light region, and the user's pulse wave from the acquired visible light image. A visible light waveform which is a waveform indicating The visible light waveform calculation unit 111 extracts a plurality of first feature points that are predetermined feature points in the visible light waveform. Then, the visible light waveform calculation unit 111 calculates the first heartbeat interval time as the feature amount of the visible light waveform (S001). Further, the visible light waveform calculation unit 111 stores the inclination from the top to the bottom of the visible light waveform at this time in the memory as the first inclination A.

  Next, the infrared light waveform calculation unit 112 acquires an infrared light image obtained by imaging a user irradiated with infrared light by the infrared light source 123 in the infrared light region, and acquires the acquired red light. An infrared light waveform, which is a waveform indicating the user's pulse wave, is extracted from the external light image. The infrared light waveform calculation unit 112 extracts a plurality of second feature points that are predetermined feature points in the infrared light waveform. Then, the infrared light waveform calculation unit 112 calculates the second heartbeat interval time as the feature amount of the infrared light waveform (S002).

  Then, the correlation calculation unit 113 determines a peak point (S003). Specifically, the correlation degree calculation unit 113 determines whether or not there is an excessively acquired peak point for the first feature point extracted in the visible light waveform. Further, the correlation calculation unit 113 determines whether or not there is an excessively acquired peak point for the second feature point extracted from the infrared light waveform. Details of the peak point determination processing by the correlation degree calculation unit 113 will be described later.

  Next, the correlation calculation unit 113 calculates the correlation between the visible light waveform and the infrared light waveform (S004). The details of the correlation degree calculation processing by the correlation degree calculation unit 113 will be described later.

  Next, the light source control unit 114 adjusts the light amount of each light source (S005). The light source control unit 114 outputs a control signal for controlling the light amount of each light source according to the result of adjusting the light amount. Details of the light amount adjustment processing by the light source control unit 114 will be described later.

  Next, the biological information calculation unit 115 calculates biological information from at least one of the feature quantity of the visible light waveform and the feature quantity of the infrared light waveform (S006).

  Next, the biological information calculation unit 115 outputs the calculated biological information to the presentation device 40 (S007).

  FIG. 24 is a flowchart showing details of peak point excess acquisition determination processing in the present embodiment.

The correlation degree calculation unit 113 calculates the standard deviation SD _RGB of the first heartbeat interval time (S101).

Next, the correlation degree calculation unit 113 determines whether or not the standard deviation SD _RGB is equal to or smaller than a fourth threshold value (S102).

When it is determined that the standard deviation SD _RGB is equal to or smaller than the fourth threshold (Yes in S102), the correlation degree calculation unit 113 calculates the standard deviation SD _IR of the second heartbeat interval time (S103).

Then, the correlation degree calculation unit 113 determines whether or not the standard deviation _SDIR is less than or equal to the fourth threshold (S104).

As described above, the correlation calculation unit 113 performs at least one of step S102 and step S104, so that the calculated standard deviation SD _RGB exceeds the fourth threshold, and the calculated standard deviation _SDIR is A second determination is made to determine whether the fourth threshold is exceeded.

When it is determined that the standard deviation _SDIR is equal to or less than the fourth threshold (Yes in S104), the correlation calculation unit 113 transmits a “False” signal to the light source control unit 114 (S105).

On the other hand, the correlation calculation unit 113 determines that the standard deviation SD _RGB exceeds the fourth threshold (No in S102), or determines that the standard deviation SD _IR exceeds the fourth threshold (S104). No), the absolute error e between the corresponding first heartbeat interval time and the second heartbeat interval time is calculated (S106).

  The correlation calculation unit 113 determines whether or not the absolute error e is smaller than −200 [ms] (S107).

  When determining that the absolute error e is smaller than −200 [ms] (Yes in S107), the correlation degree calculation unit 113 transmits a “False, RGB” signal to the light source control unit 114 (S109).

  On the other hand, when it is determined that the absolute error e is −200 [ms] or more (No in S107), the correlation calculation unit 113 determines whether or not the absolute error e is greater than 200 [ms] (S108). ).

That is, if the correlation degree calculation unit 113 determines that the standard deviation SD _RGB exceeds the fourth threshold and the standard deviation SD _IR exceeds the fourth threshold as a result of the second determination, A third determination of whether or not an absolute error e (time difference) between the first heartbeat interval time and the second heartbeat interval time corresponding to each other in the series is less than a fifth threshold, and a sixth greater than the fifth threshold A fourth determination is made as to whether or not the time difference is larger than the threshold value.

  When determining that the absolute error e is greater than 200 [ms] (Yes in S108), the correlation degree calculation unit 113 transmits a “False, IR” signal to the light source control unit 114 (S110).

  When it is determined that the absolute error e is 200 [ms] or less (No in S108), the correlation degree calculation unit 113 transmits a “False, Both” signal to the light source control unit 114 (S111).

  FIG. 25 is a flowchart showing details of correlation degree calculation processing in the present embodiment.

  First, the correlation degree calculation unit 113 calculates the degree of correlation between the plurality of first heart beat interval times and the plurality of second heart beat interval times (S201).

  The correlation degree calculation unit 113 determines whether or not the correlation degree obtained by the calculation is larger than the second threshold (S202). That is, the correlation degree calculation unit 113 performs the first determination that determines whether or not the calculated correlation degree is equal to or greater than the second threshold value.

  When determining that the degree of correlation is greater than the second threshold (Yes in S202), the correlation degree calculation unit 113 transmits a “True” signal to the light source control unit 114 (S203).

  On the other hand, when the correlation calculation unit 113 determines that the correlation is equal to or lower than the second threshold (No in S202), the correlation calculation unit 113 transmits a “False” signal to the light source control unit 114 (S204).

  FIG. 26 is a flowchart showing details of light amount adjustment processing in the present embodiment.

  In the light source control unit 114, the signal received from the correlation degree calculation unit 113 is any one of the signals “True”, “False”, “False, IR”, “False, RGB”, and “False, Both”. Is determined (S301).

  When the received signal is a “True” signal, the light source control unit 114 decreases the amount of visible light and increases the amount of infrared light (S302).

  When the received signal is a “False” or “False, IR” signal, the light source control unit 114 increases only the amount of infrared light (S303). In other words, the light source control unit 114 receives the “False, IR” signal when the correlation calculation unit 113 determines that the absolute error e is larger than the sixth threshold value. A control signal for increasing the amount of light is output to the infrared light source 123 as a second control signal.

  When the light source control unit 114 increases the amount of infrared light in step S302 or step S303, whether the second slope of the infrared light waveform is equal to the first slope A stored in the memory or not. Is determined (S304).

  If the light source control unit 114 determines that the second inclination is equal to the first inclination (Yes in S304), the light amount adjustment process ends.

When the received signal is “False, RGB”, the light source control unit 114 determines whether or not the standard deviation _SDIR is equal to or smaller than the fourth threshold (S305). That is, the light source control unit 114 determines whether or not the standard deviation _SDIR is equal to or less than the fourth threshold when the correlation calculation unit 113 determines that the absolute error e is smaller than the fifth threshold. 5. Make a decision.

If the light source control unit 114 determines that the standard deviation _SDIR is equal to or less than the fourth threshold value (Yes in S305), the light source control unit 114 performs the process of step S302. That is, the light source control unit 114, when the correlation calculation unit 113 determines that the standard deviation _SDIR is equal to or less than the fourth threshold, displays the first control signal for reducing the amount of visible light in the visible light source 121. A second control signal that outputs to the light source 121 and increases the amount of infrared light in the infrared light source 123 is output to the infrared light source 123.

When the received signal is “False, Both” or when the standard deviation _SDIR is determined to be larger than the fourth threshold (No in S305), the light source control unit 114 increases the amount of visible light. The light quantity is returned to the initial light quantity, and the light quantity of the infrared light is decreased to turn off the infrared light source 123 (S306). That is, the light source control unit 114 receives the “False, Both” signal when the correlation calculation unit 113 determines that the absolute error e is greater than or equal to the fifth threshold value and less than or equal to the sixth threshold value. A control signal for increasing the amount of visible light in the light source 121 is output to the visible light source 121 as a first control signal, and a control signal for decreasing the amount of infrared light in the infrared light source 123 is used as a second control signal. Output to the infrared light source 123. _{Alternatively,} when the correlation calculation unit 113 determines that the standard deviation _SDIR is greater than the fourth threshold, the light source control unit 114 outputs a control signal for increasing the amount of visible light in the visible light source 121 as the first control signal. And a control signal for reducing the amount of infrared light in the infrared light source 123 is output to the infrared light source 123 as a second control signal.

  If the light source control unit 114 determines in step S304 that the second inclination is different from the first inclination A (No in S304), or if step S306 ends, the process returns to step S001. That is, in this case, if the pulse wave measurement device 10 does not satisfy the determination condition of step S304 even if the visible light amount in the visible light source 121 and the infrared light amount in the infrared light source 123 are changed, the step S304 is performed. Returning to S001, the acquisition of the visible light image, the acquisition of the infrared light image, the extraction of the visible light waveform, the extraction of the infrared light waveform, and the calculation of the correlation degree are repeated, and the calculation of the correlation degree is repeatedly performed. In response, the first control signal and the second control signal are output. That is, acquisition of a visible light image, acquisition of an infrared light image, extraction of a visible light waveform, extraction of an infrared light waveform, calculation of a correlation degree, output of a first control signal, and output of a second control signal are The process is repeated until the determination condition in step S304 is satisfied. Note that the visible light image repeatedly acquired in the second and subsequent processes is the second visible light image, and the infrared light image repeatedly acquired in the second and subsequent processes is the second infrared light image. The visible light waveform repeatedly extracted by the processing is the second visible light waveform, and the infrared light waveform repeatedly extracted by the second and subsequent processing is the second infrared light waveform.

  For example, the first visible light image is a visible light image captured by the visible light imaging unit 122 before the first control signal or the third control signal is output, and the second visible light image is the first control signal. Alternatively, a visible light image captured by the visible light imaging unit 122 after the third control signal is output. The first infrared light image is an infrared light image captured by the infrared light imaging unit 124 before the second control signal or the fourth control signal is output, and the second infrared light image is It is the infrared light image imaged by the infrared light imaging part 124 after the 2nd control signal or the 4th control signal was output.

[1-4. Effect etc.]
According to the pulse wave measuring apparatus 10 according to the present embodiment, the visible light waveform obtained from the visible light image obtained by imaging the user's pulse wave and the infrared light image infrared light obtained by imaging the same pulse wave. The degree of correlation with the infrared light waveform obtained from the waveform is calculated, and the amount of infrared light emitted from the infrared light source is controlled according to the degree of correlation. For this reason, the amount of infrared light can be adjusted appropriately, and the user's biological information can be acquired even in a dark state such as during sleep. Thereby, the living body monitoring at the time of sleep without contact etc. becomes possible, without providing the living body sensor which contacts a person.

  Further, according to the pulse wave measuring device 10, the correlation degree calculation unit 113 compares the first heartbeat interval time calculated from the visible light waveform and the second heartbeat interval time calculated from the infrared light waveform, Calculate the degree of correlation. For this reason, it is possible to easily calculate the degree of correlation between the visible light waveform and the infrared light waveform.

  Further, according to the pulse wave measuring device 10, in order to compare the second inclination in the infrared light waveform after adjusting the light amount of the infrared light source and the first inclination A stored in the memory, It can be determined whether or not the amount of light from the external light source has reached an appropriate amount.

  Moreover, according to the pulse wave measuring device 10, when the absolute error e exceeds the third threshold value, the first heart beat interval time and the second heart beat interval time determined to exceed the third threshold value. The predetermined feature point that is the reference for calculating the heartbeat interval time in the waveform having the more predetermined feature points is excluded from the calculation target of the heartbeat interval time. For this reason, an excessively acquired peak point can be deleted, and an appropriate value of the first heart beat interval time or the second heart beat interval time can be obtained.

  Moreover, according to the pulse wave measuring device 10, the light amounts in the visible light source and the infrared light source are determined according to the calculated correlation and the extraction result of the predetermined feature points from the visible light waveform and the infrared light waveform. Each is determined to be increased, decreased, or maintained, and a control signal corresponding to the determination result is output to the visible light source and the infrared light source. Thereby, the light quantity of a visible light source and an infrared light source can be adjusted appropriately.

  Further, according to the pulse wave measuring device 10, predetermined feature points are obtained from the visible light waveform or the infrared light waveform acquired while the light amount of the visible light source 121 or the infrared light source 123 is controlled by the control signal. Do not extract. For this reason, predetermined feature points can be appropriately extracted, and biological information can be calculated with high accuracy.

  Further, according to the pulse wave measuring device 10, visible light is visible until two or more predetermined feature points continuous from the waveform are extracted within the second predetermined period in each of the visible light waveform and the infrared light waveform. It waits for the output of a control signal for controlling the amount of visible light in the light source 121 or a control signal for controlling the amount of infrared light in the infrared light source. For this reason, predetermined feature points can be appropriately extracted, and biological information can be calculated with high accuracy.

[1-5. Modified example]
Although not particularly mentioned in the above embodiment, the visible light source 121 may be controlled so that the light amount is set to an initial value set in advance when starting from the light amount 0. Thereby, when there is an illuminance that the user likes or an illuminance in which a pulse wave can be easily acquired by the user, the state can be immediately set.

  The visible light waveform calculation unit 111 records the light amount of the visible light source 121 when the visible light waveform can be acquired and the inclination between the vertex and the bottom of the visible light waveform is the largest, and the user records the room You may control so that the light quantity of the visible light source 121 may become the recorded value whenever it enters.

  Moreover, although the visible light source 121 in this Embodiment was implement | achieved by the fixed state, for example, a fluorescent lamp and white LED, it is not restricted to this. For example, the visible light source 121 may have a drive unit so that not only the amount of light to be irradiated but also the direction of light irradiation and the direction of imaging can be changed. An example is shown in FIG.

  FIG. 27 is a diagram for explaining the features of the pulse wave measuring apparatus 10A according to the modification. (A) of FIG. 27 is the figure seen from the side of 10 A of pulse-wave measuring devices and the user U. FIG. (B) of FIG. 27 is the figure which looked at 10A of pulse wave measuring devices and the user U from the ceiling side. FIG. 27C is a diagram showing a visible light waveform or an infrared light waveform when light from the light source of the pulse wave measuring device 10A is irradiated on the face of the user U. (D) of FIG. 27 is a figure which shows a visible light waveform or an infrared light waveform when the light by the light source of 10 A of pulse wave measuring devices is irradiated to the surface (part except the user's U skin) of a futon.

  As shown in FIGS. 27A and 27B, the pulse wave measuring device 10 </ b> A drives the drive unit 25 in the direction of irradiating visible light or infrared light with the direction along the ceiling as an axis. Can be changed. Thereby, for example, as shown in FIG. 27 (b), by driving the drive unit 25 from a state in which light is irradiated on the body side part (that is, the surface of the futon) from the face, the user U's By changing the light irradiation range and the imaging range on the face, the feature quantities of the visible light waveform and the infrared light waveform can be obtained with high accuracy. In addition, as a method for specifying the light irradiation part and the imaging part, for example, by performing face recognition on the captured image, the position of the face of the user U is specified, and the cheek of the specified face is aimed, for example You may adjust the angle of 10 A of pulse-wave measuring devices by driving the drive part 25 so that light may be irradiated.

  In addition, as shown to (a) and (b) of FIG. 27, the drive part 25 is mounted | worn with the upper side of 10 A of pulse wave measuring apparatuses, and is provided with a motor. Then, the user transmits a control signal for controlling the motor using a controller such as the portable terminal 30 to the pulse wave measuring device 10 to change the angle of the pulse wave measuring device 10 in the height direction of the user, for example. Therefore, it is possible to cope with a large shift in the irradiation position in the vertical direction. Further, the pulse wave measuring device 10 may control the motor to image the face according to the result of face recognition.

  In the example of FIG. 27, the angle control is performed only in the height direction of the user (the user's vertical direction when the user is lying down), but the present invention is not limited to this. For example, the angle in the lateral direction of the user (the left and right direction of the user when the user is lying down) may be controlled. As a result, finer adjustments can be made over a wider range. For example, even if the sleeping direction and the sleeping position differ depending on the user, the user's face can be specified and the pulse wave can be acquired.

  In this case, since the pulse wave may not be acquired depending on the irradiation direction of the visible light source 121, in the light source control unit 114, the amount of visible light exceeds a predetermined threshold, and in the visible light waveform calculation unit 111, If the feature quantity of the pulse wave is not acquired, the user's face is identified using the face recognition program for the image obtained by the visible light imaging unit 122, and the light irradiation direction is changed to the face direction. Good. At that time, when the entire face is irradiated with light, strong light enters the user's eyes. For example, the visible light source 121 is recognized as a light source that can irradiate a place like a laser beam. However, the light may be focused around the cheeks under the eyes.

  Although not specifically mentioned in the above embodiment, if the infrared light is continuously irradiated to the human eye, the visual acuity may be lowered. For this reason, the infrared light source 123 may irradiate infrared light by limiting the ROI to a region of the user's face excluding human eyes. For example, when the infrared light source 123 irradiates the user's face with light, the pulse wave is particularly easily acquired on the cheek. For this reason, the light source control unit 114 may specify, for example, a portion under the user's eyes and cause the infrared light source 123 to irradiate infrared light toward only the portion. For example, the light source control unit 114 recognizes the user's face by analyzing an image captured by the infrared light imaging unit 124, and specifies the lower part of the user's eyes using the result of the face recognition. In addition, the light source control unit 114 sets the amount of infrared light to a predetermined value when the power of the infrared light emitted from the infrared light source 123 is equal to or greater than a predetermined threshold and has elapsed for a predetermined time. You may adjust the light quantity of the infrared light source 123 so that it may be suppressed to less than light quantity. In addition, as described above, in the case of infrared light, since it affects the user's visual acuity, the location of the cheek is identified from the user's face recognition, and the irradiation region is irradiated with infrared light only on the user's cheek. You may squeeze.

  Moreover, in the said embodiment, although the pulse-wave calculating apparatus 100 in the pulse-wave measuring apparatus 10 is incorporated in the pulse-wave measuring apparatus 10, it is not restricted to this. For example, the pulse wave calculation device 100 may be realized as an external server device, may be realized by the portable terminal 30, or may be realized by an information terminal such as a PC. In other words, in this case, if the image captured from the visible light imaging unit 122 and the infrared light imaging unit 124 can be acquired and the light amount of the visible light source 121 and the infrared light source 123 can be controlled, the pulse wave The arithmetic device 100 may be realized by any device.

  Each component included in the pulse wave measuring device or the like may be a circuit. These circuits may constitute one circuit as a whole, or may be separate circuits. Each of these circuits may be a general-purpose circuit or a dedicated circuit. That is, in each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component.

  Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the display control method of each of the above embodiments is the following program.

  That is, this program is a pulse wave measurement method executed by a pulse wave measurement apparatus including a processor and a memory in a computer, and is obtained by imaging a user irradiated with visible light from a visible light source in the visible light region. The obtained visible light image is obtained, and the infrared light image obtained by imaging the user irradiated with the infrared light by the infrared light source in the infrared light region is obtained, and from the obtained visible light image Extracting a visible light waveform that is a waveform indicating the user's pulse wave, extracting an infrared light waveform that is a waveform indicating the user's pulse wave from the acquired infrared light image, and extracting the visible light A degree of correlation between the waveform and the extracted infrared light waveform is calculated, and a control signal for controlling the amount of infrared light emitted from the infrared light source according to the calculated degree of correlation is transmitted to the infrared signal. Output to light source Wherein the at least one characteristic quantity and characteristic quantity of the infrared light waveform of the visible light wave, and calculates biological information, and executes the pulse wave measuring method and outputs the calculated the biometric information.

  As mentioned above, although the pulse wave measuring device etc. which concern on the one or some aspect were demonstrated based on embodiment, this indication is not limited to this embodiment. Unless it deviates from the gist of the present disclosure, various modifications conceived by those skilled in the art have been made in this embodiment, and forms constructed by combining components in different embodiments are also within the scope of one or more aspects. May be included.

  For example, in the above-described embodiment, a process executed by a specific component may be executed by another component instead of the specific component. Further, the order of the plurality of processes may be changed, and the plurality of processes may be executed in parallel.

  The present disclosure is useful as a pulse wave measuring device that can accurately calculate biological information.

10, 10A Pulse wave measuring device 20 Case 21 Visible light LED
22 Visible Light Camera 23 Infrared LED
24 Infrared Camera 25 Drive Unit 30 Portable Terminal 40 Presentation Device 100 Pulse Wave Computing Device 101 CPU
102 Main memory 103 Storage 104 Communication IF
111 Visible Light Waveform Calculation Unit 112 Infrared Light Waveform Calculation Unit 113 Correlation Degree Calculation Unit 114 Light Source Control Unit 115 Biological Information Calculation Unit 121 Visible Light Source 122 Visible Light Imaging Unit 123 Infrared Light Source 124 Infrared Light Imaging Unit 201 CPU
202 Main memory 203 Storage 204 Display 205 Communication IF
206 Input IF

Claims (16)

A pulse wave measuring device comprising a processor and a memory,
The processor is
Obtaining a first visible light image obtained by imaging a user irradiated with visible light from a visible light source in the visible light region;
Obtaining a first infrared light image obtained by imaging the user irradiated with infrared light from an infrared light source in an infrared light region;
Extracting a first visible light waveform that is a waveform indicating the pulse wave of the user from the acquired first visible light image;
Extracting a first infrared light waveform that is a waveform indicating the pulse wave of the user from the acquired first infrared light image;
Calculating a first correlation between the extracted first visible light waveform and the extracted first infrared light waveform;
Performing a first determination to determine whether the calculated first correlation is equal to or greater than a second threshold;
As a result of the first determination, when it is determined that the calculated first correlation is equal to or greater than the second threshold, a first control signal for reducing the amount of visible light in the visible light source is provided to the visible light source. And outputting a second control signal for increasing the amount of infrared light in the infrared light source to the infrared light source,
Obtaining a second visible light image obtained by imaging the user irradiated with visible light based on the first control signal by the visible light source in a visible light region;
Acquiring a second infrared light image obtained by imaging the user irradiated with infrared light based on the second control signal by the infrared light source in an infrared light region;
From the acquired second visible light image, a second visible light waveform that is a waveform indicating the pulse wave of the user is extracted,
From the acquired second infrared light image, a second infrared light waveform that is a waveform indicating the pulse wave of the user is extracted;
Using at least one of the extracted second visible light waveform and the second infrared light waveform to calculate biological information;
A pulse wave measuring device that outputs the calculated biological information. The processor, in the calculation of the first correlation degree,
When the first visible light waveform is divided into a plurality of first unit waveforms in a pulse wave cycle unit that is a cycle of the pulse wave, each of the plurality of first unit waveforms has a maximum value in the first unit waveform. A plurality of first peak points are extracted from the first visible light waveform by extracting a first peak point that is one of the first vertex and the first bottom point that is the minimum value in the first unit waveform. Extract and
When the first infrared light waveform is divided into a plurality of second unit waveforms in the pulse wave period unit, for each of the plurality of second unit waveforms, a second vertex that is the maximum value in the second unit waveform, And extracting the second peak points from the first infrared light waveform by extracting a second peak point that is one of the second bottom points, which is the minimum value in the second unit waveform,
For each of the extracted first peak points, between the first time at the first peak point and the second time at another first peak point adjacent to the first peak point in time series Calculating a plurality of the first heart beat interval times by calculating a first heart beat interval time which is a time interval of
For each of the extracted second peak points, between the third time at the second peak point and the fourth time at another second peak point adjacent to the second peak point in time series Calculating a plurality of second heartbeat interval times by calculating a second heartbeat interval time which is a time interval of
Using the following (Equation 1), a first correlation coefficient between the plurality of first heart beat interval times and the plurality of second heart beat interval times, which correspond to each other in time series, is expressed as the first correlation. The pulse wave measuring device according to claim 1, wherein the pulse wave measuring device is calculated as a degree.

The processor further includes:
Performing a second determination to determine whether the first standard deviation exceeds a fourth threshold and the second standard deviation exceeds the fourth threshold;
As a result of the second determination, when it is determined that the first standard deviation exceeds the fourth threshold and the second standard deviation exceeds the fourth threshold,
One second corresponding to the one first heart beat interval time among the plurality of first heart beat interval times and the one first heart beat interval time among the plurality of second heart beat interval times in time series. Third determination as to whether or not the first time difference from the heartbeat interval time is less than a fifth threshold, and whether or not the first time difference is greater than a sixth threshold greater than the fifth threshold. Make 4 judgments,
As a result of the third determination and the fourth determination,
If it is determined that the first time difference is greater than the sixth threshold, the second control signal is output to the infrared light source;
When it is determined that the first time difference is not less than the fifth threshold and not more than the sixth threshold, a third control signal for increasing the amount of visible light in the visible light source is output to the visible light source, and The pulse wave measurement device according to claim 2, wherein a fourth control signal for reducing the amount of infrared light in the infrared light source is output to the infrared light source. The processor further includes:
As a result of the third determination and the fourth determination, if it is determined that the first time difference is smaller than the fifth threshold, it is determined whether the second standard deviation is equal to or less than the fourth threshold. Make a fifth decision,
As a result of the fifth determination,
If it is determined that the second standard deviation is less than or equal to the fourth threshold, the first control signal is output to the visible light source, and the second control signal is output to the infrared light source,
The third control signal is output to the visible light source and the fourth control signal is output to the infrared light source when it is determined that the second standard deviation is greater than the fourth threshold. 3. The pulse wave measuring device according to 3. The processor further includes:
A first connecting the first vertex of one of the plurality of first vertices and one first bottom point of the plurality of first bottom points immediately in the time series of the first vertex. Storing the slope of the straight line in the memory as the first slope;
When the second visible light waveform is divided into a plurality of third unit waveforms in the pulse wave period unit, for each of the plurality of third unit waveforms, a third vertex that is the maximum value in the third unit waveform, and Extracting a plurality of third peak points from the second visible light waveform by extracting a third peak point that is one of the third bottom points, which is the minimum value in the third unit waveform,
When the second infrared light waveform is divided into a plurality of fourth unit waveforms in the pulse wave period unit, for each of the plurality of fourth unit waveforms, a fourth vertex that is the maximum value in the fourth unit waveform, And extracting the fourth peak point from the second infrared light waveform by extracting a fourth peak point that is one of the fourth bottom points, which is the minimum value in the fourth unit waveform,
For each of the extracted third peak points, between the fifth time at the third peak point and the sixth time at another third peak point adjacent in time series to the third peak point Calculating a plurality of third heart beat interval times by calculating a third heart beat interval time which is a time interval of
For each of the extracted fourth peak points, between the seventh time at the fourth peak point and the eighth time at another fourth peak point adjacent in time series to the fourth peak point Calculating a plurality of the fourth heart beat interval times by calculating a fourth heart beat interval time which is a time interval of
Using the following (Equation 2), a second correlation coefficient between the plurality of third heart beat interval times and the plurality of fourth heart beat interval times corresponding to each other in time series is calculated,
Repeat the acquisition of the second visible light image, the extraction of the second visible light waveform, the acquisition of the second infrared light image, the extraction of the second infrared light waveform, and the calculation of the second correlation coefficient. Done
In the calculation of the second correlation coefficient that is repeatedly performed, the fourth vertex of the plurality of fourth vertices in the second infrared light waveform obtained by the repetition, and the plurality of the plurality of fourth vertices Of the fourth base points, a second slope that is a slope of a second straight line connecting the first base point immediately after the one fourth vertex in the time series, and the memory stored in the memory Compare with the first slope,
Outputting the second control signal to the infrared light source until the second inclination becomes the first inclination;
The pulse wave measuring device according to claim 4.

The processor further includes:
Performing a sixth determination to determine whether the third standard deviation exceeds the fourth threshold and whether the fourth standard deviation exceeds the fourth threshold;
As a result of the sixth determination, when it is determined that the third standard deviation exceeds the fourth threshold and the fourth standard deviation exceeds the fourth threshold,
One fourth heartbeat time corresponding to one third heartbeat interval time among the plurality of third heartbeat interval times and the one third heartbeat interval time among the plurality of fourth heart beat interval times in time series. Performing a seventh determination as to whether or not a second time difference from the heartbeat interval time is less than the fifth threshold, and an eighth determination as to whether or not the second time difference is greater than the sixth threshold;
As a result of the seventh determination and the eighth determination,
If it is determined that the second time difference is greater than the sixth threshold, the second control signal is output to the infrared light source;
When it is determined that the second time difference is not less than the fifth threshold and not more than the sixth threshold, the third control signal is output to the visible light source, and the fourth control signal is output to the infrared light. The pulse wave measuring device according to claim 5, wherein the pulse wave measuring device outputs to a light source. The processor further includes:
As a result of the seventh determination and the eighth determination, if it is determined that the second time difference is smaller than the fifth threshold, it is determined whether the fourth standard deviation is equal to or less than the fourth threshold. Make a ninth decision,
As a result of the ninth determination,
If it is determined that the fourth standard deviation is less than or equal to the fourth threshold, the first control signal is output to the visible light source, and the second control signal is output to the infrared light source,
The third control signal is output to the visible light source and the fourth control signal is output to the infrared light source when it is determined that the fourth standard deviation is larger than the fourth threshold. 6. The pulse wave measuring device according to 6. The processor is
In the calculation of the second correlation coefficient that is repeatedly performed,
Performing a tenth determination to determine whether the number of the third peak points or the number of the fourth peak points exceeds a first threshold in the first predetermined period;
As a result of the tenth determination, when it is determined that the number of the third peak points or the number of the fourth peak points exceeds the first threshold in the first predetermined period,
For each of the plurality of third vertices, an inflection point between the third vertex and the third bottom point immediately after the third vertex in the time series of the third vertex. Extracting a plurality of first inflection points by extracting a first inflection point;
For each of the plurality of fourth vertices, an inflection point between the fourth vertex and the fourth bottom point immediately after the fourth vertex in the time series of the plurality of fourth bottom points. Extracting a plurality of second inflection points by extracting second inflection points;
For each of the extracted first inflection points, a ninth time at the first inflection point and a tenth time at another first inflection point adjacent to the first inflection point. Calculating the time interval between them as the third heartbeat interval time,
For each of the extracted second inflection points, the eleventh time at the second inflection point and the twelfth time at another second inflection point adjacent to the second inflection point. A time interval between them as the fourth heartbeat interval time,
Between the plurality of third heart beat interval times calculated using the first inflection points and the plurality of fourth heart beat interval times calculated using the second inflection points, which correspond to each other in time series. The pulse wave measuring device according to claim 5, wherein the correlation coefficient is calculated as the second correlation coefficient by using the (Equation 2). The processor further includes:
Of the plurality of third heart beat interval times and the plurality of fourth heart beat interval times, an absolute error between the third heart beat interval time and the fourth heart beat interval time corresponding to each other in time series is a third threshold value. If it exceeds
Comparing the number of the plurality of third peak points with the number of the plurality of fourth peak points;
Of the third heartbeat interval time and the fourth heartbeat interval time determined to exceed the third threshold, the heart rate calculated from the peak point that is determined to be larger as a result of the comparison Identify the interval time,
The pulse wave measuring apparatus according to any one of claims 5 to 8, wherein a peak point that is a reference for calculating the specified heartbeat interval time is excluded from the calculation target of the heartbeat interval time. The processor further includes:
Comparing the number of the plurality of third peak points with the number of the plurality of fourth peak points;
Among the plurality of third heartbeat intervals and the plurality of fourth heartbeat intervals, the heartbeat interval calculated by the peak point that is determined to be smaller as a result of the comparison is specified,
When the standard deviation of the specified heartbeat interval time is not more than the fourth threshold value,
For each of the plurality of third vertices, an inflection point between the third vertex and the third bottom point immediately after the third vertex in the time series of the third vertex. Extracting a plurality of first inflection points by extracting a first inflection point;
For each of the plurality of fourth vertices, an inflection point between the fourth vertex and the fourth bottom point immediately after the fourth vertex in the time series of the plurality of fourth bottom points. Extracting a plurality of second inflection points by extracting second inflection points;
For each of the extracted first inflection points, a thirteenth time at the first inflection point and a fourteenth time at another first inflection point adjacent to the first inflection point. Calculating the time interval between them as the third heartbeat interval time,
For each of the extracted second inflection points, a 15th time at the second inflection point and a 16th time at another second inflection point adjacent to the second inflection point. A time interval between them as the fourth heartbeat interval time,
Between the plurality of third heart beat interval times calculated using the first inflection points and the plurality of fourth heart beat interval times calculated using the second inflection points, which correspond to each other in time series. The pulse wave measuring device according to claim 5, wherein the correlation coefficient is calculated as the second correlation coefficient by using the (Equation 2). The processor is
In the extraction of the plurality of first peak points, the plurality of first peak points from the first visible light waveform acquired in a period excluding the period in which the light amount of the visible light source is controlled by the first control signal. Extract
In the extraction of the plurality of second peak points, the plurality of second peak values are obtained from the first infrared light waveform acquired in a period excluding a period in which the light amount of the infrared light source is controlled by the second control signal. Extract peak points,
In the extraction of the plurality of third peak points, the plurality of third peak points from the second visible light waveform acquired in a period excluding the period in which the light amount of the visible light source is controlled by the third control signal. Extract
In the extraction of the plurality of fourth peak points, the plurality of fourth peaks are obtained from the second infrared light waveform acquired in a period excluding a period in which the light amount of the infrared light source is controlled by the fourth control signal. The pulse wave measuring device according to any one of claims 5 to 10, wherein a peak point is extracted. The processor is
In the output of the first control signal or the third control signal, two or more first peak points continuous from the first visible light waveform are extracted within a second predetermined period, or the second Waiting for the output of the first control signal or the third control signal until two or more third peak points continuous from the visible light waveform are extracted within a second predetermined period;
In the output of the second control signal or the fourth control signal, until two or more second peak points continuous from the first infrared light waveform are extracted within the second predetermined period, or The output of the second control signal or the fourth control signal is waited until two or more fourth peak points continuous from the second infrared light waveform are extracted within the second predetermined period. To 11. The pulse wave measuring device according to any one of 11 to 11. A pulse wave measurement method executed by a pulse wave measurement device including a processor and a memory,
Obtaining a first visible light image obtained by imaging a user irradiated with visible light from a visible light source in the visible light region;
Acquiring a first infrared light image obtained by imaging a user irradiated with infrared light by an infrared light source in an infrared light region;
Extracting a first visible light waveform that is a waveform indicating the pulse wave of the user from the acquired first visible light image;
Extracting a first infrared light waveform that is a waveform indicating the pulse wave of the user from the acquired first infrared light image;
Calculating a first correlation between the extracted first visible light waveform and the extracted first infrared light waveform;
Performing a first determination to determine whether the calculated first correlation is equal to or greater than a second threshold;
As a result of the first determination, when it is determined that the calculated first correlation is equal to or greater than the second threshold, a first control signal for reducing the amount of visible light in the visible light source is provided to the visible light source. And outputting a second control signal for increasing the amount of infrared light in the infrared light source to the infrared light source,
Obtaining a second visible light image obtained by imaging the user irradiated with visible light based on the first control signal by the visible light source in a visible light region;
Acquiring a second infrared light image obtained by imaging the user irradiated with infrared light based on the second control signal by the infrared light source in an infrared light region;
From the acquired second visible light image, a second visible light waveform that is a waveform indicating the pulse wave of the user is extracted,
From the acquired second infrared light image, a second infrared light waveform that is a waveform indicating the pulse wave of the user is extracted;
Using at least one of the extracted second visible light waveform and the second infrared light waveform to calculate biological information;
A pulse wave measuring method for outputting the calculated biological information.   A program for causing a computer to execute the pulse wave measurement method according to claim 13.   A computer-readable non-transitory recording medium storing the program according to claim 14. A pulse wave measuring method,
Acquiring a plurality of first visible light images in the visible light region of the user irradiated with the first visible light emitted from the visible light source;
Acquiring a plurality of second infrared light images in the infrared light region of the user irradiated with the first infrared light emitted from the infrared light source;
Extracting a first visible light waveform from the plurality of first visible light images;
Extracting a first infrared light waveform from the plurality of first infrared light images;
Calculating a degree of correlation between the first visible light waveform and the first infrared light waveform;
If the correlation is greater than or equal to a threshold value, the visible light source emits second visible light, and the infrared light source emits second infrared light,
The amount of the second visible light per unit time is smaller than the amount of the first visible light per unit time,
The amount of the second infrared light per unit time is larger than the amount of the first infrared light per unit time,
Obtaining a plurality of second visible light images in the visible light region of the user irradiated with the second visible light;
Acquiring a plurality of second infrared light images in the infrared light region of the user irradiated with the second infrared light;
Extracting a second visible light waveform from the plurality of second visible light images;
Extracting a second infrared light waveform from the plurality of second infrared light images;
Using at least one of the second visible light waveform and the second infrared light waveform to calculate biological information;
Outputting the biological information;
The calculation of the correlation degree includes:
(1) A plurality of first peak points at a plurality of first times included in the plurality of first unit waveforms are extracted, and the plurality of first peak points are a plurality of first peaks included in the plurality of first unit waveforms. One maximum value point or a plurality of first minimum value points included in the plurality of first unit waveforms, wherein the first visible light waveform includes the plurality of first unit waveforms, and the plurality of first unit waveforms The maximum value point and the plurality of first unit waveforms respectively correspond to each other, the plurality of first minimum value points and the plurality of first unit waveforms respectively correspond to each other, and the plurality of first unit waveforms and the plurality of first unit waveforms correspond to each other. Each hour corresponds,
(2) Extracting a plurality of second peak points at a plurality of second times included in the plurality of second unit waveforms, wherein the plurality of second peak points are a plurality of second peak points included in the plurality of second unit waveforms. Two maximum value points or a plurality of second minimum value points included in the plurality of second unit waveforms, and the second visible light waveform includes the plurality of second unit waveforms, and the plurality of second unit points. The maximum value point and the plurality of second unit waveforms respectively correspond to each other, the plurality of first minimum value points and the plurality of first unit waveforms respectively correspond to each other, and the plurality of second unit waveforms and the plurality of second unit waveforms correspond to each other. 2 hours correspond,
(3) calculating a plurality of first heartbeat intervals based on the plurality of first times, wherein the plurality of first heartbeat intervals are times between a first time and a second time; 1 hour includes the first time and the second time, the time included in the first time does not exist between the first time and the second time,
(4) calculating a plurality of second heartbeat intervals based on the plurality of second times, wherein the plurality of second heartbeat intervals is a time between a third time and a fourth time; 2 hours includes the third time and the fourth time, and the time included in the second time does not exist between the third time and the fourth time,
(5) A pulse wave measuring method for calculating the degree of correlation using the following (Formula 1).

Images