JP6167614B2 - Blood flow index calculation program, blood flow index calculation device, and blood flow index calculation method - Google Patents

Description

  The present invention relates to a blood flow index calculation program, a blood flow index calculation device, and a blood flow index calculation method.

  As part of health management, pulse wave velocity and blood pressure are measured. For example, when measuring blood pressure, wear a cuff on the subject's arm, press the arm with the cuff to occlude the artery, and then measure the blood pressure using vibration generated in the blood vessel wall in the process of decompressing the cuff. To do.

  However, the above sphygmomanometer has the disadvantages that, for example, there are many procedures for measuring blood pressure, which is troublesome, the device itself is large and difficult to carry at all times, and the arm is pressurized when measuring blood pressure, which is troublesome.

  For this reason, for example, a wristwatch type blood pressure measurement device has been proposed for the purpose of improving convenience. The wristwatch-type blood pressure measurement device incorporates a function for measuring blood pressure in a small-sized wristwatch that is routinely carried. The wristwatch-type blood pressure measurement device is worn on the left wrist, and a right fingertip is placed on a front portion in which a phototransistor and a cardiac radio wave detection electrode which is an electrode for detecting cardiac radio waves are provided side by side. The wristwatch-type blood pressure measuring device detects cardiac radio waves from the right hand fingertip placed on the cardiac radio wave detection electrode and the left wrist in contact with the cardiac radio wave detection electrode, and the right fingertip placed on the phototransistor The finger pulse wave is detected from the blood flow. Then, the wristwatch-type blood pressure measurement device measures a delay time from the detection of the heart wave to the detection of the finger pulse wave, and calculates the blood pressure based on the delay time.

Japanese Patent Laid-Open No. 4-200439 JP 2007-319246 A

  However, the above technique cannot measure blood pressure without extra hardware.

  For example, the wristwatch-type blood pressure measuring device described above uses a dedicated hardware that is not incorporated in a general wristwatch, such as a cardiac radio wave detection electrode or a phototransistor, to delay time between the cardiac radio wave and the pulse wave of the finger. Measure. For this reason, the wristwatch-type blood pressure measurement device cannot measure the delay time and blood pressure unless dedicated hardware is installed.

  Although the measurement of blood pressure has been described here, the pulse wave propagation velocity obtained from the delay time cannot be measured unless dedicated hardware is also installed.

  An object of one aspect is to provide a blood flow index calculation program, a blood flow index calculation apparatus, and a blood flow index calculation method that can calculate an index related to blood flow without extra hardware.

According to one aspect of the blood flow index calculation program, a computer detects a pulse wave of the first living body from an image of the first living body captured by the first camera , and uses the second camera to detect the first living body. calculating a delay amount of said second detecting a pulse wave of a living body, pulse wave of the first pulse wave and the second biological biological from an image in which the second organism different sites is captured and Then, an index relating to blood flow is calculated using the delay amount, an inclination of a terminal having the first camera is detected, and the first living body is imaged from the image obtained by capturing the first living body by the first camera. The distance between the living body and the first camera is calculated, and the first living body and the first camera are calculated from the inclination of the terminal and the distance between the first living body and the first camera. Difference in height between the first living body and the first camera Based on the difference between the, to execute processing for correcting an indication as to the blood flow.

  According to one embodiment, an index relating to blood flow can be calculated without extra hardware.

FIG. 1 is a block diagram illustrating a functional configuration of the blood flow index calculating apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of a photographing method. FIG. 3 is a diagram illustrating an example of the spectrum of each signal of the G signal and the R signal. FIG. 4 is a diagram illustrating an example of a spectrum of each signal of the R component multiplied by the G component and the correction coefficient k. FIG. 5 is a diagram illustrating an example of a spectrum after calculation. FIG. 6 is a block diagram illustrating a functional configuration of the first waveform detection unit. FIG. 7 is a graph showing an example of a facial pulse wave and a finger pulse wave. FIG. 8 is a flowchart illustrating a procedure of blood flow index calculation processing according to the first embodiment. FIG. 9 is a flowchart illustrating the procedure of the first detection process according to the first embodiment. FIG. 10 is a flowchart illustrating the procedure of the second detection process according to the first embodiment. FIG. 11 is a block diagram illustrating a functional configuration of the blood flow index calculation device according to the second embodiment. FIG. 12 is a diagram illustrating an example of a method for calculating the difference in height between the face and the in-camera. FIG. 13 is a flowchart illustrating a procedure of blood flow index correction processing according to the second embodiment. FIG. 14 is a graph illustrating an example of a method for changing a plurality of frame frequencies. FIG. 15 is a schematic diagram illustrating an example of a computer that executes a blood flow index calculation program according to the first to third embodiments.

  Embodiments of a blood flow index calculation program, a blood flow index calculation device, and a blood flow index calculation method disclosed in the present application will be described below in detail with reference to the drawings. Note that this embodiment does not limit the disclosed technology. Each embodiment can be appropriately combined within a range in which processing contents are not contradictory.

[Configuration of blood flow index calculation device]
First, the functional configuration of the blood flow index calculation device according to the present embodiment will be described. FIG. 1 is a block diagram illustrating a functional configuration of the blood flow index calculating apparatus according to the first embodiment. A blood flow index calculation apparatus 10 shown in FIG. 1 detects pulse waves of two parts using images obtained by imaging two different parts of a living body with two cameras, and relates to blood flow from a delay amount between the pulse waves. The blood flow index calculation process for calculating the index is executed. “Pulse wave” as used herein refers to fluctuations in blood volume, and includes so-called heart rate and heart rate waveform.

  As an aspect, the blood flow index calculation device 10 can be implemented by installing a blood flow index calculation program provided as package software or online software on a desired computer. For example, the blood flow index calculation program is installed in a mobile communication terminal that can be connected to a mobile communication network such as a smartphone, a mobile phone, or a PHS (Personal Handyphone System). Further, the blood flow index calculation program may be installed not only in the mobile communication terminal that can be connected to the mobile communication network but also in a digital camera or tablet terminal that does not have the ability to connect to the mobile communication network. Thereby, a portable terminal such as a mobile communication terminal or a tablet terminal can be caused to function as the blood flow index calculating device 10. Here, a portable terminal is illustrated as an example of implementation of the blood flow index calculating device 10, but the blood flow index calculating program can be installed in a stationary terminal device such as a personal computer.

  As shown in FIG. 1, the blood flow index calculation device 10 includes an in-camera 11a, an out-camera 11b, a first acquisition unit 12a, a second acquisition unit 12b, an extraction unit 13, and a first waveform. It has a detector 15a and a second waveform detector 15b. Furthermore, the blood flow index calculation device 10 includes a first peak detection unit 16a, a second peak detection unit 16b, a delay amount calculation unit 17, a propagation velocity calculation unit 18, and a blood pressure calculation unit 19. In addition, the blood flow index calculation device 10 may include various functional units included in known mobile terminals in addition to the functional units illustrated in FIG. For example, the blood flow index calculating apparatus 10 includes functional units such as an input / output device such as a touch panel and a display, an antenna, a wireless communication unit that performs connection with a mobile communication network, a GPS (Global Positioning System) receiver, and an acceleration sensor. You may have more.

  The in-camera 11a and the out-camera 11b are imaging devices equipped with imaging elements such as a charge coupled device (CCD) and a complementary metal oxide semiconductor (CMOS). For example, the in-camera 11a and the out-camera 11b can be equipped with three or more types of light receiving elements such as R (red), G (green), and B (blue).

  Among these, the in-camera 11a is a camera mounted on a side having a liquid crystal display (not shown), and the user of the mobile terminal projects an image of the user himself / herself on the liquid crystal display by the in-camera 11a, and changes its appearance. You can shoot while checking. On the other hand, the out camera 11b is a camera mounted on the back side of the side where the liquid crystal display is provided, and the user of the mobile terminal checks the scenery on the back side of the mobile terminal on the liquid crystal display by the out camera 11b. You can shoot subjects such as landscapes and people. The in-camera 11a may be arranged at any position on the side where the liquid crystal display is present, and the out-camera 11b may be a camera different from the in-camera 11a, and the in-camera 11a is present. It may be arranged at any position on the back side of the camera, and may be placed in parallel with the in camera on the side where the in camera 11a is present.

  The in-camera 11a and the out-camera 11b simultaneously image different biological parts by executing imaging in parallel when the blood flow index calculation program is started. Here, as an example, the following description will be made assuming that the face of the subject is photographed as the first living body by the in-camera 11a and the finger of the subject is photographed as the second living body by the out-camera 11b. .

  FIG. 2 is a diagram illustrating an example of a photographing method. As shown in FIG. 2, the in-camera 11 a places the liquid crystal display side of the blood flow index calculation device 10 in the imaging range, and images a subject existing in the imaging range. At this time, the target position that reflects the user's nose can be displayed as an aim on the liquid crystal display of the blood flow index calculation device 10 while displaying the image captured by the in-camera 11a. In this way, guidance is provided so that an image in which the user's nose is within the center of the shooting range can be captured among facial parts such as the user's eyes, ears, nose and mouth.

  On the other hand, the out camera 11b puts the back surface of the surface on which the liquid crystal display is provided in the imaging range, and images a subject existing in the imaging range. At this time, various kinds of guidance can be performed so that the user's finger is placed in the imaging range of the out camera 11b. For example, a message such as “Please place your finger on the out-camera 11b” can be superimposed on the face image of the user captured by the in-camera 11a on the liquid crystal display. It is also possible to output sound from a speaker. Thus, guidance is provided so that an image in which the user's finger is within the shooting range can be shot.

  Here, the blood flow index calculation device 10 captures an image in synchronization between the in-camera 11a and the out-camera 11b. For example, both the frame of the image captured by the in-camera 11a and the frame of the image captured by the out-camera 11b are associated using a frame number or the like. With this association, images having the same frame number are used for subsequent processing as images captured at the same time.

  Thus, the in-camera 11a and the out-camera 11b according to the present embodiment activate both of the two cameras when calculating an index relating to blood flow such as a pulse wave velocity and blood pressure. The captured images are associated with each frame. In other words, it differs from a general usage method in which one camera is exclusively activated while one camera is activated and the other camera is deactivated by switching the photographing mode.

  Thereafter, the face image captured by the in-camera 11a is output to the first acquisition unit 12a described later, and the finger image captured by the out-camera 11b is output to the second acquisition unit 12b described later. Is done. In addition, although the case where the face image and the finger image are output to the first acquisition unit 12a and the second acquisition unit 12b after imaging is illustrated here, the face image and the finger image are not necessarily displayed immediately. It is not necessary to output to the acquisition unit 12a and the second acquisition unit 12b. For example, a face image or a finger image can be temporarily stored in an auxiliary storage device such as a flash memory or a hard disk (not shown) or a removable medium such as a memory card.

  Returning to the description of FIG. 1, the first acquisition unit 12 a is a processing unit that acquires an image of the first living body. As an aspect, the first acquisition unit 12a acquires a face image captured by the in-camera 11a. As another aspect, the first acquisition unit 12a may acquire a face image from an auxiliary storage device that accumulates a face image or a removable medium. As a further aspect, the first acquisition unit 12a can also acquire a face image received from an external device via a network.

  The second acquisition unit 12b is a processing unit that acquires an image of the second living body. As an aspect, the second acquisition unit 12b acquires a finger image captured by the out camera 11b. As another aspect, the second acquisition unit 12b can acquire a finger image from an auxiliary storage device or a removable medium that accumulates the finger image, similarly to the first acquisition unit 12a. As a further aspect, the second acquisition unit 12b can acquire the image of the finger received from the external device via the network in the same manner as the first acquisition unit 12a.

  The first acquisition unit 12a and the second acquisition unit 12b execute processing using image data such as two-dimensional bitmap data or vector data obtained from an output from an image sensor such as a CCD or a CMOS. Although illustrated, it is good also as performing the following process. That is, the first acquisition unit 12a and the second acquisition unit 12b may acquire the signal output from one detector as it is and execute the subsequent processing.

  The extraction unit 13 is a processing unit that extracts a region of the first living body that is a pulse wave detection target from the image of the first living body. As one aspect, each time the face image is acquired by the first acquisition unit 12a, the extraction unit 13 performs image processing such as template matching on the image to obtain a predetermined face part, for example, a user's Extract facial regions including eyes, nose, lips, cheeks and hair.

  After extracting the face area, the extraction unit 13 performs a predetermined statistical process on the pixel value of each pixel included in the face area. For example, the extraction unit 13 averages the pixel values of the pixels included in the face area for each wavelength component. In addition to the average value, the median value and the mode value may be calculated. In addition to the weighted average, an arbitrary average process such as a weighted average or a moving average may be executed. As a result, an average value of pixel values of each pixel included in the face area is calculated for each wavelength component as a representative value representing the face area.

  The first waveform detection unit 15a has a signal other than a pulse wave frequency band in which a pulse wave can be taken between each wavelength component from a signal of a representative value for each wavelength component of each pixel included in the region extracted by the extraction unit 13. It is a processing unit that detects a waveform of a signal in which components in a specific frequency band are canceled each other.

  As an aspect, the first waveform detection unit 15a includes three wavelength components included in the image, that is, two wavelength components of R component, G component, and B component, which are different in R light absorption component and G component. The pulse waveform of the face is detected using time series data of the representative values.

  To explain this, there are capillaries on the face surface, and when the blood flow flowing through the blood vessels changes due to the heartbeat, the amount of light absorbed by the bloodstream also changes according to the heartbeat. The brightness that is produced also changes with the heartbeat. Although the amount of change in luminance is small, when the average luminance of the entire face region is obtained, the time-series data of luminance includes a pulse wave component. However, the luminance also changes due to body movement in addition to the pulse wave, and this becomes a noise component of pulse wave detection, so-called body movement artifact. Therefore, a pulse wave is detected with two or more wavelengths having different light absorption characteristics of blood, for example, a G component having a high light absorption characteristic (about 525 nm) and an R component having a low light absorption characteristic (about 700 nm). Since the heart rate is in the range of 30 bpm to 240 bpm when converted to 0.5 Hz to 4 Hz per minute, other components can be regarded as noise components. Assuming that noise does not have wavelength characteristics or is minimal even if it is present, components other than 0.5 Hz to 4 Hz should be equal between the G signal and the R signal. Is different. Therefore, by correcting the sensitivity difference between components other than 0.5 Hz to 4 Hz and subtracting the R component from the G component, the noise component can be removed and only the pulse wave component can be extracted.

  For example, the G component and the R component can be represented by the following formula (1) and the following formula (2). “Gs” in the following equation (1) indicates the pulse wave component of the G signal, “Gn” indicates the noise component of the G signal, and “Rs” in the following equation (2) indicates the R signal. “Rn” indicates the noise component of the R signal. Further, since the noise component has a sensitivity difference between the G component and the R component, the correction coefficient k for the sensitivity difference is expressed by the following equation (3).

Ga = Gs + Gn (1)
Ra = Rs + Rn (2)
k = Gn / Rn (3)

  When the sensitivity difference is corrected and the R component is subtracted from the G component, the pulse wave component S is expressed by the following equation (4). When this is transformed into the formula represented by Gs, Gn, Rs and Rn using the above formula (1) and the above formula (2), the following formula (5) is obtained, and further, the above formula (3 ) To eliminate the correction coefficient k and rearrange the equations, the following equation (6) is derived.

S = Ga-kRa (4)
S = Gs + Gn−k (Rs + Rn) (5)
S = Gs− (Gn / Rn) Rs (6)

  Here, the G signal and the R signal have different light absorption characteristics, and Gs> (Gn / Rn) Rs. Therefore, the pulse wave component S from which noise is removed can be calculated by the above equation (6).

  FIG. 3 is a diagram illustrating an example of the spectrum of each signal of the G signal and the R signal. The vertical axis of the graph shown in FIG. 3 indicates the signal intensity, and the horizontal axis indicates the frequency (bpm). As shown in FIG. 3, since the sensitivity of the image sensor differs between the G component and the R component, the signal intensities of the two differ. On the other hand, in both the R component and the G component, there is no change in that noise appears outside the range of 30 bpm to 240 bpm, particularly in a specific frequency band of 3 bpm or more and less than 20 bpm. For this reason, as shown in FIG. 3, the signal intensity corresponding to the designated frequency Fn included in the specific frequency band of 3 bpm or more and less than 20 bpm can be extracted as Gn and Rn. The sensitivity difference correction coefficient k can be derived from these Gn and Rn.

  FIG. 4 is a diagram illustrating an example of a spectrum of each signal of the R component multiplied by the G component and the correction coefficient k. In the example of FIG. 4, the result of multiplying the absolute value of the correction coefficient is shown for convenience of explanation. Also in the graph shown in FIG. 4, the vertical axis indicates the signal strength, and the horizontal axis indicates the frequency (bpm). As shown in FIG. 4, when the spectrum of each signal of the G component and the R component is multiplied by the correction coefficient k, the sensitivity is uniform between the components of the G component and the R component. In particular, the spectrum signal intensity in a specific frequency band is almost the same in most spectrum signals. On the other hand, in the peripheral region 400 of the frequency where the pulse wave is actually included, the signal intensity of the spectrum is not uniform between the G component and the R component.

  FIG. 5 is a diagram illustrating an example of a spectrum after calculation. In FIG. 5, the scale of the signal intensity, which is the vertical axis, is enlarged from the viewpoint of improving the visibility of the frequency band in which the pulse wave appears. As shown in FIG. 5, when the spectrum of the R signal after the multiplication of the correction coefficient k is subtracted from the spectrum of the G signal, a signal in which a pulse wave appears due to the difference in the light absorption characteristics between the G component and the R component. It can be seen that the noise component is reduced with the strength of the component maintained as much as possible. In this way, it is possible to detect a pulse wave waveform from which only noise components have been removed.

  Next, the functional configuration of the first waveform detection unit will be described more specifically. FIG. 6 is a block diagram illustrating a functional configuration of the first waveform detection unit. As shown in FIG. 6, the first waveform detection unit 15a includes BPF (Band-Pass Filter) 152R and 152G, extraction units 153R and 153G, LPF (Low-Pass Filter) 154R and 154G, and a calculation unit 155. And BPFs 156R and 156G, a multiplication unit 157, and a calculation unit 158. In the example of FIGS. 3 to 5, the example in which the pulse wave is detected in the frequency space has been described. However, in FIG. 6, the noise in the time series space is reduced from the viewpoint of reducing the time required for the conversion to the frequency component. The functional structure in the case of detecting a pulse wave by canceling components is shown.

  For example, the time-series data of the R signal whose signal value is the representative value of the R component pixel value of each pixel included in the face area is input from the extraction unit 13 to the first waveform detection unit 15a. The time series data of the G signal having the representative value of the G component pixel value of each pixel included in the face area as a signal value is input. Among these, the R signal of the face area is input to the BPF 152R and BPF 156R in the first waveform detector 15a, and the G signal of the face area is input to the BPF 152G and BPF 156G in the first waveform detector 15a. The

  Each of BPF 152R, BPF 152G, BPF 156R, and BPF 156G is a band-pass filter that passes only signal components in a predetermined frequency band and removes signal components in other frequency bands. These BPF 152R, BPF 152G, BPF 156R, and BPF 156G may be implemented by hardware, or may be implemented by software.

  The difference in the frequency band that the BPF passes will be described. The BPF 152R and the BPF 152G pass a signal component in a specific frequency band in which a noise component appears more noticeably than other frequency bands.

  Such a specific frequency band can be determined by comparing with a frequency band that can be taken by a pulse wave. An example of a frequency band that can be taken by a pulse wave is a frequency band of 0.5 Hz to 4 Hz, and a frequency band of 30 bpm to 240 bpm when converted per minute. From this, as an example of the specific frequency band, a frequency band of less than 0.5 Hz and more than 4 Hz that cannot be measured as a pulse wave can be employed. Further, the specific frequency band may partially overlap with the frequency band that can be taken by the pulse wave. For example, it is allowed to overlap with the frequency band that the pulse wave can take in the section of 0.7 Hz to 1 Hz that is difficult to be measured as a pulse wave, and the frequency band of less than 1 Hz and 4 Hz or more is adopted as the specific frequency band. You can also Further, the specific frequency band can be narrowed down to a frequency band in which noise is more noticeable with the frequency band of less than 1 Hz and 4 Hz or more as the outer edge. For example, noise appears more noticeably in a low frequency band lower than a frequency band that can take a pulse wave, rather than a high frequency band that is higher than a frequency band that the pulse wave can take. For this reason, a specific frequency band can also be narrowed down to a frequency band of less than 1 Hz. Further, since there are many differences in the sensitivity of the image sensor of each component in the vicinity of the direct current component where the spatial frequency is zero, the specific frequency band can be narrowed down to a frequency band of 0.05 Hz to less than 1 Hz. Furthermore, the specific frequency band can be narrowed down to a frequency band of 0.05 Hz or more and 0.3 Hz or less where noise such as flickering of ambient light other than human body movement, for example, blinking or shaking of the body, is likely to appear.

  Here, as an example, the following description will be given assuming that the BPF 152R and the BPF 152G pass a signal component in a frequency band of 0.05 Hz to 0.3 Hz as a specific frequency band. Here, the case where a bandpass filter is used to extract a signal component in a specific frequency band is illustrated, but a low-pass filter is used when a signal component in a frequency band below a certain frequency is extracted. You can also

  On the other hand, the BPF 156R and the BPF 156G pass signal components in a frequency band that can be taken by a pulse wave, for example, a frequency band of 0.5 Hz to 4 Hz. Hereinafter, a frequency band that can be taken by a pulse wave may be referred to as a “pulse wave frequency band”.

  The extraction unit 153R extracts the absolute intensity value of the signal component in the specific frequency band of the R signal. For example, the extraction unit 153R extracts the absolute intensity value of the signal component in the specific frequency band by executing an absolute value calculation process of the signal component in the specific frequency band of the R component. Further, the extraction unit 153G extracts the absolute intensity value of the signal component in the specific frequency band of the G signal. For example, the extraction unit 153G extracts the absolute intensity value of the signal component in the specific frequency band by executing an absolute value calculation process of the signal component in the specific frequency band of the G component.

  The LPF 154R and the LPF 154G are low-pass filters that perform a smoothing process that responds to time changes on time-series data of absolute intensity values in a specific frequency band. The LPF 154R and the LPF 154G are the same except that the signal input to the LPF 154R is an R signal and the signal input to the LPF 154G is a G signal. By such smoothing processing, absolute value intensities R′n and G′n in a specific frequency band are obtained.

  The calculation unit 155 divides the absolute value strength G′n of the specific frequency band of the G signal output by the LPF 154G by the absolute value strength R′n of the specific frequency band of the R signal output by the LPF 154R. n / R'n "is executed. Thereby, a correction coefficient k for the sensitivity difference is calculated.

  The multiplier 157 multiplies the signal component in the pulse wave frequency band of the R signal output from the BPF 156R by the correction coefficient k calculated by the calculator 155.

  The calculation unit 158 subtracts the signal component of the pulse wave frequency band of the R signal multiplied by the correction coefficient k by the multiplication unit 157 from the signal component of the pulse wave frequency band of the G signal output by the BPF 156G “Gs−k *”. Rs "is executed. The signal thus obtained corresponds to the waveform of the facial pulse wave, and the sampling frequency thereof corresponds to the frame frequency at which the image is captured. Hereinafter, the waveform of the pulse wave may be referred to as “pulse wave waveform”.

  Returning to the description of FIG. 1, the second waveform detection unit 15 b calculates the pulse wave frequency band from the signal of the representative value of the wavelength component of each pixel included in the image of the second living body acquired by the second acquisition unit. It is a processing part which detects the waveform of the signal from which the component of the specific frequency band other than is removed. As one aspect, the second waveform detection unit 15b calculates the representative value by executing the above statistical processing on the pixel value of each pixel included in the image of the finger. At this time, the second waveform detection unit 15b performs the statistical processing by focusing on the pixel value of the G component among the three wavelength components, that is, the R component, the G component, and the B component. Then, the second waveform detector 15b removes the signal component of the specific frequency band included in the signal of the representative value of the G component of the image of the finger using a BPF (not shown) and the like, and uses the pulse wave frequency band. Pass the signal component. Thereby, the signal component of the pulse wave frequency band is extracted. Thus, the time-series data of the pulse wave frequency band signal output by the second waveform detector 15b corresponds to the pulse wave waveform of the finger. Note that the second waveform detector 15b may extract the pulse waveform of the finger from the representative value of the G component and the representative value of the R component, similarly to the first waveform detector.

  The first peak detector 16a is a processor that detects the peak of the pulse waveform of the first living body. As one aspect, the first peak detector 16a calculates a differential waveform of the facial pulse wave by time-differentiating the facial pulse waveform detected by the first waveform detector 15a, and the differential coefficient A zero cross point where the sign changes from positive to negative, that is, a maximum point indicating a peak is detected. For example, the first peak detector 16a determines whether or not the differential coefficient of the amplitude value detected at the previous sampling point is zero each time the amplitude value of the face pulse waveform is detected. To do. At this time, when the differential coefficient of the amplitude value detected at the previous sampling point is zero, the first peak calculation unit 16a changes the sign of the differential coefficient of the amplitude value from positive to negative at the previous and subsequent sampling points. It is further determined whether or not to change. As a result, when the sign of the differential coefficient changes from positive to negative before and after, the first peak detector 16a detects the sampling point where the differential coefficient is zero as the peak of the pulse waveform of the face, The time of the sampling point at which the peak appears is registered in an internal memory (not shown). Note that the peak detection does not necessarily have to be realized by time differentiation of the pulse waveform, and may be detected from the face pulse waveform itself.

  The second peak detector 16b is a processor that detects the peak of the pulse waveform of the second living body. As one aspect, the second peak detector 16b detects the peak of the finger pulse waveform by detecting the zero-cross point from the differential waveform of the finger pulse wave, similar to the first peak detector 16a. Detect and register the time of the sampling point where the peak appears in the internal memory.

  FIG. 7 is a graph showing an example of a facial pulse wave and a finger pulse wave. The vertical axis of the graph shown in FIG. 7 indicates signal intensity (amplitude), and the horizontal axis indicates time. As shown in FIG. 7, in the pulse wave of the face, the peak appears at the time point T1, while in the pulse wave of the finger, the peak appears at the time point T2. It can be seen that the facial pulse wave and the finger pulse wave have different points in time when the peaks appear, and the peak of the finger pulse wave is later than the peak of the facial pulse wave. This is because there is a time difference in the timing at which the blood delivered from the heart reaches each part of the living body. Generally, since the finger is farther from the heart than the face, the pulse wave first propagates to the face and then propagates to the finger.

  The delay amount calculation unit 17 is a processing unit that calculates a delay amount between the pulse wave waveform of the first living body and the pulse wave waveform of the second living body. As one aspect, the delay amount calculation unit 17 detects the peak time detected from the pulse wave waveform of the face by the first peak detection unit 16a and the pulse wave waveform of the finger by the second peak detection unit 16b. The amount of delay between the pulse waves of the face and finger is calculated by calculating the time difference between the peak times. For example, in the example shown in FIG. 7, the delay amount calculation unit 17 can calculate the delay amount between the finger and the face by subtracting the time T1 from the time T2. Here, the time T1 when the peak is detected in the face pulse waveform is subtracted from the time T2 when the peak is detected in the finger pulse waveform, but T2 may be subtracted from T1. In this case, the same value can be calculated by taking an absolute value.

  The propagation velocity calculation unit 18 is a processing unit that calculates a pulse wave propagation velocity using a delay amount between the first living body and the second living body. Here, distance and time are used to calculate the speed, but the time from the distance and time until the pulse wave propagates to the finger after the pulse wave propagates to the face by the delay amount calculation unit 17. A delay amount that is a time difference is required. Therefore, by calibrating the distance difference between the distance from the heart to the finger and the distance from the heart to the face, which is the starting point of the pulse wave, The propagation speed can be calculated. In this equation (7), “Vp” refers to the pulse wave velocity, “L” refers to the distance difference between the heart-to-finger distance and the heart-to-face distance, and “Td” is Refers to the amount of delay.

  Vp = L / Td (7)

  That is, the propagation velocity calculation unit 18 substitutes the pulse wave propagation velocity Vp by substituting the delay amount Td calculated by the delay amount calculation unit 17 and the distance difference L set on the internal memory into the above equation (7). It can be calculated. As an example, the distance difference L can be initially set by the user of the blood flow index calculation device 10. For example, the value of the distance difference L itself may be input via an input device (not shown), or may be initially set by inputting a distance L1 from the heart to the face and a distance L2 from the heart to the finger. It is good. In addition, for each combination of items such as age, gender and height, the distance difference is obtained by searching for the age, gender and height input by the user from the statistical data in which the statistical value of the distance difference corresponding to the combination is associated. It is also possible to have the default setting.

  Here, the above-mentioned pulse wave velocity is an index useful for diagnosing the progress of arteriosclerosis, and for example, blood vessel age can be measured from the pulse wave velocity. Thus, the pulse wave velocity can be used as an information source for obtaining an index useful for health management by being output to an application program for calculating the degree of progression of arteriosclerosis and the age of blood vessels.

  The blood pressure calculation unit 19 is a processing unit that calculates blood pressure. As one aspect, the blood pressure calculation unit 19 calculates the blood pressure by substituting the pulse wave propagation velocity calculated by the propagation velocity calculation unit 18 into the following equation (8). The following formula (8) is an example of a blood pressure calculation formula, and the blood pressure, which is the objective variable, is approximated by a linear formula using the pulse wave velocity as an explanatory variable. In the following equation (8), “P” indicates blood pressure, and “Vp” indicates pulse wave propagation velocity. In the following formula (8), “A” indicates the slope of the linear expression, and “B” indicates the intercept of the linear expression, both of which are constants.

  P = A * Vp + B (8)

  The slope A and the intercept B are set to different values depending on the individual. For example, the slope A and the intercept B are measured using a blood pressure measurement value measured by a sphygmomanometer or the like in synchronization with the calculation of the pulse wave propagation speed, together with the pulse wave propagation speed calculated by the blood flow index calculation device 10. It can be derived by inputting. By executing regression analysis such as the least square method between the measured values of the pulse wave velocity and blood pressure, the slope A and intercept B of the above equation (8) can be set.

  Here, the blood pressure is an index useful for various diagnoses. For example, when blood pressure is high, diseases such as hypertension, renal disease, arteriosclerosis, hyperlipidemia, and cerebrovascular disease can be diagnosed. On the other hand, when the blood pressure is low, diseases such as heart failure, anemia, and major bleeding can be diagnosed. Thus, the blood pressure can be used as an information source for obtaining an index useful for health care by being output to the application program for executing the various diagnoses described above.

  For example, the pulse wave velocity and blood pressure can be output to an arbitrary output destination including a display device (not shown) included in the blood flow index calculation device 10. For example, when a measurement program for measuring blood vessel age, blood pressure, or the like using the pulse wave velocity, or a diagnostic program for diagnosing various diseases from blood pressure is installed in the blood flow index calculation device 10, a measurement program, A diagnostic program can be the output destination. In addition, a server device that provides a measurement program or a diagnostic program as a Web service can be used as an output destination. Further, a terminal device used by a person concerned of the user who uses the blood flow index calculation device 10, such as a caregiver or a doctor, can be used as the output destination. This also enables monitoring services outside the hospital, for example, at home or at home. Needless to say, the measurement results and the diagnostic results of the measurement program and the diagnostic program can be displayed on the terminal devices of the persons concerned including the blood flow index calculation device 10.

  In addition, said function part is realizable by making a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc. run a blood-flow parameter | index calculation program. Further, the above-described functional unit can be realized by a hard wired logic such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The functional unit includes the first acquisition unit 12a, the second acquisition unit 12b, the extraction unit 13, the first waveform detection unit 15a, the second waveform detection unit 15b, the first peak detection unit 16a, the first 2 peak detector 16b, delay amount calculator 17, propagation velocity calculator 18, blood pressure calculator 19, and the like.

[Process flow]
Subsequently, the flow of processing of the blood flow index calculation device 10 according to the present embodiment will be described. Here, after (1) blood flow index calculation processing executed by the blood flow index calculation device 10 is described, (2) first detection processing and (3) executed as a subroutine of blood flow index calculation processing ) The second detection process will be described.

(1) Blood Flow Index Calculation Processing FIG. 8 is a flowchart illustrating the procedure of blood flow index calculation processing according to the first embodiment. This blood flow index calculation process is started when the user of the blood flow index calculation apparatus 10 starts the blood flow index calculation program.

  As shown in FIG. 8, the first acquisition unit 12a, the extraction unit 13, and the first waveform detection unit 15a detect the pulse wave of the face from the face image captured by the in-camera 11a. "Is executed (step S101).

  Thereafter, the first peak detector 16a determines whether or not the zero cross point where the sign of the differential coefficient changes from positive to negative, that is, the maximum point indicating the peak, can be detected from the pulse wave waveform of the face detected in step S101. Determination is made (step S102). If no peak can be detected from the pulse wave waveform of the face (No in step S102), the process proceeds to step S104 without performing step S103.

  At this time, when a peak can be detected from the pulse wave waveform of the face (Yes in step S102), the first peak detector 16a does not illustrate the time of the sampling point at which the peak of the pulse wave of the face appears in step S102. Register in the internal memory (step S103).

  Subsequently, the second acquisition unit 12b and the second waveform detection unit 15b execute “second detection processing” for detecting a finger pulse wave from the finger image captured by the out-camera 11b (step S104). .

  Thereafter, the second peak detector 16b determines whether or not the zero-cross point where the sign of the differential coefficient changes from positive to negative, that is, the maximum point indicating the peak, can be detected from the pulse wave waveform of the finger detected in step S104. Determination is made (step S105). If no peak can be detected from the pulse wave of the finger (No in step S105), the process returns to step S101 without performing the process in step S103, and the first detection process is executed.

  At this time, when a peak can be detected from the pulse waveform of the finger (Yes in step S105), the delay amount calculation unit 17 determines whether or not the peak time of the facial pulse wave has been registered in the internal memory. (Step S106). Even when the peak time of the face pulse wave has not been registered (No in step S106), the process returns to step S101 and the first detection process is executed.

  Here, when the peak time of the facial pulse wave has been registered (step S106 Yes), the delay amount calculation unit 17 is the time difference of the peak time between the facial pulse wave and the finger pulse wave. A delay amount Td is calculated (step S107).

  Subsequently, the propagation velocity calculation unit 18 calculates the pulse wave propagation velocity Vp by substituting the distance difference L set in the internal memory together with the delay amount calculated in step S107 into the above equation (7) ( Step S108). Furthermore, the blood pressure calculator 19 calculates the blood pressure by substituting the pulse wave velocity Vp calculated in step S108 into the above equation (8) (step S109).

  After that, the pulse wave velocity Vp calculated in step S108 and the blood pressure P calculated in step S109 are output to a predetermined output destination, the peak time registered in the internal memory is deleted, and the process is terminated ( Step S110). Here, the case where the process is terminated after the peak time is deleted has been illustrated, but the process may return to step S101 and repeatedly output the pulse wave velocity Vp and blood pressure P.

  In the flowchart shown in FIG. 8, the case where the second detection process is executed after the first detection process is executed is illustrated, but the order of these processes may be reversed, or both may be performed in parallel. It may be executed. In this case, when the facial pulse wave peak time and the finger pulse wave peak time have already been registered in the internal memory, the processing after step S107 may be executed.

  In the flowchart shown in FIG. 8, the case where both the pulse wave velocity and the blood pressure are calculated has been illustrated, but it is not always necessary to calculate both, and it is also possible to calculate only one of them.

(2) First Detection Process FIG. 9 is a flowchart illustrating the procedure of the first detection process according to the first embodiment. This process is a process corresponding to step S101 shown in FIG. 8, and is started when the blood flow index calculation program is started by the user of the blood flow index calculation apparatus 10.

  As shown in FIG. 9, the in-camera 11a captures the face of the user (step S201), and the first acquisition unit 12a acquires the face image captured in step S201 (step S202).

  Subsequently, the extraction unit 13 performs image processing such as template matching on the image acquired in step S201 to thereby include a face area including predetermined face parts such as the user's eyes, nose, lips, cheeks and hair. Is extracted (step S203).

  Then, the extraction unit 13 calculates the representative value of the pixel value of each pixel included in the face area extracted in step S203 for each wavelength component of the G component and the R component (step S204). As a result, an R signal that is a representative value of the R component of the face area is output to the BPF 152R and the BPF 156R, and a G signal that is a representative value of the G component of the face area is output to the BPF 152G and the BPF 156G.

  Subsequently, the BPF 152R extracts a signal component of a specific frequency band of the R signal, for example, a frequency band of 0.05 Hz to 0.3 Hz, while the BPF 152G extracts a signal component of a specific frequency band of the G signal ( Step S205A).

  The extraction unit 153R extracts the absolute intensity value of the signal component of the specific frequency band of the R signal, while the extraction unit 153G extracts the absolute intensity value of the signal component of the specific frequency band of the G signal (step S206). ).

  After that, the LPF 154R performs a smoothing process for responding to a time change on the time-series data of the absolute intensity value in the specific frequency band of the R signal, while the LPF 154G has the absolute intensity value in the specific frequency band of the G signal. A smoothing process for responding to time changes is performed on the time series data (step S207).

  Subsequently, the calculation unit 155 divides the absolute value strength G′n of the specific frequency band of the G signal output by the LPF 154G by the absolute value strength R′n of the specific frequency band of the R signal output by the LPF 154R. The correction coefficient k is calculated by executing “G′n / R′n” (step S208).

  In parallel with the processing in step S205A, the BPF 156R extracts a pulse wave frequency band of the R signal, for example, a signal component in a frequency band of 0.5 Hz to 4 Hz, while the BPF 156G is a pulse wave frequency of the G signal. A band signal component is extracted (step S205B).

  Thereafter, the multiplication unit 157 multiplies the signal component in the pulse wave frequency band of the R signal extracted in step S205B by the correction coefficient k calculated in step S208 (step S209). The calculation unit 158 then subtracts the signal component in the pulse wave frequency band of the R signal multiplied by the correction coefficient k in step S209 from the signal component in the pulse wave frequency band of the G signal extracted in step S205B. Gs-k * Rs "is executed (step S210), and the process ends.

  As described above, the time-series data of the signal obtained by the calculation in step S210 corresponds to the pulse wave waveform of the face. The amplitude value of the facial pulse wave signal is input to the first peak detector 16a at a sampling rate corresponding to the frame frequency.

(3) Second Detection Process FIG. 10 is a flowchart illustrating the procedure of the second detection process according to the first embodiment. This process is a process corresponding to step S104 shown in FIG. 8, and is started after the first detection process is completed. Note that the second detection process is not limited to after the first detection process is completed, and may be executed prior to the first detection process or may be executed simultaneously with the first detection process.

  As shown in FIG. 10, the out-camera 11b captures the user's finger (step S301), and the second acquisition unit 12b acquires the finger image captured in step S301 (step S302). Subsequently, the second waveform detection unit 15b calculates the representative value of the G component of the pixel value of each pixel of the finger image acquired in step S302 (step S303).

  Thereafter, the second waveform detector 15b extracts a signal component in the pulse wave frequency band of the G signal having the representative value of the G component as an amplitude value (step S304). The time-series data of the signal obtained by the extraction in step S304 corresponds to the waveform of the finger pulse wave. Thus, the 2nd waveform detection part 15b detects the pulse wave of a finger | toe, and complete | finishes a process.

[Effect of Example 1]
As described above, the blood flow index calculating apparatus 10 according to the present embodiment detects the pulse wave of the face from the image of the in-camera 11a out of the two cameras and detects the pulse wave of the finger from the image of the out-camera 11b. Then, an index relating to blood flow is calculated from the amount of delay between the two pulse waves. For this reason, the blood flow index calculation device 10 according to the present embodiment can calculate the pulse wave velocity and blood pressure by using the hardware of a general mobile terminal and starting the delay amount. Therefore, according to the blood flow index calculation device 10 according to the present embodiment, it is possible to calculate a blood flow index without extra hardware.

  Furthermore, in the blood flow index calculating apparatus 10 according to the present embodiment, the blood flow index is calculated using two images captured by the in-camera 11a and the out-camera 11b. It is possible to suppress the troublesomeness caused by wearing or the like than when measuring the. For example, in the blood flow index calculation device 10 according to the present embodiment, it is possible to suppress troublesomeness such as a trouble of wearing the cuff on the subject's arm and a feeling of pressure that the arm is compressed by the cuff, as in a general blood pressure monitor. In addition, since the blood flow index calculating device 10 according to the present embodiment does not need to be equipped with extra hardware, the device scale can be kept at a scale that is always easy to carry and the cost of the device can be reduced.

  In the first embodiment, the case where the pulse wave velocity and the blood pressure are calculated using the delay amount between the face and the finger is exemplified. However, depending on the posture that the user takes when photographing the face image and the finger image. In some cases, the relative height between the face and the finger deviates, and the pulse wave velocity and blood pressure are calculated higher or lower than the original values. Therefore, in the present embodiment, a case will be described in which the pulse wave velocity and the blood pressure are corrected based on the height difference between the face and the finger.

[Configuration of blood flow index calculation device]
FIG. 11 is a block diagram illustrating a functional configuration of the blood flow index calculation device according to the second embodiment. Compared with the blood flow index calculation device 10 shown in FIG. 1, the blood flow index calculation device 20 shown in FIG. 11 has an acceleration sensor 21, an inclination detection unit 22, a distance calculation unit 23, a height calculation unit 24, and a correction unit. 25 is further different. In FIG. 11, the same reference numerals are given to functional units that exhibit the same functions as the functional units illustrated in FIG. 1, and description thereof is omitted.

  The acceleration sensor 21 is a sensor that detects the acceleration of the blood flow index calculation device 20. As an aspect of the acceleration sensor 21, a triaxial acceleration sensor that measures acceleration in the X axis direction, the Y axis direction, and the Z axis direction can be employed. Thus, the sensor values in the three-axis directions measured by the acceleration sensor 21 are converted into digital values by an A / D converter (not shown) and then output to the inclination detection unit 22. As an acceleration measurement method, any method such as a semiconductor method, a mechanical method, an optical method, or the like can be adopted.

  The inclination detection unit 22 is a processing unit that detects the inclination of the blood flow index calculation device 20. The “tilt” here refers to the tilt of the installation surface of the in-camera 11a with respect to the vertical direction, that is, the liquid crystal display surface. For example, the inclination takes a positive value when the blood flow index calculation device 20 is inclined from the vertical toward the installation surface of the in-camera 11a, and is inclined from the vertical toward the back surface where the out-camera 11b is located. In some cases, a negative value is taken. As one aspect, every time acceleration is detected by the acceleration sensor 21, the inclination calculation unit 22 uses the acceleration to update the inclination stored in the internal memory, thereby holding the latest inclination in the internal memory. .

  The distance calculation unit 23 is a processing unit that calculates the distance between the first living body and the in-camera 11a. As an aspect, the distance calculation unit 23 calculates the distance D between the user's face and the in-camera 11a by calculating the following equation (9).

  D = D0 * Q / Q0 (9)

  Here, “D0” and “Q0” in the above equation (9) are parameters to be calibrated in advance. Among these, “D0” indicates the distance between the face and the in-camera 11a. In addition, “Q0” indicates the number of pixels between a plurality of feature points on the face image captured by the in-camera 11a when the distance between the face and the in-camera 11a is the distance D0. For example, a combination of arbitrary facial parts such as a pair of eyes, a pair of nose and mouth, a pair of both ears, and the like can be adopted as feature points. As an example, when the number of pixels between the feature points of both eyes is obtained, a pattern matching algorithm that can execute extraction of feature points including the center point of the right eye and the center point of the left eye as the feature points of both eyes is used as the extraction unit 13. Adopt. Then, the above-described “Q0” is obtained by counting the number of pixels between the center point of the right eye and the center point of the left eye extracted from the face image taken by the face and the in-camera 11a at a distance D0. be able to. In this way, D0 and Q0 are calibrated in advance.

  Further, “Q” in the above equation (9) represents the number of pixels between a plurality of feature points on the face image captured by the in-camera 11a when the distance between the face and the in-camera 11a is the distance D. Point to. The “Q” is not a pre-calibrated value as in “Q0”, but is a face image used for calculating the pulse wave velocity and blood pressure when correcting the pulse wave velocity and blood pressure. To the number of pixels between the feature points.

  Thus, D0 and Q0 among the parameters of the above equation (9) are calibrated in advance, and Q can be measured from the face image used for calculating the pulse wave velocity and blood pressure by the extraction unit 13. it can. Therefore, the distance calculation unit 23 obtains the distance Q between the feature points at the stage where the biological region is extracted from the face image by the extraction unit 13, and substitutes it into the above equation (9) to thereby determine the The distance D between them can be calculated.

  The height calculation unit 24 is a processing unit that calculates a difference in height between the first living body and the in-camera 11a. As one aspect, the height calculation unit 24 uses the inclination θ of the installation surface of the in-camera 11a detected by the inclination detection unit 22 and the distance D calculated by the distance calculation unit 23, and the face and the in-camera 11a. The difference H in height is calculated.

  FIG. 12 is a diagram illustrating an example of a method for calculating a difference in height between the face and the in-camera 11a. FIG. 12 shows a scene where the in-camera 11a is raised higher than the position of the user's eyes and a face image and a finger image are captured. The in-camera 11a is positioned higher than the position of the user's eyes. The calculation method in the case of lowering is the same. Note that “θ” shown in FIG. 12 indicates the inclination of the installation surface of the in-camera 11a, “D” indicates the distance between the user's face and the in-camera 11a, and “H” indicates the user. The height difference between the face and the in-camera 11a.

  As shown in FIG. 12, when the user photographs the user himself, it can be assumed that the user installs the in-camera 11a with the installation surface facing the user's face. In this case, the inclination θ from the vertical direction of the installation surface of the in-camera 11a can be regarded as equal to the angle formed by the straight line connecting the user's face and the in-camera 11a and the horizontal line. Therefore, the height difference H between the user's face and the in-camera 11a is obtained by taking the sine of the trigonometric ratio, as shown in the following equation (10). It can be represented by the product of the distance D between and sin θ.

  H = D * Sinθ (10)

  Using the expression (10) derived in this way, the height calculation unit 24 can calculate the height difference H between the face and the in-camera 11a. That is, the height calculation unit 24 substitutes the inclination θ of the installation surface of the in-camera 11a detected by the inclination detection unit 22 and the distance D calculated by the distance calculation unit 23 into the above equation (10). It can be calculated. For example, when measuring the pulse wave velocity and blood pressure while lying down, the user captures a face image with the blood flow index calculating device 20 raised above the face. In this case, since the inclination θ has a positive value, the height difference H also has a positive value. On the other hand, when the pulse wave velocity and blood pressure are measured while the user is sitting, the face image is photographed while looking at the liquid crystal display while holding the blood flow index calculating device 20 around the chest. In this case, since the inclination θ has a negative value, the height difference H also takes a negative value.

  The correction unit 25 is a processing unit that corrects the value of an index related to blood flow. As an aspect, the correction unit 25 corrects the blood pressure value calculated by the blood pressure calculation unit 19 using the height difference H calculated by the height calculation unit 24. For example, the correction unit 25 can correct the blood pressure value by calculating the following equation (11). In the following equation (11), “Ps” indicates a blood pressure value after correction, and “P” indicates a blood pressure value before correction. “C” indicates a correction coefficient, and “H” indicates a height difference between the user's face and the in-camera 11a. The correction coefficient “C” is set to a value (constant) obtained by performing calibration in advance.

  Ps = P + C * H (11)

  Here, in the above equation (11), the value calculated by the blood pressure calculation unit 19 is substituted for the blood pressure P before correction, and the value calculated by the height calculation unit 24 is substituted for the height difference H. Is substituted. The correction coefficient “C” is also a known value set in advance. For this reason, the correction | amendment part 25 can calculate the value of the blood pressure after correction | amendment by said Formula (11). Although the blood pressure correction formula is exemplified here, for the same purpose, after correction using the following formula (12), which multiplies the pulse wave velocity Vp before correction by a correction coefficient E calibrated in advance. It is also possible to calculate the pulse wave propagation velocity Vps.

  Vps = Vp + E * H (12)

  For example, when measuring the pulse wave velocity and blood pressure while lying down, the user captures a face image with the blood flow index calculating device 20 raised above the face. In such a state, the arm pressure is reduced as compared with the case where the height of the face and the finger is the same, so the pulse wave velocity and blood pressure may be calculated lower. In such a case, the value obtained by multiplying the height difference H for which a positive value is calculated by the correction coefficient C is added to the blood pressure P before correction, so that the height of the face and the finger are the same. It can be corrected to a value equivalent to the calculated blood pressure. In addition, since the value obtained by multiplying the height difference H for which a positive value is calculated by the correction coefficient E is added to the pulse wave velocity Vp before correction, the height of the face and the finger is the same. Correction can be made to a value equivalent to the calculated pulse wave velocity.

  On the other hand, when the pulse wave velocity and blood pressure are measured while the user is sitting, the face image is photographed while looking at the liquid crystal display while holding the blood flow index calculating device 20 around the chest. In such a state, the arm pressure increases as compared with the case where the height of the face and the finger is the same, and thus the pulse wave velocity and blood pressure may be calculated higher. In such a case, a value obtained by multiplying the height difference H for which a negative value is calculated by the correction coefficient C is subtracted from the blood pressure P before correction, so that the height of the face and the finger are the same. It can be corrected to a value equivalent to the calculated blood pressure. In addition, since the value obtained by multiplying the height difference H calculated as a negative value by the correction coefficient E is subtracted from the pulse wave velocity Vp before correction, the height of the face and the finger is the same. Correction can be made to a value equivalent to the calculated pulse wave velocity.

  In addition, said distance calculation part 23, the height calculation part 24, and the correction | amendment part 25 are realizable by making CPU, MPU, etc. run a blood-flow parameter | index calculation program. Each functional unit described above can also be realized by a hard-wired logic such as ASIC or FPGA.

[Process flow]
Subsequently, a flow of processing of the blood flow index calculating device 20 according to the present embodiment will be described. The blood flow index calculation process executed by the blood flow index calculation device 20 is the same as the blood flow index calculation process according to the first embodiment, and thus the description thereof will be omitted, and the blood flow executed after the blood flow index calculation process will be omitted. The index correction process will be described.

  FIG. 13 is a flowchart illustrating a procedure of blood flow index correction processing according to the second embodiment. This blood flow index correction process is started when the pulse wave velocity and blood pressure are calculated by the blood flow index calculation process.

  As shown in FIG. 13, when the pulse wave velocity and blood pressure are calculated by the blood flow index calculation process, the inclination detection unit 22 uses the acceleration calculated by the acceleration sensor 21 from the vertical of the in-camera 11a. The inclination θ is detected (step S401). Thereafter, the distance calculation unit 23 substitutes the distance Q between the feature points extracted from the face image by the extraction unit 13 into the equation (9), thereby calculating the distance D between the user's face and the in-camera 11a. Calculate (step S402).

  Subsequently, the height calculation unit 24 substitutes the inclination θ detected in step S401 and the distance D calculated in step S402 into the above equation (10), whereby the user's face, the in-camera 11a, The difference H in height is calculated (step S403).

  Thereafter, the correcting unit 25 substitutes the pulse wave velocity Vps after correction by substituting the height difference H calculated in step S403 and the pulse wave velocity Vp before correction into the above equation (12). Calculate (step S404). Subsequently, the correcting unit 25 calculates the corrected blood pressure by substituting the height difference H calculated in step S403 and the uncorrected blood pressure P into (11) above (step S405). The process ends.

[Effect of Example 2]
As described above, the blood flow index calculation device 20 according to the present embodiment is between the inclination of the blood flow index calculation device 20 and the first living body calculated from the image of the first living body and the in-camera 11a. The pulse wave velocity and blood pressure are corrected using the difference in height between the two. For this reason, in the blood flow index calculation device 20 according to the present embodiment, even when the heights of the first living body and the second living body are different, the pulse wave velocity and blood pressure detection accuracy do not deteriorate. Therefore, the blood flow index calculating device 20 according to the present embodiment can calculate the pulse wave velocity and blood pressure with high accuracy.

  Although the embodiments related to the disclosed apparatus have been described above, the present invention may be implemented in various different forms other than the above-described embodiments. Therefore, another embodiment included in the present invention will be described below.

[Calculation of delay amount]
In the first embodiment and the second embodiment described above, the case where the delay amount is calculated from the face image and the finger image captured at the constant frame frequency is exemplified. However, from the face image and the finger image captured at the variable frame frequency, The amount of delay can also be calculated.

  That is, since the time resolution increases as the frame frequency of the in-camera 11a and the out-camera 11b is increased, the detection accuracy of pulse waves of the face and fingers can be improved, but the number of images to be calculated increases accordingly. Therefore, the processing load also increases. In particular, when calculating the pulse wave velocity and blood pressure on a device that has lower performance than a stationary information processing device and requires operation with low power consumption, such as a portable terminal, the increase in processing load is This leads to an increase in power consumption, which in turn leads to a decrease in the battery driving time of the terminal.

  Therefore, in this embodiment, in order to suppress an increase in processing load by increasing the frame frequency, the vicinity of the peak where the peak appears in the pulse wave waveform is changed to a higher frame frequency while the peak frequency is increased. The frame frequency is changed to a lower value except in the vicinity.

  FIG. 14 is a graph illustrating an example of a method for changing a plurality of frame frequencies. FIG. 14 shows two waveforms, a facial pulse wave waveform detected from an image captured by the in-camera 11a and a differential waveform of the facial pulse wave. The vertical axis of the graph shown in FIG. 14 indicates signal strength (amplitude), and the horizontal axis indicates time. In the example of FIG. 14, it is assumed that the in-camera 11a and the out-camera 11b have an ability to capture an image at a first frame frequency Tc of about 15 Hz to 20 Hz and a second frame frequency Tf of about 60 Hz. The following explanation will be given.

  The blood flow index calculating device 10 or 20 detects two continuous peaks from the pulse wave waveform as a preparation stage for predicting the appearance of future peaks. In this case, since it is only necessary to be able to predict an approximate pulse wave cycle, the in-camera 11a is caused to image the user's face at the first frame frequency Tc. In addition, since it is sufficient to detect only one of the pulse wave of the face or the pulse wave of the finger in order to predict the pulse wave cycle, here, a case where the pulse wave of the face is used for prediction of the pulse wave cycle is illustrated.

  Then, the blood flow index calculation device 10 or 20 detects the face pulse waveform and the differential waveform of the face pulse waveform shown in FIG. 14 from the face image captured at the first frame frequency Tc. In this case, the peak of the differential waveform of the facial pulse waveform appears at the time Tp1 and then appears at the time Tp2. Thus, when at least two consecutive peaks are detected, the time Tp3 at which a peak appears in the future is predicted using the following equation (13). In the following formula (13), “Tp1” indicates the time when the first peak appears, and “Tp2” indicates the time when the second peak appears. Therefore, “Tp2-Tp1” indicates the pulse wave period. Means. “N” is an arbitrary integer. For example, by substituting 1 for N, the time for which a peak appears after one cycle of Tp2 can be obtained.

  Tp3 = Tp2 + N * (Tp2-Tp1) (13)

  For example, in the above formula (13), the value detected by the first peak detector 16a is substituted into Tp1 and Tp2, and an integer “1” is substituted into N, so that a peak appears one cycle after Tp2. The predicted appearance time Tp3 can be calculated.

  After the predicted peak appearance time Tp3 is calculated in this manner, the blood flow index calculating device 10 or 20 outputs a predetermined range of the peak predicted appearance time Tp3 every time the amplitude value of the pulse wave waveform is output. That is, it is determined whether or not it is within Tp3 ± α. At this time, when the time is not within the peak predicted appearance time Tp3 ± α, the blood flow index calculation device 10 or 20 displays the face image and the finger image at the first frame frequency Tc on the in-camera 11a and the out-camera 11b. Let's take an image.

  On the other hand, when the time is within the peak predicted appearance time Tp3 ± α, the blood flow index calculation device 10 or 20 displays the face image and the finger image at the second frame frequency Tf on the in-camera 11a and the out-camera 11b. Let's take an image. In this case, the face pulse wave and the finger pulse wave can be detected from the face image and the finger image captured with a temporal resolution higher than the first frame frequency Tc, and further, the delay amount, the pulse wave can be detected. The propagation speed and blood pressure can be calculated. In the example shown in FIG. 14, the peak is actually detected at a time Tp3f that is one frame different from the time Tp3 at which the peak was predicted by the above equation (13). However, the time Tp3f is also predicted. Since it is included in the appearance time Tp3 ± α, a peak can be detected with a high time resolution.

  In this way, the vicinity of the peak where the peak appears in the pulse wave waveform changes the frame frequency to a higher value, while the frame frequency is changed to a lower value except for the vicinity of the peak. The processing load that increases with the improvement can be suppressed.

[Blood flow index calculation program]
The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a smartphone, a mobile phone, or a PHS. In the following, an example of a computer that executes a blood flow index calculation program having the same function as that of the above embodiment will be described with reference to FIG.

  FIG. 15 is a schematic diagram illustrating an example of a computer that executes a blood flow index calculation program according to the first to third embodiments. As illustrated in FIG. 15, the computer 100 includes an operation unit 110 a, a speaker 110 b, an in camera 110 c, an out camera 110 d, a display 120, and a communication unit 130. Further, the computer 100 includes a CPU 150, a ROM 170, and a RAM 180. These units 110 to 180 are connected via a bus 140.

  As shown in FIG. 15, the ROM 170 has the first acquisition unit 12a, the second acquisition unit 12b, the extraction unit 13, the first waveform detection unit 15a, and the second waveform detection shown in the first embodiment. A blood flow index calculation program 170a that performs the same functions as the unit 15b, the blood pressure calculation unit 17, the propagation velocity calculation unit 18, and the blood pressure calculation unit 19 is stored in advance. The ROM 170 also includes the first acquisition unit 12a, the second acquisition unit 12b, the extraction unit 13, the first waveform detection unit 15a, the second waveform detection unit 15b, and the delay amount described in the second embodiment. A blood flow index calculation program 170a that performs the same functions as the calculation unit 17, the propagation velocity calculation unit 18, the blood pressure calculation unit 19, the inclination detection unit 22, the distance calculation unit 23, the height calculation unit 24, and the correction unit 25 is stored in advance. It does not matter if it is done. The blood flow index calculation program 170a may be integrated or separated as appropriate, similarly to each component of each functional unit shown in FIGS. That is, it is not always necessary that all data stored in the ROM 170 is stored in the ROM 170, and only data necessary for processing is stored in the ROM 170.

  Then, the CPU 150 reads the blood flow index calculation program 170 a from the ROM 170 and expands it in the RAM 180. Thus, as shown in FIG. 15, the blood flow index calculation program 170a functions as a blood flow index calculation process 180a. The blood flow index calculation process 180a expands various data read from the ROM 170 in an area allocated to itself on the RAM 180 as appropriate, and executes various processes based on the expanded various data. The blood flow index calculation process 180a includes processing executed by the functional units shown in FIGS. 1 and 11, for example, processing shown in FIGS. 8 to 10 and FIG. In addition, each processing unit virtually realized on the CPU 150 does not always require that all processing units operate on the CPU 150, and only a processing unit necessary for the processing needs to be virtually realized.

  Note that the blood flow index calculation program 170a is not necessarily stored in the ROM 170 from the beginning. For example, each program is stored in a “portable physical medium” such as a memory card inserted into the computer 100. Then, the computer 100 may acquire and execute each program from these portable physical media. In addition, each program is stored in another computer or server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, etc., and the computer 100 acquires and executes each program from these. It may be.

DESCRIPTION OF SYMBOLS 10 Blood flow parameter | index calculation apparatus 11a In camera 11b Out camera 12a 1st acquisition part 12b 2nd acquisition part 13 Extraction part 15a 1st waveform detection part 15b 2nd waveform detection part 16a 1st peak detection part 16b 1st 2 peak detection units 17 delay amount calculation unit 18 propagation speed calculation unit 19 blood pressure calculation unit

Claims (5)

On the computer,
Detecting a pulse wave of the first living body from an image of the first living body captured by the first camera;
A pulse wave of the second living body is detected from an image of a second living body that is different from the first living body by a second camera;
Calculating a delay amount between the pulse wave of the first living body and the pulse wave of the second living body;
Using the delay amount to calculate an index related to blood flow ,
Detecting the tilt of the terminal having the first camera;
Calculating a distance between the first living body and the first camera from an image of the first living body captured by the first camera;
Calculating a difference in height between the first living body and the first camera from an inclination of the terminal and a distance between the first living body and the first camera;
A blood flow index calculation program for executing a process of correcting an index relating to the blood flow based on a difference in height between the first living body and the first camera . In the computer,
From the pulse wave of the first living body or the pulse wave of the second living body, calculate the waveform of the pulse wave or the peak of the differential waveform of the pulse wave,
Further executing a process of predicting the pulse wave of the first living body and the peak of the pulse wave of the second living body from the peak of the waveform of the pulse wave or the peak of the differential waveform of the pulse wave,
As a process for detecting the pulse wave of the first living body,
When the peak is within a predetermined range from the predicted time, the first time resolution is higher than the first time resolution used by the first camera when the peak is outside the predetermined range. Detecting a pulse wave of the first living body from an image of the first living body captured by the camera of
As a process for detecting the pulse wave of the second living body,
When the peak is within a predetermined range from the predicted time, the second time resolution is higher than the first time resolution used by the second camera when the peak is out of the predetermined range. Detecting a pulse wave of the second living body from an image of the second living body captured by the camera of
As a process of calculating the delay amount,
The amount of delay between the pulse wave of the first living body detected from the image captured with the second time resolution and the pulse wave of the second living body detected from the image captured with the second time resolution. The blood flow index calculation program according to claim 1 , wherein the blood flow index calculation program is calculated. As a process for detecting the pulse wave of the first living body,
Extracting the first living body region included in an image of the first living body captured by the first camera;
Extracting a component of a specific frequency band other than a frequency band in which a pulse wave can be taken between each wavelength component from a representative value signal for each wavelength component of each pixel included in the first biological region,
Correction by which the signal of the representative value is multiplied when the difference of the signal of the representative value is calculated between the wavelength components by comparing the magnitude of the component of the specific frequency band between the wavelength components Calculating a correction coefficient that is a coefficient and is minimized after the component of the specific frequency band is calculated;
Multiplying at least one of the signals of representative values of each wavelength component by the correction coefficient,
Claim 1 or, characterized in that to detect the waveform of the signal component of the specific frequency band is offset to one another by calculating a difference between the signal of the representative value among the wavelength components after the multiplication of the correction coefficient 2. A blood flow index calculation program according to 2. A first waveform detector that detects a pulse wave of the first living body from an image obtained by imaging the first living body by the first camera;
A second waveform detector that detects a pulse wave of the second living body from an image obtained by imaging a second living body that is different from the first living body by the second camera;
A delay amount calculation unit for calculating a delay amount between the pulse wave of the first living body and the pulse wave of the second living body;
A blood flow index calculation unit that calculates an index related to blood flow using the delay amount ;
An inclination detecting unit for detecting an inclination of a terminal having the first camera;
A distance calculation unit that calculates a distance between the first living body and the first camera from an image obtained by capturing the first living body by the first camera;
A height calculation unit that calculates a difference in height between the first living body and the first camera from an inclination of the terminal and a distance between the first living body and the first camera;
A blood flow index calculation apparatus comprising: a correction unit that corrects an index related to the blood flow based on a difference in height between the first living body and the first camera . Computer
Detecting a pulse wave of the first living body from an image of the first living body captured by the first camera;
A pulse wave of the second living body is detected from an image of a second living body that is different from the first living body by a second camera;
Calculating a delay amount between the pulse wave of the first living body and the pulse wave of the second living body;
Using the delay amount to calculate an index related to blood flow ,
Detecting the tilt of the terminal having the first camera;
Calculating a distance between the first living body and the first camera from an image of the first living body captured by the first camera;
Calculating a difference in height between the first living body and the first camera from an inclination of the terminal and a distance between the first living body and the first camera;
A blood flow index calculation method , comprising: executing a process of correcting an index relating to the blood flow based on a difference in height between the first living body and the first camera .

Images