JP2021108752A - Biological information measuring device, biological information measuring system, control method of biological information measuring device, and control program - Google Patents

Abstract

To suppress deterioration in measurement accuracy of biological information due to a change in an imaging environment.SOLUTION: A biological information measuring device includes: a storage control unit for causing a spectrum of first irradiation light that the skin is irradiated with to be stored in a learning phase for acquiring a video of the skin captured for a model to be learned as a video for learning; a light source control unit for controlling a light source for irradiating the skin with light on the basis of a spectrum of second irradiation light that the skin is irradiated with and the spectrum of the first irradiation light in a prediction phase for predicting biological information including information related to a living body by using the model; and a prediction unit for predicting biological information on the basis of prediction information based on a video for prediction imaging the skin that is irradiated with controlled light which is emitted from the controlled light source, and the model.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to a biometric information measuring device, a biometric information measuring system, a control method of the biometric information measuring device, and a control program.

Patent Document 1 describes a pulse wave measuring device that measures a pulse wave from an image of photographed skin. Further, Patent Document 1 describes a pulse wave arithmetic unit that calculates biological information from the measured pulse wave. Here, the pulse wave arithmetic unit described in Patent Document 1 determines the optimum installation position of the pulse wave measuring device based on the direction in which the disturbance light is located and the brightness characteristic of the disturbance light.

Japanese Unexamined Patent Publication No. 2018-066431

By the way, the absorbance of a substance (hemoglobin or the like) in the body differs depending on the wavelength of the irradiation light on the skin. Then, the color of the photographed skin changes as the absorbance changes. Therefore, when the light applied to the skin changes, the color of the photographed skin changes. As a result, when the imaging environment changes, the pulse wave and biological information measured from the image of the skin photographed change.

In the technique described in Patent Document 1, the position of the pulse wave measuring device is moved according to the direction in which the disturbance light is located and the brightness characteristic of the disturbance light. Here, in the technique described in Patent Document 1, if the pulse wave measuring device cannot be moved to an appropriate position, the measurement accuracy of biological information may decrease due to a change in the imaging environment.

Therefore, one aspect of the present disclosure is, for example, to provide a biometric information measuring device capable of suppressing a decrease in the measurement accuracy of biometric information due to a change in the imaging environment.

The biological information measuring device according to one aspect of the present disclosure obtains the spectrum of the first irradiation light applied to the skin in the learning phase of acquiring the image of the skin taken to train the model as the learning image. The spectrum of the second irradiation light irradiated to the skin and the first irradiation light in the prediction phase of predicting the biological information including the information related to the living body by using the storage control unit to be stored and the model. Based on the spectrum of the above, a light source control unit that controls a light source that irradiates the skin with light, and a prediction that the controlled light is emitted from the controlled light source and the skin irradiated with the controlled light is photographed. It is provided with a prediction unit that predicts the biological information based on the prediction information based on the video and the model.

The biometric information measurement system according to one aspect of the present disclosure includes a biometric information measuring device, a light source, and a photographing device, and the biological information measuring device is for learning an image of skin photographed for learning a model. In the learning phase of acquiring as an image, in the storage control unit that stores the spectrum of the first irradiation light applied to the skin, and in the prediction phase of predicting biological information including information related to the living body using the model. A light source control unit that controls the light source that irradiates the skin with light based on the spectrum of the second irradiation light applied to the skin and the spectrum of the first irradiation light, and the controlled light source. It is provided with a prediction unit that predicts the biological information based on the prediction information based on the prediction image obtained by photographing the skin irradiated with the controlled light and the model.

The control method of the biological information measuring device according to one aspect of the present disclosure is the first irradiation light applied to the skin in the learning phase in which the image of the skin taken to train the model is acquired as the learning image. In the memory control step for storing the spectrum of the above and the prediction phase for predicting the biological information including the information related to the living body using the model, the spectrum of the second irradiation light applied to the skin and the first A light source control step that controls a light source that irradiates the skin with light based on the spectrum of the irradiation light, and a photograph of the skin that is irradiated with the controlled light by emitting controlled light from the controlled light source. Includes a prediction step for predicting the biological information based on the prediction information based on the predicted video and the model.

The control program according to one aspect of the present disclosure is the first irradiation applied to the skin in the learning phase in which the biological information measuring device acquires the image of the skin taken to train the model as the learning image. In the memory control function for storing the spectrum of light and the prediction phase for predicting biological information including information related to the living body using the model, the spectrum of the second irradiation light applied to the skin and the first A light source control function that controls a light source that irradiates the skin with light based on the spectrum of the irradiation light, and the skin that is irradiated with the controlled light by emitting controlled light from the controlled light source. The prediction function for predicting the biological information is realized based on the prediction information based on the captured prediction image and the model.

It is a schematic diagram which shows an example of the biological information measurement system which concerns on 1st Embodiment. It is a figure which shows an example of the image generated by the photographing apparatus. It is a block diagram which shows an example of the functional structure of the biological information measuring apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the relationship between a pixel value and a spectrum of light. It is a flowchart which shows an example of the process of estimating the spectrum of the light applied to the skin based on the image. It is a flowchart which shows an example of the operation of the said biometric information measuring apparatus in each of a learning phase and a prediction phase. It is a flowchart which shows an example of the operation of the said biometric information measuring apparatus in a prediction phase. It is a graph which shows an example of the result of having calculated the adjusted spectrum difference. It is a block diagram which shows an example of the functional structure of the biological information measuring apparatus which concerns on the modification of 1st Embodiment.

[First Embodiment]
Hereinafter, the first embodiment will be described with reference to FIGS. 1 to 9. In the drawings, the same or equivalent elements are designated by the same reference numerals, and duplicate description will be omitted.

(Overall configuration of biometric information measurement system 100)
FIG. 1 is a schematic view showing an example of the biological information measurement system 100.

The biological information measurement system 100 is a system for measuring the biological information of the subject 104 in a non-contact manner. Here, the biological information measured by the biological information measurement system 100 is, for example, blood pressure, heart rate, stress level, and the like. The biological information measurement system 100 includes, for example, a light source 101, an imaging device 102, and a biological information measurement device 103.

The light source 101 irradiates the skin of the subject (living body) 104 with light. In the following description, an example in which the subject 104 is a human will be described, but the subject 104 may be an animal or the like.

Further, the light source 101 emits light having a wavelength band of 3 or more. The light source 101 emits light within a predetermined range from, for example, the wavelengths of the peak spectrum of 600, 500, and 450 nm. In other words, the light source 101 emits red, green, and blue light. The light source 101 includes, for example, an LED (Light Emitting Diode) and the like.

Further, the light source 101 includes a mechanism for changing the timing of emitting light for each wavelength band. The biological information measuring device 103 controls the timing and changes the intensity of light emitted from the light source 101 (hereinafter referred to as “intensity of the light source 101”) for each wavelength band.

The photographing device 102 is a device that photographs an area including the skin of the subject 104. The photographing device 102 has sensitivity in the wavelength band of the light emitted from the light source. That is, the photographing device 102 may be, for example, an RGB camera having sensitivity in the wavelength bands of red, green, and blue light. Since the light emitted to the skin is the combined light of the light emitted from the light source 101 and the external light, the image taken by the photographing device 102 is affected by the combined light.

The biological information measuring device 103 estimates a pulse wave from the change in skin color photographed by the photographing device 102. Here, the pulse wave is a signal indicating a change in the volume of peripheral blood vessels. Since the blood pressure and volume of peripheral blood vessels change with the beating of the heart, which changes the color of the skin, the biological information measuring device 103 can estimate the pulse wave from the change. Details of the method by which the biological information measuring device 103 estimates the pulse wave will be described later.

The biological information measuring device 103 trains a model using a combination of a learning pulse wave estimated from a learning image of the skin and the learning biological information as learning data. In the following description, the stage of training the model is referred to as a "learning phase". Further, the learning image is an image generated by the biological information measuring device 103 in the learning phase, and the learning pulse wave is a pulse wave estimated from the learning image. Further, the learning biometric information is biometric information actually measured using, for example, a measuring device such as a blood pressure monitor.

When the model is trained, the biological information measuring device 103 estimates the spectrum of light radiated to the skin from the light source (hereinafter referred to as “learning spectrum”) and stores the spectrum. The details of the method by which the biological information measuring device 103 estimates the spectrum of light will be described later with reference to FIG. 7.

In addition, the biological information measuring device 103 estimates the predictive pulse wave from the predictive image of the skin, and predicts the biological information from the predictive pulse wave based on the trained model. That is, the biological information measuring device 103 measures the biological information of the subject 104 in a non-contact manner by prediction using the trained model.

In the following description, the stage of predicting biological information using the trained model is referred to as a "prediction phase". Further, the prediction image is an image generated by the biological information measuring device 103 in the prediction phase, and the prediction pulse wave is a pulse wave estimated from the prediction image.

In the prediction phase as well, the biological information measuring device 103 estimates the spectrum of the light emitted from the light source to the skin (hereinafter referred to as “prediction spectrum”) as in the learning phase. Next, the intensity of the light source 101 is controlled based on the learning spectrum, and the prediction spectrum is corrected so as to cancel the influence of the external light. As a result, the shooting conditions can be aligned between the learning phase and the prediction phase.

Then, the biological information measurement system 100 irradiates the skin with light having the corrected predicted spectrum, and predicts the biological information again. Therefore, the biometric information measurement system 100 and the biometric information measurement device 103 can suppress a decrease in the measurement accuracy of the biometric information due to a change in the imaging environment.

(How to acquire learning video and prediction video)
FIG. 2 is a diagram showing an example of an image generated by the photographing device 102. An image generated by the photographing apparatus 102 will be described with reference to FIG. In the following description, a case where the photographing device 102 photographs the face of the subject 104 will be illustrated and described.

For example, the photographing device 102 photographs the face of the subject 104 at a predetermined frame rate (for example, 300 fps (frames per second)) for a predetermined time (for example, 30 seconds to 60 seconds), and captures a time-series photographed image. Generate. The captured image includes a plurality of frame images 201.

The biological information measuring device 103 extracts the region of interest (so-called ROI (Region Of Interest) 202) including the skin from the frame image 201 included in the captured image, and generates an image of the extracted region of interest 202. For example, when the size of the frame image 201 is 640 vertical × 480 horizontal pixels, the size of the region of interest 202 may be 20 vertical × 20 horizontal pixels.

As illustrated in FIG. 2, when the image 201 includes a face region, the region of interest 202 may be a forehead portion. At this time, the biological information measuring device 103 acquires a time-series image showing the region of interest 202 including the skin of the forehead portion as a learning image and a prediction image.

(Functional configuration of biological information measuring device 103)
FIG. 3 is a block diagram showing an example of the functional configuration of the biological information measuring device 103. The biological information measuring device 103 includes, for example, a storage unit 310, a control unit 320, and the like.

The storage unit 310 stores information necessary for controlling the entire biological information measuring device 103. The storage unit 310 is realized by a ROM (Read Only Memory), a RAM (Random Access Memory), or the like, and the details thereof are not limited.

The storage unit 310 stores the model 311 and the learning spectrum 312. The model 311 may be a mathematical model that approximates the input / output relationship between the learning pulse wave and the biological information, and may be, for example, a linear model, a support vector machine, a fully connected neural network, a convolutional neural network, or the like.

When the model 311 is a fully connected neural network, for example, its input may be a pre-extracted pulse wave feature (for example, pulse wave amplitude). Further, when the model 311 is a convolutional neural network, for example, the input may be a pulse wave waveform corresponding to a plurality of frames.

The control unit 320 controls the entire biometric information measuring device 103. The control unit 320 includes, for example, an image acquisition unit 321, a first spectrum estimation unit 322, a first pulse wave calculation unit 323, a biological information acquisition unit 324, a learning unit 325, a memory control unit 326, a second spectrum estimation unit 327, and a second. Two pulse wave calculation unit 328, spectrum difference calculation unit (calculation unit) 329, light source control unit 330, prediction unit 331 and the like are included.

The control unit 320 is realized by, for example, a processor such as a CPU (Central Processing Unit). The processor, for example, reads and executes a program stored in the storage unit 310 to make each unit included in the control unit 320 function.

The first spectrum estimation unit 322, the first pulse wave calculation unit 323, the biological information acquisition unit 324, and the learning unit 325 execute the processing related to the learning phase. The second spectrum estimation unit 327, the second pulse wave calculation unit 328, the spectrum difference calculation unit 329, the light source control unit 330, and the prediction unit 331 execute the processing related to the prediction phase.

The image acquisition unit 321 acquires time-series captured images. Further, the image acquisition unit 321 extracts the region of interest 202 from each frame image 201 included in the captured image.

The image acquisition unit 321 generates a learning image in the learning phase, and outputs the generated learning image to the first pulse wave calculation unit 323. Further, the image acquisition unit 321 outputs one frame image included in the learning image to the first spectrum estimation unit 322 as a learning image in the learning phase.

Further, the image acquisition unit 321 generates a prediction image in the prediction phase, and outputs the generated prediction image to the second pulse wave calculation unit 328. Further, the image acquisition unit 322 outputs one frame image included in the prediction image to the second spectrum estimation unit 327 as a prediction image in the prediction phase.

The first spectrum estimation unit 322 estimates the spectrum of the light (first irradiation light) irradiated to the skin in the learning phase as the learning spectrum 312 based on the color information included in the learning image. The first spectrum estimation unit 322 stores the estimated learning spectrum 312 in the storage unit 310. The color information included in the learning image is the pixel value of each pixel included in the learning image. In the present embodiment, the pixel value includes the densities of each of the R (Red), G (Green), and B (Blue) colors. In the following description, for convenience of explanation, the color information included in the learning video is referred to as "learning color information".

The first pulse wave calculation unit 323 calculates the learning pulse wave based on the learning color information. The first pulse wave calculation unit 323 may calculate the learning pulse wave based on, for example, time-series changes in pixel values, luminance values, and the like in a plurality of frames included in the learning video.

The biological information acquisition unit 324 acquires the biological information of the subject 104 as the learning biological information in the learning phase.

The learning unit 325 causes the model 311 to learn the input / output relationship between the learning pulse wave and the learning biological information.

The memory control unit 326 stores the learning spectrum 312 in the learning phase. Further, the storage control unit 326 stores the trained model 311 in the storage unit 310 after the learning phase.

The second spectrum estimation unit 327 estimates the spectrum of the light (second irradiation light) irradiated to the skin in the prediction phase as the prediction time spectrum based on the color information included in the prediction image. The color information included in the prediction image is a pixel value of each pixel included in the prediction image. In the following description, for convenience of explanation, the color information included in the prediction image is referred to as "prediction color information".

The second pulse wave calculation unit 328 calculates the prediction pulse wave from the prediction image.

The spectrum difference calculation unit 329 calculates the difference between the value based on the learning color information and the value based on the prediction color information. The value based on the learning color information includes, for example, an average value of pixel values corresponding to a plurality of pixels included in the learning image. Further, the value based on the prediction color information includes, for example, an average value of pixel values corresponding to a plurality of pixels included in the prediction image. In the following description, for convenience of explanation, the above difference calculated by the spectrum difference calculation unit 329 will be referred to as “adjusted spectral difference”. Details of the method for calculating the adjusted spectral difference will be described later.

In the prediction phase, the light source control unit 330 controls the light source 101 based on the prediction time spectrum and the learning time spectrum 312. Specifically, the light source control unit 330 reduces the adjusted spectral difference by controlling the intensity of the light source 101. For example, the light source control unit 330 controls the light source 101 to keep the adjusted spectrum difference within a predetermined range.

The prediction unit 331 predicts the biological information based on the prediction information based on the prediction image and the model 311. The prediction information is information estimated from the prediction image, and is, for example, a prediction pulse wave estimated from the prediction image when the skin irradiated with the controlled light is photographed. The controlled light is the light emitted from the light source 101 controlled by the light source control unit. Specifically, the controlled light is the light emitted from the light source 101 after the light source control unit 330 reduces the adjusted spectral difference.

Next, the operation of the biological information measuring device 103 will be described.

Regarding the operation of the biological information measuring device 103, first, a method of estimating the spectrum of light will be described in detail.

(Estimation method of light spectrum)
The relationship between the pixel values of the pixels included in the image of the skin and the spectrum of the light applied to the skin will be described. FIG. 4 is a diagram showing an example of the relationship between the pixel value and the spectrum of light and the like.

With respect to the wavelength bands of light corresponding to each of the R, G, and B colors, the spectrum of the light emitted from the light source 101, the spectrum of the external light, the absorption coefficient of the substance related to the skin pigment, and the spectral sensitivity of the photographing apparatus 102. The multiplied value correlates with the pixel value for each of R, G, and B.

The spectrum of light emitted from the light source 101 correlates with the intensity of the light source 101. Further, the external light is different from the light emitted from the light source 101 in the skin photographing environment. The external light may be, for example, sunlight. In addition, substances related to skin pigments are substances related to skin color. Substances related to skin pigments include melanin, hemoglobin and the like. The absorption coefficient of substances related to skin pigments varies depending on the wavelength band of the light radiated to the skin. The spectral sensitivity of the photographing apparatus 102 is the spectral sensitivity with respect to the wavelength band of light corresponding to each of the R, G, and B colors.

Here, when the photographing device 102 photographs the skin for a predetermined time (for example, 30 seconds to 60 seconds), the case where the spectral sensitivity of the photographing device 102 is steady will be described as an example. In that case, the pixel values for each of R, G, and B correlate with the spectrum of the light applied to the skin with respect to the wavelength band of the light corresponding to each of the colors R, G, and B. As described above, the light applied to the skin is a composite light of the light emitted from the light source 101 and the external light. Therefore, the spectrum of the light applied to the skin correlates with the spectrum of the light emitted from the light source 101 and the spectrum of the external light.

Therefore, in the learning phase, the learning spectrum 312 in the wavelength band of light corresponding to each color of R, G, and B correlates with the pixel value for each of R, G, and B in the learning video. Similarly, in the prediction phase, the spectrum at the time of prediction in the wavelength band of light corresponding to each color of R, G, and B correlates with the pixel value for each of R, G, and B in the prediction image.

From the above, the pixel value of the pixel included in the image of the skin is correlated with the spectrum of the light applied to the skin. Therefore, the biological information measuring device 100 estimates the spectrum of the light applied to the skin based on the image.

Next, the details of the process of estimating the spectrum of the light applied to the skin based on the image will be described. FIG. 5 is a flowchart showing an example of a process of estimating the spectrum of light applied to the skin based on an image. Specifically, FIG. 5 is a flowchart showing an example of step S603 (described later) shown in FIG. 6 and an example of processing of step S703 (described later) shown in FIG. 7.

In step S501, the biometric information measuring device 103 calculates the average value of the pixel values of the plurality of pixels included in the video for each of R, G, and B. Specifically, in the learning phase, the first spectrum estimation unit 322 calculates the average value of the pixel values of the plurality of pixels included in the learning video for each of R, G, and B. Further, in the prediction phase, the second spectrum estimation unit 327 calculates the average value of the pixel values of the plurality of pixels included in the prediction image for each of R, G, and B.

In step S502, the biological information measuring device 103 normalizes the average value of the pixel values calculated for each of R, G, and B.

In step S503, the biological information measuring device 103 estimates the normalized average value of the pixel values for each of R, G, and B as the spectrum of the light applied to the skin. In the learning phase, the first spectrum estimation unit 322 estimates the normalized average value of the pixel values for each of R, G, and B as the learning spectrum 312. In other words, the first spectrum estimation unit 322 estimates a value obtained by normalizing the average value of the pixel values of each color included in the learning image as a learning spectrum. Then, the storage control unit 326 stores the learning spectrum 312 in the storage unit 310 in the learning phase. Further, in the prediction phase, the second spectrum estimation unit 327 estimates the normalized average value of the pixel values for each of R, G, and B as the prediction time spectrum. In other words, the second spectrum estimation unit 327 estimates a value obtained by normalizing the average value of the pixel values of each color included in the prediction image as the prediction time spectrum.

Next, the operation of the biometric information measuring device 103 in each of the learning phase and the prediction phase will be described. FIG. 6 is a flowchart showing an example of the operation of the biological information measuring device 103 in each of the learning phase and the prediction phase. Steps S601 to S606 of FIG. 6 show an example of the operation of the biological information measuring device 103 in the learning phase. Steps S611 to S612 in FIG. 6 show an example of the operation of the biometric information measuring device 103 in the prediction phase.

(Operation of biometric information measuring device 103 in the learning phase)
First, the operation of the biological information measuring device 103 in the learning phase will be described.

In step S601, the light source control unit 330 causes the light source 101 to emit light.

In step S602, the image acquisition unit 321 causes the photographing device 102 to photograph the skin of the subject 104. The photographing device 102 photographs the skin of the subject 104 in response to an instruction from the image acquisition unit 321 and generates a captured image. Here, when the photographing device 102 photographs the skin of the subject 104, the light emitted from the light source 101 irradiates the skin. Then, the video acquisition unit 321 acquires the learning video. Specifically, the video acquisition unit 321 acquires the video of the region of interest 202 from the captured video as a learning video.

In step S603, the first spectrum estimation unit 322 estimates the learning spectrum 312. Specifically, the video acquisition unit 321 extracts one frame image 201 included in the time-series learning video as a learning image. The learning image may be any frame image 201 in the time-series learning video. The first spectrum estimation unit 322 estimates the learning spectrum 312 based on the learning image.

In step S604, the first pulse wave calculation unit 323 calculates the learning pulse wave based on the learning image. For example, the first pulse wave calculation unit 323 extracts a feature amount from each frame image 201 included in the learning video. The feature amount is information related to the color information for learning, and may be, for example, the pixel value of each pixel for each of R, G, and B, the brightness of each pixel, and the like. Then, the first pulse wave calculation unit 323 calculates a signal indicating a temporal change of the feature amount over a predetermined time. The first pulse wave calculation unit 323 determines a signal output by performing independent component analysis, a low-pass filter, or the like on the signal as a learning pulse wave.

In step S605, the biometric information acquisition unit 324 acquires the biometric information for learning. Specifically, the biological information acquisition unit 324 acquires the biological information corresponding to the time when the learning pulse wave is acquired as the learning biological information. For example, the biometric information acquisition unit 324 may acquire the learning biometric information in synchronization with the time when the learning video is generated. The details of the method for measuring biological information are not limited. For example, a device that measures biological information (for example, a sphygmomanometer or the like) may measure biological information (for example, blood pressure). Then, the biological information acquisition unit 324 may acquire biological information from the device.

In step S606, the learning unit 325 causes the model 311 to learn the input / output relationship between the learning pulse wave and the learning biological information. Then, the storage control unit 326 stores the trained model 311 in the storage unit 310.

After the learning phase is completed, the biometric information measuring device 103 adjusts the intensity of the light source 101 in the prediction phase (step S611). Then, the biometric information measuring device 103 predicts the biometric information based on the trained model 311 (step S612).

(Operation of biometric information measuring device 103 in the prediction phase)
Next, the operation of the biological information measuring device 103 in the prediction phase will be described in detail. FIG. 7 is a flowchart showing an example of the operation of the biological information measuring device 103 in the prediction phase. Steps S701 to 706 shown in FIG. 7 are detailed processes of step S411 shown in FIG. Further, steps S707 to 708 shown in FIG. 7 are detailed processes of step S612 shown in FIG.

In step S701, the light source control unit 330 sets an initial value of the intensity of the light source 101. For example, the initial value may be the same as the intensity of the light source 101 in the learning phase.

Then, the light source control unit 330 causes the light source 103 to emit light. Then, the photographing device 102 photographs the skin of the subject 104 in response to the instruction from the image acquisition unit 321 and generates a captured image. Then, the video acquisition unit 321 acquires the prediction video. Specifically, the image acquisition unit 321 acquires the image of the region of interest 202 from the captured image as the prediction image. Here, the shooting environment in the learning phase and the shooting environment in the prediction phase may be different. Further, the intensity of the light source 101 in the learning phase and the intensity of the light source 101 in the prediction phase may be different.

In step S702, the light source control unit 330 determines whether or not the number of updates of the intensity of the light source 101 is a predetermined upper limit number. The update number is the number of times that the light source control unit 330 updates the intensity of the light source 101 in the prediction phase. When the number of updates of the intensity of the light source 101 is a predetermined upper limit number (step S702: Yes), the control transitions to step S703.

In step S703, the second spectrum estimation unit 327 estimates the spectrum at the time of prediction based on the prediction image. Specifically, the second spectrum estimation unit 327 executes the processes of steps S601 to S603 shown in FIG. 6 by inputting the prediction image.

In step S704, the spectrum difference calculation unit 329 calculates the difference between the learning spectrum 312 and the predicted spectrum as the adjusted spectrum difference. Specifically, the spectrum difference calculation unit 329 calculates the difference between the learning spectrum 312 and the predicted spectrum as the adjusted spectrum difference in each of R, G, and B.

Here, the learning spectrum 312 is a normalized average value of the pixel values of a plurality of pixels included in the learning video for each of R, G, and B. The prediction spectrum is a normalized average value of the pixel values of a plurality of pixels included in the prediction image for each of R, G, and B. Therefore, the spectrum difference calculation unit 329 calculates the difference between the normalized average value calculated for the learning video and the normalized average value calculated for the prediction video as the adjusted spectral difference.

In step S705, the spectral difference calculation unit 329 determines whether or not the adjusted spectral difference is in a predetermined range. For example, −0.1 <= predetermined range <= 0.1 may be used.

For example, it is assumed that the spectrum difference calculation unit 329 calculates the adjusted spectral difference as a value obtained by subtracting the normalized average value calculated for the learning video from the normalized average value calculated for the prediction video. .. In that case, when the difference between the spectrums to be adjusted is larger than the upper limit of the predetermined range, the light source control unit 330 increases the intensity of the light source 101. On the other hand, when the difference between the spectrums to be adjusted is smaller than the lower limit of the predetermined range, the light source control unit 330 lowers the intensity of the light source 101.

When the adjusted spectrum difference is not in the predetermined range (step S705: No), the light source control unit 330 updates the intensity of the light source 101 (step S706). Specifically, the light source control unit 330 controls the intensity of the light source 101 based on the adjusted spectrum difference. For example, the light source control unit 330 may update the intensity of the light source 101 by increasing or decreasing the value corresponding to the difference in the spectrum to be adjusted for each of R, G, and B with respect to the intensity of the set light source 101. ..

When the light source control unit 330 updates the intensity of the light source, the control returns to step S702 and continues the process. That is, the spectrum difference calculation unit 329 recalculates the adjusted spectral difference after the light source 101 is controlled (step S703 and step S704). Then, the light source control unit 330 controls the light source 101 again and keeps the adjusted spectrum difference calculated again within a predetermined range. As a result, the biological information measuring device 103 can reduce the difference between the learning spectrum 312 and the predicted spectrum.

When the adjusted spectrum difference is in a predetermined range (step S705: Yes), the second pulse wave calculation unit 328 calculates the prediction pulse wave from the prediction image (step S707).

Specifically, when the adjusted spectrum difference is in a predetermined range, the image acquisition unit 321 causes the photographing device 102 to re-photograph the skin of the subject 104. That is, in a state where the light source 101 emits the controlled light, the photographing device 102 re-photographs the skin of the subject 104 and generates a photographed image. Then, the image acquisition unit 321 generates a prediction image based on the captured image. Then, the second pulse wave calculation unit 328 calculates the prediction pulse wave from the generated prediction image. That is, the second pulse wave calculation unit 328 calculates the prediction pulse wave from the prediction image in which the skin is re-photographed after adjusting the intensity of the light source 101.

For example, the second pulse wave calculation unit 328 extracts the feature amount from each frame image 201 included in the prediction image. The feature quantity is related to the color information for prediction. The feature amount may be, for example, a pixel value for each of R, G, and B. Then, the second pulse wave calculation unit 328 generates a time-series signal based on the extracted feature amount. The second pulse wave calculation unit 328 performs processing such as independent component analysis and a low-pass filter on the generated time series signal, for example. The second pulse wave calculation unit 328 determines the signal output by performing the processing as the prediction pulse wave.

In step S708, the prediction unit 331 predicts biological information from the prediction pulse wave based on the trained model 311. For example, the prediction unit 331 calculates the correlation value between the plurality of learning pulse waves included in the trained model 311 and the prediction pulse wave. The prediction unit 331 identifies the learning pulse wave corresponding to the maximum correlation value among the calculated correlation values. Then, the prediction unit 331 may predict the biological information associated with the specified learning pulse wave as the biological information corresponding to the prediction pulse wave.

In the prediction phase, biometric information may be predicted multiple times in succession. In that case, the light source control unit 330 may control the light source 101 based on the intensity of the light source 101 set for the first time. Alternatively, the spectral difference calculation unit 329 may calculate the adjusted spectral difference each time before the prediction unit 331 predicts the biological information. Then, the light source control unit 330 may control the light source 101 each time based on the calculated spectral difference to be adjusted.

(Calculation result of adjusted spectrum difference)
Next, an example of the result of calculating the adjusted spectrum difference will be described. FIG. 8 is a graph showing an example of the result of calculating the adjusted spectrum difference. The vertical axis shows the adjusted spectral difference calculated by the spectral difference calculation unit 329. The horizontal axis shows the number of updates. Here, before the spectral difference calculation unit 329 calculates the adjusted spectral difference illustrated in FIG. 8, the light source control unit 330 controls the light intensities corresponding to the R, G, and B colors in the same priority order. It is assumed that As illustrated in FIG. 8, the adjusted spectral difference becomes smaller each time the intensity of the light source 101 is changed.

Due to the wavelength dependence of the absorbance of hemoglobin, among the light corresponding to each of the R, G, and B colors, the light in the wavelength band corresponding to each G color has the greatest influence on the shape of the pulse wave. Therefore, the light source control unit 330 may control the light intensity corresponding to each G color with priority over the light intensity corresponding to each of the R and B colors. As a result, the biological information measuring device 103 can reduce the number of times the intensity of the light source 101 is updated as compared with the case where the intensity of light corresponding to each of the R, G, and B colors is controlled by the same priority.

Further, the light source control unit 330 sets the amount of change according to the adjusted spectrum difference corresponding to the color of G to be larger than the amount of change corresponding to the adjusted spectrum difference corresponding to each of the R and B colors, and sets the light source 101. You may control it. As a result, the biological information measuring device 103 has a larger amount of change in the spectrum at the time of prediction than in the case where the light intensity corresponding to each of the R, G, and B colors is controlled by the same amount of change for each of the R, G, and B colors. can.

(effect)
From the above, the biological information measurement system 100 can reduce the adjusted spectral difference by adjusting the intensity of the light source 101. In other words, the biological information measurement system 100 can bring the predicted spectrum closer to the learning spectrum by adjusting the intensity of the light source 101. As a result, the biological information measurement system 100 can bring the imaging environment at the time of prediction closer to the imaging environment at the time of learning by adjusting the intensity of the light source 101. As a result, when the biological information measurement system 100 predicts biological information from a pulse wave based on an image of the skin, it is possible to suppress a decrease in prediction accuracy due to a change in the imaging environment. As a result, the biometric information measurement system 100 can improve the prediction accuracy of biometric information when the imaging environment at the time of learning and the imaging environment at the time of prediction are different.

(Modification example 1)
As a modification 1 of the biological information measuring device 103, the biological information measuring device 901 will be described. FIG. 9 is a block diagram showing an example of the functional configuration of the biological information measuring device 901. The biological information measuring device 901 includes a spectrum acquisition unit (acquisition unit) 902 instead of the first spectrum estimation unit 322 and the second spectrum estimation unit 327. Further, the biological information measuring device 901 includes a spectrum difference calculation unit 903 instead of the spectrum difference calculation unit 329.

The spectrum acquisition unit 902 acquires the measured value of the spectrum of the irradiation light to the skin. The measured value may be, for example, a spectrum of irradiation light to the skin measured by a spectroscope. Specifically, the spectrum acquisition unit 902 acquires the measured value of the spectrum during learning in the learning phase. The storage control unit 326 stores the measured value of the learning spectrum 312 in the storage unit 310. Further, the spectrum acquisition unit 902 acquires the measured value of the spectrum at the time of prediction in the prediction phase. In other words, the spectrum acquisition unit 902 acquires the measured value of the learning spectrum or the measured value of the predicted spectrum.

The spectrum difference calculation unit 903 calculates the difference between the measured value of the learning spectrum and the measured value of the predicted spectrum as the adjusted spectrum difference. Therefore, the biological information measuring device 901 can calculate the adjusted spectrum difference more accurately. Therefore, the biological information measuring device 901 contributes to reducing the number of times the intensity of the light source 101 is updated as compared with the biological information measuring device 103.

Assuming that the light source 101 emits light in a wavelength band of 3 or more, for example, an example in which light having a peak spectrum wavelength within a predetermined range from 600, 500, 450 nm has been described, but the present invention is not limited to this. The light source 101 may emit near-infrared light in addition to red, green, and blue. In that case, the photographing apparatus 102 may be, for example, an RGB + IR camera having sensitivity in the wavelength band of red, green, blue, and near infrared light. Further, the photographing device 102 may be composed of a combination of two cameras, an RGB camera and an IR camera.

(Additional notes)
The above-described embodiment may be described as the following embodiment, but is not limited to the following.

(Form 1) In the learning phase in which an image of the skin taken to train the model is acquired as a learning image, a storage control unit that stores the spectrum of the first irradiation light applied to the skin, and the model. In the prediction phase of predicting biological information including information related to the living body, the skin is based on the spectrum of the second irradiation light applied to the skin and the spectrum of the first irradiation light. The light source control unit that controls the light source that irradiates the light, and the prediction information and the model based on the prediction image in which the controlled light is emitted from the controlled light source and the skin irradiated with the controlled light is photographed. A biometric information measuring device including a prediction unit that predicts the biometric information based on the above.

(Form 2) The biological information measuring device according to Form 1, further comprising a first spectrum estimation unit that estimates the spectrum of the first irradiation light based on the color information included in the learning image.

(Form 3) The biometric information measuring device according to Form 1 or 2, further comprising a second spectrum estimation unit that estimates the spectrum of the second irradiation light based on the color information included in the prediction image.

(Form 4) The light source control unit further includes a calculation unit that calculates a difference between a value based on the color information included in the learning image and a value based on the color information included in the prediction image, and the light source control unit is the light source. The biometric information measuring device according to any one of the first to third forms, which controls the above-mentioned difference within a predetermined range.

(Form 5) The calculation unit recalculates the difference after the light source is controlled, and the light source control unit controls the light source again to bring the recalculated difference into the predetermined range. The biometric information measuring device according to the fourth embodiment.

(Form 6) A learning unit for learning the input / output relationship between the pulse wave calculated from the learning image and the information related to the living body is further provided in the model, and the prediction unit has the input / output relationship. The biometric information measuring device according to any one of forms 1 to 5, which predicts the biometric information from a pulse wave calculated from the prediction image based on the above.

(Form 7) The biometric information measuring device according to any one of forms 1 to 6, wherein the light source control unit controls a light source having a wavelength band of three or more.

(Form 8) The living body according to any one of forms 1 to 7, further comprising an acquisition unit for acquiring the measured value of the spectrum of the first irradiation light or the measured value of the spectrum of the second irradiation light. Information measuring device.

(Form 9) In the learning phase, which includes a biological information measuring device, a light source, and a photographing device, the biological information measuring device acquires an image of the skin photographed for learning a model as a learning image. In the memory control unit that stores the spectrum of the first irradiation light irradiated to the skin and the prediction phase that predicts the biological information including the information related to the living body using the model, the second irradiation to the skin is performed. Based on the spectrum of the irradiation light and the spectrum of the first irradiation light, the controlled light is emitted from the light source control unit that controls the light source that irradiates the skin with light, and the controlled light source, and the controlled light is emitted. A biological information measurement system including a prediction unit that predicts the biological information based on the prediction information based on the prediction image obtained by photographing the skin irradiated with the controlled light and the model.

(Form 10) In the learning phase in which an image of the skin taken to train the model is acquired as a learning image, a memory control step for storing the spectrum of the first irradiation light applied to the skin and the model. In the prediction phase of predicting biological information including information related to the living body using the above, the skin is subjected to the spectrum of the second irradiation light irradiated to the skin and the spectrum of the first irradiation light. A light source control step that controls a light source that irradiates light, prediction information based on a prediction image in which controlled light is emitted from the controlled light source, and the skin that is irradiated with the controlled light, and the model. A method for controlling a biometric information measuring device, which includes a prediction step for predicting the biometric information based on the above.

(Form 11) A memory for storing the spectrum of the first irradiation light applied to the skin in the learning phase in which the biological information measuring device acquires the image of the skin taken to train the model as a learning image. In the control function and the prediction phase of predicting biological information including information related to the living body using the model, the spectrum of the second irradiation light irradiated to the skin and the spectrum of the first irradiation light are set. Based on the light source control function that controls the light source that irradiates the skin with light, and the prediction image that the controlled light is emitted from the controlled light source and the skin that is irradiated with the controlled light is photographed. A control program that realizes a prediction function that predicts the biological information based on the prediction information and the model.

The present disclosure is not limited to the above embodiment, and is replaced with a configuration that is substantially the same as the configuration shown in the above embodiment, a configuration that exhibits the same action and effect, or a configuration that can achieve the same purpose. You may.

[Example of realization by software]
The control block (particularly, the control unit 320) of the biological information measuring device 103 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC (Integrated Circuit) chip) or the like, or software using a CPU. May be realized by. In the latter case, the biometric information measuring device 103 is a CPU that executes instructions of a program that is software that realizes each function, a ROM or a storage device in which the program and various data are readablely recorded by a computer (or CPU). These are referred to as "recording media"), and a RAM or the like for developing the above program is provided. As the recording medium, a "non-temporary tangible medium", for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, the program may be supplied to the computer via an arbitrary transmission medium (communication network, broadcast wave, etc.) capable of transmission. Specifically, in the program according to the embodiment of the present disclosure, the computer mounted on the biological information measuring device 103 is equipped with an image acquisition unit 321 and a first spectrum estimation unit 322, a first pulse wave calculation unit 323, and biological information. The acquisition unit 324, the learning unit 325, the storage control unit 326, the second spectrum estimation unit 327, the second pulse wave calculation unit 328, the spectrum difference calculation unit 329, the light source control unit 330, and the prediction unit 331 are realized, respectively. The same applies to the spectrum acquisition unit 902 and the spectrum difference calculation unit 903, and detailed description thereof will be omitted.

100 Biometric information measurement system, 101 Light source, 102 Imaging device, 103 Biometric information measurement device, 104 Subject, 201 frame image, 202 Area of interest, 310 Storage unit, 311 model, 312 Learning spectrum, 320 Control unit, 321 Image acquisition unit 322 1st spectrum estimation unit, 323 1st spectrum wave calculation unit, 324 biological information acquisition unit, 325 learning unit, 326 memory control unit, 327 second spectrum estimation unit, 328 second spectrum wave estimation unit, 329 spectrum difference estimation Unit, 330 Light source control unit, 331 Prediction unit, 901 Biological information measurement device, 902 Spectrum acquisition unit, 903 Spectrum difference calculation unit

Claims (11)

In the learning phase in which the image of the skin taken to train the model is acquired as the image for learning, the storage control unit that stores the spectrum of the first irradiation light applied to the skin, and the storage control unit.
In the prediction phase of predicting biological information including information related to the living body using the model, based on the spectrum of the second irradiation light applied to the skin and the spectrum of the first irradiation light, A light source control unit that controls the light source that irradiates the skin with light,
A prediction unit that predicts the biological information based on the prediction information based on the prediction image obtained by photographing the skin irradiated with the controlled light by emitting the controlled light from the controlled light source and the model. Biometric information measuring device equipped with. The biometric information measuring device according to claim 1, further comprising a first spectrum estimation unit that estimates the spectrum of the first irradiation light based on the color information included in the learning image. The biometric information measuring device according to claim 1 or 2, further comprising a second spectrum estimation unit that estimates the spectrum of the second irradiation light based on the color information included in the prediction image. Further, a calculation unit for calculating the difference between the value based on the color information included in the learning video and the value based on the color information included in the prediction video is provided.
The biometric information measuring device according to any one of claims 1 to 3, wherein the light source control unit controls the light source to keep the difference within a predetermined range. After the light source is controlled, the calculation unit recalculates the difference.
The biometric information measuring device according to claim 4, wherein the light source control unit controls the light source again and puts the recalculated difference within the predetermined range. A learning unit for learning the input / output relationship between the pulse wave calculated from the learning image and the information related to the living body is further provided.
The biometric information measuring device according to any one of claims 1 to 5, wherein the prediction unit predicts the biometric information from a pulse wave calculated from the prediction video based on the input / output relationship. The biometric information measuring device according to claim 1, wherein the light source control unit controls a light source having a wavelength band of three or more. The biometric information measuring apparatus according to any one of claims 1 to 7, further comprising an acquisition unit for acquiring the measured value of the spectrum of the first irradiation light or the measured value of the spectrum of the second irradiation light. .. Biometric information measuring device and
Light source and
Including shooting equipment
The biometric information measuring device is
In the learning phase in which the image of the skin taken to train the model is acquired as the image for learning, the storage control unit that stores the spectrum of the first irradiation light applied to the skin, and the storage control unit.
In the prediction phase of predicting biological information including information related to the living body using the model, the said, based on the spectrum of the second irradiation light applied to the skin and the spectrum of the first irradiation light. A light source control unit that controls the light source that irradiates the skin with light,
A prediction unit that predicts the biological information based on the prediction information based on the prediction image obtained by photographing the skin irradiated with the controlled light by emitting the controlled light from the controlled light source and the model. Biometric information measurement system equipped with. In the learning phase in which the image of the skin taken to train the model is acquired as the image for learning, a memory control step for storing the spectrum of the first irradiation light applied to the skin, and a memory control step.
In the prediction phase of predicting biological information including information related to the living body using the model, the said, based on the spectrum of the second irradiation light applied to the skin and the spectrum of the first irradiation light. A light source control step that controls the light source that irradiates the skin with light,
A prediction step of predicting the biological information based on the prediction information based on the prediction image obtained by photographing the skin irradiated with the controlled light by emitting the controlled light from the controlled light source and the model. Control method of biometric information measuring device including. For biometric information measuring devices
In the learning phase of acquiring the image of the skin taken to train the model as a learning image, a memory control function for storing the spectrum of the first irradiation light applied to the skin, and
In the prediction phase of predicting biological information including information related to the living body using the model, the said, based on the spectrum of the second irradiation light applied to the skin and the spectrum of the first irradiation light. A light source control function that controls the light source that irradiates the skin with light,
A prediction function that predicts the biological information based on the prediction information based on the prediction image obtained by photographing the skin irradiated with the controlled light by emitting the controlled light from the controlled light source and the model. A control program that realizes.

Images