JP7431584B2 - Biological information measuring device, biological information measuring system, operating method of biological information measuring device, and control program - Google Patents

Description

The present disclosure relates to a biological information measuring device, a biological information measuring system, an operating method of the biological information measuring device, and a control program.

Patent Document 1 describes a pulse wave measuring device that measures a pulse wave from an image of the skin. Further, Patent Document 1 describes a pulse wave calculation device that calculates biological information from a measured pulse wave. Here, the pulse wave calculation device described in Patent Document 1 determines the optimal installation position of the pulse wave measuring device based on the direction in which the disturbance light is located and the brightness characteristics of the disturbance light.

JP2018-068431A

By the way, the absorbance of substances in the body (such as hemoglobin) differs depending on the wavelength of light irradiated to the skin. As the absorbance changes, the color of the photographed skin changes. Therefore, when the light irradiating the skin changes, the color of the photographed skin changes. As a result, if the shooting environment changes, the pulse wave and biological information measured from the skin image will change.

In the technique described in Patent Document 1, the position of the pulse wave measuring device is moved depending on the direction in which the disturbance light is located and the brightness characteristics of the disturbance light. Here, in the technique described in Patent Document 1, if the pulse wave measuring device cannot be moved to an appropriate position, there is a risk that the measurement accuracy of biological information will decrease due to a change in the imaging environment.

Therefore, an object of one aspect of the present disclosure is to provide a biological information measuring device and the like that can suppress a decrease in the measurement accuracy of biological information due to changes in the imaging environment, for example.

A biological information measuring device according to an aspect of the present disclosure, in a learning phase in which a skin image photographed for learning a model is acquired as a learning image, the spectrum of the first irradiation light irradiated to the skin is In a prediction phase of predicting biological information including information related to a living body using a storage control unit to store and the model, a spectrum of the second irradiation light irradiated to the skin and the first irradiation light. a light source control unit that controls a light source that irradiates the skin with light based on the spectrum of the light source; and a prediction that the controlled light is emitted from the controlled light source and the skin that is irradiated with the controlled light is photographed. a prediction unit that predicts the biological information based on the prediction information based on the video for use and the model.

A biological information measurement system according to an aspect of the present disclosure includes a biological information measurement device, a light source, and a photographing device, and the biological information measurement device uses a skin image photographed for learning a model. a storage control unit that stores the spectrum of the first irradiation light irradiated onto the skin in a learning phase that is acquired as an image; and a prediction phase that uses the model to predict biological information including information related to the biological body; a light source control unit that controls the light source that irradiates the skin with light based on the spectrum of the second irradiation light irradiated to the skin and the spectrum of the first irradiation light; and the controlled light source. and a prediction unit that predicts the biological information based on the model and prediction information based on a prediction video taken of the skin irradiated with the controlled light.

A method for controlling a biological information measuring device according to an aspect of the present disclosure includes a first irradiation light irradiated to the skin in a learning phase in which an image of skin photographed for learning a model is acquired as a learning image. in the storage control step of storing the spectrum of the second irradiation light irradiated to the skin and the prediction phase of predicting biological information including information related to the living body using the model. a light source control step of controlling a light source that irradiates the skin with light based on the spectrum of the irradiation light; controlled light is emitted from the controlled light source; and the skin irradiated with the controlled light is photographed. and a prediction step of predicting the biological information based on the model and prediction information based on the prediction video obtained.

A control program according to an aspect of the present disclosure causes a biological information measuring device to perform first irradiation applied to the skin in a learning phase in which a skin image photographed in order to learn a model is acquired as a learning image. a storage control function that stores a spectrum of light; and a prediction phase that uses the model to predict biological information including information related to a living body; a light source control function that controls a light source that irradiates the skin with light based on a spectrum of the irradiated light; and a light source control function that controls a light source that irradiates the skin with light, and a controlled light is emitted from the controlled light source, and the skin that is irradiated with the controlled light is controlled. A prediction function for predicting the biological information is realized based on the prediction information based on the captured prediction video and the model.

FIG. 1 is a schematic diagram showing an example of a biological information measurement system according to a first embodiment. FIG. 2 is a diagram showing an example of a video generated by a photographing device. FIG. 1 is a block diagram showing an example of a functional configuration of a biological information measuring device according to a first embodiment. FIG. 3 is a diagram illustrating an example of the relationship between pixel values and light spectra. 12 is a flowchart illustrating an example of a process for estimating the spectrum of light irradiated onto the skin based on an image. It is a flow chart which shows an example of operation of the above-mentioned biological information measuring device in each of a learning phase and a prediction phase. It is a flowchart which shows an example of operation of the above-mentioned biological information measuring device in a prediction phase. It is a graph which shows an example of the result of calculating the adjusted spectral difference. FIG. 2 is a block diagram showing an example of a functional configuration of a biological information measuring device according to a modification of the first embodiment.

[First embodiment]
The first embodiment will be described below based on FIGS. 1 to 9. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and duplicate explanations will be omitted.

(Overall configuration of biological information measurement system 100)
FIG. 1 is a schematic diagram showing an example of a biological information measurement system 100.

The biological information measurement system 100 is a system for measuring biological information of a subject 104 in a non-contact manner. Here, the biological information measured by the biological information measurement system 100 includes, for example, blood pressure, heart rate, and stress level. The biological information measurement system 100 includes, for example, a light source 101, an imaging device 102, and a biological information measurement device 103.

A light source 101 irradiates the skin of a subject (living body) 104 with light. Note that in the following description, an example will be described in which the subject 104 is a human, but the subject 104 may be an animal or the like.

Further, the light source 101 emits light in three wavelength bands or more. The light source 101 emits light having a peak spectrum wavelength within a predetermined range of 600, 500, and 450 nm, for example. In other words, the light source 101 emits red, green, and blue light. The light source 101 includes, for example, an LED (Light Emitting Diode).

Furthermore, the light source 101 includes a mechanism that changes the timing of emitting light for each wavelength band. The biological information measuring device 103 controls the timing and changes the intensity of the light emitted from the light source 101 (hereinafter referred to as "intensity of the light source 101") for each wavelength band.

The photographing device 102 is a device that photographs a region including the skin of the subject 104. The imaging device 102 is sensitive to the wavelength band of light emitted from the light source. That is, the photographing device 102 may be, for example, an RGB camera that is sensitive to wavelength bands of red, green, and blue light. Note that since the light irradiated onto the skin is a composite light of the light emitted from the light source 101 and external light, the image photographed by the photographing device 102 is affected by the composite light.

The biological information measuring device 103 estimates the pulse wave from the change in skin color photographed by the photographing device 102. Here, the pulse wave is a signal indicating a change in the volume of a peripheral blood vessel. The blood pressure and volume of peripheral blood vessels change with the heartbeat, which changes the color of the skin, so the biological information measuring device 103 can estimate the pulse wave from the changes. Details of the method by which the biological information measuring device 103 estimates the pulse wave will be described later.

The biological information measuring device 103 trains a model using a combination of a learning pulse wave estimated from a learning video of the skin and the learning biological information as learning data. Note that in the following description, the stage in which the model is trained will be referred to as a "learning phase." Further, the learning video is a video generated by the biological information measuring device 103 in the learning phase, and the learning pulse wave is a pulse wave estimated from the learning video. Further, the learning biometric information is, for example, biometric information actually measured using a measuring device such as a blood pressure monitor.

When the biological information measuring device 103 trains the model, it estimates the spectrum of the light irradiated onto the skin from the light source (hereinafter referred to as the "learning spectrum"), and stores the spectrum. Note that details of the method by which the biological information measuring device 103 estimates the spectrum of light will be described later with reference to FIG. 7.

Furthermore, the biological information measuring device 103 estimates a predictive pulse wave from a predictive image of the skin, and predicts biological information from the predictive pulse wave based on a learned model. That is, the biological information measuring device 103 measures the biological information of the subject 104 in a non-contact manner by making predictions using a trained model.

Note that in the following description, the stage of predicting biological information using a learned model will be referred to as a "prediction phase." Further, the prediction video is a video generated by the biological information measuring device 103 in the prediction phase, and the prediction pulse wave is a pulse wave estimated from the prediction video.

In the prediction phase as well, the biological information measuring device 103 estimates the spectrum of light irradiated onto the skin from the light source (hereinafter referred to as "prediction spectrum"), similarly to the learning phase. Next, the intensity of the light source 101 is controlled based on the learning spectrum, and the prediction spectrum is corrected to cancel the influence of external light. Thereby, the imaging conditions can be made the same in the learning phase and the prediction phase.

Then, the biological information measurement system 100 irradiates the skin with light having the corrected prediction spectrum to predict the biological information again. Therefore, the biological information measurement system 100 and the biological information measurement device 103 can suppress a decrease in the measurement accuracy of biological information due to a change in the imaging environment.

(Method for acquiring learning videos and prediction videos)
FIG. 2 is a diagram showing an example of a video generated by the imaging device 102. An image generated by the imaging device 102 will be described based on FIG. 2. In addition, in the following description, the case where the imaging device 102 photographs the face of the subject 104 is illustrated and demonstrated.

For example, the photographing device 102 photographs the face of the subject 104 at a predetermined frame rate (for example, 300 fps (frames per second)) for a predetermined period of time (for example, 30 seconds to 60 seconds), and creates a time-series photographed image. generate. The photographed video includes a plurality of frame images 201.

The biological information measurement device 103 extracts regions of interest (so-called ROIs) 202 including skin from frame images 201 included in captured images, and generates images of the extracted regions of interest 202 . For example, if the size of the frame image 201 is 640 pixels in height x 480 pixels in width, the size of the region of interest 202 may be 20 pixels in height x 20 pixels in width.

Note that, as illustrated in FIG. 2, when the image 201 includes a face region, the region of interest 202 may be the forehead region. At this time, the biological information measuring device 103 acquires a time-series video showing the region of interest 202 including the skin of the forehead region as a learning video and a prediction video.

(Functional configuration of biological information measuring device 103)
FIG. 3 is a block diagram showing an example of the functional configuration of the biological information measuring device 103. The biological information measuring device 103 includes, for example, a storage section 310, a control section 320, and the like.

The storage unit 310 stores information necessary for controlling the entire biological information measuring device 103. The storage unit 310 is realized by a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and its details are not limited.

The storage unit 310 stores a model 311, a learning spectrum 312, and the like. The model 311 may be a mathematical model that approximates the input/output relationship between the learning pulse wave and biological information, and may be, for example, a linear model, a support vector machine, a fully connected neural network, a convolutional neural network, or the like.

When the model 311 is a fully connected neural network, its input may be, for example, a pulse wave feature extracted in advance (eg, pulse wave amplitude, etc.). Furthermore, when the model 311 is a convolutional neural network, its input may be pulse wave waveforms corresponding to a plurality of frames, for example.

The control unit 320 controls the entire biological information measuring device 103. The control unit 320 includes, for example, a video acquisition unit 321, a first spectrum estimation unit 322, a first pulse wave calculation unit 323, a biological information acquisition unit 324, a learning unit 325, a storage control unit 326, a second spectrum estimation unit 327, and a first pulse wave calculation unit 323. It includes a two-pulse wave calculation section 328, a spectral difference calculation section (calculation section) 329, a light source control section 330, a prediction section 331, and the like.

The control unit 320 is realized by, for example, a processor such as a CPU (Central Processing Unit). The processor causes each unit included in the control unit 320 to function, for example, by reading and executing a program stored in the storage unit 310.

Note that the first spectrum estimation section 322, first pulse wave calculation section 323, biological information acquisition section 324, and learning section 325 execute processing related to the learning phase. The second spectrum estimation section 327, the second pulse wave calculation section 328, the spectral difference calculation section 329, the light source control section 330, and the prediction section 331 execute processing related to the prediction phase.

The video acquisition unit 321 acquires time-series captured videos. Furthermore, the video acquisition unit 321 extracts a region of interest 202 from each frame image 201 included in the captured video.

The video acquisition unit 321 generates a learning video in the learning phase, and outputs the generated learning video to the first pulse wave calculation unit 323. Furthermore, in the learning phase, the video acquisition unit 321 outputs one frame image included in the learning video to the first spectrum estimating unit 322 as a learning image.

Further, the video acquisition unit 321 generates a prediction video in the prediction phase, and outputs the generated prediction video to the second pulse wave calculation unit 328. Furthermore, in the prediction phase, the video acquisition unit 322 outputs one frame image included in the prediction video to the second spectrum estimation unit 327 as a prediction image.

The first spectrum estimation unit 322 estimates the spectrum of the light (first irradiation light) irradiated onto the skin in the learning phase as the learning spectrum 312 based on the color information included in the learning video. The first spectrum estimation unit 322 causes the storage unit 310 to store the estimated learning spectrum 312. The color information included in the learning image is the pixel value of each pixel included in the learning image. In this embodiment, the pixel value includes the density of each color of R (Red), G (Green), and B (Blue). Note that in the following description, for convenience of explanation, the color information included in the learning video will be referred to as "learning color information."

The first pulse wave calculation unit 323 calculates a learning pulse wave based on the learning color information. The first pulse wave calculation unit 323 may calculate the learning pulse wave based on, for example, time-series changes in pixel values, brightness values, etc. of pixels in a plurality of frames included in the learning video.

The biological information acquisition unit 324 acquires the biological information of the subject 104 as learning biological information in the learning phase.

The learning unit 325 causes the model 311 to learn the input/output relationship between the learning pulse wave and the learning biological information.

The storage control unit 326 stores the learning spectrum 312 in the learning phase. Further, the storage control unit 326 stores the learned model 311 in the storage unit 310 after the learning phase.

The second spectrum estimation unit 327 estimates the spectrum of the light irradiated onto the skin (second irradiation light) in the prediction phase as the prediction spectrum based on the color information included in the prediction video. The color information included in the prediction image is the pixel value of each pixel included in the prediction image. In the following description, for convenience of explanation, the color information included in the prediction video will be referred to as "prediction color information."

The second pulse wave calculation unit 328 calculates a predictive pulse wave from the predictive video.

The spectral difference calculation unit 329 calculates the difference between the value based on the learning color information and the value based on the prediction color information. The value based on the learning color information includes, for example, an average value of pixel values corresponding to a plurality of pixels included in the learning image. Further, the value based on the prediction color information includes, for example, an average value of pixel values corresponding to a plurality of pixels included in the prediction image. In the following description, for convenience of explanation, the difference calculated by the spectral difference calculation unit 329 will be referred to as "adjusted spectral difference." Details of the method for calculating the adjusted spectral difference will be described later.

The light source control unit 330 controls the light source 101 in the prediction phase based on the prediction spectrum and the learning spectrum 312. Specifically, the light source control unit 330 reduces the adjusted spectral difference by controlling the intensity of the light source 101. For example, the light source control unit 330 controls the light source 101 to keep the adjusted spectral difference within a predetermined value range.

The prediction unit 331 predicts biological information based on the prediction information based on the prediction video and the model 311. The prediction information is information estimated from the prediction video, and is, for example, a prediction pulse wave estimated from the prediction video when the skin irradiated with the controlled light is photographed. Note that the controlled light is light emitted from the light source 101 controlled by the light source control unit. Specifically, the controlled light is light emitted from the light source 101 after the light source control unit 330 reduces the adjusted spectrum difference.

Next, the operation of the biological information measuring device 103 will be explained.

Regarding the operation of the biological information measuring device 103, first, a method for estimating the spectrum of light will be described in detail.

(Method for estimating light spectrum)
The relationship between the pixel values of pixels included in an image of skin and the spectrum of light irradiated onto the skin will be explained. FIG. 4 is a diagram illustrating an example of the relationship between pixel values and light spectra.

Regarding the wavelength bands of light corresponding to R, G, and B colors, the spectrum of light emitted from the light source 101, the spectrum of external light, the absorption coefficient of substances related to skin pigment, and the spectral sensitivity of the photographing device 102 are calculated. The multiplied value and the pixel values for each of R, G, and B are correlated.

The spectrum of light emitted from the light source 101 and the intensity of the light source 101 are correlated. Further, the external light is light different from the light emitted from the light source 101 in the skin photographing environment. The external light may be, for example, sunlight. Further, the substance related to skin pigment is a substance related to skin color. Substances related to skin pigment include melanin, hemoglobin, and the like. The absorption coefficient of substances related to skin pigments differs depending on the wavelength band of light irradiated to the skin. Further, the spectral sensitivity of the photographing device 102 is the spectral sensitivity with respect to the wavelength bands of light corresponding to each of R, G, and B colors.

Here, an example will be described in which the spectral sensitivity of the imaging device 102 is constant when the imaging device 102 photographs the skin for a predetermined period of time (for example, 30 seconds to 60 seconds). In that case, the pixel values for each of R, G, and B correlate with the spectrum of light irradiated onto the skin with respect to the wavelength bands of light corresponding to each of the R, G, and B colors. As described above, the light irradiated onto the skin is a composite light of the light emitted from the light source 101 and external light. Therefore, the spectrum of the light irradiated onto the skin, the spectrum of the light emitted from the light source 101, and the spectrum of external light are correlated.

Therefore, in the learning phase, the learning spectrum 312 in the wavelength band of light corresponding to each color of R, G, and B correlates with the pixel values of each of R, G, and B in the learning video. Similarly, in the prediction phase, the prediction spectra in the wavelength bands of light corresponding to each color of R, G, and B are correlated with the pixel values of each of R, G, and B in the prediction video.

As described above, there is a correlation between the pixel values of pixels included in an image of the skin and the spectrum of the light irradiated onto the skin. Therefore, the biological information measuring device 100 estimates the spectrum of light irradiated onto the skin based on the image.

Next, details of the process of estimating the spectrum of light irradiated onto the skin based on an image will be explained. FIG. 5 is a flowchart illustrating an example of a process for estimating the spectrum of light irradiated onto the skin based on an image. Specifically, FIG. 5 is a flowchart showing an example of step S603 (described later) shown in FIG. 6 and an example of the process of step S703 (described later) shown in FIG.

In step S501, the biological information measuring device 103 calculates the average value of the pixel values of a plurality of pixels included in the video for each of R, G, and B. Specifically, in the learning phase, the first spectrum estimation unit 322 calculates the average value of the pixel values of a plurality of pixels included in the learning video for each of R, G, and B. Furthermore, in the prediction phase, the second spectrum estimation unit 327 calculates the average value of the pixel values of a plurality of pixels included in the prediction video for each of R, G, and B.

In step S502, the biological information measuring device 103 normalizes the average value of the pixel values calculated for each of R, G, and B.

In step S503, the biological information measuring device 103 estimates the normalized average value of the pixel values for each of R, G, and B as the spectrum of the light irradiated to the skin. In the learning phase, the first spectrum estimation unit 322 estimates the normalized average value of the pixel values for each of R, G, and B as the learning spectrum 312. In other words, the first spectrum estimation unit 322 estimates a value obtained by normalizing the average value of pixel values of each color included in the learning video as the learning spectrum. Then, the storage control unit 326 stores the learning spectrum 312 in the storage unit 310 in the learning phase. In the prediction phase, the second spectrum estimation unit 327 estimates the normalized average value of the pixel values for each of R, G, and B as the prediction spectrum. In other words, the second spectrum estimation unit 327 estimates a value obtained by normalizing the average value of pixel values of each color included in the prediction video as the prediction spectrum.

Next, the operation of the biological information measuring device 103 in each of the learning phase and the prediction phase will be explained. FIG. 6 is a flowchart showing an example of the operation of the biological information measuring device 103 in each of the learning phase and the prediction phase. Steps S601 to S606 in FIG. 6 show an example of the operation of the biological information measuring device 103 in the learning phase. Steps S611 to S612 in FIG. 6 show an example of the operation of the biological information measuring device 103 in the prediction phase.

(Operation of biological information measuring device 103 in learning phase)
First, the operation of the biological information measuring device 103 in the learning phase will be explained.

In step S601, the light source control unit 330 causes the light source 101 to emit light.

In step S602, the image acquisition unit 321 causes the photographing device 102 to photograph the skin of the subject 104. The photographing device 102 photographs the skin of the subject 104 in response to instructions from the video acquisition unit 321 and generates a photographed video. Here, when the photographing device 102 photographs the skin of the subject 104, the skin is irradiated with light emitted from the light source 101. The video acquisition unit 321 then acquires the learning video. Specifically, the video acquisition unit 321 acquires the video of the region of interest 202 from the captured video as a learning video.

In step S603, the first spectrum estimation unit 322 estimates the learning spectrum 312. Specifically, the video acquisition unit 321 extracts one frame image 201 included in the time-series learning video as a learning image. The learning image may be any frame image 201 in the time-series learning video. The first spectrum estimation unit 322 estimates the learning spectrum 312 based on the learning image.

In step S604, the first pulse wave calculation unit 323 calculates a learning pulse wave based on the learning video. For example, the first pulse wave calculation unit 323 extracts feature amounts from each frame image 201 included in the learning video. The feature amount is information related to the learning color information, and may be, for example, the pixel value of each pixel for each of R, G, and B, the brightness of each pixel, and the like. Then, the first pulse wave calculation unit 323 calculates a signal indicating a temporal change in the feature amount over a predetermined period of time. The first pulse wave calculation unit 323 determines a signal outputted by performing processing such as independent component analysis and a low-pass filter on the signal as a learning pulse wave.

In step S605, the biological information acquisition unit 324 acquires learning biological information. Specifically, the biological information acquisition unit 324 acquires biological information corresponding to the time point when the learning pulse wave was acquired as the learning biological information. For example, the biological information acquisition unit 324 may acquire the learning biological information in synchronization with the time when the learning video is generated. Note that the details of the method for measuring biological information do not matter. For example, a device that measures biological information (such as a blood pressure monitor) may measure biological information (such as blood pressure). Then, the biological information acquisition unit 324 may acquire biological information from the device.

In step S606, the learning unit 325 causes the model 311 to learn the input/output relationship between the learning pulse wave and the learning biological information. Then, the storage control unit 326 stores the learned model 311 in the storage unit 310.

After completing the learning phase, the biological information measuring device 103 adjusts the intensity of the light source 101 in the prediction phase (step S611). Then, the biological information measuring device 103 predicts biological information based on the learned model 311 (step S612).

(Operation of biological information measuring device 103 in prediction phase)
Next, the operation of the biological information measuring device 103 in the prediction phase will be described in detail. FIG. 7 is a flowchart showing an example of the operation of the biological information measuring device 103 in the prediction phase. Steps S701 to S706 shown in FIG. 7 are detailed processing of step S411 shown in FIG. Further, steps S707 to S708 shown in FIG. 7 are detailed processing of step S612 shown in FIG.

In step S701, the light source control unit 330 sets an initial value of the intensity of the light source 101. For example, the initial value may be the same as the intensity of the light source 101 in the learning phase.

The light source control unit 330 then causes the light source 103 to emit light. Then, the photographing device 102 photographs the skin of the subject 104 in response to an instruction from the video acquisition unit 321, and generates a photographed video. Then, the video acquisition unit 321 acquires the prediction video. Specifically, the image acquisition unit 321 acquires an image of the region of interest 202 from the captured image as a prediction image. Here, the imaging environment in the learning phase and the imaging environment in the prediction phase may be different. Further, the intensity of the light source 101 in the learning phase may be different from the intensity of the light source 101 in the prediction phase.

In step S702, the light source control unit 330 determines whether the number of updates of the intensity of the light source 101 is a predetermined upper limit number of times. The number of updates is the number of times the light source control unit 330 updates the intensity of the light source 101 in the prediction phase. When the number of updates of the intensity of the light source 101 is the predetermined upper limit number of times (step S702: Yes), the control transitions to step S703.

In step S703, the second spectrum estimation unit 327 estimates a prediction spectrum based on the prediction video. Specifically, the second spectrum estimation unit 327 executes the processes of steps S601 to S603 shown in FIG. 6 with the prediction video as input.

In step S704, the spectrum difference calculation unit 329 calculates the difference between the learning spectrum 312 and the prediction spectrum as an adjusted spectrum difference. Specifically, the spectrum difference calculation unit 329 calculates the difference between the learning spectrum 312 and the prediction spectrum for each of R, G, and B as the adjusted spectrum difference.

Here, the learning spectrum 312 is a normalized average value of the pixel values of a plurality of pixels included in the learning video for each of R, G, and B. Further, the prediction spectrum is a normalized average value of the pixel values of a plurality of pixels included in the prediction video for each of R, G, and B. Therefore, the spectral difference calculation unit 329 calculates the difference between the normalized average value calculated for the learning video and the normalized average value calculated for the prediction video as an adjusted spectral difference.

In step S705, the spectral difference calculation unit 329 determines whether the adjusted spectral difference is within a predetermined value range. For example, -0.1<=predetermined value range<=0.1.

For example, assume that the spectral difference calculation unit 329 calculates, as the adjusted spectral difference, a value obtained by subtracting the normalized average value calculated for the learning video from the normalized average value calculated for the prediction video. . In that case, if the adjusted spectral difference is larger than the upper limit of the predetermined range, the light source control unit 330 increases the intensity of the light source 101. On the other hand, if the adjusted spectral difference is smaller than the lower limit of the predetermined range, the light source control unit 330 reduces the intensity of the light source 101.

If the adjusted spectral difference is not within the predetermined value range (step S705: No), the light source control unit 330 updates the intensity of the light source 101 (step S706). Specifically, the light source control unit 330 controls the intensity of the light source 101 based on the adjusted spectral difference. For example, the light source control unit 330 may update the intensity of the light source 101 by increasing or decreasing a value corresponding to the adjusted spectrum difference for each of R, G, and B. .

If the light source control unit 330 updates the intensity of the light source, control returns to step S702 and continues the process. That is, after the light source 101 is controlled, the spectral difference calculation unit 329 calculates the adjusted spectral difference again (step S703 and step S704). Then, the light source control unit 330 controls the light source 101 again to keep the adjusted spectral difference calculated again within a predetermined value range. Thereby, the biological information measuring device 103 can reduce the difference between the learning spectrum 312 and the prediction spectrum.

If the adjusted spectral difference is within a predetermined range (step S705: Yes), the second pulse wave calculation unit 328 calculates a predictive pulse wave from the predictive video (step S707).

Specifically, when the adjusted spectral difference is within a predetermined value range, the image acquisition unit 321 causes the photographing device 102 to re-photograph the skin of the subject 104. That is, in a state where the light source 101 emits controlled light, the imaging device 102 re-images the skin of the subject 104 and generates a captured image. Then, the video acquisition unit 321 generates a prediction video based on the captured video. Then, the second pulse wave calculation unit 328 calculates a predictive pulse wave from the generated predictive video. That is, the second pulse wave calculation unit 328 calculates a predictive pulse wave from a predictive image obtained by re-photographing the skin after adjusting the intensity of the light source 101.

For example, the second pulse wave calculation unit 328 extracts feature amounts from each frame image 201 included in the prediction video. The feature amount is related to predictive color information. The feature amount may be a pixel value for each of R, G, and B, for example. Then, the second pulse wave calculation unit 328 generates a time-series signal based on the extracted feature amount. The second pulse wave calculation unit 328 performs processing such as independent component analysis and low-pass filtering on the generated time-series signal, for example. The second pulse wave calculation unit 328 determines the signal outputted by performing the processing as the predictive pulse wave.

In step S708, the prediction unit 331 predicts biological information from the predictive pulse wave based on the learned model 311. For example, the prediction unit 331 calculates a correlation value between a plurality of learning pulse waves included in the learned model 311 and a prediction pulse wave. The prediction unit 331 identifies the learning pulse wave corresponding to the maximum correlation value among the calculated correlation values. The prediction unit 331 may then predict the biological information associated with the identified learning pulse wave as the biological information corresponding to the predictive pulse wave.

Note that in the prediction phase, biometric information may be predicted multiple times in succession. In that case, the light source control unit 330 may control the light source 101 based on the intensity of the light source 101 that is set for the first time. Alternatively, the spectral difference calculation unit 329 may calculate the adjusted spectral difference each time before the prediction unit 331 predicts the biological information. The light source control unit 330 may control the light source 101 each time based on the calculated adjusted spectrum difference.

(Calculation result of adjusted spectral difference)
Next, an example of the result of calculating the adjusted spectral difference will be described. FIG. 8 is a graph showing an example of the result of calculating the adjusted spectral difference. The vertical axis indicates the adjusted spectral difference calculated by the spectral difference calculation unit 329. The horizontal axis indicates the number of updates. Here, before the spectral difference calculation unit 329 calculates the adjusted spectral difference illustrated in FIG. It shall be assumed that As illustrated in FIG. 8, the adjusted spectral difference becomes smaller each time the intensity of the light source 101 is changed.

Note that due to the wavelength dependence of the absorbance of hemoglobin, among the lights corresponding to the R, G, and B colors, light in the wavelength band corresponding to each G color has the greatest influence on the shape of the pulse wave. Therefore, the light source control unit 330 may control the intensity of light corresponding to each G color with priority over the intensity of light corresponding to each color R and B. Thereby, the biological information measuring device 103 can reduce the number of times the intensity of the light source 101 is updated than when controlling the intensity of light corresponding to each color of R, G, and B using the same priority order.

Further, the light source control unit 330 sets the amount of change according to the adjusted spectral difference corresponding to the G color to be larger than the amount of change according to the adjusted spectral difference corresponding to each color of R and B, and controls the light source 101. May be controlled. As a result, the biological information measuring device 103 increases the amount of change in the spectrum at the time of prediction than when controlling the intensity of light corresponding to each color of R, G, and B using the same amount of change for each color. can.

(effect)
As described above, the biological information measurement system 100 can reduce the adjusted spectral difference by adjusting the intensity of the light source 101. In other words, the biological information measurement system 100 can bring the prediction spectrum closer to the learning spectrum by adjusting the intensity of the light source 101. As a result, by adjusting the intensity of the light source 101, the biological information measurement system 100 can bring the imaging environment at the time of prediction closer to the imaging environment at the time of learning. Thereby, when the biological information measurement system 100 predicts biological information from a pulse wave based on an image of the skin, it is possible to suppress a decrease in prediction accuracy due to a change in the photographing environment. As a result, the biological information measurement system 100 can improve the prediction accuracy of biological information when the photographing environment at the time of learning and the photographing environment at the time of prediction are different.

(Modification 1)
As a first modification of the biological information measuring device 103, a biological information measuring device 901 will be described. FIG. 9 is a block diagram showing an example of the functional configuration of the biological information measuring device 901. The biological information measuring device 901 includes a spectrum acquisition section (acquisition section) 902 instead of the first spectrum estimation section 322 and the second spectrum estimation section 327. Furthermore, the biological information measuring device 901 includes a spectral difference calculation section 903 instead of the spectral difference calculation section 329.

The spectrum acquisition unit 902 acquires the measured value of the spectrum of the light irradiated onto the skin. The measured value may be, for example, a spectrum of light irradiated to the skin measured by a spectrometer. Specifically, the spectrum acquisition unit 902 acquires the measured value of the learning spectrum in the learning phase. The storage control unit 326 causes the storage unit 310 to store the measured value of the learning spectrum 312. In addition, the spectrum acquisition unit 902 acquires the measured value of the prediction spectrum in the prediction phase. In other words, the spectrum acquisition unit 902 acquires the measured value of the learning spectrum or the measured value of the prediction spectrum.

The spectral difference calculation unit 903 calculates the difference between the measured value of the learning spectrum and the measured value of the prediction spectrum as an adjusted spectral difference. Therefore, the biological information measuring device 901 can calculate the adjusted spectral difference even more accurately. Therefore, the biological information measuring device 901 contributes to reducing the number of updates of the intensity of the light source 101 more than the biological information measuring device 103.

Note that the light source 101 has been described as emitting light in three wavelength bands or more, for example, in which the wavelength of the peak spectrum is within a predetermined range of 600, 500, and 450 nm, but the present invention is not limited thereto. The light source 101 may emit near-infrared light in addition to red, green, and blue light. In that case, the imaging device 102 may be, for example, an RGB+IR camera that is sensitive to wavelength bands of red, green, blue, and near-infrared light. Furthermore, the photographing device 102 may be configured with a combination of two cameras: an RGB camera and an IR camera.

(Additional notes)
The above embodiment may be described as the following form, but is not limited to the following form.

(Form 1) In a learning phase in which a skin image photographed to train a model is acquired as a learning image, a storage control unit stores a spectrum of the first irradiation light irradiated to the skin, and the model In the prediction phase of predicting biological information including information related to a living body using a light source control unit that controls a light source that irradiates light; a light source control unit that controls a light source that emits light from the controlled light source; and prediction information based on a predictive image taken of the skin that is irradiated with the controlled light; and the model. A prediction unit that predicts the biological information based on the biological information measuring device.

(Form 2) The biological information measuring device according to Form 1, further comprising a first spectrum estimation unit that estimates a spectrum of the first irradiation light based on color information included in the learning video.

(Form 3) The biological information measuring device according to Form 1 or 2, further comprising a second spectrum estimation unit that estimates a spectrum of the second irradiation light based on color information included in the prediction video.

(Form 4) The light source control unit further includes a calculation unit that calculates a difference between a value based on color information included in the learning video and a value based on color information included in the prediction video, and the light source control unit The biological information measuring device according to any one of modes 1 to 3, wherein the biological information measuring device controls the difference to keep the difference within a predetermined value range.

(Form 5) The calculation unit recalculates the difference after the light source is controlled, and the light source control unit controls the light source again to bring the recalculated difference into the predetermined value range. The biological information measuring device according to Form 4.

(Form 6) The learning unit further includes a learning unit that causes the model to learn an input-output relationship between the pulse wave calculated from the learning video and information related to the living body, and the prediction unit The biological information measuring device according to any one of aspects 1 to 5, wherein the biological information is predicted from a pulse wave calculated from the prediction video based on the prediction video.

(Embodiment 7) The biological information measuring device according to any one of embodiments 1 to 6, wherein the light source control unit controls a light source of three or more wavelength bands.

(Form 8) The living body according to any one of Modes 1 to 7, further comprising an acquisition unit that acquires a measured value of the spectrum of the first irradiation light or a measured value of the spectrum of the second irradiation light. Information measurement device.

(Mode 9) The biological information measuring device includes a biological information measuring device, a light source, and a photographing device, and the biological information measuring device acquires a skin image photographed as a learning video in order to train a model. a storage control unit that stores the spectrum of the first irradiation light irradiated onto the skin; and a second irradiation light irradiated onto the skin in a prediction phase that uses the model to predict biological information including information related to the living body. a light source control unit that controls the light source that irradiates the skin with light based on the spectrum of the irradiation light and the spectrum of the first irradiation light; A biological information measurement system comprising: a prediction unit that predicts the biological information based on the model and predictive information based on a predictive image taken of the skin irradiated with controlled light.

(Form 10) In the learning phase of acquiring a skin image photographed to train the model as a learning image, a storage control step of storing the spectrum of the first irradiation light irradiated to the skin, and the model In the prediction phase of predicting biological information including information related to a living body using a light source control step of controlling a light source that irradiates light, a controlled light is emitted from the controlled light source, and prediction information based on a predictive image of the skin irradiated with the controlled light and the model; a prediction step of predicting the biological information based on the biological information measuring device.

(Form 11) Storage for storing the spectrum of the first irradiation light irradiated on the skin in the learning phase in which the biological information measuring device acquires a skin image photographed as a learning image in order to train the model. In the control function and the prediction phase of predicting biological information including information related to the biological body using the model, the spectrum of the second irradiation light irradiated to the skin and the spectrum of the first irradiation light a light source control function that controls a light source that irradiates the skin with light based on the control function; and a predictive image of the controlled light emitted from the controlled light source and the skin irradiated with the controlled light. A control program that realizes a prediction function of predicting the biological information based on prediction information and the model.

The present disclosure is not limited to the above-described embodiments, and is replaced with a structure that is substantially the same as the structure shown in the above-described embodiment, a structure that has the same effect, or a structure that can achieve the same purpose. You can.

[Example of implementation using software]
The control block (in particular, the control unit 320) of the biological information measuring device 103 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC (Integrated Circuit) chip) or by software using a CPU. It can be realized by In the latter case, the biological information measuring device 103 includes a CPU that executes instructions of a program that is software that implements each function, and a ROM or storage device (ROM or storage device) in which the programs and various data are recorded so as to be readable by the computer (or CPU). These are referred to as "recording media"), RAM, etc. for developing the above programs. As the recording medium, a "non-temporary tangible medium" such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, etc. can be used. Further, the program may be supplied to the computer via any transmittable transmission medium (communication network, broadcast waves, etc.). Specifically, the program according to the embodiment of the present disclosure includes an image acquisition unit 321, a first spectrum estimation unit 322, a first pulse wave calculation unit 323, and biological information in a computer installed in the biological information measurement device 103. The acquisition unit 324, the learning unit 325, the storage control unit 326, the second spectrum estimation unit 327, the second pulse wave calculation unit 328, the spectrum difference calculation unit 329, the light source control unit 330, and the prediction unit 331 are realized, respectively. Note that the same applies to the spectrum acquisition section 902 and the spectrum difference calculation section 903, and detailed description thereof will be omitted.

Reference Signs List 100 biological information measurement system, 101 light source, 102 imaging device, 103 biological information measurement device, 104 subject, 201 frame image, 202 region of interest, 310 storage unit, 311 model, 312 learning spectrum, 320 control unit, 321 image acquisition unit , 322 first spectrum estimation section, 323 first pulse wave calculation section, 324 biological information acquisition section, 325 learning section, 326 storage control section, 327 second spectrum estimation section, 328 second pulse wave estimation section, 329 spectral difference estimation section, 330 light source control section, 331 prediction section, 901 biological information measuring device, 902 spectrum acquisition section, 903 spectral difference calculation section

Claims (10)

a storage control unit that stores the spectrum of the first irradiation light that is irradiated onto the skin in a learning phase that acquires an image of the skin photographed to train the model as a learning image;
In a prediction phase of predicting biological information including information related to a living body using the model, a spectrum of the first irradiation light and a spectrum of a second irradiation light irradiated to the skin in the prediction phase are determined. a spectral difference calculation unit that calculates the difference between the two as an adjusted spectral difference;
In the prediction phase, a light source control unit that controls a light source that emits light to irradiate the skin so that the adjusted spectral difference falls within a predetermined value range;
Controlled light is emitted from the controlled light source, and a prediction unit that predicts the biological information based on the model and prediction information based on a prediction video taken of the skin irradiated with the controlled light; A biological information measuring device equipped with The biological information measuring device according to claim 1, further comprising a first spectrum estimator that estimates a spectrum of the first irradiation light based on color information included in the learning video. The biological information measuring device according to claim 1 or 2, further comprising a second spectrum estimator that estimates a spectrum of the second irradiation light based on color information included in the prediction video. The calculation unit calculates the adjusted spectral difference again after the light source is controlled,
The biological information measuring device according to any one of claims 1 to 3, wherein the light source control unit controls the light source again to keep the recalculated adjusted spectral difference within the predetermined value range. further comprising a learning unit that causes the model to learn an input/output relationship between the pulse wave calculated from the learning video and information related to the living body,
The biological information measuring device according to any one of claims 1 to 4, wherein the prediction unit predicts the biological information from a pulse wave calculated from the prediction video based on the input/output relationship. The biological information measuring device according to any one of claims 1 to 5, wherein the light source control section controls a light source of three wavelength bands or more. The biological information measuring device according to any one of claims 1 to 6, further comprising an acquisition unit that acquires a measured value of the spectrum of the first irradiation light or a measured value of the spectrum of the second irradiation light. . A biological information measuring device,
a light source and
including a photographic device;
The biological information measuring device includes:
a storage control unit that stores the spectrum of the first irradiation light that is irradiated onto the skin in a learning phase that acquires an image of the skin photographed to train the model as a learning image;
In a prediction phase of predicting biological information including information related to a living body using the model, a spectrum of the first irradiation light and a spectrum of a second irradiation light irradiated to the skin in the prediction phase are determined. a spectral difference calculation unit that calculates the difference between the two as an adjusted spectral difference;
In the prediction phase, a light source control unit that controls the light source that emits light to irradiate the skin so that the adjusted spectral difference falls within a predetermined value range;
Controlled light is emitted from the controlled light source, and a prediction unit that predicts the biological information based on the model and prediction information based on a prediction video taken of the skin irradiated with the controlled light; A biological information measurement system equipped with A method for operating a biological information measuring device, the method comprising:
a storage control step of storing a spectrum of the first irradiation light irradiated to the skin in a learning phase in which the biological information measuring device acquires an image of the skin photographed as a learning image in order to train the model; ,
In a prediction phase in which the biological information measuring device predicts biological information including information related to a living body using the model, the spectrum of the first irradiation light and the first irradiation light irradiated on the skin in the prediction phase are measured. a step of calculating a difference from the spectrum of the irradiated light of No. 2 as an adjusted spectral difference;
a light source control step in which the biological information measuring device controls a light source that emits light to irradiate the skin so as to keep the adjusted spectral difference within a predetermined value range in the prediction phase;
The biological information measuring device is configured to emit controlled light from the controlled light source and calculate the biological information based on the prediction information based on a predictive image of the skin irradiated with the controlled light and the model. A method of operating a biological information measuring device including a prediction step of predicting information. For biological information measuring devices,
a storage control function that stores the spectrum of the first irradiation light irradiated to the skin in a learning phase in which a skin image photographed to train the model is acquired as a training video;
In a prediction phase of predicting biological information including information related to a living body using the model, a spectrum of the first irradiation light and a spectrum of a second irradiation light irradiated to the skin in the prediction phase are determined. a spectral difference calculation function that calculates the difference between the two as an adjusted spectral difference;
In the prediction phase, a light source control function that controls a light source that emits light to irradiate the skin so that the adjusted spectral difference falls within a predetermined value range;
Controlled light is emitted from the controlled light source, and a prediction function that predicts the biological information based on the model and prediction information based on a prediction video taken of the skin irradiated with the controlled light; A control program that realizes this.

Images