JP7298728B2 - System, method and program - Google Patents

Description

The present disclosure relates to a psychosomatic state estimation device, a psychosomatic state estimation method, and a program.

A method for evaluating a person's stress level is disclosed in Patent Document 1, for example. As described in Patent Literature 1, the stress level of a person is obtained by measuring information (for example, pulse rate (heart rate), etc.) that enables evaluation of the stress level in a resting state.

However, maintaining a resting state imposes a burden on the user. Patent Literature 2 discloses stably detecting fatigue of a user from a fatigue detection device attached to the user even when the user cannot maintain a resting state.

Further, Patent Documents 3 and 4 disclose a method of measuring a user's pulse waveform without the user's awareness.

Japanese Patent No. 3054708 WO2016/088819 WO2017/085894 WO2017/085896

The technique described in Patent Document 2 requires the user to wear the device. Therefore, wearing such a device causes annoyance to the user (person to be measured).

Further, in the techniques of Patent Literatures 3 and 4, pulse waveforms for determining whether or not there is an arrhythmia are obtained from multiple sites of the subject. Therefore, when a plurality of sites are not imaged by a single imaging device, it may not be possible to determine whether or not the person to be measured has arrhythmia for which the plurality of sites are not imaged. In order to accurately determine whether or not there is an arrhythmia, it is necessary to ask the person to be measured to take images of a plurality of sites, which may impose a burden on the person to be measured.

The present disclosure has been made in view of the above problems, and its purpose is to provide a technique for estimating the subject's mental and physical conditions, such as stress, with high accuracy without imposing a burden on the subject. be.

A psychosomatic state estimation apparatus according to an aspect of the present disclosure includes a first estimation method using a plurality of luminance values of a first region including a part of a face among a plurality of time-series face images. and the second pulse rate in the first period estimated by a second estimation method different from the first estimation method, a first a calculating means for calculating a similarity; and a subject using at least one of the first pulse rate, the second pulse rate, and a plurality of face images in the first period according to the calculated first similarity. an estimating means for estimating the psychosomatic state of the measurer.

In addition, a psychosomatic state estimation method according to an aspect of the present disclosure provides a first region including a face part of a plurality of time-series face images, a first region using a plurality of luminance values of the first region. A degree of similarity between a first pulse rate in a first period estimated by an estimation method and a second pulse rate in the first period estimated by a second estimation method different from the first estimation method. calculating a first similarity, and measuring using at least one of the first pulse rate, the second pulse rate, and a plurality of facial images in the first period according to the calculated first similarity Estimate the psychosomatic state of a person.

A computer program for realizing each of the above devices or methods by a computer, and a computer-readable non-transitory recording medium storing the computer program are also included in the scope of the present disclosure.

Advantageous Effects of Invention According to the present disclosure, it is possible to accurately estimate the subject's mental and physical conditions such as stress without imposing a burden on the subject.

1 is a block diagram showing an example of the configuration of a psychosomatic state estimation device according to a first embodiment; FIG. FIG. 5 is a diagram showing an example of a pulse estimation result acquired by an acquisition unit; It is a figure which shows an example of the calculation result of similarity. FIG. 10 is a diagram showing an example of an estimation result of a subject's psychosomatic state stored in a storage unit; 4 is a flow chart showing an example of the operation flow of the psychosomatic state estimation device according to the first embodiment; FIG. 11 is a block diagram showing an example of a configuration of a psychosomatic state estimation device according to a second embodiment; FIG. 9 is a flow chart showing an example of the operation flow of the psychosomatic state estimation device according to the second embodiment. 1 is a diagram exemplifying a hardware configuration of a computer (information processing device) capable of implementing each embodiment; FIG.

<First Embodiment>
A first embodiment of the present disclosure will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of a psychosomatic state estimation device 100 according to this embodiment. As shown in FIG. 1 , the psychosomatic state estimation device 100 includes an acquisition unit 110 , a calculation unit 120 , an estimation unit 130 , an output unit 140 and a storage unit 150 .

Acquisition unit 110 acquires a pulse estimation result. The pulse estimation result is information indicating the pulse (heartbeat) estimated from a plurality of time-series face images (time-series face images). In this embodiment, the pulse estimation result is described as an estimated pulse rate (heart rate), but it may be a pulse waveform.

A face image is an image of a person to be measured during daily life such as at work or while driving. A face image is obtained by being imaged by an imaging device equipped with an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging device is equipped with a plurality of light receiving elements such as R (Red), G (Green), and B (Blue), for example.

A plurality of time-series facial images are obtained by capturing temporally continuous facial images with this imaging device. The time-series facial images may be still images or moving images. The imaging device may be a digital camera, a web camera, or a mobile terminal equipped with a camera.

Here, an example of a method of estimating the pulse rate from a plurality of time-series face images captured by an imaging device will be described. The pulse rate may be estimated by the psychosomatic state estimation device 100 or by a device different from the psychosomatic state estimation device 100 . In the following description, it is assumed that a pulse rate estimating device provided inside or outside the psychosomatic state estimating device 100 estimates the pulse rate.

A pulse estimation device detects the position and size of a person's face from a plurality of time-series face images, and extracts a face region containing the face. Any face detection technique that detects the position and size of a target face is used for face detection. The pulse estimation device detects feature points from the extracted image of the face area. Feature points to be detected are, for example, the eyes, nose, mouth, and contour of the face. An arbitrary feature point detection algorithm is used to detect the feature points. Then, the pulse estimation device determines a region surrounding the detected feature point as a region of interest (ROI). Here, one or more regions of interest may be determined.

After that, the pulse estimating device estimates the pulse rate in each determined region of interest using the R, G, and B luminance values (gradation values of luminance) in each pixel of the image of the region of interest. . The pulse estimation device estimates the pulse rate for one region of interest (first region) using at least two estimation methods.

For example, the pulse estimation device estimates the pulse rate in a first period for one region of interest by a method using a plurality of luminance values in the region of interest (first estimation method), and for the same region of interest Therefore, the pulse rate in the first period may be estimated by a second estimation method different from the first estimation method.

As a first estimation method, the pulse estimating device uses, for example, a pulse rate calculated by emphasizing the luminance value of the green component among the plurality of luminance values more than the luminance values of the other components. You may output as a pulse rate. Hemoglobin contained in blood has the property of absorbing green light. Therefore, among the plurality of color components, it is highly likely that changes caused by blood flow appear most in the green component. Therefore, the inventors found that the estimation accuracy of the pulse rate obtained by emphasizing the luminance value of the green component more than the luminance values of the other components among the luminance values of the components of a plurality of colors is high. acquisition unit 110 acquires the estimated pulse rate as the first pulse rate. Thereby, the obtaining unit 110 can obtain a pulse rate with high accuracy.

The first period is an imaging period of a plurality of time-series face images used when the pulse estimating device estimates the first pulse rate. The pulse estimator estimates the pulse rate (second pulse rate) by the second estimation method using an image of the same region of interest in the same period (first period) as the first pulse rate. The second estimation method may be a method different from the first estimation method, and may be a method of estimating the pulse using brightness values of a plurality of colors. A method of estimating the pulse using only the luminance value may be used.

FIG. 2 is a diagram showing an example of the pulse estimation result acquired by the acquisition unit 110. As shown in FIG. In FIG. 2, the pulse estimation result is represented graphically. In FIG. 2, the horizontal axis indicates time, and the vertical axis indicates pulse rate (bpm: beats per minute). FIG. 2 shows a first pulse rate and a second pulse rate for one region of interest. In FIG. 2, the first pulse rate is represented by circles with symbols b10 to b16, and the second pulse rates are represented by diamonds with symbols b20 to b26. Here, for example, the first pulse rate b11 at time t1 indicates the pulse rate estimated using a plurality of face images where the time T at which the face image was captured satisfies t0<T<=t1. Although the first pulse rate and the second pulse rate shown in FIG. 2 are estimated at equal time intervals, they may be estimated at arbitrary intervals.

The first period described above may have a predetermined length, or may be variable. Also, one or more pulse rate is sufficient for the first period. That is, in FIG. 2, the first period may be, for example, t0<T<=t1, which is the period in which the face image used for estimating the first pulse rate b11 is captured, or the first pulse rate b11 may be It may be t0<T<=t3, which is the period in which the face images used to estimate the numbers b11, b12, and b13 were captured.

In this way, the acquiring unit 110 acquires the first pulse rate and the second pulse rate for each region of interest estimated for each arbitrary period as a pulse estimation result together with information indicating the region of interest. Then, the acquisition unit 110 supplies the acquired pulse estimation result to the calculation unit 120 .

The calculator 120 calculates a similarity (first similarity) between the first pulse rate and the second pulse rate. Based on the pulse estimation result supplied from the acquisition unit 110, the calculation unit 120 calculates the degree of similarity between the first pulse rate and the second pulse rate in an arbitrary period for each region of interest.

For example, when the first period is t0<T<=t1, the calculator 120 calculates the degree of similarity between the first pulse rate b11 and the second pulse rate b21 shown in FIG. The degree of similarity calculated by the calculation unit 120 may represent the closeness between pulse rates using any method.

For example, when the first period is t0<T<=t3, the calculator 120 calculates the closeness between the first pulse rate b11 and the second pulse rate b21 shown in FIG. The closeness to the pulse rate b22 and the closeness between the first pulse rate b13 and the second pulse rate b23 are respectively calculated, and the calculated statistic of the plurality of similarities (for example, average value) is calculated as the degree of similarity. You may Further, the calculator 120 may calculate the distance between the graph represented by the first pulse rates b11 to b13 and the graph represented by the second pulse rates b21 to b23 as the degree of similarity.

Further, when the acquisition unit 110 acquires a pulse waveform as the pulse estimation result, the calculation unit 120 may calculate the degree of similarity between the pulse waveforms. The method of calculating the degree of similarity between waveforms is not particularly limited, and the degree of similarity of the entire shape may be used as the degree of similarity. may be used as the degree of similarity. Note that the method by which the calculation unit 120 calculates the degree of similarity from the pulse waveform is not limited to this.

Note that when the pulse rate estimated for a certain period and a certain region of interest is 3 or more (that is, when the pulse estimation result includes a third pulse rate different from the first pulse rate and the second pulse rate), The calculation unit 120 may calculate all similarities, or may calculate similarities between combined pulse rates. It should be noted that the combination of pulse rates may be all combinations that can be created, or an arbitrary number of combinations.

The calculation unit 120 calculates the calculated similarity as information indicating the first pulse rate and the second pulse rate for which the similarity is to be calculated, and the imaging period of the face image for which the first pulse rate and the second pulse rate are to be estimated. and information indicating the region of interest, and supplied to the estimation unit 130 . FIG. 3 is a diagram showing an example of a similarity calculation result supplied to the estimation unit 130 by the calculation unit 120. As shown in FIG. As shown in FIG. 3, the similarity calculation result 31 includes a region of interest 32 that is information indicating a region of interest, a period 33 that indicates a face image imaging period for which the first pulse rate and the second pulse rate are estimated, A first pulse rate 34 estimated for the period 33, a second pulse rate 35 estimated for the period 33, and a similarity 36 indicating the similarity between the first pulse rate 34 and the second pulse rate 35. include. Note that the similarity calculation result 31 is not limited to this.

The estimation unit 130 calculates (a) the first pulse rate 34, (b) the second pulse rate 35, and (c) a plurality of time-series using at least one of the facial images in period 33 of the facial images in .

For example, when the similarity 36 is equal to or greater than a predetermined threshold, the estimating unit 130 detects (a) the first pulse rate 34, (b) the second pulse rate 35, and (c) the face in the period 33 associated with the similarity 36. At least one of the images is used to estimate the psychosomatic state. In this case, if the similarity 36 is lower than the predetermined threshold, the estimating unit 130 determines that at least one of the estimated first pulse rate and second pulse rate contains a lot of noise, and It is not necessary to estimate the psychosomatic state. As a result, the psychosomatic state can be estimated using the pulse rate during a period with little noise, so the accuracy of the estimation result can be improved.

Here, the estimation of the psychosomatic state performed by the estimation unit 130 will be described. In normal cases, heart rate and pulse rate correspond. Therefore, the heart rate can be estimated from the estimated pulse rate. Therefore, the estimation unit 130 calculates an RR interval (RRI: RR Interval), which is an interval between R waves, from the estimated heart rate. If the calculated RRIs are not evenly spaced in time, processing such as cubic spline interpolation may be performed.

Then, using the calculated RRI, the estimating unit 130 performs, for example, (A) estimation targeting the time domain and (B) estimation targeting the frequency domain, thereby Estimate the state.

Estimation unit 130, for example, (A) as an estimation targeting the time domain, from the RRI, for example, calculates the following (1) to (6), etc., based on the calculated results, estimates the psychosomatic state may

(1) ECG RR interval variation coefficient (CVRR: Coefficient of Variation of RR intervals)
(2) RRI median (3) RRI moving average (4) RRI change rate (5) RRI time variation (6) Lorenz plot For example, (3) RRI moving average is calculated with a window of about 30 seconds is. In addition, (4) the RRI rate of change is obtained in advance by obtaining the RRI average value, median value, standard deviation, variance, etc. for each of the stress state and the relaxed state, and calculating the difference between the relaxed state and the stress state as the rate of change. It is what I did. Also, (5) the amount of RRI time variation is obtained by calculating the variance and standard deviation of the RRI, for example, with a window of one minute.

For example: (1) CVRR tends to be low under high stress conditions. Similarly, the values of (2) to (5) also tend to be low when the stress is high. Also, under high stress conditions, (6) the area obtained by the Lorenz plot tends to be smaller. Thus, the estimating section 130 estimates the psychosomatic state of the subject according to the characteristics (1) to (6).

Note that the estimating unit 130 may calculate the time variation of the heart rate as (A) estimation targeting the time domain, and estimate the psychosomatic state based on the calculated value. For example, the estimating unit 130 may estimate the psychosomatic state from the time variation of the heart rate based on the tendency that the time variation of the heart rate increases in a state of high stress.

Estimating section 130 also transforms the RRI into the frequency domain by Fourier transforming it. Then, for example, as (B) estimation targeting the frequency domain, the estimation unit 130 extracts a heart rate variability low frequency band component (LF) and a heart rate variability high frequency band component (LF) from the RRI transformed into the frequency domain. HF: High frequency component) may be extracted, and the subject's psychosomatic state may be estimated based on LF and HF.

In general, LF is determined by the integrated value of the spectrum in the range of 0.04 to 0.15 Hz, and HF is determined by the integrated value of the spectrum in the range of 0.15 to 0.4 Hz. A relatively large HF indicates that the parasympathetic nerves are dominant, and a relatively small HF indicates that the sympathetic nerves are dominant. Therefore, the estimating section 130 may estimate the subject's psychosomatic state by estimating the state of relaxation or stress using the HF. In addition, the estimation unit 130 may calculate LF/HF and estimate the psychosomatic state based on the calculated value. The estimation unit 130 may also calculate the HF ratio (HFR: HF Ratio) using the following formula (1), and estimate the psychosomatic state based on the calculated HFR.

HFR=HF/(LF+HF) (1)
This HFR tends to have a low value in a stressful state and a high value in a relaxed state.

Note that the method by which the estimation unit 130 estimates the physical and mental condition of the subject using the heart rate (the first pulse rate and/or the second pulse rate) and the RRI is not limited to this. The estimation unit 130 may employ any method to estimate the psychosomatic state.

Note that the estimation unit 130 may estimate the psychosomatic state using the similarity 36, for example. For example, the degree of similarity 36 in a certain period indicates that the reliability of the first pulse rate and the second pulse rate in that period is high. Therefore, the estimation unit 130 estimates the psychosomatic state by the above-described estimation method for each period, and quantifies the estimated psychosomatic state. After that, the estimation unit 130 accumulates the value obtained by multiplying the quantified psychosomatic state by the similarity 36 in the storage unit 150 or the like, which will be described later. The estimating unit 130 uses the accumulated results to further estimate the psychosomatic state of the accumulated period, thereby estimating the psychosomatic state with higher accuracy than the estimation using only the data with the similarity 36 higher than the predetermined threshold. can do. Note that the estimation unit 130 may supplement the psychosomatic state estimated for a period in which the degree of similarity 36 is smaller than a predetermined threshold, according to estimation results before and after that period.

Also, the estimation unit 130 may change the method of estimating the psychosomatic state according to the similarity 36, for example. For example, if the similarity 36 is less than a predetermined threshold, the estimation unit 130 performs (A) estimation targeting the time domain, and if the similarity 36 is greater than or equal to the predetermined threshold, (B) the frequency domain. You may make an estimation based on

LF/HF and the like are generally calculated from an electrocardiogram that can measure heartbeats with high accuracy. A high degree of similarity 36 between the first and second pulse rates obtained from the face image indicates that the estimated first and second pulse rates are highly reliable, as described above. Therefore, the estimating section 130 can accurately estimate the psychosomatic state of the subject by using highly reliable data.

In addition, even when the degree of similarity 36 is small (reliability is low), a numerical value within a predetermined range is calculated for the average value of the pulse rate and the like. Therefore, even if the degree of similarity 36 is smaller than the predetermined threshold, (A) the estimation unit 130 can estimate the psychosomatic state of the subject over many periods by performing estimation targeting the time domain. can do.

Note that the first pulse rate, the second pulse rate, or the face image used by the estimation unit 130 to estimate the psychosomatic state may be a region different from the region of interest used to calculate the degree of similarity. For example, if the similarity 36 is the similarity 3 of the first pulse rate and the second pulse rate estimated for the region of interest on the forehead, the estimating unit 130 calculates the first pulse rate for the region of interest other than the forehead and the entire face image. The physical and mental condition of the subject may be estimated using the first pulse rate, the second pulse rate, or the facial image itself.

Also, the first pulse rate and the second pulse rate used by the estimation unit 130 to estimate the psychosomatic state may be pulse rates related to a plurality of regions of interest. For example, if the regions of interest whose similarity 36 is higher than a predetermined threshold are the “forehead”, the “right cheek” and the “left cheek”, the estimator 130 calculates the first pulse rate and the second pulse rate associated with each region of interest. At least one of the numbers may be used to estimate the psychosomatic state.

The estimation unit 130 supplies the estimation result of the subject's psychosomatic state to the output unit 140 .

The output unit 140 outputs the estimation result supplied from the estimation unit 130 . The output unit 140 may store the estimation result in a storage device such as the storage unit 150, or may output the result to a display device, a printer, or the like.

The storage unit 150 stores the estimation result of the subject's psychosomatic state. The storage unit 150 may be realized by a storage device different from the psychosomatic state estimation device 100 . FIG. 4 is a diagram showing an example of the estimation result of the subject's psychosomatic state stored in the storage unit 150. As shown in FIG. As shown in FIG. 4 , the estimation result 41 includes a period 42 indicating the period during which the face image was captured and the psychosomatic state 43 indicating the estimated psychosomatic state. Note that the estimation result 41 may also include other information such as information indicating the region of interest.

By accumulating the estimation results 41 accumulated in the storage unit 150 in this way, for example, for a predetermined period of time, it is possible to collect data used for analyzing changes in the subject's mental and physical conditions. As a result, the estimation results 41 accumulated in the storage unit 150 can be effectively used when the subject's psychosomatic condition is observed on a weekly basis, a monthly basis, or the like.

Next, the operation flow of psychosomatic state estimation apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 5 is a flowchart showing an example of the operation flow of psychosomatic state estimation apparatus 100 according to the present embodiment.

As shown in FIG. 5, the obtaining unit 110 obtains a pulse estimation result including at least the first pulse rate and the second pulse rate (step S51). The first pulse rate is estimated by a first estimation method using a plurality of luminance values of the region of interest for each region of interest (first region) including facial parts in a plurality of time-series face images. is the pulse rate. The second pulse rate is the pulse rate estimated by a second estimation method different from the first estimation method for each region of interest for which the first pulse rate has been estimated. The first pulse rate and the second pulse rate are pulse rates estimated for the same period (first period).

Next, the calculation unit 120 calculates the first similarity indicating the similarity between the estimated pulse rates (between the first pulse rate and the second pulse rate) with respect to the same period (first period) as the region of interest. is calculated for each (step S52).

Then, the estimation unit 130 estimates the physical and mental condition of the subject using at least one of the first pulse rate, the second pulse rate, and the plurality of face images in the first period according to the first degree of similarity. (Step S53).

After that, the output unit 140 outputs the estimation result to, for example, the storage unit 150 (step S54).

With the above, the psychosomatic state estimation device 100 ends the processing.

In this way, in order for the psychosomatic state estimation apparatus 100 to estimate the psychosomatic state of the subject using the pulse rate estimated from the face image, the subject is made to wear a device or the like for measuring the pulse rate. It is possible to estimate the psychosomatic condition of the person to be measured without imposing a load on the subject.

In addition, since the psychosomatic state estimation apparatus 100 estimates the psychosomatic state of the person to be measured according to the similarity of a plurality of pulse rates estimated with respect to the region of interest (first region), a single imaging device The psychosomatic state of the person to be measured can be estimated even if the region is not imaged. As a result, psychosomatic state estimation apparatus 100 according to the present embodiment can estimate the psychosomatic state of the person to be measured without asking the person to be measured to view multiple parts.

In addition, since the psychosomatic state estimating apparatus 100 estimates the psychosomatic state of the person to be measured according to the degree of similarity of a plurality of pulse rates, the psychosomatic state is estimated using, for example, a pulse rate that is assumed to contain no noise. can do. As a result, the psychosomatic state estimation apparatus 100 can estimate the subject's psychosomatic state with high accuracy.

Also, when capturing a face image including multiple regions of interest, multiple imaging devices may be required depending on the orientation of the subject's face. However, according to the psychosomatic state estimation apparatus 100 according to the present embodiment, the psychosomatic state is estimated using the similarity of a plurality of pulse rates estimated with respect to the region of interest (first region). It may be one. As a result, the psychosomatic state estimation apparatus 100 can reduce the number of imaging devices installed for estimating the psychosomatic state, and can reduce installation costs.

Note that the estimation unit 130 may estimate the psychosomatic state according to the parts included in the first region. The first region is the region of interest, as described above. Regions of interest include parts such as the mouth, right cheek, left cheek, and forehead. Each part of the face has different pixel values depending on the thickness of the skin. In other words, a site where the pixel value is likely to change is a site where the pulse can be estimated with high accuracy. Therefore, when there are a plurality of regions of interest (first regions), the estimating unit 130 may, for example, calculate the first pulse rate, the second pulse rate, and the region of interest associated with the region of interest including a site where the pulse can be estimated with high accuracy. At least one of the images may be used to estimate the psychosomatic state.

In addition, the estimation unit 130 may estimate the psychosomatic state using different estimation methods depending on the part. For example, if the first region includes a part such as the forehead where the pulse can be estimated with high accuracy, the estimating unit 130 estimates the psychosomatic state in the frequency domain, and the first region can estimate the pulse with high accuracy. When parts other than parts are included, the psychosomatic state may be estimated with the time domain as the target.

In this manner, the estimation unit 130 performs estimation according to the parts of the face included in the first region, so that the psychosomatic state estimation apparatus 100 can more accurately estimate the subject's psychosomatic state.

Further, if the difference between the first similarity and the second similarity indicating the similarity calculated for a second period different from the first period is within a predetermined range, estimating section 130 The physical and mental condition of the subject may be estimated using at least one of the first pulse rate, the second pulse rate, and the plurality of face images in the second period.

For example, if the first period is t0<T<=t3 shown in FIG. 3, the second period is t3<T<=t6, for example. As such, the second period is preferably a period temporally adjacent (continuous) to the first period. Note that the length of the second period may be the same as or different from that of the first period.

Estimating section 130 compares the similarity between the same region of interest for the first period and the similarity for the second period. When the region of interest 32 is the “forehead”, the estimation unit 130 calculates the similarity 36 of the “forehead” to the first period of “0.84” and the similarity 36 of the “forehead” to the second period of “0.91”. Compare with Then, if the difference between the degree of similarity for the first period and the degree of similarity for the second period is within a predetermined range, estimating section 130 determines that the first pulse rates b11 to 16, the second pulse rates b21 to b26, and t0 = At least one of the face images captured at t6 may be used to estimate the psychosomatic state.

Thus, the psychosomatic state estimating apparatus 100 changes the pulse rate and/or the time window of the face image used for estimating the psychosomatic state according to the difference between similarities. As a result, when there is a high possibility that the pulse rate is being estimated with high accuracy, for example, when the subject is in a resting state, the psychosomatic state estimation apparatus 100 obtains the data for a long period instead of the data for a short period. This data is used to estimate the psychosomatic state of the person being measured. Also, if the estimation of the pulse rate varies depending on the estimation method, there is a high possibility that the person to be measured is not in a resting state. In such a case, the psychosomatic state estimation apparatus 100 estimates the subject's psychosomatic state using short-term data. In this way, since the period for estimating the psychosomatic state can be changed according to the difference in the degree of similarity, the psychosomatic state estimation apparatus 100 can more accurately estimate the psychosomatic state of the subject.

Further, in the first period, the calculation unit 120 calculates the pulse rate estimated for the second region different from the first region, the first pulse rate or the second pulse rate estimated for the first region, and may be calculated as a third degree of similarity indicating the degree of similarity of the . For example, if the first region is the “forehead” region of interest 32, the second region of the region of interest 32 in FIG. 3 is the “right cheek” or “left cheek” region of interest 32 . Here, if the second region is the region of interest 32 of the “right cheek” and the first period is t0<T<=t3, the calculation unit 120 calculates the first Pulse rates b11-b13 or second pulse rates b21-b23 and first pulse rates c11-c13 or second pulse rates c21-c23 estimated from the region of interest 32 of the "right cheek" 3 similarity may be calculated.

Then, the estimating section 130 may estimate the subject's psychosomatic state based on the third degree of similarity in addition to the first degree of similarity. For example, the estimation unit 130 may estimate the psychosomatic state of the subject when the third degree of similarity is higher than a predetermined threshold. If the similarity between the estimated pulse rates is high, there is a high possibility that the pulse rates do not contain noise or the like. In other words, it can be said that the estimated pulse rate is highly reliable. Therefore, by estimating the psychosomatic state using the third degree of similarity between the plurality of regions (the first region and the second region), psychosomatic state estimation apparatus 100 can estimate the psychosomatic state more reliably than when using only the first degree of similarity. The psychosomatic state can be estimated using a high pulse rate, and the estimation accuracy of the psychosomatic state can be improved.

Further, the calculation unit 120 calculates the first similarity based on the amount of movement of the feature points in the first region included in each of the plurality of face images in the first period or the amount of change in the orientation of the face of the subject. can be calculated. The amount of movement of the feature points and the amount of change in the face orientation may be calculated by the calculation unit 120, or may be calculated by another device and may be acquired by the acquisition unit 110. FIG.

Even when the similarity between pulse rates is high, the estimated pulse rate may contain noise if the amount of movement of the feature points or the amount of change in the orientation of the face is large. Therefore, the calculating unit 120 may calculate the first similarity by multiplying the similarity between the pulse rates by a value corresponding to, for example, the amount of movement of the feature point or the amount of change in the orientation of the face. Accordingly, the calculation unit 120 can output the value obtained by evaluating the reliability of the estimated pulse rate as the first degree of similarity. Therefore, for example, the estimating unit 130 can estimate the psychosomatic state using the first pulse rate, the second pulse rate, or the face image having the high first similarity, so that the accuracy of estimating the psychosomatic state can be improved. can.

In addition, the estimating unit 130 estimates the psychosomatic state based on the amount of movement of the feature points in the first region included in each of the plurality of face images in the first period or the amount of change in the orientation of the face of the subject. may As a result, the estimating unit 130 can estimate the psychosomatic state by taking into consideration various changes in the subject's condition, such as a small change in the pulse rate but a large movement amount of the feature point.

Further, the calculation unit 120 may calculate, as the fourth degree of similarity, the amount of change (difference) between the pulse rate in a certain region of interest in the first period and the pulse rate in the second period. The fourth degree of similarity may be the amount of change between the frequency corresponding to the pulse rate in the frequency domain in the first period and the frequency corresponding to the pulse rate in the frequency domain in the second period. Then, the estimation unit 130 may estimate the psychosomatic state depending on whether the fourth degree of similarity is greater than, for example, a predetermined threshold. Note that the predetermined threshold value may be set to 12 bpm, for example, based on the physiological property that a pulse rate change of 12 bpm or more is unlikely to occur in one second. If the fourth degree of similarity is greater than the predetermined threshold, there is a possibility that noise is included in at least one of the pulse rate in the first period and the pulse rate in the second period. Therefore, the estimating unit 130 estimates the psychosomatic state using, for example, the pulse rate and the face image related to the fourth similarity smaller than the predetermined threshold, so that the psychosomatic state estimation apparatus 100 can estimate the psychosomatic state with accuracy can increase

<Second Embodiment>
A second embodiment of the present disclosure will be described with reference to FIGS. 6 and 7. FIG. FIG. 6 is a block diagram showing an example of the configuration of the psychosomatic state estimation device 10 according to this embodiment. As shown in FIG. 6 , the psychosomatic state estimation device 10 includes a calculator 12 and an estimator 13 .

The calculation unit 12 calculates a first period estimated by a first estimation method using a plurality of luminance values of the first region for a first region including a part of the face among the plurality of time-series face images. A first similarity is calculated that indicates a similarity between the first pulse rate and the second pulse rate in the first period estimated by a second estimation method different from the first estimation method. The calculator 120 described above is an example of the calculator 12 .

The first area may be any area that includes facial parts (eg, eyes, nose, mouth, etc.). Also, the first estimation method and the second estimation method are the same as in the above-described first embodiment. Similarly to the calculation unit 120, the calculation method of the degree of similarity by the calculation unit 12 is not particularly limited.

The estimating unit 13 uses at least one of the first pulse rate, the second pulse rate, and the plurality of face images in the first period to evaluate the body and mind of the subject according to the first similarity calculated by the calculating unit 12. Estimate the state. The estimation unit 130 described above is an example of the estimation unit 13 . The method for estimating the psychosomatic state of the person to be measured may be any method as described above, and is not particularly limited.

FIG. 7 is a flowchart showing an example of the operation flow of the psychosomatic state estimation device 10 according to the present embodiment. As shown in FIG. 7, the calculation unit 12 calculates a first similarity indicating the similarity between the first pulse rate and the second pulse rate (step S71). Then, the estimating unit 13 estimates the psychosomatic state of the subject using at least one of the first pulse rate, the second pulse rate, and the plurality of face images in the first period according to the first similarity ( step S72). The psychosomatic state estimation device 10 ends the processing.

As described above, in order for the psychosomatic state estimating apparatus 10 to estimate the psychosomatic state of the subject using the pulse rate estimated from the face image, the subject is made to wear a device or the like for measuring the pulse rate. It is possible to estimate the psychosomatic condition of the person to be measured without imposing such a burden on them.

In addition, the psychosomatic state estimation apparatus 10 estimates the psychosomatic state of the person to be measured according to the similarity of a plurality of pulse rates estimated for a certain region (first region). The psychosomatic state of the person to be measured can be estimated even if the region is not imaged. As a result, the psychosomatic state estimating apparatus 10 according to the present embodiment can estimate the psychosomatic state of the person to be measured without asking the person to be measured to see multiple parts.

In addition, since the psychosomatic state estimating apparatus 10 estimates the psychosomatic state of the person to be measured according to the similarity of a plurality of pulse rates, the psychosomatic state is estimated using, for example, a pulse rate that is assumed to contain no noise. can do. As a result, the psychosomatic state estimation device 10 can estimate the subject's psychosomatic state with high accuracy.

(About hardware configuration)
In each embodiment of the present disclosure, each component of each device represents a functional unit block. A part or all of each component of each device is realized by an arbitrary combination of an information processing device 900 and a program as shown in FIG. 8, for example. FIG. 8 is a block diagram showing an example of the hardware configuration of an information processing device 900 that implements each component of each device. The information processing apparatus 900 includes, as an example, the following configuration.

- CPU (Central Processing Unit) 901
・ROM (Read Only Memory) 902
・RAM (Random Access Memory) 903
・Program 904 loaded into RAM 903
- Storage device 905 for storing program 904
A drive device 907 that reads and writes the recording medium 906
- A communication interface 908 that connects to the communication network 909
- An input/output interface 910 for inputting/outputting data
A bus 911 connecting each component
Each component of each device in each embodiment is implemented by the CPU 901 acquiring and executing a program 904 that implements these functions. A program 904 that implements the function of each component of each device is stored in advance in, for example, the storage device 905 or the ROM 902, and is loaded into the RAM 903 and executed by the CPU 901 as necessary. The program 904 may be supplied to the CPU 901 via the communication network 909 or may be stored in the recording medium 906 in advance, and the drive device 907 may read the program and supply it to the CPU 901 .

There are various modifications in the implementation method of each device. For example, each device may be realized by any combination of the information processing device 900 and a program that are separate for each component. Also, a plurality of components included in each device may be realized by any combination of one information processing device 900 and a program.

Also, part or all of each component of each device is implemented by other general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be composed of a single chip, or may be composed of multiple chips connected via a bus.

A part or all of each component of each device may be realized by a combination of the above-described circuits and the like and programs.

When part or all of each component of each device is implemented by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally arranged or distributed. good too. For example, the information processing device, circuits, and the like may be realized as a form in which each is connected via a communication network, such as a client-and-server system, a cloud computing system, or the like.

The above-described embodiments are preferred embodiments of the present disclosure, and the scope of the present disclosure is not limited only to the above-described embodiments. can be modified or substituted for each of the above-described embodiments, and various modified forms can be constructed.

REFERENCE SIGNS LIST 10 psychosomatic state estimation device 12 calculation unit 13 estimation unit 100 psychosomatic state estimation device 110 acquisition unit 120 calculation unit 130 estimation unit 140 output unit 150 storage unit

Claims (9)

A first estimation using the luminance value of the first region for a first region showing the skin of at least one of the forehead, right cheek, left cheek, mouth, eyes, and nose of the person to be measured in the image. and a second pulse rate estimated for the first area by a second estimation method different from the first estimation method, the first pulse rate and Calculation means for calculating a degree of similarity representing the closeness of the second pulse rate ;
A method for estimating the stress-related state of the person to be measured using first information, which is information on heartbeat variability calculated by performing estimation targeting the time domain, and performing estimation targeting the frequency domain. changing means for switching to a method determined by the degree of similarity and a threshold among the methods of estimating the stress-related state of the person to be measured using the second information, which is information about the frequency of the heartbeat variability calculated by ,
A system with When the degree of similarity is smaller than a predetermined threshold, the stress-related state of the subject is estimated by a method of estimating the stress-related state of the subject using the first information.
The system of claim 1. When the degree of similarity is equal to or greater than a predetermined threshold, the stress-related state of the subject is estimated by a method of estimating the stress-related state of the subject using the second information.
3. A system according to claim 1 or 2. the computer
A first estimation using the luminance value of the first region for a first region showing the skin of at least one of the forehead, right cheek, left cheek, mouth, eyes, and nose of the person to be measured in the image. and a second pulse rate estimated for the first area by a second estimation method different from the first estimation method, the first pulse rate and Calculate the similarity representing the closeness of the second pulse rate ,
A method for estimating the stress-related state of the person to be measured using first information, which is information on heartbeat variability calculated by performing estimation targeting the time domain, and performing estimation targeting the frequency domain. of the method of estimating the state of stress of the person to be measured using the second information, which is the information of the frequency related to the heartbeat variability calculated by the method , switching to the method determined by the similarity and the threshold ;
Method. the computer
If the degree of similarity is smaller than a predetermined threshold, the stress-related state of the subject is estimated by a method of estimating the stress-related state of the subject using the first information.
5. The method of claim 4. the computer
estimating the stress-related state of the subject by a method of estimating the stress-related state of the subject using the second information when the similarity is equal to or greater than a predetermined threshold ;
6. A method according to claim 4 or 5. A first estimation using the luminance value of the first region for a first region showing the skin of at least one of the forehead, right cheek, left cheek, mouth, eyes, and nose of the person to be measured in the image. and a second pulse rate estimated for the first area by a second estimation method different from the first estimation method, the first pulse rate and a process of calculating a degree of similarity representing the closeness of the second pulse rate ;
A method for estimating the stress-related state of the person to be measured using first information, which is information on heartbeat variability calculated by performing estimation targeting the time domain, and performing estimation targeting the frequency domain. A process of switching to a method determined by the similarity and a threshold from among the methods of estimating the stress -related state of the subject using the second information, which is information about the frequency of the heartbeat variability calculated by
A program that makes a computer run A process of estimating the stress-related state of the subject by a method of estimating the stress-related state of the subject using the first information when the similarity is smaller than a predetermined threshold.
8. The program according to claim 7, causing the computer to execute: A process of estimating the stress-related state of the subject by a method of estimating the stress-related state of the subject using the second information when the degree of similarity is equal to or greater than a predetermined threshold .
9. The program according to claim 7 or 8, causing the computer to execute:

Images