JP2020120908A - Mental state estimation system, mental state estimation method, and program - Google Patents

Abstract

To provide a mental state estimation system capable of accurately estimating a mental state.SOLUTION: A mental state estimation system 100 comprises: a first output unit 21 for outputting information for prompting an object person U to make a predetermined expression; an image acquisition unit 22 for acquiring an image obtained by capturing the object person U with the predetermined expression; an extraction unit 23 for extracting an expression index of the object person U from the image; a storage unit 24 for storing expression indices at a plurality of points of time before an object point of time, as first time series data; and an estimation unit 25 for estimating the object person U's mental state at the object point of time on the basis of an expression index at the object point of time extracted by the extraction unit 23 and the first time series data stored in the storage unit 24.SELECTED DRAWING: Figure 1

Description

The present invention relates to a mental state estimation system, a mental state estimation method, and a program.

In recent years, the number of patients with mental illnesses such as depression who have suffered from mental health disorders has been increasing. Early detection is important in the treatment of mental illness, but it is the actual situation that it is difficult for ordinary people to notice signs and symptoms of mental illness.

Therefore, Patent Document 1 discloses a health management assisting device that estimates the mental state (mental health state) of the subject based on the facial expression recognized from the image of the image of the face of the subject.

International Publication No. 2016/129193

However, in Patent Document 1, the mental state may not be accurately estimated depending on the facial expression of the imaged subject.

Therefore, the present invention provides a mental state estimation system, a mental state estimation method, and a program capable of accurately estimating a mental state.

A mental state estimation system according to an aspect of the present invention acquires a first output unit that outputs information for urging a subject to have a predetermined facial expression, and an image obtained by capturing the subject with the predetermined facial expression. An image acquisition unit, an extraction unit that extracts the target person's facial expression index from the image, a storage unit that stores the facial expression index at a plurality of time points before the target time point as temporary series data, and the extraction unit extracts An estimation unit that estimates the mental state of the subject at the target time point based on the facial expression index at the target time point and the temporary series data stored in the storage unit.

A mental state estimation method according to an aspect of the present invention includes a step of outputting information for prompting a target person to have a predetermined facial expression, a step of acquiring an image of the target person having the predetermined facial expression, and the image. From the target person's facial expression index, storing a plurality of the facial expression indexes at a plurality of time points before the target time point in the storage unit as time series data, the facial expression index at the target time point, and the storage Estimating the mental state of the subject based on the time series data stored in the section.

A program according to an aspect of the present invention is a program for causing a computer to execute the mental state estimation method described above.

According to one aspect of the present invention, it is possible to realize a mental state estimation system, a mental state estimation method, and a program capable of accurately estimating a mental state.

FIG. 1 is a block diagram showing the configuration of a mental state estimation system according to an embodiment. FIG. 2 is a flowchart showing an example of the operation of the mental state estimation system according to the embodiment. FIG. 3 is a diagram showing an example of time-series data based on a facial expression index according to the embodiment. FIG. 4 is a block diagram showing a configuration of a mental state estimation system according to a modified example of the embodiment. FIG. 5 is a diagram showing an example of an application example of the mental state estimation system according to the embodiment.

Hereinafter, embodiments will be specifically described with reference to the drawings. It should be noted that each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of constituent elements, steps, order of steps, and the like shown in the following embodiments are examples, and are not intended to limit the present invention. Further, among the constituent elements in the following embodiments, constituent elements not described in independent claims are described as arbitrary constituent elements.

It should be noted that each drawing is a schematic diagram and is not necessarily strictly illustrated. Further, in each drawing, the same reference numerals are given to substantially the same configurations, and overlapping description may be omitted or simplified.

In the following specification, the image will be described as an example of a moving image (video), but it may be a still image. Further, the image may include characters and numbers. The moving image is composed of a plurality of still images.

(Embodiment)
Hereinafter, the mental state estimation system and the like according to the present embodiment will be described with reference to FIGS. 1 to 3. The mental state estimation system according to the present embodiment is a device for estimating the mental health of the subject based on the image captured of the subject.

[1. Constitution]
First, the configuration of the mental state estimation system 100 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing the configuration of a mental state estimation system 100 according to this embodiment.

As shown in FIG. 1, the mental state estimation system 100 includes an imaging device 10, a mental state estimation device 20, and a display device 30. The mental state estimation system 100 may include at least the mental state estimation device 20. Further, FIG. 1 also shows a terminal device 40 to which the result estimated by the mental state estimation device 20 is output.

The imaging device 10 captures an image including at least the face of the target person U. Specifically, the imaging device 10 captures an image including the face of the target person U when performing predetermined information described below. The image captured by the image capturing apparatus 10 may be a monochrome image or a color image. The imaging device 10 is, for example, a digital video camera or a digital still camera including an image sensor.

The mental state estimation device 20 estimates the mental state of the target person U based on the image acquired from the imaging device 10. The mental state estimation device 20 includes a first output unit 21, an image acquisition unit 22, an extraction unit 23, a storage unit 24, an estimation unit 25, a threshold value calculation unit 26, and a second output unit 27.

The first output unit 21 is communicably connected to the display device 30 and outputs information (hereinafter, also referred to as instruction information) for prompting the target person U to have a predetermined facial expression, to the display device 30.

Note that, in the following, an example in which the predetermined facial expression is a smile will be described. Good. Further, in the present embodiment, the instruction information is information for prompting a smile, and in the example of FIG. 1, information for displaying a text such as “Please make a smile”.

The image acquisition unit 22 is communicably connected to the imaging device 10 and acquires an image captured by the imaging device 10. In the present embodiment, the image acquisition unit 22 acquires a moving image of the face of the target person U as the image captured by the imaging device 10.

The extraction unit 23 extracts the facial expression index of the target person U from the image acquired by the image acquisition unit 22 by the image sensing process. In the present embodiment, the extraction unit 23 extracts a smile degree as a facial expression index. The extraction unit 23 may convert, for example, the degree of smile into an index indicating the mental state of the subject U (for example, the degree of stress indicating the degree of stress). The index indicating the degree of smile and the mental state is an example of the first feature amount. In the following, an example in which the first feature amount is a smile degree will be described. The degree of smile is an index indicating the degree of smile, and is indicated by 0 to 100%. Note that an example will be described in which the higher the degree of smile (closer to 100%), the more smile, but the invention is not limited to this. Further, the extraction of the degree of smile is not particularly limited, and a conventional technique may be used. Therefore, a detailed description of extracting the smile degree will be omitted.

The facial expression index is not limited to the smile degree, and may be any index that can estimate the mental state of the target person U. The facial expression index may be, for example, a smile formation speed, a change amount of at least one of the mouth and eyes, the order in which the mouth and eyes change, left-right symmetry of the face, face angle, and skin color.

The smile formation speed is the speed at which the target person U smiles, and is, for example, the time from when the first output unit 21 outputs the instruction information until the target person U smiles. If the mental state of the subject U is deteriorated (for example, the subject U is under stress), the speed at which a smile is formed is not deteriorated (for example, the mental state is normal). ) It will be slower than the case. That is, it takes a long time for the target person U to smile. Therefore, the mental state of the target person U can be estimated from the smile formation rate.

Note that the time until a smile here is, for example, the change in the corners of the mouth or the corners of the eyes (for example, the inclination of the corners of the mouth or the corners of the corners of the eyes of the subject U after the first output unit 21 outputs the instruction information) Change) is the maximum time. The extraction unit 23 can extract the smile formation speed by the image acquisition unit 22 acquiring the moving image from the imaging device 10. The deterioration of mental state means deterioration of mental health condition. For example, it means that a sign of a mental illness such as depression appears. In addition, the mental state can be rephrased as the energy level or the activity level.

The amount of change in at least one of the mouth and eyes is, for example, the amount of change in the corner of the eyes and the angle of the mouth from when the first output unit 21 outputs the instruction information until the subject U smiles. When the mental state of the subject U deteriorates, the amount of change decreases. Therefore, the mental state of the subject U can be estimated from the change amount of at least one of the mouth and the eye. From the viewpoint of estimating the mental state with higher accuracy, it is preferable to use the change amounts of both the mouth and the eyes.

The order in which the mouth and the eyes change indicates, for example, from the mouth and the eyes when the face of the target person U becomes a smile after the first output unit 21 outputs the instruction information. When the eyes of the mouth and eyes move first, the subject U is likely to smile from the bottom of his heart. That is, it is highly possible that the mental state of the subject U has not deteriorated. On the other hand, when the mouth of the mouth and eyes moves first, the smile of the target person U is likely to be a make-up smile. In other words, the mental state of the subject U is likely to have deteriorated. Therefore, the mental state of the subject U can be estimated from the order in which the mouth and eyes change.

The left-right symmetry of the face is, for example, the left-right symmetry of the face when the subject U smiles. If the right and left sides of the face are symmetrical, the subject U is highly likely to smile from the heart. That is, it is highly possible that the mental state of the subject U has not deteriorated. On the other hand, if the right and left sides of the face are asymmetric, the smile of the subject U is likely to be a make-up smile. In other words, the mental state of the subject U is likely to have deteriorated. Therefore, the mental state of the subject U can be estimated from the bilateral symmetry of the face. The symmetry may be output depending on whether or not it is symmetric, or may be output as a symmetry degree (for example, a numerical value). Further, it is assumed that the left and right are asymmetrical, for example, when a smiling face is tight.

Similarly, regarding the face angle and the skin color, the extracted results are different between when the mental state is deteriorated and when the mental state is normal. Therefore, the mental state of the subject U can be estimated from the face angle and the skin color. .. The face angle means the angle of the face of the target person U with respect to the imaging device 10 (for example, the lens unit of the imaging device 10).

In addition, from the viewpoint of improving the estimation accuracy of the mental state, the extraction unit 23 may extract at least two or more facial expression indexes from the above. In other words, the facial expression index used for estimating the mental state may include at least two of a smile degree, a smile formation rate, mouth and eye positions, facial symmetry, face angle, and skin color.

The storage unit 24 is a storage device that stores a control program executed by each component of the mental state estimation device 20 and a smile degree recognized by the extraction unit 23. The storage unit 24 stores the time-series data of the degree of smile extracted by the extraction unit 23. The storage unit 24 stores, for example, a plurality of smile degrees based on a plurality of images captured at different image capturing times before the target time as time series data. The storage unit 24 stores, for example, smile degrees at a plurality of time points before the target time point as time series data (an example of first temporary series data). The storage unit 24 is realized by a semiconductor memory or the like. The target time point means a time point when the mental state of the target person U at that time point is determined. The target time point may be, for example, the current time point (present) or a past time point.

The estimation unit 25 estimates the current mental state of the subject U based on the currently acquired smile degree and the time-series data of the smile degree of the subject U stored in the storage unit 24. That is, the estimation unit 25 estimates the current mental state of the target person U from the currently acquired smile degree of the target person U by using the smile levels of the target person U acquired at a plurality of past times. The estimation unit 25 has a good mental state, for example, when the current smile degree is one of the threshold values calculated by the threshold calculation unit 26 and less than or equal to the threshold value, and the smile degree at the target time point is the threshold value or more and less than the threshold value. If it is, it is presumed to be in a stressed state. Note that the present time is an example of a target time point.

When the extraction unit 23 extracts at least two facial expression indices, the estimation unit 25 estimates the mental state of the subject U based on the at least two facial expression indices.

In addition, when the extraction unit 23 converts the degree of smile into an index indicating the mental state of the target person U, the estimation unit 25 estimates the mental state of the target person U based on the index and a threshold for the index. Good. The estimation unit 25 estimating the mental state of the target U based on the index indicating the mental state of the target U is included in estimating the mental state of the target U based on the degree of smile.

The threshold calculation unit 26 calculates the threshold from the time-series data of the degree of smile of the target person U stored in the storage unit 24. That is, the threshold calculation unit 26 calculates the threshold according to the target person U. The degree of smile varies from person to person, and the degree of smile at normal times (for example, when the mental state is stable) differs from person to person. Therefore, if the threshold value is uniformly determined, the mental state of the subject U may not be accurately estimated. Therefore, the mental state estimation device 20 according to the present embodiment calculates the threshold value according to the target person U from the time-series data of the degree of smile of the target person U as described above. With this, the estimation unit 25 can estimate the mental state at the target time point based on the smile personality of the target person U, and thus can more accurately estimate the mental state of the target person U. Note that the threshold value here is a value for estimating the mental state of the target person U, and for example, for estimating whether or not the mental state has deteriorated, or for estimating the degree of deterioration or improvement of the mental state. It is a value.

The method of calculating the threshold value by the threshold value calculation unit 26 is not particularly limited, but is calculated based on, for example, a statistical value of the smile degree calculated from time series data of the smile degree (for example, an average value or a median value). May be. The threshold value calculation unit 26 may calculate the threshold value by calculating a predetermined numerical value for the statistical value. The operation includes at least one of addition, subtraction, multiplication, and division. The threshold may be calculated as appropriate according to the usage of the mental state estimation system 100 and the like. In addition, the number of smile degrees included in the time-series data for calculating the threshold is preferably large. For example, the time-series data may include the degree of smile of the target person U for several weeks.

The second output unit 27 outputs the estimation result of the mental state of the target person U estimated by the estimation unit 25 to the terminal device 40.

As described above, the mental state estimation device 20 causes the target person U to make a facial expression change to a predetermined facial expression (for example, a smile), and based on the image of the target person U having the predetermined facial expression, the target person U is changed. It is characterized by estimating the mental state of the person U. In addition, the target person U having a predetermined facial expression will be described by taking a smiling face as an example. The target person U who becomes a smiling face, and the target person U who becomes a smiling face or is in the middle of returning from the smiling face (for example, the facial expression is changed) Including the subject U). Further, the mental state estimation device 20 can be configured by a computer including a CPU or a processor, a memory, an auxiliary storage device, an input device, a communication device, and the like.

The display device 30 is a device that acquires instruction information from the mental state estimation device 20 and outputs the acquired instruction information as an image. The image includes a photograph, an illustration, characters, or the like. The display device 30 is a liquid crystal display or the like. The image output by the display device 30 is visually recognized by the subject U. The display device 30 is an example of an instruction device. The mental state estimation system 100 may include a device that outputs a sound as an instruction device together with the display device 30 or in place of the display device 30. Moreover, the mental state estimation system 100 may include a device such as a projector that displays instruction information on an object (for example, a screen) as an instruction device.

The terminal device 40 is a device that presents the estimation result of the mental state of the target person U to the target person U. The terminal device 40 has a liquid crystal display or the like, and may present the estimation result as an image. Further, the terminal device 40 has a microphone and may present the estimation result by voice. The terminal device 40 is not particularly limited as long as it is a device that is communicatively connected to the mental state estimation device 20 and that can present an estimation result. The terminal device 40 is, for example, a mobile terminal owned by the target person U, and may be a smartphone or a tablet. Further, the terminal device 40 may be a stationary device (for example, a stationary computer) or the like. Further, the terminal device 40 may be a server device that manages the estimation result.

[2. motion]
Next, the processing executed by the mental state estimation system 100 will be described with reference to FIG. FIG. 2 is a flowchart showing an example of the operation of mental state estimation system 100 according to the present embodiment. FIG. 2 shows an example of the operation of the mental state estimation device 20. In addition, in FIG. 2, the time series data of the degree of smile of the target person U is already stored in the storage unit 24, and a process of estimating the mental state of the target person U using the currently acquired image will be described.

As shown in FIG. 2, first, the first output unit 21 outputs information (instruction information) for prompting the target person U to smile (S10). The timing at which the first output unit 21 outputs the instruction information is not particularly limited. For example, when the first output unit 21 acquires the preparation completion information indicating that the mental state estimation system 100 is ready to estimate the mental state, the first output unit 21 reads the instruction information from the storage unit 24 and displays the read instruction information. It may be output to the device 30. The preparation completion information may be acquired by an operation performed by the target person U or may be acquired by a sensing result of a sensor or the like.

For example, the preparation completion information may be information based on an operation (for example, an operation indicating that the user arrives in front of the image capturing apparatus 10) received from a user by an operation unit (not shown), or a motion sensor or the like may be used as the image capturing apparatus. It may be information indicating that the presence of the target person U before 10 is detected. In addition, the first output unit 21 detects the target person U in the image captured by the imaging device 10 when the imaging device 10 is performing imaging before the first output unit 21 outputs the instruction information. The instruction information may be output to. In addition, the first output unit 21 knows that the target person U is at a position where the display of the display device 30 can be seen for a long period of time (for example, the target person U is at an office desk, and the display device 30 is on the desk). (In the case where is arranged), the instruction information may be output to the display device 30 at a predetermined time interval or a predetermined time.

The timing at which the imaging device 10 starts imaging is not particularly limited. The imaging device 10 may perform imaging before the instruction information is output in step S10. For example, the imaging device 10 may start imaging when the target person U approaches the imaging device 10 within a predetermined distance by a human sensor or the like (not shown). When the first output unit 21 outputs information indicating that the instruction information has been output to the image pickup apparatus 10, the image pickup apparatus 10 may start image pickup when the information is acquired. When capturing a moving image, the image capturing apparatus 10 includes an image including a facial expression before the target person U sees the display of the display device 30 and the facial expression of the target person U changes (a facial expression during normal times, for example, a straight face). Should be able to be imaged. In addition, the imaging device 10 performs first period imaging when capturing a moving image. The first period includes a period during which the target person U usually smiles from his facial expression. For example, the first period may include a period during which the subject U smiles from the facial expression in normal times and returns to the facial expressions in normal times.

In addition, when capturing a still image, the image capturing apparatus 10 may capture the image, for example, after a second period has elapsed after the first output unit 21 outputs the instruction information. The second period may be the time from when the subject U sees the display on the display device 30 until he/she smiles, and is, for example, 1 second.

Then, the image acquisition unit 22 acquires an image captured including the time when the target person U is smiling from the imaging device 10 (S20). The imaging device 10 outputs the acquired image to the extraction unit 23. The timing at which the image acquisition unit 22 acquires an image is not particularly limited. For example, the image acquisition unit 22 may acquire an image each time the imaging device 10 captures an image, or may acquire an image at predetermined time intervals. You may get it.

Then, the extraction unit 23 extracts the degree of smile of the target person U using the image acquired from the image acquisition unit 22 (S30), and stores the extracted degree of smile in the storage unit 24 (S40). The extraction unit 23 also outputs the extracted smile degree to the estimation unit 25. The extraction unit 23 detects, for example, the face of the target person U from the image, extracts the smile degree from the detected face, and stores the extracted smile degree in the storage unit 24 together with the time stamp information. When the smile degree extracted by the extraction unit 23 is converted into an index indicating the mental state of the target person U, the index may be stored in the storage unit 24 instead of the smile degree or together with the smile degree.

In addition, when the threshold value calculation unit 26 acquires the time-series data of the degree of smile of the target person U for the threshold value calculation unit 26 to calculate the threshold value, the mental state estimation device 20 repeatedly executes the processing from steps S10 to S40.

Then, the threshold value calculation unit 26 calculates the threshold value of the smile degree from the time series data of the smile degree of the target person U stored in the storage unit 24 (S50). The estimation unit 25 determines whether the smile degree acquired from the extraction unit 23 is equal to or greater than the threshold value acquired from the threshold value calculation unit 26 (S50). When the smile degree is equal to or higher than the threshold value (Yes in S60), the estimation unit 25 estimates that the mental state of the subject U is deteriorated (S70). That is, the estimation unit 25 estimates that the mental state of the subject U is unstable and the subject U is in a stressed state. Further, when the smile degree is lower than the threshold value (No in S60), the estimation unit 25 estimates that the mental state of the target person U has not deteriorated (S80). That is, the estimation unit 25 estimates that the mental state of the subject U is stable, the subject U is in a state of less stress, and the mental health is maintained. The estimation unit 25 outputs the estimation result to the second output unit 27.

Then, the second output unit 27 outputs the estimation result acquired from the estimation unit 25 to the terminal device 40 (S90). For example, when the terminal device 40 is a smartphone owned by the target person U, the target person U can know the estimation result only by operating the smartphone. In step S90, the second output unit 27 may output at least the estimation result estimated by the estimation unit 25 that the mental state has deteriorated.

The process of step S90 may be executed each time the estimation unit 25 estimates the mental state of the target person U, or may be executed at predetermined time intervals. For example, the second output unit 27 may collectively output a plurality of estimation results to the terminal device 40.

Further, in step S90, the second output unit 27 may output not only the estimation result but also a graph showing a temporal change of the mental state as shown in FIG. FIG. 3 is a diagram showing an example of time-series data based on a facial expression index according to the present embodiment. FIG. 3 shows time-series data of the degree of stress of the subject U from January to the time point when Yes is determined in step S60. The stress level can be calculated by performing a predetermined calculation on the smile level, and is calculated by the extraction unit 23, for example. Further, it is assumed that the threshold for the stress level is set in advance. The threshold is calculated by the threshold calculator 26, for example. The higher the stress level, the lower the smile level.

By outputting the graph as shown in FIG. 3 to the terminal device 40, the target person U can intuitively understand the tendency of good and bad mental states from the graph. In addition, the subject U can know when the mental state has deteriorated, and thus can be used to identify the cause of deterioration of the mental state. The broken line shown in FIG. 3 indicates the prediction of the change in the stress level when no measures are taken when the stress level exceeds the threshold value. In this way, by showing the prediction, it is possible to urge the target person U to take measures. The second output unit 27 may output a graph showing a temporal change in the degree of smile, instead of the graph showing a temporal change in the degree of stress.

In addition to the estimation result, the second output unit 27 may output the countermeasure plan and the advice when the mental state is deteriorated. For example, the second output unit 27 may output information such as “take a rest” and “go to the hospital” when the mental state is deteriorated, in addition to the estimation result. The countermeasure plan and the advice may be stored in the storage unit 24, for example.

In the above, the example in which the threshold value is preset has been described, but the present invention is not limited to this. For example, the currently acquired smile degree may be used to calculate a threshold value when the smile degree is acquired next time. The threshold calculation unit 26 may update the threshold each time the degree of smile for estimating the mental state of the target person U is acquired.

[3. Effects, etc.]
As described above, the mental state estimation system 100 according to the present embodiment includes the first output unit 21 that outputs information for urging the target person U to smile (an example of a predetermined facial expression), and a smile. An image acquisition unit 22 that acquires an image of the target person U, an extraction unit 23 that extracts the degree of smile of the target person U (an example of a facial expression index) from the image, and a plurality of time points before the present (an example of a target time point) The storage unit 24 that stores the degree of smile in the time series data (an example of the first temporary series data), the current smile degree extracted by the extraction unit 23, and the time series data stored in the storage unit 24. And an estimation unit 25 that estimates the current mental state of the target person U.

For example, a typical symptom of "depression", which is one of the mental disorders, is depression and the like, which appears as a facial expression. In particular, it appears in smiles that show positive emotions. Therefore, in order to accurately estimate the mental state of the target person U, it is preferable to use an image of a facial expression suitable for the estimation. For example, Patent Document 1 described above discloses the use of images taken at fixed time intervals or at fixed times, but in that case, the facial expression of the target person U may differ depending on the imaging timing. That is, in Patent Document 1, there is a problem in the determination accuracy for determining the mental state of the target person U.

On the other hand, the mental state estimation system 100 according to the present embodiment includes the first output unit 21 that outputs information for prompting the target person U to smile. As a result, the target person U makes a smile, so the image acquired by the image acquisition unit 22 includes an image in which the target person U is smiling. That is, the mental state estimation system 100 can acquire an image of a facial expression suitable for estimating the mental state. The estimation unit 25 also determines the current mental state of the target person U based on the past smile level and the current smile level of the target person U. For example, as compared with the case where the estimation unit 25 estimates the mental state of the target person U based on a predetermined reference value of the degree of smile, it is possible to suppress a decrease in estimation accuracy due to individual differences in the degree of smile. Therefore, from these, the mental state estimation system 100 can accurately estimate the mental state.

Further, it further includes a threshold value calculation unit 26 that calculates a threshold value from the time-series data stored in the storage unit 24. Then, the estimation unit 25 determines that the mental state is good when the current smile degree is at least one of the threshold value and less than the threshold value, and the stress state is when the current smile degree is at least one of the threshold value and less than the threshold value. presume.

Thereby, even when the index such that the value in the normal state is different such as the smile degree is used, the mental state can be estimated by the threshold value according to the target person U. That is, it is possible to suppress a decrease in estimation accuracy due to individual differences. Therefore, the mental state can be estimated more accurately.

In addition, the facial expression index includes at least two of a smile degree, a smile formation speed, mouth and eye positions, facial symmetry, face angle, and skin color. Then, the estimation unit 25 estimates the mental state based on at least two facial expression indexes.

As a result, the mental state can be estimated more accurately than in the case where there is only one facial expression index.

Further, the second output unit 27 that outputs the estimation result of the estimation unit 25 is further included.

Thereby, the estimation result can be notified. For example, the second output unit 27 outputs the estimation result to the terminal device 40 owned by the target person U, so that the target person U can know the estimation result and can take countermeasures.

Further, as described above, the mental state estimation method according to the present embodiment includes a step (S10) of outputting information for prompting the target person U to smile (an example of a predetermined facial expression), and the target person U to smile. A step (S20) of obtaining an image of the captured image, a step (S30) of extracting the degree of smile of the target person U (an example of a facial expression index) from the image, and Based on the step of storing the smile degree as time series data in the storage unit 24 (S40), the current smile degree extracted in step S30, and the time series data stored in the storage unit 24, the target person U Estimating the mental state of (S60 to S80). Further, it may be a program for causing a computer to execute the above mental state estimation method.

As a result, the same effect as the mental state estimation system 100 is obtained.

(Modification of Embodiment)
Hereinafter, the mental state estimation system and the like according to the present modification will be described with reference to FIG. FIG. 4 is a block diagram showing the configuration of a mental state estimation system 100a according to this modification. In this modification, differences from the embodiment will be mainly described.

As shown in FIG. 4, the mental state estimation system 100a includes an imaging device 10, a mental state estimation device 20a, and a display device 30. The mental state estimation device 20a according to the present modification further includes a behavior index acquisition unit 28 that acquires a behavior index in the mental state estimation device 20 of the embodiment.

The action index acquisition unit 28 acquires the action index of the target person U based on the image acquired by the image acquisition unit 22. The action index includes, for example, at least one of the posture of the subject U, walking speed, stride, physiological data, number of human contacts, personal appearance, and the like.

The posture is the figure of the body when the target person U is smiling. When a person's mental condition deteriorates, a person tends to be stooped. Therefore, the mental state of the target person U can be estimated from the posture. The posture may be represented by an angle indicating the roundness of the back, for example.

The walking speed is a speed at which the target person U is heading toward the imaging device 10, and can be acquired when the imaging device 10 is capturing a moving image. Walking speed tends to decrease when the mental state deteriorates. Therefore, the mental state of the target person U can be estimated from the walking speed.

The stride is a stride when the subject U heads toward the imaging device 10, and can be acquired when the imaging device 10 is capturing a moving image. When the mental condition deteriorates, the stride tends to be narrow. Therefore, the mental state of the target person U can be estimated from the stride length.

The physiological data is data such as the heartbeat of the subject U and can be acquired when the imaging device 10 is capturing a moving image. When the mental state deteriorates, the fluctuation of the heart rate tends to increase (that is, the heart rhythm is disturbed). Therefore, the mental state of the subject U can be estimated from the physiological data.

The number of human contacts is the number of times that the target person U is estimated to have communicated with another person, and can be acquired, for example, when the imaging device 10 is capturing an image including the surroundings of the target person U. Deterioration of mental status tends to reduce the frequency of communication with other people. Therefore, the mental state of the target person U can be estimated from the number of human contacts.

The headlines are the beard, hairstyle, makeup, etc. of the subject U. Deterioration of mental status tends to result in poor grooming. Therefore, it is possible to estimate the mental state of the subject U from the appearance.

The acquisition method (specifically, the image analysis method) by which the behavior index acquisition unit 28 acquires the above-mentioned behavior index does not need to be particularly limited, and a conventional technique may be used. Therefore, detailed description of the acquisition method for acquiring the behavior index will be omitted.

The action index acquisition unit 28 is not limited to acquiring the action index of the target person U based on the image acquired by the image acquisition unit 22. For example, the physiological data of the subject U may be acquired from a vital sensor that measures the physiological data of the subject U. For example, the behavior index acquisition unit 28 may acquire the physiological data based on the measurement result of the wearable device worn by the subject U. In addition, for example, when the target person U is in the office of the company, the number of human contacts of the target person U is obtained from an infrared sensor (an example of a motion sensor) installed in the office (for example, the ceiling of the office). Good. Specifically, the action index acquisition unit 28 may acquire the number of times the target person U has contacted the person based on the position of the person obtained from the sensing result of the infrared sensor or the like.

In addition, the behavior index acquisition unit 28 may acquire the number of people who have come into contact with the target person U, for example, based on the information of the position positioning system that is provided in the office and measures the position of the person in the office. Good. The position positioning system includes, for example, an acquisition device provided on the ceiling of an office and a mobile device that each person in the office possesses and authenticates the person. The acquisition device has a communication unit (for example, a wireless module) that wirelessly communicates with the mobile device. The communication unit transmits a beacon signal to the mobile device and receives a signal transmitted from the mobile device in response to the beacon signal. Specifically, the communication unit transmits, as a beacon signal, a signal instructing to return the identification information of the person who owns the mobile device or the identification information of the mobile device to the mobile device, and the signal transmitted from the mobile device. As, a signal containing the identification information of the person who owns the mobile device or the identification information of the mobile device is received. Accordingly, the acquisition device can detect the number of people who have contacted the target person U in the beacon signal transmission area. The acquisition device may determine that, for example, a person within a predetermined distance range with respect to the target person U is a person who has contacted the target person U.

Further, the action index may include information about the eyes of the target person U. For example, the behavior index may include information about the bear of the eye of the target person U. When a person's mental condition deteriorates, he/she easily becomes insomnia, which tends to cause a bear around the eyes. Therefore, the mental state of the subject U can be estimated from the bear of the eyes.

The behavior index acquisition unit 28 outputs the acquired behavior index to the storage unit 24 and the estimation unit 25. The storage unit 24 stores time-series data of an action index (an example of second time-series data). The storage unit 24 stores, as time-series data (an example of second time-series data), a plurality of action indexes acquired at different times before the present (an example of a target time).

The threshold calculation unit 26 calculates a threshold for estimating the mental state of the subject U from the time-series data of the smile degree and the action index stored in the storage unit 24. That is, the time-series data of the action index is also used in estimating the mental state of the action index.

The estimation unit 25 estimates the mental state of the target person U based on the action index acquired by the action index acquisition unit 28 in addition to the facial expression index (for example, smile degree). In the present modification, the estimation unit 25 further includes, in addition to the facial expression index and the temporary series data, the current activity index acquired by the activity index acquisition unit 28 and the activity index stored in the storage unit 24. The mental state of the subject U is estimated based on the time series data.

The estimation unit 25 estimates the mental state of the target person U, for example, based on the smile degree acquired from the extraction unit 23, the action index acquired from the action index acquisition unit 28, and the threshold value acquired from the threshold value calculation unit 26. .. The estimation unit 25 calculates an index (for example, a stress level) indicating the mental state of the target person U based on the degree of smile and the behavior index, and compares the calculated index with a threshold value to determine the target person U. May estimate the mental state of. The estimation unit 25 may calculate the index indicating the mental state by adding a predetermined weight to the smile degree and the action index, for example. In addition, the estimation unit 25 may estimate the mental state of the target person U based on the result of comparison between the smile degree and the threshold value and the result of comparison between the action index and the threshold value.

As described above, the mental state estimation system 100a according to the present modification further includes the action index acquisition unit 28 that acquires the action index of the target person U. Then, the estimation unit 25 further estimates the mental state of the target person U based on the action index acquired by the action index acquisition unit 28.

Accordingly, the estimation unit 25 can estimate the mental state of the target person U based on the facial expression index and the behavioral index, and thus the mental state of the target person U based on only the facial expression index among the facial expression index and the behavioral index. The mental state can be estimated more accurately than the case of estimating. For example, a typical symptom of "depression", which is one of the mental illnesses, is depression and the like, which appears in behavior in addition to facial expressions. Therefore, in order to more accurately estimate the mental state of the target person U, the information regarding the behavior of the target person U as described above may be used.

In addition, the storage unit 24 further stores a plurality of behavior indexes acquired at different points in time before the present (an example of a target point of time) as time-series data (an example of second time-series data). Then, the estimating unit 25 further determines the mental state of the subject U based on the current action index acquired by the action index acquiring unit 28 and the time-series data of the action index stored in the storage unit 24. presume.

Accordingly, the estimation unit 25 can further estimate the current mental state of the subject U based on the past action index and the current action index of the subject U. For example, as compared with the case where the estimation unit 25 estimates the mental state of the target person U based on a predetermined reference value of the action index, it is possible to suppress a decrease in estimation accuracy due to individual differences in the action index. Therefore, the mental state estimation system 100a can estimate the mental state more accurately.

In addition, the behavior index includes at least one of the posture of the subject U, the walking speed, the stride, and the physiological data.

This makes it possible to further improve the determination accuracy of the mental state by using the action index that can be acquired from the image acquired by the image acquisition unit 22. That is, it is possible to further improve the determination accuracy of the mental state without providing an additional configuration.

(Example of application of mental state estimation system)
Next, an application example of the mental state estimation systems 100 and 100a described in the above embodiment and modification will be described with reference to FIG. Note that, although FIG. 5 shows an example of an application example of the mental state estimation system 100 according to the embodiment, the same can be applied to the mental state estimation system 100a according to the modified example of the embodiment. FIG. 5: is a figure which shows an example of the application example of the mental state estimation system 100 which concerns on embodiment.

As shown in FIG. 5, the mental state estimation system 100 according to the embodiment may be installed in a conference room R of a company or the like. The mental state estimation system 100 captures the facial expressions of all the target persons U (for example, attendees of the conference) in the conference room R before the conference is held in the conference room R, and the smile degree extracted from the captured image, for example. Based on, the mental state of each subject U is estimated. The extraction unit 23 extracts the current smile degree of each of the target persons U from the image, and the estimation unit 25 determines the target smile degree of each of the target persons U based on the current smile degree of each of the target persons U and the threshold value set for each of the target persons U. Estimate the mental state of each person U. With this, it is possible to know whether or not some of the attendees have a worsened mental state before the start of the meeting. It is possible to take measures to prevent the mental condition from being further deteriorated by the stress at the meeting.

Then, the second output unit 27 outputs the estimation result corresponding to each of the target persons U. For example, each of the subjects U can confirm only the estimation result of the mental state of himself/herself. Thereby, the privacy of each target person U is protected.

In addition, the estimation unit 25 calculates a representative value of the smile degree of a plurality of target persons U attending the conference (hereinafter, also referred to as a representative smile degree) from the current smile degree of each of the target persons U. Good. The estimation unit 25 may calculate the representative smile degree by performing a predetermined calculation on the current smile degree of each of the target persons U, and, for example, a statistical value (for example, an average value or the average value of the current smile degree of each of the target persons U). The median value) may be used as the representative smile degree. Further, the estimation unit 25 standardizes the current smile degree for each of the target persons U based on the current smile degree of the target person U and the time-series data of the smile degree of the target person U. May be calculated, and the statistical value of the standard smile degree of each subject U may be used as the representative smile degree. As a result, it is possible to know the mental states of all the target persons U who are attending the conference. For example, when a meeting is held with the target person U who belongs to one department, the departmental trend of the mental state (whether the department is active, etc.) can be known at an early stage by performing the above processing. For example, it is possible to promptly take an appropriate response such as improving the working style in the department from the departmental trend of the mental state.

Moreover, the mental state estimation system 100 according to the embodiment may be installed in an office of a company or the like. In this case, the imaging device 10 may be provided for each individual. Further, at this time, the number of times of contact with a person may be acquired as the action index. As a result, it is possible to obtain the number of contacts of people in the entire office, and it is possible to estimate whether or not the office is active (that is, whether or not the person in the office is in a good mental state). Become.

Further, the mental state estimation system 100 according to the embodiment may be used as an entry/exit management system (for example, attendance system) when entering or leaving an office in a company or the like. For example, the imaging device 10 and the display device 30 are arranged near the door, and the subject U makes a smile in order to unlock the door. Accordingly, it is possible to acquire the record of entry and exit of the subject U and the estimation result of the mental state.

Moreover, the mental state estimation system 100 according to the embodiment may be used as a control system for controlling an electric device (for example, a lighting device). For example, the target person U makes a smile in order to turn on and off the lighting device. Thereby, the control of the electric device and the estimation result of the mental state can be acquired. Further, the output of the electric device (for example, dimming and color adjustment of the lighting device) may be controlled according to the estimation result of the mental state.

Moreover, the mental state estimation system 100 according to the embodiment may be used in a school, a nursing facility, or the like. For example, by using the mental state estimation system 100 in a school, a student whose mental state has deteriorated due to bullying or the like can be detected early. Further, for example, by being used in a care facility or the like, the mental state estimation system 100 can be used as a watching system. For example, the terminal device 40 owned by the family (for example, a child) of the target person U who lives apart from the target person U based on the estimation result of the mental state of the target person U (for example, a parent) who lives in a nursing home. By outputting to, the family U can watch the target person U according to the estimation result even if the family lives in a distant place.

(Other embodiments)
As described above, the mental state estimation system, the mental state estimation method, and the program according to one or more aspects of the present invention are described based on the embodiments and modifications (hereinafter, also referred to as embodiments, etc.). However, the present invention is not limited to the above embodiments and the like.

For example, in the above-described embodiments and the like, the mental state estimation device does not have the imaging device and the display device, but may have at least one of the imaging device and the display device. In this case, the imaging device functions as an imaging unit that is a part of the mental state estimation device, and the display device functions as a display unit that is a part of the mental state estimation device. When the mental state estimation device has a display unit, the second output unit may output the estimation result to the display unit.

Further, the estimation unit in the above-described embodiments and the like may further estimate the mental state of the subject based on a questionnaire (for example, stress check) regarding the subject's mental state, attendance information, and the like.

In addition, the estimation unit in the above-described embodiment or the like greatly decreases the current smile degree with respect to the time-series data (for example, the latest smile degree acquired) (that is, the mental state deteriorates sharply). ), and when it is determined that the current smile degree is lower than the threshold value, the second output unit may output information indicating that the mental state is rapidly deteriorating to the terminal device. That is, the estimation unit may estimate the mental state based on the degree of change in the degree of smile. The most recently acquired smile degree is, for example, the smile degree for several times acquired at the time closest to the present time. As a result, it is possible to notify the subject that the mental condition is rapidly deteriorating, although the threshold has not been reached.

Further, in the above-described embodiment and the like, the second output unit outputs whether the mental state is good or bad, but the present invention is not limited to this. The second output unit may output, for example, the result of digitizing the mental state (for example, 0 to 100).

Further, in the above-described embodiments and the like, an example in which the second output unit outputs the estimation result to the terminal device has been described, but the present invention is not limited to this. The second output unit may output the estimation result to the display device. Thereby, the subject can know the estimation result without operating the terminal device.

Further, in the above embodiment, an example in which the threshold value calculation unit calculates one threshold value has been described, but the present invention is not limited to this. The threshold calculator may calculate two or more thresholds. For example, the estimation unit can estimate the degree of deterioration of the mental state of the target person by calculating two or more thresholds when the threshold calculation unit has a low degree of smile. Further, for example, the threshold calculation unit calculates the thresholds both when the smile level is high and when the smile level is low, so that the estimation unit determines that the mental state of the subject is deteriorated and that the mental state of the subject is normal. It can be estimated that the situation is better than time.

Further, in the above-described embodiments and the like, the example in which the mental state estimation system includes one image pickup device, one display device, and one terminal device has been described, but the number is not particularly limited and may be two or more. ..

Moreover, the order of the processes described in the flowcharts of the above-described embodiments and the like is an example. The order of the plurality of processes may be changed, or the plurality of processes may be executed in parallel.

Further, the communication method between the devices in the above-described embodiments and the like is not particularly limited. Wireless communication may be performed between devices, or wired communication may be performed. In addition, wireless communication and wired communication may be combined between the devices. Further, when two devices communicate with each other in the above embodiment, a relay device (not shown) may be interposed between the two devices.

Further, in the above-described embodiments and the like, each component may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

Further, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may constitute one circuit as a whole or may be separate circuits. Further, each of these circuits may be a general-purpose circuit or a dedicated circuit.

Further, the mental state estimation system has been described as an example realized as a single device (for example, the mental state estimation device according to the embodiment), but may be realized by a plurality of devices. When the mental state estimation system is realized by a plurality of devices, the components included in the mental state estimation system may be distributed to the plurality of devices in any way.

Further, the general or specific aspects of the present invention may be realized by a recording medium such as a system, a device, a method, an integrated circuit, a computer program, or a computer-readable CD-ROM. Further, the system, the device, the method, the integrated circuit, the computer program, and the recording medium may be implemented in any combination.

For example, the present invention may be realized as a program for causing a computer to execute a mental state estimation method executed by a computer such as a mental state estimation system. When the mental state estimation system is implemented as a general-purpose mobile terminal, the present invention may be implemented as an application program installed in the mobile terminal in order to operate the general-purpose mobile terminal as the mental state estimation system. .. Further, the present invention may be realized as a computer-readable non-transitory recording medium in which such a program is recorded.

In addition, it is realized by making various modifications to those skilled in the art that are conceivable to those skilled in the art, or by arbitrarily combining the components and functions in the embodiments without departing from the spirit of the present invention. The form is also included in the present invention.

21 1st output part 22 Image acquisition part 23 Extraction part 24 Storage part 25 Estimation part 26 Threshold value calculation part 27 Second output part 28 Behavior index acquisition part 100,100a Mental state estimation system U Target person

Claims (9)

A first output unit for outputting information for prompting the subject to have a predetermined facial expression,
An image acquisition unit that acquires an image of the target person having the predetermined facial expression,
An extraction unit that extracts the facial expression index of the subject from the image,
A storage unit that stores the facial expression index at a plurality of time points before the target time point as first temporary series data,
An estimation unit that estimates the mental state of the subject at the target time point based on the facial expression index at the target time point extracted by the extraction unit and the temporary series data stored in the storage unit. And a mental state estimation system. Further comprising an action index acquisition unit that acquires the action index of the target person,
The mental state estimation system according to claim 1, wherein the estimation unit further estimates the mental state of the subject based on the action index acquired by the action index acquisition unit. The storage unit further stores a plurality of the behavior indicators acquired at different time points before the target time point as second time series data,
The estimation unit further, based on the behavior index at the target time point acquired by the behavior index acquisition unit, and the second time series data stored in the storage unit, the mental state of the target person. The mental state estimation system according to claim 2. The mental state estimation system according to claim 2, wherein the behavior index includes at least one of a posture, a walking speed, a stride, and physiological data of the subject. Further comprising a threshold value calculation unit for calculating a threshold value from the temporary series data stored in the storage unit,
The estimation unit, if the facial expression index at the target time point is one of the threshold value or more and less than the threshold value, the mental state is good, and the facial expression index at the target time point is the threshold value or more and less than the threshold value on the other side. The mental state estimation system according to any one of claims 1 to 4, wherein it is estimated that the person is in a stressed state. The facial expression index includes at least two of a smile degree, a smile formation rate, mouth and eye positions, facial symmetry, face angle, and skin color,
The mental state estimation system according to claim 1, wherein the estimation unit estimates a mental state based on at least two facial expression indexes. The mental state estimation system according to claim 1, further comprising a second output unit that outputs the estimation result of the estimation unit. A step of outputting information for prompting the target person to have a predetermined facial expression,
Acquiring an image of the target person having the predetermined facial expression;
Extracting the facial expression index of the subject from the image,
Storing a plurality of the facial expression indexes at a plurality of time points before the target time point in the storage unit as time series data,
Estimating the mental state of the subject based on the facial expression index at the target time point and the time-series data stored in the storage unit. A program for causing a computer to execute the mental state estimation method according to claim 8.

Images