JP7843663B2 - Estimation device, estimation system, and estimation method - Google Patents

Description

This invention relates to a technique for estimating emotions.

Conventionally, devices are known that estimate a person's emotions based on physical quantities acquired by sensors such as cameras and heart rate monitors, and then present the estimated emotions (see, for example, Patent Document 1).

Japanese Patent Publication No. 2017-73107

Incidentally, biological information is easily influenced by environmental factors and physical condition. This tendency is particularly pronounced in electroencephalography (EEG). Therefore, it is possible that a person being subjected to emotion estimation may be in an unsuitable state due to environmental influences. When emotion estimation is performed in an unsuitable state, the likelihood of inaccurate emotion estimation increases.

In view of the above points, the present invention aims to provide a technology that enables emotion estimation in a state suitable for emotion estimation.

An exemplary estimation device of the present invention comprises a controller. The controller provides the user with multiple types of tasks. The controller acquires index values based on biosignals during the execution of each of the multiple types of tasks. Based on the relationships between the multiple index values, the controller estimates the user's suitability for emotion estimation.

According to an exemplary version of the present invention, emotion estimation can be performed in a state suitable for emotion estimation.

A diagram showing an example of the estimation system configuration. A diagram showing an example of an emotion estimation model (psychological plane). This figure shows an example of the configuration of the estimation device according to the first embodiment. A diagram showing an example of a sensor table. A diagram showing an example of a psychological planar table. A diagram showing an example of a special task table. This figure schematically shows the time-dependent changes in indicator values of physiological responses during the execution of the first and second tasks. A diagram that simulates the time evolution of physiological responses. A schematic diagram showing the time evolution of the estimated upper limit of the neutral region. A diagram showing an example of a neutral region table. A diagram showing an example of a psychological plane that includes a neutral zone. A flowchart showing an example of the preliminary stage of the estimation process performed by the estimation device of the first embodiment. A flowchart showing an example of the later stage of the estimation process performed by the estimation device of the first embodiment. A diagram illustrating a screen displaying the results of an emotion estimation. A diagram illustrating a screen displaying the results of an emotion estimation. A flowchart illustrating another example of the preliminary stage of the estimation process performed by the estimation device of the first embodiment. This figure shows a schematic configuration of a modified example of the estimation system of the first embodiment. This figure shows an example configuration of the estimation device according to the second embodiment. A diagram showing an example of a task table for aptitude estimation. A flowchart showing an example of the suitability estimation process performed by the estimation device of the second embodiment. A flowchart showing a modified example of the suitability estimation process performed by the estimation device of the second embodiment. A diagram showing an example of an aptitude level table. Example screen for informing users that they are unsuitable for emotion estimation. This diagram shows an example of displaying the estimated emotion on the screen. A flowchart showing other variations of the suitability estimation process performed by the estimation device of the second embodiment.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

<1. First Embodiment>
[1-1. Estimation System]
Figure 1 shows an example of the configuration of the estimation system 1 according to the first embodiment of the present invention. In this embodiment, user U1 is an e-sports player. User U1 is the person whose emotions are to be estimated. That is, the estimation system 1 is configured as a system for estimating the emotions of e-sports player U1.

Note that while Figure 1 shows only one user U1, there may be multiple users U1. Furthermore, user U1 may not be an e-sports player. For example, user U1 could be a patient in a medical institution, a student in an educational institution, a vehicle driver, or a viewer of content such as video or music.

As shown in Figure 1, the estimation system 1 comprises a server 10, a terminal device 20, a biosensor 30, and a game device 40. The server 10 and the terminal device 20 are connected via a network N. Network N is, for example, the internet or an intranet. The terminal device 20, the biosensor 30, and the game device 40 are provided to communicate via wired or wireless connections. The terminal device 20, the biosensor 30, and the game device 40 are connected according to communication standards such as Wi-Fi® or Bluetooth®.

Server 10 may be a physical server or a virtual server. In this embodiment, Server 10 constitutes an estimation device for estimating emotions. That is, the estimation system 1 includes the estimation device 10. Hereafter, Server 10 will be referred to as the estimation device 10. Details of the estimation device 10 will be described later. Note that the estimation device 10 may be composed of one server or multiple servers.

The terminal device 20 is, for example, a personal computer, a smartphone, or a tablet computer. In this embodiment, the terminal device 20 is used by operator O1. However, the terminal device 20 may also be configured for use by user U1. In this case, the terminal device 20 may also function as the game device 40. Furthermore, there may be multiple terminal devices 20, for example, if there are multiple users U1.

The biosensor 30 is attached to user U1 and detects user U1's biological signals as sensor signals. That is, the estimation system 1 includes a biosensor 30 that measures biological signals. In this embodiment, the biosensor 30 includes a first sensor 31 and a second sensor 32. The first sensor 31 is a headgear-type electroencephalogram (EEG) sensor. The second sensor 32 is a wristband-type optical heart rate (pulse) sensor. However, the first sensor 31 and the second sensor 32 may be changed to other biosensors depending on the biological information to be acquired. Other biosensors may be, for example, an electrocardiogram (ECG) heart rate sensor, a blood pressure monitor, or a NIRS (Near Infrared Spectroscopy) device.

The game device 40 comprises an operation unit 41 for user U1 to perform game-related operations, and a display unit 42 having a display screen. The game device 40 appropriately transmits information about user U1 operating the device to the terminal device 20. The game device 40 also appropriately receives information from the terminal device 20.

Now, referring to Figure 1, we will explain the processing flow of the estimation system 1.

The first sensor 31 and the second sensor 32 measure the user U1's biosignals and output the measurement results as sensor signals to the terminal device 20 (step S1). The terminal device 20 transmits the input sensor signals to the estimation device (server) 10. The estimation device 10 estimates the user U1's emotional state based on the input sensor signals (step S3).

Here, we will explain the overview of emotion estimation in step S3. The estimation device 10 generates index values, which are values of indicators (physiological responses) that represent the mental and physical state, based on the input sensor signals. In this embodiment, the estimation device 10 generates index values for two indicators of mental and physical state related to brain waves and heart rate. For example, the indicator of mental and physical state related to brain waves is the level of central nervous system arousal (hereinafter simply referred to as arousal), and its index value can be given as "beta waves/alpha waves of the brain wave." Also, for example, the indicator of mental and physical state related to heart rate is the activity level of the autonomic nervous system, and its index value can be given as "standard deviation of the heart rate LF component." The index values are calculated using a calculation model (calculation formula or conversion data table) that is pre-stored in memory or the like.

Furthermore, the first sensor 31 and the second sensor 32 may be configured to have calculation functions (a configuration with a built-in computer, etc.), and the sensors 31 and 32 may be configured to calculate the index value. Alternatively, the terminal device 20 may be configured to calculate the index value based on the sensor signals input from the first sensor 31 and the second sensor 32.

The estimation device 10 estimates the user U1's emotions using the calculated index values and an emotion estimation model pre-stored in memory or similar. The emotion estimation model is created based on medical evidence (such as research papers).

Figure 2 shows an example of an emotion estimation model (psychological plane). According to various medical evidence related to psychology, psychology can be estimated based on two types of indicators that represent physical states. The psychological plane shown in Figure 2, which uses two types of mental and physical state indicators as axes, is an example of an emotion estimation model in accordance with the aforementioned technological concept. In Figure 2, for example, the vertical axis represents "arousal level (arousal - unarousal)," and the horizontal axis represents "autonomic nervous system activity level (sympathetic nervous system activity (strong emotion) - parasympathetic nervous system activity (weak emotion))."

In the psychological plane shown in Figure 2, each of the four quadrants separated by the vertical and horizontal axes is assigned a corresponding psychological state. The distance from each axis indicates the intensity of the corresponding psychological state. In the example in Figure 2, the psychological states of "happy, joyful, angry, and sad" are assigned to the first quadrant. The psychological state of "depressed" is assigned to the second quadrant. The psychological state of "relaxed and calm" is assigned to the third quadrant. The psychological state of "anxious, fearful, and unpleasant" is assigned to the fourth quadrant.

By plotting two types of mental and physical state indicators obtained based on the measurement results of biosignals onto a psychological plane, it is possible to estimate the psychological state from the resulting coordinates. Specifically, the psychological state and its intensity can be estimated based on which quadrant the plotted coordinates lie in on the psychological plane and their distance from the origin.

In this embodiment, the emotion estimation model shown in Figure 2 is calibrated, and the emotion is estimated using the calibrated emotion estimation model. This will be described later. Also, in the example shown in Figure 2, the emotion estimation model is a two-dimensional plane, but it may be a three-dimensional or higher space.

Returning to Figure 1, the estimation device 10 performs emotion estimation and then provides (transmits) the obtained estimation result to the terminal device 20 (step S4). For example, the terminal device 20 displays the received emotion information of user U1 on its screen based on the operation of operator O1. Alternatively, the terminal device 20 displays the received emotion information of user U1 on the display unit 42 of the game device 40. Furthermore, the terminal device 20 provides the received emotion information of user U1 to an external game system.

The results of the emotion estimation can be used, for example, for the mental training of user U1 in esports. For instance, if user U1 experiences unfavorable emotions (anxiety, anger, etc.) during video game play, it is determined that intensive training corresponding to that emotional state is necessary. In this training, the emotion information of user U1 estimated by estimation system 1 is used.

Furthermore, various types of esports include those where multiple users (U1) cooperate and those where multiple users (U1) compete against each other. In these types of esports, if the emotional state of each player is displayed, it becomes possible to perform advanced gameplay, such as changing game tactics based on the emotional state. It also becomes possible for esports spectators to understand the emotional state of each player while watching the game, increasing the enjoyment of watching the game.

Furthermore, if the target of emotion estimation by estimation system 1 (user U1) is a patient in a medical institution, the estimated emotion can be used for examinations and treatments. For example, medical institution staff can understand that the patient is feeling anxious and implement countermeasures such as counseling. Note that the medical institution staff may be just one example of operator O1 (see Figure 1).

Furthermore, if the target of emotion estimation by estimation system 1 (user U1) is a student in an educational institution, the estimated emotion can be used to improve the lesson content. For example, a teacher can understand that a student is finding the lesson boring and improve the lesson content to make it more interesting for the student. Note that the teacher may be an example of operator O1 (see Figure 1).

Furthermore, if the target of emotion estimation by the estimation system 1 (user U1) is the vehicle driver, the estimated emotion can be used to promote safe driving. For example, the in-vehicle device can detect that the driver is not feeling adequate tension while driving and output a message encouraging them to concentrate on driving. Note that the in-vehicle device may be an example of a terminal device 20.

Furthermore, if the target of emotion estimation by estimation system 1 (user U1) is a viewer of content such as video or music, the estimated emotions can be used to create further content. For example, a video content distributor can create a highlight video by compiling scenes that viewers found enjoyable. Note that the video content distributor may be just one example of operator O1 (see Figure 1).

In the above example, the server 10 is shown as the estimation device, but for example, the terminal device 20 or the game device 40 used by the e-sports player may also be the estimation device. Furthermore, the estimation device may consist of multiple devices. For example, the server 10, terminal device 20, and game device 40, which are connected via communication, may distribute the processing for estimating emotions.

[1-2. Estimation device]
Figure 3 shows an example configuration of an estimation device 10 according to the first embodiment of the present invention. In Figure 3, only the components necessary to explain the features of this embodiment are shown, and descriptions of general components are omitted. As shown in Figure 3, the estimation device 10 includes a communication unit 11 and a storage unit 12. The estimation device 10 also includes a controller 13. The estimation device 10 may be a so-called computer device. The estimation device 10 may also be configured to include an input device such as a keyboard and an output device such as a display.

The communication unit 11 is an interface for communicating data with other devices via the network N. The communication unit 11 is, for example, a NIC (Network Interface Card).

The storage unit 12 is configured to include volatile memory and non-volatile memory. The volatile memory may include, for example, RAM (Random Access Memory). The non-volatile memory may include, for example, ROM (Read Only Memory), flash memory, or a hard disk drive. The non-volatile memory stores programs and data that can be read by a computer. At least a portion of the programs and data stored in the non-volatile memory may be acquired from other computer devices connected via wired or wireless connections, or from a portable recording medium.

As shown in Figure 3, in this embodiment, the storage unit 12 includes a table (data table) 121. More specifically, table 121 includes multiple tables for various processing. For example, table 121 includes a sensor table 121a and a psychological plane table 121b. Figure 4A shows an example of the sensor table 121a. Figure 4B shows an example of the psychological plane table 121b.

As shown in Figure 4A, the sensor table 121a includes the following items: "Sensor ID," "Sensor Type," "Biometric Signal Type," "Corresponding Indicator ID," "Corresponding Indicator Type," and "Indicator Conversion Information." Note that the table items correspond to data storage cells (storage frames).

The "Sensor ID" field in sensor table 121a stores the sensor ID data, which is identification information used to identify data records in sensor table 121a. The sensor ID data also serves as the primary key for the data records in sensor table 121a. In other words, in sensor table 121a, a data record is created for each sensor ID data, and the data for each item associated with the sensor ID data is stored in that data record.

The "Sensor Type" item in sensor table 121a stores information to identify the sensor type. In this example, the sensor name (or model number) is stored.

The "Biological Signal Type" item in sensor table 121a stores the type of measurement value based on the biological signal detected by the sensor. This biological signal type data is correlated with the corresponding index type data. Academically, it is recognized that the corresponding index type data can be estimated (calculated) by obtaining the corresponding biological signal type data.

The "Corresponding Indicator ID" item in sensor table 121a stores identification information for identifying the mental/physical state index generated (calculated) based on the signal from the sensor that detects biological signals. The "Corresponding Indicator Type" item in sensor table 121a stores the type (name, etc.) of the index.

The "Index Conversion Information" item in sensor table 121a stores conversion information (such as calculation formulas and conversion data tables) for calculating index values based on signals obtained from sensors that detect biological signals. In other words, by converting the biological signals detected by the sensor corresponding to the sensor ID data according to the index conversion information, the index value of the mental and physical state index identified by the corresponding index ID is estimated (calculated).

For example, the data record for sensor ID "SN01" in the sensor table 121a shown in Figure 4A contains the following information: The output signal of "EEG sensor BA" measures "beta/alpha waves of the brainwave." Then, by converting these "beta/alpha waves of the brainwave" using the index conversion information of "FX01," an index value for "alertness level" is obtained.

As shown in Figure 4B, the psychological plane table 121b is a two-dimensional matrix table where the indicator type (specifically, indicator ID data) is used as the parameter on the vertical and horizontal axes. In the psychological plane table 121b, data for psychological plane types usable with two types of indicator data is stored in memory cells (memory frames) determined by the two types of indicator types. For example, if the indicator types used are indicator type VSm and indicator type VSn, the psychological plane used for emotion estimation will be psychological plane mn. Information for processing using psychological plane mn is then read out and used for emotion estimation processing.

Furthermore, a common index ID is used in both the sensor table 121a and the psychological plane table 121b. That is, based on the sensor table 121a and the psychological plane table 121b, the psychological plane mn corresponding to the two types of sensors attached to user U1 can be determined. For example, suppose the sensor types attached to user U1 are "EEG sensor BA (index ID: VS01)" and "heart rate sensor HA (index ID: VS02)". In this case, the index ID data corresponding to "EEG sensor BA (index ID: VS01)" and "heart rate sensor HA (index ID: VS02)" is determined based on the sensor table 121a. Then, based on the psychological plane table 121b, psychological planes 01-02, using "alertness level" (VS01) and "autonomic nervous system activity level" (VS02) as indices, are determined as the psychological planes used for emotion estimation.

Returning to Figure 3, the controller 13 includes a processor that performs calculations and other processing. The processor may include, for example, a CPU (Central Processing Unit). The controller 13 may consist of one processor or multiple processors. If it consists of multiple processors, they should be connected to each other in a way that allows them to communicate. Note that if the estimation device 10 is configured as a cloud server, the CPU constituting the processor may be a virtual CPU.

As shown in Figure 3, the controller 13 comprises an acquisition unit 131, a region determination unit 132, an emotion estimation unit 133, and a provision unit 134. In this embodiment, the functions of the controller 13 are realized by the processor executing calculations according to the program stored in the storage unit 12.

Furthermore, the scope of this embodiment may include a computer program that enables a processor (computer) to implement at least some of the functions of the estimation device 10. The scope of this embodiment may also include a computer-readable non-volatile recording medium for recording such a computer program. The non-volatile recording medium may be, for example, the non-volatile memory described above, an optical recording medium (e.g., an optical disc), a magneto-optical recording medium (e.g., a magneto-optical disc), a USB memory, or an SD card.

Furthermore, each functional unit 111-114 may be implemented by a single program, but for example, each functional unit may be implemented by a separate program. Also, each functional unit 111-114 may be implemented as a separate server. Furthermore, each functional unit 111-114 may be implemented by having a processor execute a program, i.e., by software, as described above, but may also be implemented by other methods. At least a portion of each functional unit 111-114 may be implemented using, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). In other words, each functional unit 111-114 may be implemented by hardware using a dedicated IC, etc. Also, each functional unit 111-114 may be implemented using a combination of software and hardware. Furthermore, each functional unit 111-114 is a conceptual component. The functions performed by one component may be distributed among multiple components. Also, the functions of multiple components may be integrated into a single component.

The acquisition unit 131 receives data (sensor signals) measured by the first sensor 31 and the second sensor 32 from the terminal device 20 via the communication unit 11. The acquisition unit 131 stores the received data in the storage unit 12 as needed for subsequent processing.

The domain determination unit 132 performs preparatory processing for calibration to make the emotion estimation model (in this example, the psychological plane) suitable for estimating the emotions of each user U1. Before explaining the specific details of this preparatory processing, we will first explain why calibration is necessary.

In the psychological plane shown in Figure 2 above, each index based on biological signals (vertical and horizontal axes) is divided into two regions. When using such a psychological plane, for example, the following occurs: The index value representing arousal is classified into either "aroused" or "non-aroused," and the index representing autonomic nervous system activity is classified into either "sympathetic nervous system activity (strong emotion)" or "parasympathetic nervous system activity (weak emotion)." However, taking arousal as an example, a person's mental and physical state includes not only states of arousal and non-arousal, but also a neutral state that cannot be classified as either. Furthermore, the range of this neutral state generally varies from person to person. The same can be said for autonomic nervous system activity. Therefore, simply using the psychological plane shown in Figure 2 to estimate emotions may lead to errors in the estimation. Considering these points, this embodiment uses a configuration in which the psychological plane, which is an emotion estimation model, is calibrated before use.

The region determination unit 132, as a preparatory process for calibration, divides each index based on biological signals into three regions: a neutral region, a positive region, and a negative region. In other words, the region determination unit 132 is configured to perform a region determination process to determine the neutral region in order to divide each physiological response constituting the vertical and horizontal axes of the psychological plane into the three regions described above.

The neutral region is a region where physiological responses (physical and mental states) are ambiguous and cannot be specifically identified. The positive region is the region opposite to the neutral region, where the index value is higher than that of the neutral region. The negative region is the region opposite to the neutral region, where the index value is lower than that of the neutral region. Physiological responses are clear in both the positive and negative regions. For example, if the type of physiological response is arousal level, the positive region indicates an aroused state, and the negative region indicates a non-aroused state. For example, if the type of physiological response is autonomic nervous system activity, the positive region indicates a state where the sympathetic nervous system is active, and the negative region indicates a state where the parasympathetic nervous system is active.

The following describes, using two examples, the region determination process by which the region determination unit 132 determines the neutral region for each indicator (physiological response).

In the first example of the region determination process, user U1 is instructed to perform a special task, and the neutral region is determined according to the measurement results of the biological signals during the execution of that task. The special tasks are determined based on medical evidence and are pre-prepared for each type of physiological response. The pre-prepared special tasks are stored in the storage unit 12 as a special task table 121c. That is, in this embodiment, table 121 (see Figure 3) includes the special task table 121c.

Figure 5 shows an example of a special task table 121c. As shown in Figure 5, the items in the special task table 121c include "Task ID," "Sensor Type," "Corresponding Indicator Type," "Task Type," and "Task Content." The items "Sensor Type" and "Corresponding Indicator Type" are the same as those in Figure 4A above, so their explanation is omitted. Note that "Corresponding Indicator Type" may also be expressed as "Corresponding Physiological Response Type."

The "Task ID" field in the special task table 121c stores ID data, which is identification information used to identify task information in the special task table 121c.

The "Task Type" item in Special Task Table 121c stores whether the task is a first-class task (which elicits a large physiological response) or a second-class task (which elicits a small physiological response). A large physiological response results in a large index value, while a small physiological response results in a small index value. Whether a task elicits a large or small physiological response is determined based on medical evidence. For example, if the physiological response type is arousal level, tasks that increase arousal are first-class tasks, and tasks that decrease arousal are second-class tasks. Similarly, if the physiological response type is autonomic nervous system activity level, tasks that activate the sympathetic nervous system are first-class tasks, and tasks that activate the parasympathetic nervous system (i.e., deactivate the sympathetic nervous system) are second-class tasks.

The "Task Content" item in the special task table 121c stores the specific details of the task to be performed by user U1. The task content is determined based on medical evidence along with the task type mentioned above. For example, if the type of physiological response is arousal level, the task content of the first task is "mentally add the displayed numbers." The number of mental additions may be, for example, 10. Also, for example, if the type of physiological response is arousal level, the task content of the second task is "remain still and clench your fist." The time spent clenching your fist and remaining still may be, for example, 3 minutes. Also, for example, if the type of physiological response is autonomic nervous system activity level, the task content of the first task is "standing test." In the standing test, for example, the user is required to remain still in a supine position with eyes open for a predetermined time, and then raise their upper body and remain still in a long sitting position with eyes open for a predetermined time. The predetermined time may be, for example, 3 minutes. Also, for example, if the type of physiological response is autonomic nervous system activity level, the task content of the second task is "Aschnell test." The Aschnell test, for example, requires the subject to gently press one eyelid with the pads of their index and middle fingers for a predetermined time while their eyes are closed and at rest. Another variation of the Aschnell test may involve wearing an eye mask for a predetermined time, such as three minutes.

As can be seen from the contents of the special task table 121c, determining the neutral region for each physiological reaction requires the user U1 to execute a first task and a second task for each type of physiological reaction. The region determination unit 132 (i.e., the controller 13) performs the following processing when determining the neutral region for each physiological reaction. First, the region determination unit 132 instructs user U1 to execute a first task that is expected to increase the index value obtained during task execution when compared to the second task, and a second task that is expected to decrease it. Then, the region determination unit 132 determines the neutral region based on the index values obtained from the execution of the first and second tasks. Once the neutral region is determined, the positive and negative regions are automatically determined. In other words, the physiological reaction (index) can be divided into three regions. Furthermore, there is a time difference between the timing of the first and second tasks performed for each type of physiological reaction, and the physiological reaction is measured for each task.

In this configuration, before emotion estimation, user U1 is required to perform a special task, and the positive, neutral, and negative regions are determined based on the results. This allows for the division of physiological responses (indicators) into three regions, appropriately reflecting user U1's current environment. In other words, it is expected that emotion estimation will be performed appropriately.

A specific example of the neutral region determination method described above will be explained with reference to Figure 6. Figure 6 schematically shows the time change of the index value of the physiological response during the execution of the first and second tasks. In Figure 6, the horizontal axis represents time, and the vertical axis represents the physiological response (index). In Figure 6, signal α corresponds to the time change of the index value obtained during the execution of the first task. Signal β corresponds to the time change of the index value obtained during the execution of the second task. In Figure 6, the special tasks are executed in the order of the first task followed by the second task, but this is an example, and the special tasks may also be executed in the order of the second task followed by the first task. Also, in Figure 6, the region indicated by arrow NR (a part of the vertical axis) corresponds to the neutral region.

The region determination unit 132 statistically processes the time-varying data of the index value obtained during the execution of the first task to determine the upper limit of the neutral region (NR). For example, the region determination unit 132 smooths the time-varying data of the index value obtained during the execution of the first task, and sets the minimum value of the data after smoothing as the upper limit of the neutral region (NR). Furthermore, the region determination unit 132 statistically processes the time-varying data of the index value obtained during the execution of the second task to determine the lower limit of the neutral region (NR). For example, the region determination unit 132 smooths the time-varying data of the index value obtained during the execution of the second task, and sets the maximum value of the data after smoothing as the lower limit of the neutral region (NR). Note that the smoothing process may be performed, for example, to remove noise components such as minute peak signals.

In other words, the region determination unit 132 (i.e., the controller 13) determines the boundary between the neutral region and the positive region based on the index value obtained during the execution of the first task. Furthermore, the region determination unit 132 determines the boundary between the neutral region and the negative region based on the index value obtained during the execution of the second task. This configuration is suitable when two tasks (the first task and the second task) with large differences in the magnitude of physiological responses can be prepared.

The region determination unit 132 determines the upper and lower limits of the neutral region for each type of physiological response. For this purpose, in this embodiment, the user U1 is requested to perform special tasks consisting of a first task and a second task for each of the two physiological responses (arousal level and autonomic nervous system activity level). Then, for each of the two physiological responses, the neutral region determination process is performed according to the measurement results of the biological signals during the execution of the special task.

Furthermore, the above assumes that the first fluctuation range (the range of fluctuation of the index value obtained during the execution of the first task) and the second fluctuation range (the range of fluctuation of the index value obtained during the execution of the second task) are separate, and the area between the two fluctuation ranges is defined as the neutral region (NR) (see Figure 6). The fluctuation range of the index value refers to the range of value fluctuations associated with the time change of the index value. However, for example, if the difference in the magnitude of physiological responses between the first and second tasks is small, the first and second fluctuation ranges may overlap. In such cases, for example, the minimum value of the data obtained by smoothing the time-varying data of the index value obtained during the execution of the first task may be used as the lower limit of the neutral region (NR). Alternatively, for example, the maximum value of the data obtained by smoothing the time-varying data of the index value obtained during the execution of the second task may be used as the upper limit of the neutral region (NR).

In other words, the region determination unit 132 (i.e., the controller 13) may define the boundary between the neutral region and the positive region based on the index value obtained during the execution of the second task. The region determination unit 132 may also define the boundary between the neutral region and the negative region based on the index value obtained during the execution of the first task. To put it another way, the region where the first and second variation ranges overlap may be determined as the neutral region NR. Alternatively, the neutral region NR may be determined based on the region where the first and second variation ranges overlap. By adopting this configuration, the constraints on determining the first and second tasks can be relaxed, making it easier to set up special tasks.

Next, we will describe a second example of the region determination process for determining the neutral region. In this second example, we do not request user U1 to perform any special tasks. In this second example, we determine the neutral region by focusing on the fact that, as time-series data of physiological responses accumulates, the upper and lower limits of the neutral region estimated by statistical processing converge to a certain value. Once the neutral region is determined, the positive and negative regions are automatically determined. That is, once the neutral region is determined, the physiological response can be divided into three regions.

In other words, in the second example, the region determination unit 132 (i.e., the controller 13) accumulates time-series data of the indicator values and determines the neutral region based on the statistical processing results of the time-series data. With this configuration, there is no need to request the user U1 to perform a special task before emotion estimation, thus reducing the burden on the user U1.

A method for determining the neutral region without special tasks will be explained with reference to Figures 7 and 8. Figure 7 is a schematic diagram showing the temporal change of a physiological response. In Figure 7, the horizontal axis represents time, and the vertical axis represents the physiological response (indicator). Figure 8 is a schematic diagram showing the temporal change of the estimated upper limit of the neutral region (see Figure 7). In Figure 8, the horizontal axis represents time, and the vertical axis represents the estimated upper limit of the neutral region. Note that the temporal change of the estimated lower limit of the neutral region shows a similar temporal change to the upper limit shown in Figure 8. That is, the vertical axis of the graph shown in Figure 8 can be interpreted as the lower limit.

As shown in Figure 7, a person's physiological responses (which can also be called their mental and physical states) change in various states over time (based on changes in the environment, etc., that occur over time), even without being given a special task. More specifically, physiological responses fluctuate between high, low, or ambiguous states (neutral states) over time. Because physiological responses shift up and down due to external stimuli, the distribution of physiological response values becomes statistically distinctive. For example, since there are often no external stimuli, physiological response values usually fall within the neutral region. When stimuli are present, physiological response values tend to fluctuate sharply. By accumulating time-series data of physiological responses and performing cluster analysis to divide the accumulated data into the three states mentioned above, the upper and lower limits of the neutral region can be estimated. Any known method can be used for cluster analysis. For example, k-means, Otsu's multi-level model, and Gaussian mixture models can be used as cluster analysis methods.

In Figure 8, it is assumed that the amount of accumulated time-series data on physiological responses increases over time. That is, in Figure 8, the amount of data used to estimate the upper limit of the neutral region using statistical methods (cluster analysis in a detailed example) is assumed to increase over time. As the amount of data increases over time, the estimated upper limit converges to a constant value, as shown in Figure 8. This trend is also observed for the lower limit, as described above. The neutral region identified by the converged upper and lower limits is considered highly reliable. Therefore, in this example, the neutral region is determined when it is determined that the estimated upper and lower limits have converged.

Furthermore, whether or not the upper limit has converged can be determined based on a comparison between the currently estimated upper limit and the previously estimated upper limit. For example, if the ratio of the two is within a predetermined range relative to 1, it can be determined that convergence has occurred. In this example as well, the region determination unit 132 determines the upper and lower limits of the neutral region for each type of physiological response.

The region determination unit 132 stores the determined upper and lower limits in the storage unit 12, either by performing a special task or without performing a special task. Specifically, the region determination unit 132 stores the determined upper and lower limits in a data table in the storage unit 12. That is, table 121 (see Figure 3) includes a neutral region table 121d containing neutral region information for each user U1.

Figure 9 shows an example of the neutral region table 121d. As shown in Figure 9, the items in the neutral region table 121d include "User ID," "Sensor Type," "Corresponding Indicator Type," "Upper Limit," "Lower Limit," and "Acquisition Date and Time." The items "Sensor Type" and "Corresponding Indicator Type" are the same as those in Figures 4A and 5 above, so their explanation is omitted.

The "User ID" field in the neutral area table 121d stores user ID data, which is identification information used to identify user information in the neutral area table 121d. The user ID data is transmitted, for example, from the game device 40 to the estimation device 10 via the terminal device 20.

The "Upper Limit" and "Lower Limit" fields in the neutral area table 121d store the upper and lower limit information determined by the area determination unit 132. The "Acquisition Date and Time" field in the neutral area table 121d stores the date and time information for which the upper and lower limit information was acquired. The upper and lower limit values, determined for each item such as the user ID, are updated each time the latest information is acquired. Furthermore, when the upper and lower limit information is updated, the acquisition date and time are also updated.

Returning to Figure 3, the emotion estimation unit 133 selects a model to use for emotion estimation based on the data acquired by the acquisition unit 131. Specifically, the emotion estimation unit 133 compares the acquired data with the sensor table 121a and extracts the "corresponding index type" data (index type data) corresponding to the sensor type and biosignal type included in the acquired data. Then, the emotion estimation unit 133 compares the "index ID" of the extracted index type data with the psychological plane table 121b and selects a model (psychological plane) to use for emotion estimation. For example, if the sensor IDs are "SN01" (sensor type: electroencephalogram sensor BA) and "SN02" (sensor type: heart rate sensor HA), the corresponding index IDs are "VS01" (index type: arousal level) and "VS02" (index type: autonomic nervous system activity level) (see Figure 4A). Based on these, the emotion estimation unit 133 selects psychological planes 01-02 as the emotion estimation model to use.

The emotion estimation unit 133, upon selecting the psychological plane to be used, extracts the upper and lower limits of the neutral region corresponding to the index type of the selected psychological plane from the neutral region table 121d using the user ID of user U1, the target of the emotion estimation. Then, the emotion estimation unit 133 processes the selected psychological plane to include the neutral region using the extracted upper and lower limits. In other words, the emotion estimation unit 133 performs calibration against a pre-prepared emotion estimation model.

Figure 10 shows an example of a psychological plane including the neutral region. The psychological plane including the neutral region shown in Figure 10 is a modified version of the psychological plane shown in Figure 2. In Figure 10, the shaded region NR represents the neutral region. The upper limit of the "Arousal Level" index determines the boundary between the neutral and positive regions on the arousal level (vertical axis). The lower limit of the "Arousal Level" index determines the boundary between the neutral and negative regions on the arousal level (vertical axis). The upper limit of the "Autonomic Nervous System Activity Level" index determines the boundary between the neutral and positive regions on the autonomic nervous system activity level (horizontal axis). The lower limit of the "Autonomic Nervous System Activity Level" index determines the boundary between the neutral and negative regions on the autonomic nervous system activity level (horizontal axis).

In the processed psychological plane, the first neutral region NR1, determined by the upper and lower limits of the indicator type "arousal level," is a band-shaped region extending parallel to the horizontal axis. Similarly, in the processed psychological plane, the second neutral region NR2, determined by the upper and lower limits of the indicator type "autonomic nervous system activity level," is a band-shaped region extending parallel to the vertical axis. The neutral region NR on the psychological plane consists of the first neutral region NR1 and the second neutral region NR2, and its shape is cross-shaped. The widths of the band-shaped first neutral region NR1 and the second neutral region NR2 may be the same or different. Furthermore, in the example shown in Figure 10, the first neutral region NR1 and the second neutral region NR2 overlap with the vertical axis or the horizontal axis, respectively. Here, the vertical and horizontal axes refer to the axes initially predicted (set) from the biosignal values. However, it is possible that at least one of the first neutral region NR1 and the second neutral region NR2 may be located off-center from the axis (the initially planned axis). In other words, determining the neutral region NRs ultimately determines the vertical and horizontal axes (passing through the origin). The midpoint between each neutral region NR1 and NR2 may be considered the vertical and horizontal axes.

The emotion estimation unit 133 generates a calibrated psychological plane and then performs emotion estimation processing using that psychological plane. The emotion estimation unit 133 converts information based on biosignals acquired by the acquisition unit 131 into index values using index conversion information in the sensor table 121a. In this embodiment, the index values for arousal level and autonomic nervous system activity are determined using data obtained from the first sensor 31 (sensor type: electroencephalogram sensor BA) and the second sensor 32 (sensor type: heart rate sensor HA).

The emotion estimation unit 133 then estimates emotions according to the position of the coordinates obtained by plotting each index value on a psychological plane including the neutral region NR obtained by the previous processing. In this embodiment, the emotional estimation is performed according to the position of the coordinates obtained by plotting the index values of the arousal level and the index values of the autonomic nervous system activity on a psychological plane including the neutral region (see Figure 10), with the arousal level on the vertical axis and the autonomic nervous system activity level on the horizontal axis.

The emotion estimation unit 133 determines that if the coordinate position obtained from plotting the index values is within the neutral region NR, it cannot determine the type of emotion of user U1. That is, the emotion estimation unit 133 determines that the emotion is in an undetermined state. On the other hand, if the coordinate position obtained from plotting the index values is outside the neutral region NR, the unit estimates (specifies) that the type of emotion of user U1 corresponds to the emotion at that coordinate position. For example, if the coordinate position is above the first neutral region NR1 in Figure 10 and to the right of the second neutral region NR2 (i.e., in the first quadrant), the unit estimates (specifies) that the user has emotions such as "happy, joyful, angry, or sad."

As can be seen from the above, the controller 13 identifies which of the three regions—the neutral region, the positive region on one side of the neutral region, and the negative region on the other side of the neutral region—the index value based on the biological signal belongs to. The controller 13 then determines that the individual is in an emotionally indecisive state if the index value belongs to the neutral region. This determination of an emotionally indecisive state may be a direct determination, such as setting an "emotionally indecisive state flag." Furthermore, this determination of an emotionally indecisive state may include not only the direct determination described above, but also indirect determinations, such as changing the subsequent processing content if the index value belongs to the neutral region.

Furthermore, the controller 13 calculates multiple types of index values. The controller 13 identifies which of the three regions described above each of these index values belongs to. If all of the multiple index values are outside the neutral region, the controller 13 uses the multiple index values and a pre-prepared emotion estimation model to identify the type of emotion. In this embodiment, there are two types of multiple values. However, the configuration may involve three or more index values. In this case, the emotion estimation model may be a three-dimensional or higher space, rather than a two-dimensional plane.

As in this embodiment, by configuring the system to perform emotion estimation by providing a neutral region, the possibility of incorrect estimation can be reduced in emotion estimation processing, which tends to have many error factors and is prone to misjudgment. Furthermore, a neutral region is created for each user U1. The position and size (width) of the neutral region typically differ for each user U1. By providing a neutral region, emotion estimation can be tailored to the characteristics of each user, thereby improving the accuracy of emotion estimation.

Returning to Figure 3, the provisioning unit 134 provides the emotion information estimated by the emotion estimation unit 133 to the terminal device 20 via the network N. Operator O1 then uses the emotion information provided by the provisioning unit 134, for example, by viewing it. Operator O1 may also provide the emotion information to user U1, who is an e-sports player.

[1-3. Estimation method]
Next, we will describe the estimation method performed by the estimation device 10.

Furthermore, the computer program that implements the estimation method of this embodiment in a computer device is included within the scope of this embodiment. Also, the computer-readable non-volatile recording medium that records such a computer program is included within the scope of this embodiment. Moreover, the computer program that implements the estimation method of this embodiment in a computer device may consist of only one program, or it may consist of multiple programs.

(1-3-1. When performing special tasks)
First, we will describe an example of processing when emotion estimation is performed after determining the neutral region by performing a special task. Figure 11 is a flowchart showing an example of the preliminary part of the estimation process performed by the estimation device 10 of the first embodiment. Figure 12 is a flowchart showing an example of the final part of the estimation process performed by the estimation device 10 of the first embodiment. Here, the preliminary part of the estimation process corresponds to the region determination process that determines the neutral region as described above. In the example shown in Figure 11, the preliminary part of the estimation process corresponds to the region determination process when performing a special task. The final part of the estimation process corresponds to the emotion estimation process using an emotion estimation model.

The process shown in Figure 11 is initiated, for example, when the acquisition unit 131 of the estimation device 10 starts acquiring signals from the biosensor 30, which includes the first sensor 31 and the second sensor 32. This acquisition may be initiated, for example, by a user U1 instruction. The process shown in Figure 12 is performed following the process shown in Figure 11. It is preferable that the preceding process shown in Figure 11 is performed before the e-sports event begins.

In step S11, the region determination unit 132 identifies the target indicator type based on the information from the biosensor 30 acquired by the acquisition unit 131 and the sensor table 121a. In this embodiment, "alertness level" and "autonomic nervous system activity level" are identified as indicator types. Once the indicator types are identified, the process proceeds to the next step S12.

In step S12, the domain determination unit 132 determines whether the upper and lower limits of the neutral domain for each indicator type have been determined for user U1, the target of emotion estimation. Whether the upper and lower limits of the neutral domain for user U1, the target of emotion estimation, have been determined is determined by the information contained in the neutral domain table 121d (see Figure 9). In this embodiment, it is confirmed whether the upper and lower limits of the neutral domain for "arousal level" and "autonomic nervous system activity level" for user U1, the target of emotion estimation, have been determined. If they have been determined (Yes in step S12), the process proceeds to step S20 (see Figure 12). If they have not been determined (No in step S12), the process proceeds to step S13.

Furthermore, the process in step S12 is not required. That is, regardless of whether the upper and lower limits of the neutral region for each indicator type have been determined in the past, the process of determining the upper and lower limits of the neutral region for each indicator type may be performed. The neutral region may fluctuate depending on factors such as the environment in which user U1 is placed or the physical condition of user U1. For this reason, it is preferable to determine the upper and lower limits of the neutral region before each esports session begins.

Furthermore, in this embodiment, if there are any indicator types identified in step S11 for which the upper and lower limits of the neutral region have not yet been determined, the process of determining the upper and lower limits is performed for all previously identified indicator types. However, this is merely an example. That is, the configuration may be such that the process of determining the upper and lower limits of the neutral region is performed only for indicator types for which the upper and lower limits of the neutral region have not yet been determined.

In step S13, the region determination unit 132 sets variable x to 1. In this embodiment, if variable x is 1, it is determined that the processing for the first indicator type out of the two indicator types identified in step S11 will be performed. If variable x is 2, it is determined that the processing for the second indicator type out of the two indicator types identified in step S11 will be performed. When variable x is set to 1, the process proceeds to the next step S14.

In step S14, the region determination unit 132 requests user U1 to execute the first task of the first indicator type because the variable is 1. The task request is made, for example, by displaying the request on the display unit 42 of the game device 40 used by user U1. The task request may also be made by voice guidance in addition to, or instead of, a screen display. Preferably, the request includes the specific content of the first task. Information regarding the content of the first task is contained in the special task table 121c (see Figure 5). In this embodiment, for example, the first indicator type is "awareness level." The content of the first task is "mentally add the numbers displayed on the display unit 42." Once the request to execute the first task of the first indicator type is made, the process proceeds to the next step S15.

In step S15, the region determination unit 132 determines the upper limit of the neutral region based on the signal obtained from the first sensor 31 during the execution of the first task of the first index type. The start and end of the user U1's task may be determined, for example, by requiring the user U1 to operate an operating member at the start and end of the task, and the determination being made based on the operation information of the operating member. The operating member may, for example, be included in the operation unit 41 of the game device 40. Alternatively, the start and end of the user U1's task may be determined by information obtained from a camera that photographs the user U1, which is positioned in the game device 40 or the like. In this embodiment, the region determination unit 132 determines the upper limit of the neutral region in the arousal level of the user U1 who is the target of emotion estimation. The determined upper limit is stored in the neutral region table 121d (see Figure 9). Once the upper limit of the neutral region is determined, the process proceeds to the next step S16.

In step S16, the domain determination unit 132 requests user U1 to execute the second task of the first indicator type. The processing related to the task request may be the same as for the first task, and a detailed explanation is omitted here. In this embodiment, since the first indicator type is "alertness," the content of the second task is "rest and clench your fist." Once the request to execute the second task of the first indicator type is made, the process proceeds to the next step S17.

In step S17, the region determination unit 132 determines the lower limit of the neutral region based on the signal obtained from the first sensor 31 during the execution of the second task of the first indicator type. The determination of the start and end of the user U1's task may be the same as in the case of the first task. In this embodiment, the region determination unit 132 determines the lower limit of the neutral region in terms of arousal level for user U1, who is the target of emotion estimation. The determined lower limit is stored in the neutral region table 121d. Once the lower limit of the neutral region is determined, the process proceeds to the next step S18.

In this example, the special tasks are executed in the order of Task 1 followed by Task 2, but this order can be reversed.

In step S18, the region determination unit 132 performs the operation of adding 1 to the variable x. Once the addition operation on variable x is complete, the process proceeds to the next step S19.

In step S19, the region determination unit 132 determines whether variable x is 3 or greater. If variable x is less than 3 (No in step S19), the process returns to step S14, and the processing from step S14 onward is repeated. However, at this stage, variable x is 2. Therefore, processing related to the second indicator type is performed. In this embodiment, the second indicator type is "autonomic nervous system activity level." For this reason, the content of the first task required in step S14 is the "standing test." In step S15, the upper limit of the neutral region in the autonomic nervous system activity level of user U1, the target of emotion estimation, is determined. Also, the content of the second task required in step S16 is the "Aschnell test." In step 17, the lower limit of the neutral region in the autonomic nervous system activity level of user U1, the target of emotion estimation, is determined. If variable x is 3 or greater (Yes in step S19), the process proceeds to step S20 (see Figure 12).

In this embodiment, the upper and lower limits of the neutral region are determined in the order of the first indicator type followed by the second indicator type. However, this is merely an example, and the upper and lower limits of the neutral region may be determined in the order of the second indicator type followed by the first indicator type.

In step S20, the emotion estimation unit 133 processes the pre-stored psychological plane using the neutral region information (upper and lower limits) for each index type obtained in the previous region determination process. Then, the emotion estimation unit 133 performs emotion estimation using the psychological plane including the neutral region. In this embodiment, for example, the processed psychological plane shown in Figure 10 is used to estimate the emotion. As described above, there are cases where the emotion is determined to be in an undefined state and the type of emotion is not specified, and cases where the type of emotion is specifically identified. Once the emotion estimation process using the psychological plane is complete, the process proceeds to the next step S21.

In step S21, the provisioning unit 134 provides the emotion estimation result to an external source. For example, the provisioning unit 134 provides the emotion estimation result to the terminal device 20. The emotion estimation result may be displayed on the display screen of the terminal device 20 as appropriate. Furthermore, the operator O1 of the terminal device 20 may perform processing using the emotion estimation result as appropriate. The result of the emotion estimation processing provided to the terminal device 20 may be output to the game device 40 automatically from the terminal device 20, or by operation of the operator O1 of the terminal device 20. The emotion estimation result output to the game device 40 may then be displayed on, for example, the display unit 42.

Figures 13A and 13B illustrate screens displaying the emotion estimation results. In the examples shown in Figures 13A and 13B, an emotion map is displayed on the screen (specifically on the right side of the screen) to allow for easy recognition of the emotion estimation results. The emotion map is, for example, graphic information including a graph plotting the measurement results (coordinate positions) on a psychological plane. Furthermore, in the examples shown in Figures 13A and 13B, an area is provided on the screen to display the emotion estimation results as text.

Figure 13A shows the results when the coordinate positions identified by the two index values are outside the neutral region (NR). In this case, the type of emotion can be identified, and the result of identifying the type of emotion is shown. Figure 13B shows the results when the coordinate positions identified by the two index values are within the neutral region (NR). In this case, since the type of emotion cannot be identified due to an indeterminate emotional state, this is indicated by text. Note that in the example shown in Figure 13B, if we focus only on autonomic nervous system activity (horizontal axis), the plotted position is not within the neutral region (NR). For this reason, for example, a message such as "The autonomic nervous system activity is estimated to be in a state of 'weak emotion'" may be displayed.

(1-3-2. When no special tasks are performed)
Next, we will describe an example of processing when emotion estimation is performed after determining the neutral region without performing a special task. Figure 14 is a flowchart showing another example of the preliminary part of the estimation process performed by the estimation device 10 of the first embodiment. In the example shown in Figure 14, the preliminary part of the estimation process corresponds to the region determination process when no special task is performed. In this example as well, once the preliminary part of the estimation process is completed, the subsequent part of the estimation process shown in Figure 12 is performed.

The process shown in Figure 14 is initiated, for example, when the acquisition unit 131 of the estimation device 10 begins acquiring signals from the biosensor 30, including the first sensor 31 and the second sensor 32. It is preferable that the process shown in Figure 14 be initiated immediately before the start of the e-sports event (e.g., 20 minutes before). This allows for the determination of the neutral region to be as close as possible to the actual environment in which the e-sports event is played. For example, the user may be made to warm up before starting the e-sports event (by running the game's warm-up mode), and the data accumulated during this process may be used to perform the process shown in Figure 14. However, the process shown in Figure 14 may be performed several hours or more before the start of the e-sports event. Furthermore, the process shown in Figure 14 may be performed while the user U1 is going about their daily life. Also, in some cases, the process shown in Figure 14 may be performed after the start of the e-sports event.

In step S31, the region determination unit 132 identifies the target index type based on the information from the biosensor 30 acquired by the acquisition unit 131. This process is the same as step S11 shown in Figure 11, so a detailed explanation is omitted. Once the index type is identified, the process proceeds to the next step, S32.

In step S32, the domain determination unit 132 determines whether the upper and lower limits of the neutral domain for each indicator type have been determined for user U1, the target of emotion estimation. If they have been determined (Yes in step S32), the process proceeds to step S20 (see Figure 12). If they have not been determined (No in step S32), the process proceeds to step S33. The process in step S32 is the same as that in step S12 shown in Figure 11, so a detailed explanation is omitted. Note that, as in the example shown in Figure 11, the process in step S32 may be omitted.

In step S33, the region determination unit 132 estimates the upper and lower limits of the neutral region for each indicator type using statistical processing methods such as cluster analysis. The data subject to statistical processing such as cluster analysis is the time-series data of indicator values accumulated for each indicator type. In this embodiment, the upper and lower limits of the neutral region are estimated for each indicator type using the time-series data of the indicator values for "arousal level" and "autonomic nervous system activity level." Once the upper and lower limits for each indicator type are estimated, the process proceeds to the next step S34.

In step S34, the region determination unit 132 determines whether the upper and lower limits of the neutral region have converged for each indicator type. In this embodiment, it is determined whether the upper and lower limits of the neutral region for "alertness level" have converged. It is also determined whether the upper and lower limits of the neutral region for "autonomic nervous system activity level" have converged. If it is determined that the upper and lower limits of the neutral region for all indicator types have converged (Yes in step S34), the process proceeds to the next step S35. On the other hand, if there are indicator types for which the upper and lower limits of the neutral region have not converged (No in step S34), the process returns to step S33, and the processing from step S33 onward is carried out.

In step S35, the region determination unit 132 determines the upper and lower limits of the neutral region for each indicator type, using the upper and lower limits that were previously determined to have converged. These determined upper and lower limits are stored in the neutral region table 121d. Once the upper and lower limits of the neutral region for each indicator type are determined, the process proceeds to step S20 shown in Figure 12, where the emotion estimation process is performed. This emotion estimation process is omitted from this explanation as it has been described above.

Furthermore, if the criteria (thresholds) for determining when the upper and lower limits have converged are set too strictly, the time it takes for the upper and lower limits to converge will increase. Therefore, in the early stages when the amount of accumulated time-series data is small, the criteria can be set more loosely to determine the neutral region as quickly as possible. While this reduces the accuracy of setting the neutral region, it allows for a state where emotion estimation can be performed in a short time. Then, while securing a state where emotion estimation can be performed early, the processing from step S33 onwards, as shown in Figure 14, can be continued while changing the criteria according to the amount of accumulated data. This allows for a gradual increase in the accuracy of setting the neutral region and a gradual increase in the accuracy of emotion estimation.

The configuration described above, which gradually increases the accuracy of the neutral zone settings, can be used in applications such as esports, for example, as follows: At the beginning of a game, when the accuracy of the neutral zone settings is low, the system informs user U1 of the low accuracy of emotion estimation and displays the estimated emotion results on the screen. Once the accuracy of the neutral zone settings increases and the emotion estimation accuracy reaches a predetermined level or higher, the notification about the low accuracy is removed, and the official emotion estimation results are displayed on the screen.

Furthermore, the configuration described above, which gradually increases the accuracy of the neutral zone setting, can be used in, for example, an autonomous vehicle in the following way: At the stage where the accuracy of the neutral zone setting is low, only the emotion estimation result is displayed to the driver. Once the accuracy of the neutral zone setting increases and the emotion estimation accuracy reaches a predetermined level or higher, the system automatically switches between autonomous and manual driving according to the emotion estimation result. For example, if the emotion estimation result determines that the driver is irritated, the system can be judged as dangerous and switch from manual to autonomous driving.

[1-4. Variations]
Next, a modified example of the first embodiment will be described.

(1-4-1. First variation)
In the above configuration, the neutral region determined by the region determination unit 132 is used directly to calibrate the emotion estimation model (psychological plane). Alternatively, the controller 13 may adjust the neutral region depending on the intended use of the emotion estimation results. Specifically, adjusting the neutral region means adjusting the width of the neutral region determined by the region determination unit 132. With this configuration, the possibility of identifying the type of emotion through the emotion estimation process can be varied according to the intended use.

For example, if the goal is to inform user U1 of the estimated emotion type as much as possible, even at the expense of some accuracy in emotion estimation, the width of the neutral region, which forms a band on the psychological plane, can be narrowed. This adjustment reduces the likelihood that the index value obtained from the biosignal will fall within the neutral region. As a result, the probability of identifying the emotion type can be increased. An example of a situation where the goal is to inform user U1 of the estimated emotion type as much as possible, even at the expense of some accuracy in emotion estimation, is simply to display the emotion type on the screen without performing any control processing based on the emotion type.

Furthermore, if you want to identify the type of emotion as accurately as possible, you can adjust the settings so that the width of the neutral region, which forms a band on the psychological plane, is wider. With this adjustment, if the index value obtained from the biosignals is not sufficiently large, it will fall into the neutral region. As a result, the possibility of misidentifying the type of emotion can be reduced. Examples of situations where you want to identify the type of emotion as accurately as possible include using the emotion estimation results for mental training, or using emotion types to control vehicle operation.

(1-4-2. Second variation)
The above configuration uses only an emotion estimation model to estimate emotions, but this is merely an example. The controller 13 may take into account the emotion estimation results derived from information other than biological signals when estimating emotions using the emotion estimation model. By configuring it in this way, the type of emotion can be estimated more appropriately.

Figure 15 shows a schematic configuration of a modified version of the estimation system 1 of the first embodiment. As shown in Figure 15, the estimation device (server) 10A of the modified version is also connected to the terminal device 20A via a network N, similar to the first embodiment described above. However, this modified version differs from the above embodiment in that, in addition to biological signals, image signals are input to the terminal device 20A, and these image signals are transmitted to the estimation device 10A. The biological signals are, more precisely, sensor signals obtained by measuring biological signals with a biological sensor 30. The image signals are signals output from a camera (not shown) that photographs the user U1. This camera may, for example, be built into the game device 40. Alternatively, it may be configured to be installed separately from the game device 40.

The estimation device 10A, similar to the first embodiment described above, performs emotion estimation using the emotion estimation model shown in Figure 10, etc. Here, let's assume that the emotion estimation model estimates the user U1 to have one of the following emotions: "enjoyment, happiness, anger, or sadness" (corresponding to the first quadrant of the psychological plane). In this case, the estimation device 10A performs image processing of the user U1's face image, analyzes the facial expression, and classifies the type of emotion. For example, it uses image information to classify the emotion into one of the three categories: "enjoyment, happiness," "anger," or "sadness." In other words, by incorporating image information, the type of emotion estimated by the emotion estimation model can be classified in more detail.

The above example illustrates the case where emotions in the first quadrant are estimated using an emotion estimation model (psychological plane). However, classification using image information may also be performed when emotions in other quadrants are estimated. For example, if the types of emotions estimated using the emotion estimation model include both positive and negative emotions, classification using image information may be performed. Furthermore, while image information was used above, other types of information, such as audio information, may also be used.

Let's explain using the example of estimating the emotions of a vehicle driver. In this case, for example, an in-vehicle camera for a dashcam can be used as the camera that obtains the image information mentioned above. If information other than image information is used, for example, audio information obtained from a microphone in an in-vehicle device such as a dashcam may be used. When using audio information, the driver may be asked predetermined questions, and their responses may be analyzed to classify their emotions. Furthermore, as an example of using information other than image information, CAN (Controller Area Network) information or driving behavior information may also be used.

Examples of CAN information used to estimate emotions include accelerator information, brake information, and steering angle information. For instance, accelerator, brake, and steering angle operations tend to be rougher when a driver is angry or frustrated, and analyzing this information allows for emotional classification. Furthermore, driving behavior information can be obtained from at least one of the following: CAN information, radar information, LiDAR (Light Detection and Ranging) information, and image information capturing the vehicle's surrounding environment. For example, if a tendency to yield to pedestrians or other vehicles is observed, it can be determined that the driver is not experiencing anger or sadness. Conversely, if a short following distance is observed, it can be determined that the driver is experiencing anger or frustration.

<2. Second Embodiment>
Next, a second embodiment will be described. In describing the second embodiment, components similar to those in the first embodiment will be denoted by the same reference numerals, and explanations will be omitted unless they require special explanation.

[2-1. Estimation System]
The estimation system of the second embodiment has the same configuration as the estimation system 1 of the first embodiment. For this reason, a detailed description of the estimation system of the second embodiment will be omitted. The estimation system of the second embodiment includes an estimation device 10B (see Figure 16, described later) and a biosensor 30 for measuring biological signals (see Figure 1). The biosensor 30 includes a first sensor 31 configured as an electroencephalogram sensor and a second sensor 32 configured as a heart rate sensor.

[2-2. Estimation device]
Figure 16 shows an example of the configuration of the estimation device 10B according to the second embodiment of the present invention. The configuration of the estimation device 10B is generally the same as that of the estimation device 10 of the first embodiment (see Figure 3). Similar to the first embodiment, the estimation device 10B includes a communication unit 11 and a storage unit 12. The estimation device 10B also includes a controller 13B.

The configuration of controller 13B is the same as that of controller 13 in the first embodiment. Furthermore, the functions of controller 13B are generally the same as those of controller 13 in the first embodiment. However, controller 13B differs from controller 13 in that it includes an aptitude estimation unit 135. The aptitude estimation unit 135, like the other functional units 131 to 134, is implemented, for example, by software. The aptitude estimation unit 135 may also be implemented by hardware using a dedicated IC, or by a combination of software and hardware.

Furthermore, the estimation device 10B of this embodiment may be configured to include the acquisition unit 131 and the aptitude estimation unit 135 among the functional units 131 to 135, but may not include the domain determination unit 132, the emotion estimation unit 133, and the provision unit 134. In this case, the estimation device 10B may be a device that prepares for emotion estimation. In addition, an emotion estimation device comprising at least the emotion estimation unit 133 among the domain determination unit 132, the emotion estimation unit 133, and the provision unit 134 may be provided separately from the estimation device 10B.

Before explaining the aptitude estimation unit 135, let's explain why it is included. Information obtained based on biosignals is easily influenced by environmental factors and physical condition. Therefore, at the time of emotion estimation based on biosignals, the person being estimated may be in an unsuitable state due to environmental influences. If emotion estimation is performed in such a state, there is a high possibility of errors in the emotion estimation, potentially leading to a decrease in the accuracy of the emotion estimation. In this embodiment, the aptitude estimation unit 135 is provided to suppress such a decrease in emotion estimation accuracy.

The aptitude estimation unit 135 estimates the user U1's aptitude for emotion estimation based on biosignals. Specifically, the aptitude estimation unit 135 (i.e., the controller 13B) assigns user U1 multiple tasks. The aptitude estimation unit 135 also acquires index values based on biosignals during the execution of each of these tasks. Furthermore, the aptitude estimation unit 135 estimates user U1's aptitude for emotion estimation based on the relationships between these index values.

With this configuration, by having user U1 perform multiple tasks before sentiment estimation, it is possible to estimate user U1's suitability for sentiment estimation. As a result, the likelihood of performing sentiment estimation in a suitable state increases, thereby improving the accuracy of sentiment estimation.

The aptitude estimation unit 135, which has the functions described above, will now be explained in more detail.

Each of the multiple tasks that the aptitude estimation unit 135 has user U1 perform is a task that can evoke a predetermined state of physiological response (e.g., level of arousal) based on medical evidence. Furthermore, the magnitude (index value) of the physiological response that each of these tasks can evoke differs from one another. In other words, a hierarchy emerges in the magnitude (index value) of the physiological response obtained when user U1 performs each of the multiple tasks.

In this embodiment, table 121 (see Figure 16) includes a task table 121e for aptitude estimation. The task table 121e for aptitude estimation contains information about the multiple types of tasks described above. Figure 17 shows an example of the task table 121e for aptitude estimation. As shown in Figure 17, the items in the task table 121e for aptitude estimation include "Task ID," "Sensor Type," "Corresponding Indicator Type," "Ranking," and "Task Content." The items "Sensor Type" and "Corresponding Indicator Type" are the same as those in Figure 4A, etc., described above, so their explanation is omitted. Note that "Corresponding Indicator Type" may also be expressed as "Corresponding Physiological Response Type."

The "Task ID" item in the aptitude estimation task table 121e stores ID data, which is identification information used to identify task information in the aptitude estimation task table 121e.

The "Ranking" item in the aptitude estimation task table 121e stores ranking information of the magnitude of physiological responses (index values) when user U1 performs the various tasks described above. This ranking information is determined based on medical evidence. For example, if the type of physiological response (index type) is arousal level, the ranking is determined by the magnitude of the index value obtained when performing the task. In the aptitude estimation table 121e, tasks judged to have larger index values based on medical evidence are ranked higher (the ranking number decreases).

The "Task Content" item in the aptitude estimation task table 121e stores the specific details of the task to be performed by user U1. The task content is determined based on medical evidence, along with the aforementioned hierarchy. For example, if the type of physiological response is arousal level, the highest-ranking task content is "applying tenderness." Specifically, "applying tenderness" means applying pressure that everyone would find painful, such as a foot massage, for a predetermined period of time. Similarly, if the type of physiological response is arousal level, the second-ranking task content is "mentally adding displayed numbers." Furthermore, if the type of physiological response is arousal level, the third-ranking task content is "resting and clenching a fist." Finally, if the type of physiological response is arousal level, the fourth-ranking task content is "remaining with eyes closed (not thinking about anything)."

The aptitude estimation unit 135, for example, compares the sensor type information of the biosensor 30 worn by user U1 with the task table 121e for aptitude estimation to determine multiple types of tasks for user U1 to perform. The sensor type information is acquired by the acquisition unit 131 via a terminal device 20 that is communicated with the biosensor 30. Furthermore, if user U1 wears multiple types of biosensors 30, aptitude estimation may be performed using each type of biosensor 30. Alternatively, if user U1 wears multiple types of biosensors 30, aptitude estimation may be performed using only one type of biosensor 30.

For example, in this embodiment, the biosensors 30 worn by user U1 consist of two sensors: a first sensor 31 and a second sensor 32. If the aptitude estimation task table 121e contains aptitude estimation tasks corresponding to these two sensors 31 and 32, aptitude estimation may be performed using both sensors 31 and 32, or aptitude estimation may be performed using only one of the sensors. When aptitude estimation is performed using only one of the sensors, the type of sensor used for aptitude estimation may be determined, for example, according to pre-prepared priority information. Furthermore, if the aptitude estimation task table 121e contains only aptitude estimation tasks corresponding to one of the sensors, aptitude estimation may be performed using that sensor.

For example, let's consider the case where the sensor type is an electroencephalogram (EEG) sensor BA. The aptitude estimation unit 135 has user U1 perform four tasks. The order in which the four tasks are performed is not particularly limited, but for example, they are performed starting with the lowest-ranking task (i.e., the one with the lowest level of arousal). The aptitude estimation unit 135 acquires the index value (magnitude of arousal) for each of the four tasks performed by user U1. The index value is determined using the signal obtained from the EEG sensor BA, which measures user U1's brainwaves, and the index conversion information contained in the sensor table 121a (see Figure 4A). The aptitude estimation unit 135 determines the order of the four acquired index values according to their magnitudes.

The aptitude estimation unit 135 then determines whether the sequence of index values obtained when user U1 performs each task matches the sequence in the aptitude estimation task table 121e. If they match, the aptitude estimation unit 135 estimates that user U1 is in a state suitable for emotion estimation. If they do not match, the aptitude estimation unit 135 estimates that user U1 is in an unsuitable state for emotion estimation.

In other words, the controller 13B estimates that user U1 is in a state suitable for emotion estimation when the order of magnitudes of the indicator values obtained during the execution of each of the multiple types of tasks matches a pre-set order. Conversely, the controller 13B estimates that user U1 is in an unsuitable state for emotion estimation when the order of magnitudes of the indicator values obtained during the execution of each of the multiple types of tasks differs from the pre-set order. This configuration allows for easy determination of whether user U1 is in a state suitable for emotion estimation based on the order of magnitudes of physiological responses determined based on medical evidence.

In the example shown above, when the type of physiological response is arousal level, there are four types of tasks required of user U1 for aptitude estimation (see Figure 17). However, this is merely an example. User U1 may be required to perform multiple types of tasks for aptitude estimation; for example, there may be only two. However, if the system is configured to have user U1 perform only two tasks with significantly different magnitudes of physiological responses, the order obtained during task execution may not change from the set order even if user U1's physical condition or environment changes. In other words, the significance of aptitude estimation may be diminished. Taking this into consideration, it is preferable to determine the types of tasks to be performed by user U1 in a way that ensures appropriate aptitude estimation (so that the order obtained during task execution changes appropriately in response to environmental changes, etc.).

In this embodiment, when user U1 is estimated to be in a state suitable for emotion estimation, the region determination unit 132 and emotion estimation unit 133 perform emotion estimation similar to that in the first embodiment described above. That is, when user U1 is estimated to be in a state suitable for emotion estimation, the controller 13B performs emotion estimation using index values based on biosignals. The controller 13B identifies which of the three regions—neutral, positive, and negative—the index value based on biosignals belongs to. If the index value belongs to the neutral region, the controller 13B determines that the user is in an undetermined emotional state. The controller 13B obtains multiple index values, and if all of these index values are outside the neutral region, it estimates the type of emotion using the multiple index values and a pre-prepared emotion estimation model.

Furthermore, the emotion estimation method in the second embodiment may be completely different from the emotion estimation method in the first embodiment. The emotion estimation method may also be a known method.

Furthermore, in this embodiment, the emotion estimation result is provided to the terminal device 20 via the network N by the providing unit 134.

[2-3. Estimation method]
Next, we will describe the estimation method performed by the estimation device 10B.

Furthermore, the computer program that implements the estimation method of this embodiment in a computer device is included within the scope of this embodiment. Also, the computer-readable non-volatile recording medium that records such a computer program is included within the scope of this embodiment. Moreover, the computer program that implements the estimation method of this embodiment in a computer device may consist of only one program, or it may consist of multiple programs.

Figure 18 is a flowchart illustrating an example of the aptitude estimation process performed by the estimation device 10B of the second embodiment. The process shown in Figure 18 begins, for example, when the acquisition unit 131 of the estimation device 10B starts acquiring signals from the biosensor 30, including the first sensor 31 and the second sensor 32. It is preferable that the aptitude estimation process shown in Figure 18 is performed before the start of the e-sports event.

In step S41, the aptitude estimation unit 135 sets the variable n to zero. Note that the variable n represents the number of times. Once the variable n is set to zero, the process proceeds to the next step, S42.

In step S42, the aptitude estimation unit 135 identifies target index values based on the information from the biosensor 30 acquired by the acquisition unit 131 and the aptitude estimation table 121e. In this embodiment, for example, an electroencephalogram (EEG) sensor is identified as the sensor type, and the level of arousal is identified as the index. The aptitude estimation unit 135 then sequentially assigns tasks corresponding to the identified index to the user U1, and acquires index values based on biosignals while the user is performing the assigned tasks.

In this embodiment, for example, the following tasks are sequentially given to user U1: "close eyes and rest (thinking nothing)," "rest and clench your fist," "mentally add the displayed numbers," and "apply pressure to the user." Each time user U1 performs a task, an index value (an index value of alertness) based on electroencephalography (EEG) is acquired. The tasks may be given, for example, using the screen display of the game device 40 used by user U1. Alternatively, tasks may be given via voice guidance instead of screen display, or in addition to screen display. Once the aptitude estimation unit 135 acquires the index value for each given task, it proceeds to the next step S43.

In step S43, the aptitude estimation unit 135 arranges the multiple (four in this embodiment) indicator values obtained in response to user U1's task execution in order of ranking, from largest to smallest. The aptitude estimation unit 135 then determines whether the ranking obtained from user U1's task execution is the same as the pre-set ranking (the ranking shown in the "Ranking" item) stored in the aptitude estimation table 121e. If the two rankings are the same (Yes in step S43), the aptitude estimation unit 135 proceeds to the next step, S44. On the other hand, if the two rankings are different (No in step S43), the aptitude estimation unit 135 returns to step S42.

In step S44, the aptitude estimation unit 135 adds 1 to the variable n. Once the process of adding 1 to variable n is complete, the process proceeds to the next step, S45.

In step S45, the aptitude estimation unit 135 determines whether the variable n is greater than or equal to a pre-prepared threshold number nth. The threshold number nth is determined, for example, through experimentation. If the variable n is greater than or equal to the threshold number (Yes in step S45), the system estimates that user U1 is in a suitable state for emotion estimation, and the process proceeds to step S11 shown in Figure 11 or step S31 shown in Figure 14. After the completion of the process shown in Figure 11 or Figure 14, the process proceeds to step S20 shown in Figure 12 to perform emotion estimation. Note that the calibration preparation process shown in Figure 11 or Figure 14 may be omitted. On the other hand, if the variable n is less than the threshold number (No in step S45), the process returns to step S42.

As can be seen from the above, in this embodiment, when the number of times the sequence obtained by the execution of user U1's task matches the pre-prepared setting sequence reaches a predetermined number of times, user U1 is presumed to be in a state suitable for emotion estimation. However, this is an example, and user U1 may be presumed to be in a state suitable for emotion estimation if the sequence obtained by the execution of user U1's task matches the pre-prepared setting sequence even once.

Furthermore, in this embodiment, if the number of matches between the sequence obtained by user U1's task execution and the pre-prepared setting sequence does not reach a predetermined number of times, user U1 is presumed to be in an unsuitable state for emotion estimation, and the suitability estimation task is provided indefinitely. However, this is an example. For example, when the number of mismatches between the sequence obtained by user U1's task execution and the pre-prepared setting sequence reaches a predetermined number, user U1 may be presumed to be in an unsuitable state for emotion estimation, and the suitability estimation process may be terminated.

[2-4. Modified Examples]
Next, a modified example of the second embodiment will be described.

(2-4-1. First variation)
Figure 19 is a flowchart showing a modified example of the suitability estimation process performed by the estimation device 10B of the second embodiment.

In step S51, the aptitude estimation unit 135 identifies the target indicators based on the information from the biosensor 30 acquired by the acquisition unit 131 and the aptitude estimation table 121e. The aptitude estimation unit 135 then sequentially assigns tasks corresponding to the identified indicators to the user U1, and acquires indicator values based on biosignals while the user performs the assigned tasks. These processes are the same as those in step S42 of Figure 18 described above. Once the aptitude estimation unit 135 has acquired the indicator values for each assigned task, it proceeds to the next step, S52.

In step S52, the aptitude estimation unit 135 arranges the multiple index values obtained in response to user U1's task execution in order of ranking, from largest to smallest. The aptitude estimation unit 135 then compares the ranking obtained from user U1's task execution with the pre-set ranking (shown in the "Ranking" item) stored in the aptitude estimation table 121e. For example, this comparison may result in the two rankings being completely identical. Alternatively, it may result in one instance of a reversal of relative order between the two rankings. For example, let's consider the case where the physiological response (index) is arousal level. An example of a reversal of relative order might be when the index value obtained from "resting and clenching a fist" is ranked 2nd, and the index value obtained from "mentally adding the displayed numbers" is ranked 3rd. Once the comparison results are obtained, the process proceeds to the next step, S53.

In step S53, the aptitude estimation unit 135 estimates the aptitude level according to the comparison results obtained earlier. The aptitude level table 121f (see Figure 20) is used to estimate the aptitude level. Figure 20 shows an example of the aptitude level table 121f. As shown in Figure 20, the items in the aptitude level table 121f include "Aptitude Estimation Task ID," "Sensor Type," "Corresponding Indicator Type," "Comparison Status," and "Aptitude Level." The items "Sensor Type" and "Corresponding Indicator Type" are the same as those in Figure 17, etc., so their explanation is omitted.

The "Aptitude Task ID" item in the aptitude level table 121f stores information corresponding to the "Task ID" item in the aptitude estimation task table 121e (see Figure 17).

The "Comparison Status" item in the aptitude level table 121f stores information to be compared with the comparison result in step S52. Note that the number of tasks may differ depending on the type of physiological response. Therefore, the correspondence between "Comparison Status" and "Aptitude Level" is not necessarily unique. Considering this point, in this example, a "Comparison Status" is provided for each type of physiological response. For example, if the type of physiological response (indicator type) is arousal level, the comparison status information includes "Exact Match," "One Change in the Order of Sequences," and "Other." "Other" refers to cases where the comparison result is neither "Exact Match" nor "One Change in the Order of Sequences."

Next, we will explain using the example of a physiological response type being arousal level. "Other" includes cases where there is one instance of a two-rank swap. For example, when "resting and clenching a fist" is performed, the index value obtained is ranked 1st, and when user U1 performs "tendering pressure," the index value obtained is ranked 3rd. "Other" also includes cases where there are two instances of rank swapping. For example, when both of the following cases are met: Case 1 is when, for example, "resting and clenching a fist" is performed, the index value obtained is ranked 4th, and when "closed-eyes and rest" is performed, the index value obtained is ranked 3rd. Case 2 is when, for example, "tendering pressure" is performed, the index value obtained is ranked 2nd, and when "mentally adding the displayed numbers" is performed, the index value obtained is ranked 1st.

The "Aptitude Level" column in Aptitude Level Table 121f contains information about the aptitude level determined by the comparison results. In the example shown in Figure 20, the aptitude level is "Appropriate" when the comparison result is "Exact Match." The aptitude level is "Somewhat Unstable" when the comparison result is "Other." The aptitude level is "Unsuitable" when the comparison result is "Other." Note that in this example, there are three aptitude levels, but this is an example; there could be four or more levels.

Once the aptitude level is estimated, the process proceeds to the next step, S54.

In step S54, the aptitude estimation unit 135 determines whether or not to perform emotion estimation of user U1. For example, in a configuration where the aptitude level is estimated in three stages, as shown in Figure 20, emotion estimation may be performed if the user falls into the top two levels. The top two levels are "suitable" and "somewhat unstable." Conversely, emotion estimation may not be performed if the user falls into the lowest level. The lowest level is "unsuitable."

If it is determined that emotion estimation should be performed (Yes in step S54), the process proceeds to step S11 shown in Figure 11 or step S31 shown in Figure 14. After the completion of the process shown in Figure 11 or Figure 14, the process proceeds to step S20 shown in Figure 12, where emotion estimation is performed. If it is determined that emotion estimation should not be performed (No in step S54), the process proceeds to step S55.

In step S55, the aptitude estimation unit 135 performs a process to notify user U1 that they are unsuitable for emotion estimation. For example, the aptitude estimation unit 135 displays a notification on the screen of the game device 40 (see Figure 1) used by user U1, indicating that user U1 is unsuitable for emotion estimation. Figure 21 shows an example of a screen notifying user U1 that they are unsuitable for emotion estimation. As shown in Figure 21, if user U1 is unsuitable for emotion estimation, they may be prompted to perform an aptitude check again, for example, after 20 minutes.

As can be seen from the above, in this modified version, the controller 13B estimates the user's level of suitability for emotion estimation based on the degree of discrepancy between the order of magnitude of the index values obtained during the execution of each of the multiple types of tasks and the pre-set order for the multiple types of tasks. With this configuration for estimating the suitability level, the controller 13B can change the way the emotion estimation results are used according to the level.

For example, if the aptitude level is estimated to be "appropriate," the emotion estimation result may be displayed normally as shown in Figures 13A and 13B above. On the other hand, if the aptitude level is estimated to be "somewhat unstable," the emotion estimation result may be displayed provisionally, as shown in Figure 22. Figure 22 shows an example of provisionally displaying the emotion estimation result on the screen. In Figure 22, a cautionary note is included in the screen display to inform the user U1 that it is a provisional display. Furthermore, taking the case of autonomous driving control as an example, if the aptitude level is "appropriate," the system may be configured to perform autonomous driving-related control according to the emotion estimation result. On the other hand, if the aptitude level is "somewhat unstable," the emotion estimation result may be simply displayed on the in-vehicle device screen for reference only and not used for autonomous driving-related control.

Furthermore, the controller 13B may adjust the neutral region NR (see Figure 10) according to the aptitude level. This configuration can suppress the possibility of errors in emotion estimation. For example, when the aptitude level is "somewhat unstable," the width of the neutral region may be adjusted to be wider than when the aptitude level is "appropriate."

(2-4-2. Second variation)
Figure 23 is a flowchart showing another modified example of the aptitude estimation process performed by the estimation device 10B of the second embodiment. The flowchart shown in Figure 23 is generally similar to the flowchart shown in Figure 19. For this reason, we will focus on explaining the differences. In the example shown in Figure 23, if it is determined that emotion estimation should not be performed (No in step S54), the process does not proceed immediately to step S55, but rather to step S56. This is different from the flowchart shown in Figure 19. In this modified example, instead of estimating the aptitude level in step S53, an estimation of whether or not the state is suitable for emotion estimation may be performed.

In step S56, the suitability estimation unit 135 determines whether there are other means for estimating emotions. These other means may be, for example, other biosensors 30 worn by user U1. For example, even if the first sensor 31, which is an electroencephalogram (EEG) sensor, determines that the user is in an unsuitable state for emotion estimation, the second sensor 32, which is a heart rate sensor, may determine that the user is in a suitable state for emotion estimation. In such cases, emotion estimation may be performed using the heart rate sensor instead of the EEG sensor. Alternatively, for example, the other means may be a camera (an example of a sensor) that captures an image of user U1's face. That is, emotion estimation may be performed using the face image of user U1 captured by the camera.

As can be seen from the above, when the controller 13B estimates that user U1 is in an unsuitable state for emotion estimation, it may perform emotion estimation using information from a different sensor than the one that provided the information leading to that estimation. By configuring it in this way, the possibility of a situation where no emotion estimation result can be obtained can be suppressed.

<3. Things to keep in mind>
The various technical features disclosed in the embodiments for carrying out the invention as described herein can be modified in various ways without departing from the spirit of the technical creation. Furthermore, the multiple embodiments and modifications disclosed in the embodiments for carrying out the invention as described herein may be combined to the extent possible.

1... Estimation system 10, 10A, 10B... Estimation device 13, 13B... Controller 30... Biosensor

Claims (10)

An estimation device equipped with a controller,
The aforementioned controller,
Give the user multiple types of tasks,
Obtain index values based on biosignals during the execution of each of the aforementioned multiple tasks,
An estimation device that estimates the user's suitability for emotion estimation based on the relationships between a plurality of the aforementioned index values. The aforementioned controller,
When the order of magnitude of the index values obtained during the execution of each of the aforementioned tasks matches the pre-set order for the aforementioned tasks, it is estimated that the user is in a state suitable for emotion estimation.
The estimation device according to claim 1, wherein if the order of magnitudes of the index values obtained during the execution of each of the aforementioned multiple tasks differs from the set order, the device estimates that the user is in a state unsuitable for emotion estimation. The estimation device according to claim 1, wherein the controller estimates the level of the user's suitability for emotion estimation according to the degree of discrepancy between the order of magnitudes of the index values obtained during the execution of each of the multiple types of tasks and the pre-set order which is the order set for the multiple types of tasks. The estimation device according to claim 1, wherein the controller performs emotion estimation using index values based on biosignals when the user is estimated to be in a state suitable for emotion estimation. The aforementioned controller,
If the user is estimated to be in a state suitable for emotion estimation, emotion estimation is performed using index values based on biosignals.
The estimation device according to claim 3, wherein the form of using the emotion estimation result is changed according to the level. The aforementioned controller,
If the user is estimated to be in a state suitable for emotion estimation, emotion estimation is performed using index values based on biosignals.
The system identifies which of the three regions—the neutral region, the positive region on one side of the neutral region, and the negative region on the other side of the neutral region—the index value based on the biological signal belongs to.
The estimation device according to claim 3, which determines that the state of emotion is indecisive when the index value falls within the neutral range. The estimation device according to claim 6, wherein the controller adjusts the neutral region according to the level. The estimation device according to claim 4, wherein, when the controller is estimated to be in a state unsuitable for emotion estimation, it performs emotion estimation using information from a sensor different from the one that led to the estimation of unsuitability. An estimation system comprising an estimation device for estimating emotions and a biosensor for measuring biological signals,
The estimation device is,
Give the user multiple types of tasks,
Obtain index values based on biosignals during the execution of each of the aforementioned multiple tasks,
Based on the relationships between multiple aforementioned index values, the suitability of the user for emotion estimation is estimated.
An estimation system that performs emotion estimation using index values based on biosignals when the user is estimated to be in a state suitable for emotion estimation. Give the user multiple types of tasks,
Obtain index values based on biosignals during the execution of each of the aforementioned multiple tasks,
Based on the relationships between multiple aforementioned index values, the suitability of the user for emotion estimation is estimated.
An estimation method in which a device performs the processing.