JP7224032B2 - Estimation device, estimation program and estimation method - Google Patents

Description

TECHNICAL FIELD The present invention relates to an estimation device, an estimation program, and an estimation method, and more particularly to an estimation device, an estimation program, and an estimation method for estimating, for example, the degree of difficulty recognized by a subject regarding a topic of conversation.

An example of this type of estimating device is disclosed in Patent Document 1. Patent Literature 1 discloses a stimulus presentation system that determines a user's mental state by measuring a user's line of sight and brain function, and changes the content of task presentation based on the determined mental state.

WO2013/186911

The stimulus presentation system of Cited Document 1 measures the user's line of sight in order to determine the user's mental state. When this stimulus presentation system is applied to a system for estimating the cognitive load of a subject when the subject converses with a communication target, the measured line of sight information can be used as a factor for estimating the cognitive load. Unable to use. This is because there is no relationship between whether or not the person is looking at the communication target and whether or not the person is listening to or understanding the conversation of the communication target.

SUMMARY OF THE INVENTION Therefore, a primary object of the present invention is to provide a novel estimation device, estimation program and estimation method.

Another object of the present invention is to provide an estimation device, an estimation program, and an estimation method capable of estimating the degree of cognitive load of a subject during conversation based on brain activity.

A first invention is an estimation device for estimating the degree of cognitive load perceived by a subject for a topic based on the cerebral blood flow of the subject who has a conversation with a communication target, wherein the brain of the subject who performs a cognitive task Acquisition means for acquiring a cerebral blood flow signal that is a signal about blood flow; calculation means for calculating a vector of differential entropy of the cerebral blood flow signal acquired by the acquisition means for each first predetermined time; A construction means for constructing a cluster space by clustering the calculated differential entropy vector into a predetermined number of classes with different levels of cognitive load for the cognitive task using the K-means method, and an acquisition means. Acquiring a cerebral blood flow signal when a subject converses with a communication target, calculating a differential entropy vector of the acquired cerebral blood flow signal by a calculating means, and calculating a predetermined number to which the differential entropy vector belongs in the cluster space estimating means for estimating the degree of cognitive load regarding the content of the subject's conversation by determining one of the classes , the calculating means calculating the cerebral blood flow signal for a first predetermined time An estimator that separates into non-overlapping segments of a second predetermined time interval and calculates a differential entropy vector for a given probability distribution using the variance of the subject's prefrontal cortex activity for each segment.

A second invention is according to the first invention, wherein the acquisition means acquires a cerebral blood flow signal in a left side portion of the prefrontal cortex of the subject.

A third invention is according to the first or second invention, and further comprises information storage means for storing information on the degree of cognitive load estimated by the estimation means in relation to the content of the conversation.

A fourth invention belongs to the third invention, wherein the communication target is the conversation agent, and further comprising topic selection means for selecting a topic provided by the conversation agent based on the information stored by the information storage means.

A fifth invention is an estimation program executed by a computer for estimating the degree of cognitive load perceived by a subject with respect to a topic based on the cerebral blood flow of the subject who has a conversation with a communication target, the computer processor an acquisition step of acquiring a cerebral blood flow signal, which is a signal about the cerebral blood flow of a subject performing a cognitive task; A construction step of constructing a cluster space by clustering the differential entropy vector calculated in the calculation step into a predetermined number of classes with different levels of cognitive load for the cognitive task using the K-means method, and obtaining In the step, acquiring a cerebral blood flow signal when the subject converses with the communication target in constructing the cluster space, calculating a differential entropy vector of the acquired cerebral blood flow signal in the calculating step, and calculating the differential entropy vector An estimating step of estimating the degree of cognitive load regarding the content of the conversation of the subject by determining one class among a predetermined number of classes belonging to the cluster space , and the calculating step includes a first predetermined Separating the cerebral blood flow signal for the hour into non-overlapping sections of a second predetermined time interval, and calculating a differential entropy vector for a predetermined probability distribution using the variance of the subject's prefrontal cortex activity for each section. It is an estimation program.

A sixth invention is a computer estimation method for estimating the degree of cognitive load perceived by a subject for a topic based on the cerebral blood flow of the subject who has a conversation with a communication target, comprising: (a) performing a cognitive task; (b) calculating a differential entropy vector of the cerebral blood flow signal obtained in step (a) for each first predetermined time period; (c) constructing a cluster space by clustering the vector of differential entropy calculated in step (b) into a predetermined number of classes with different levels of cognitive load for the cognitive task using the K-means method; and (d) acquiring a cerebral blood flow signal when the subject in constructing the cluster space converses with the communication target in step (a), and converting the differential entropy vector of the acquired cerebral blood flow signal into step (b). ), and estimating the degree of cognitive load for the content of the conversation of the subject by determining one class out of a predetermined number of classes to which the differential entropy vector belongs in the cluster space step (b) separates the cerebral blood flow signal for the first predetermined time period into non-overlapping sections of the second predetermined time period, and uses the variance of the prefrontal cortex activity of the subject for each section to obtain a predetermined probability distribution It is an estimation method that computes the differential entropy vector for .

According to the present invention, it is possible to estimate the magnitude of the subject's cognitive load during conversation based on brain activity.

The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of the embodiments with reference to the drawings.

FIG. 1 is a block diagram showing an example of the electrical configuration of the estimation system of the first embodiment. FIG. 2(A) is an external view of the brain activity measuring device, and FIG. 2(B) is a diagram showing the positional relationship between the light emitting section and the light receiving section of the brain activity measuring device. FIG. 3 is a diagram showing the positional relationship when the subject and the robot converse. FIG. 2 is a diagram showing the arrangement positions of a plurality of electrodes provided on a brain cap that functions as an electroencephalogram detection device, and the arrangement positions of the irradiation unit and the light reception unit of the electroencephalography activity measurement device superimposed on the arrangement positions of the electrodes. FIG. 5 shows the entire period of the cerebral blood flow signal detected to construct the cluster space, the length of the unit for calculating the differential entropy vector, and the length of the unit for calculating the variance when calculating the differential entropy vector. It is a figure for explaining. 6 is a diagram showing an example of a memory map of the RAM shown in FIG. 1. FIG. FIG. 7 is a flowchart showing an example of construction processing of a CPU incorporated in the estimation apparatus shown in FIG. 1; FIG. 8 is a flowchart showing an example of estimation processing of a CPU incorporated in the estimation apparatus shown in FIG. 1; FIG. 9 is a flow chart showing an example of speech control processing of a CPU incorporated in the estimation device shown in FIG. FIG. 10 is a block diagram showing an example of the electrical configuration of the estimation system of the second embodiment.

<First embodiment>
FIG. 1 is a block diagram showing the electrical configuration of the estimation system 10 of the first embodiment. Estimation system 10 includes estimator 12 , which is connected to display 14 , A/D converter 16 and robot 20 . Also, the A/D converter 16 is connected to a brain activity measuring device 18 . As will be described later, the robot 20 included in the estimation system 10 is provided for conversing with the subject, and outputs voice according to the topic and utterance content determined by the estimation device 12 . That is, the estimation system 10 also functions as a conversation system, and the estimation device 12 also functions as a conversation control device.

Note that the electrical configuration of the estimation system 10 shown in FIG. 1 is merely an example and should not be limited. For example, estimation device 12 may include display device 14 . In addition, as will be described later, since the differential entropy vector calculated based on the cerebral blood flow signal may be input to the estimating device 12, the A/D converter 16 and the brain activity measuring device 18 are connected to the estimating device 12. It does not have to be connected.

The estimation device 12 is a computer such as a general-purpose personal computer or a workstation, and includes a CPU 30. The CPU 30 connects an HDD 32, a RAM 34, a display control unit 36, an input device 38, an input/output interface (hereinafter referred to as (referred to as “input/output I/F”) 40 and a communication circuit 42 . The display device 14 is connected to the display controller 36 and the A/D converter 16 is connected to the input/output I/F 40 .

The HDD 32 is the main storage device of the estimation device 12 and stores an operating system and various application programs. A RAM 34 is a volatile memory and is used as a working area and a buffer area for the CPU 30 .

The display control unit 36 is a driver of the display device 14, and under the control of the CPU 30, the degree of cognitive load (level), specifically, the perceived difficulty of the content of communication (chatter or conversation). Outputs a display signal for displaying the estimated estimation result.

Input device 38 means at least one of a keyboard, computer mouse and touch pad (or touch panel).

The input/output I/F 40 receives signals from the A/D converter 16 . The A/D converter 16 amplifies the analog signal input from the brain activity measuring device 18 , converts it into a digital signal, and inputs it to the CPU 30 via the input/output I/F 40 . However, in this first example, the brain activity is that of the prefrontal cortex. (The analog signal input from the brain activity measuring device 18 is a signal about cerebral blood flow (hereinafter referred to as "cerebral blood flow signal").

1 shows that the A/D converter 16 and the input/output I/F 40 are connected by wire, the A/D converter 16 is incorporated in the brain activity measuring device 18, and the brain activity measuring device 18 and the input/output I/F 40 may be wirelessly connected.

The brain activity measuring device 18 is a device for measuring the state of brain activity using near-infrared spectroscopy (NIRS), and irradiates near-infrared light, which is highly permeable to living tissue, from the scalp to measure changes in hemoglobin concentration. measure. Briefly, the near-infrared light emitted from the brain activity measuring device 18 is scattered by living tissue, and part of the scattered near-infrared light passes through the cerebral cortex existing inside the skull of the subject. , reaches the scalp again as reflected light. This reaching position is known to be about 30 mm away from the light irradiation position in the case of adults (Reference 1: Li, T., Luo, Q. & Gong, H. Gender-specific hemodynamics in prefrontal cortex during a verbal working memory task by near-infrared spectroscopy. Behavioral Brain Research, 209, 148-153 (2010)). Here, since the near-infrared light is absorbed by hemoglobin in the blood, the change in the reflected light intensity detected at the arrival position described above reflects the change in the concentration of hemoglobin due to brain activity. Thereby, the cerebral blood flow is detected.

As shown in FIG. 2A, the brain activity measuring device 18 of the first embodiment has an appearance like a C-shaped hair band when viewed from above or below. A light-emitting portion for near-infrared light and a light-receiving portion for reflected light are provided on the surface (see FIG. 2B).

The brain activity measuring device 18 is attached to the subject's forehead so that the irradiation unit and the light receiving unit are in contact with or face each other (see FIGS. 2A, 2B, and 3). As shown in FIG. 2B, the brain activity measuring device 18 is provided with two irradiation units TL and TR and four light receiving units L1, L3, R1 and R3. The distance of the light receiving part L1 from the irradiation part TL is 10 mm, and the distance of the light receiving part L3 from the irradiation part TL is 30 mm. Further, the distance of the light receiving section R1 from the irradiation section TR is 10 mm, and the distance of the light receiving section R3 from the irradiation section TR is 30 mm. Reflected near-infrared light emitted from the irradiation part TL is received by the light receiving parts L1 and L3, and reflected near-infrared light emitted from the irradiation part TR is received by the light receiving parts R1 and R3.

As the brain activity measuring device 18, a portable brain activity measuring device (wearable optical topography system) “HOT-1000” manufactured by NeU Co., Ltd. can be used. In this case, one light receiving section is provided on each side, and the measurement position (distance from the irradiation section) is adjusted by a slide system.

FIG. 4 shows the arrangement positions of a plurality of electrodes provided on a brain cap functioning as an electroencephalogram detection device, and the arrangement positions of the irradiation units TL and TR and the light receiving units L1, L3, R1 and R3 superimposed on the arrangement positions of the electrodes. It is a diagram. However, in FIG. 4, the electrodes provided on the brain cap are indicated by circles, and the irradiation parts TL, TR and the light receiving parts L1, L3, R1, R3 are indicated by squares.

The multiple electrodes are arranged according to the International 10-20 method, as shown in FIG. FIG. 4 schematically shows the arrangement of a plurality of electrodes when the subject wears the brain cap, viewed from above the subject's head. In FIG. 4, the irradiation unit TL is arranged diagonally above the electrode F7 to the right, and the irradiation unit TR is arranged diagonally above the electrode F8 to the left. The light receiving portion L1 is arranged on the left side of the electrode FP1, and the light receiving portion L3 is arranged on the right side of the electrode FP1. The light receiving portion L1 is arranged on the left side of the electrode FP1, and the light receiving portion L3 is arranged on the right side of the electrode FP1.

A cerebral blood flow signal detected by the brain activity measuring device 18 is amplified and digitally converted by the A/D converter 16 and supplied to the estimating device 12 as described above.

In this first example, frontal lobe activity is recorded by detecting total blood flow by emitting laser light with a wavelength of 810 nm at a sampling rate of 10.0 Hz.

Studies on the activation of brain regions during memory and language processing suggest a more specific left lateral activation in both genders, especially females (Ref. 2: Cooper, R., Selb, J., Gagnon, L., Phillip, D., Schytz, H.W., Iversen, H.K., Ashina, M. & Boas, D.A. A systematic comparison of motion artifact correction techniques for functional near-infrared spectroscopy, Frontiers in Neuroscience, 6, 147 (2012)).

In this first embodiment, a signal based on the output of the light receiving section L3 is detected as the cerebral blood flow signal based on the references 1 and 2 above. That is, the cerebral blood flow in the left side portion of the prefrontal cortex is detected.

The detected cerebral blood flow signal is amplified and digitally converted by the A/D converter 16, and the amplified and digitally converted data (cerebral blood flow data) is preprocessed. In this first embodiment, as preprocessing, normalization processing, filtering with a first-order polynomial Butterworth filter, linear detrending, principal component analysis, etc. are applied (executed). Specifically, from the cerebral blood flow data, the mean value of the 1-minute rest period is subtracted as a baseline. This normalizes the cerebral blood flow data. The normalized cerebral blood flow data is also filtered with a first order polynomial Butterworth filter using 0.01 Hz for the low pass and 0.6 Hz for the high pass. Further, the filtered cerebral blood flow data are subjected to linear detrending and then principal component analysis. That is, the cerebral blood flow data are normalized and artifact-removed. However, the artifacts are physiological artifacts such as heartbeat, respiration, and motion artifacts such as head movement. The robot 20 is a robot capable of executing at least voice communication. Although detailed description is omitted, in the first embodiment, the robot 20 includes at least a control unit, a communication unit, and an audio output unit, and receives and outputs audio of utterance content on the topic determined by the estimation device 12. Output and have a conversation with the subject. In other words, the robot 20 is a subject's communication target.

As the robot 20, RoboBee (registered trademark) developed by the institution to which the applicant of this application belongs can be used.

As shown in FIG. 3, the robot 20 as a communication target and the subject are positioned facing each other at a predetermined distance (1.4 m in this first embodiment). The robot 20 is placed on the floor, desk, or table, and adjusted so that the part corresponding to the face of the robot 20 faces the face of the subject sitting on the chair.

Topics are contents that give cognitive load to subjects, and examples include tourist attractions in Kyoto (temples), tourist attractions in Kyoto (guidance to temples), and the like. For example, in the topic of sightseeing spots (temples) in Kyoto, the robot 20 is made to utter "Please tell me three famous temples in Kyoto", "Please tell me a temple in Kyoto where autumn leaves are beautiful", and the like. In addition, in the topic of sightseeing spots in Kyoto (guidance to temples), the robot 20 was asked, "What kind of transportation should I take from JR Kyoto Station to Kiyomizu Temple?", "From Sanjo Keihan Station to Yasaka Shrine." Please tell me the walking route."

It is thought that the topic of tourist attractions in Kyoto (temples) and the topic of tourist attractions in Kyoto (guidance to temples) have a lower cognitive load than the latter topic. In this way, a plurality of topics with different difficulty levels, ie, cognitive load levels, are prepared for the content of the conversation, and the estimating device 12 selects a topic according to a predetermined rule, which will be described later, and causes the robot 20 to speak.

In this first embodiment, the cerebral blood flow rate of a subject who is conversing with the robot 20 is detected, a feature amount is extracted from the cerebral blood flow rate, and the magnitude of cognitive load (hereinafter referred to as "cognitive load ) is estimated (or judged), and the topic is changed or maintained according to the degree of cognitive load. However, cognitive load means cognitive resources that load working memory, ie, working memory. Working memory means temporary storage of information related to the performance of cognitive tasks.

In addition, in the first embodiment, as a discriminator for estimating the degree of cognitive load of the subject, data with similar feature amounts extracted from cerebral blood flow are collected, and in a space in which several groups (classes) are formed. A cluster space is constructed. In order to construct the cluster space, a speech N-back task is used as a cognitive task. The N-back task is also called a continuous processing task, and is a task to answer the stimulus presented N times before. In the N-back task, it is considered that it is difficult to memorize the task and that the change in cerebral blood flow due to habituation is unlikely to occur. In addition, since the difficulty level increases as the value of N increases, the difficulty level of the task can be easily set. In this first embodiment, N=1 or 2 is set. That is, the difficulty level of the N-back task is set to two levels (low, high).

The estimating device 12 acquires the cerebral blood flow signal of the subject when the subject performs the N-back task, and based on the feature amount (differential entropy in the first embodiment) of the acquired cerebral blood flow signal, cluster space to build.

The reason for adopting differential entropy (that is, the average amount of information of continuous random variables) as a feature amount is that the inventors' research (references 3 and 4) shows that differential entropy is the prefrontal cortex (PFC) of human subjects. This is because we show that it is significantly superior to the feature spaces mainly used for classifying time-series data by functional near-infrared spectroscopy of activity.

However, Reference 3 is "Keshmiri S., Sumioka, H., Yamazaki, R. & Ishiguro, H. A Non-parametric Approach to the Overall Estimate of Cognitive Load Using NIRS Time Series, Frontiers in Human Neuroscience, 11, p.15 (2017)." Reference 4 is "Keshmiri S., Sumioka, H., Yamazaki, R. & Ishiguro, H. Differential Entropy Preserves Variational Information of Near-Infrared Spectroscopy Time Series Associated with Working Memory , Frontiers in Neuroinformatics, 12 (2018).”

Now, in this first embodiment, the method of constructing the cluster space will be described in detail. In this first embodiment, after a 1-minute rest period, the subject is asked to perform the N-back task for a first predetermined time (5 minutes). A task for 1 back or 2 back is performed, and the subject, while listening to the sound of a series of numbers (numbers 0 to 9), repeats the number one or two before from the number displayed on the screen. Select and click. A cerebral blood flow signal of the subject at this predetermined time is recorded.

The cerebral blood flow data obtained by A/D converting the recorded cerebral blood flow signal is preprocessed as described above. The normalized and artifact-removed cerebral blood flow data, which has been preprocessed, is separated into non-overlapping segments every second predetermined time period (eg, 20 seconds) as shown in FIG. , a differential entropy vector is calculated according to Equations 1 and 2.

As shown in FIG. 5, each segment is separated into non-overlapping sub-segments every third predetermined time (e.g., 5 seconds), and the vector of differential entropy has a predetermined number M (4 in this example) of elements consists of That is, in this first embodiment, the differential entropy vector for each segment is a four-dimensional vector. In Expression 1, H is the differential entropy vector of the segment X of interest, and Xi means the i-th subsegment of the segment X of interest.

[Number 2]

In this first embodiment, all differential entropy vectors calculated for each segment are clustered by the K-means method. Specifically, all differential entropy vectors are divided into two classes: a class with low cognitive load (sometimes called “class C1”) and a class with high cognitive load (sometimes called “class C2”) for the N-back task. classified into one class (or cluster). In other words, according to the cognitive load induced by the N-back (N = 1, 2) task with speech, the time-series data of the feature amount of the prefrontal cortex activity of the subject detected by near-infrared spectroscopy (NIRS) was K-meaned. An algorithm is used to classify into K (2 in this second example) clusters. That is, a cluster space having two clusters is generated. However, the norm of the vector of the center (or center of gravity) of each of the two clusters is calculated, and the class with the smaller value is determined as class C1, and the class with the larger value is determined as class C2.

In the K-means method, vectors with close spatial distances are classified into K clusters. , and the vector of the differential entropy obtained when performing the N-back task with a high degree of difficulty may belong to class C1.

When the cluster space is generated (constructed), as described above, using this cluster space, the estimating device 12 determines the content (or topic) of the conversation between the subject and the robot 20. Estimate or determine the difficulty or cognitive load perceived by

Here, in the first embodiment, a method for estimating the degree of cognitive load of the subject conversing with the robot 20 will be described in detail. However, the same processing as the construction of the cluster space described above will be briefly described.

Before conversing with the robot 20, the subject remains quiet for 1 minute. That is, there is a rest period of one minute, similar to the construction process. After one minute has passed, the robot 20 selects the content (topic) of conversation with the subject according to a predetermined rule, and provides the topic by voice. In this first embodiment, the predetermined rule is that, for subjects who have a history of conversation, a topic estimated to have a high degree of cognitive load (a topic estimated to be difficult) is selected, and for subjects who have no history of conversation, selects topics at random.

During the conversation with the robot 20 as a communication target, the cerebral blood flow of the subject is measured by the brain activity measuring device 18 and the cerebral blood flow signal is input to the estimation device 12 via the A/D converter 16 . The estimating device 12 acquires the cerebral blood flow signal for a second predetermined time period (20 seconds in this first embodiment), and further acquires the cerebral blood flow signal for a third predetermined time period (20 seconds in this first embodiment). In the example, it is divided every 5 seconds), the variance for each divided sub-segment is calculated according to Equation 2, and the differential entropy for each sub-segment is calculated according to Equation 1. Therefore, a differential entropy vector for the second predetermined time period is calculated. It is then determined to which class C1 or C2 in the cluster space the differential entropy vector belongs. Specifically, it is determined whether the position of the differential entropy vector in the cluster space is closer to the center of gravity of class C1 or the center of gravity of class C2, and it is determined that it belongs to the closer class C1 or C2. During the conversation, it is determined that each segment belongs to class C1 or C2, and when the estimation results for a predetermined number of times (in this first embodiment, nine times) are obtained, the topic of the conversation (that is, the current Estimate the degree of difficulty perceived by the subject, that is, the degree of cognitive load. A majority vote is taken for the estimation results for a predetermined number of times, and the larger estimation result is determined as the degree of cognitive load for the current topic. That is, in the first embodiment, the degree of cognitive load on the current topic is estimated when the current topic is spoken for 3 minutes.

In addition, the estimation device 12 compares the current estimation result with the past estimation result for the current topic, and rewrites (that is, updates) the estimation result when the estimation results are different. However, for new topics for which past estimation results are not described, the current estimation results are recorded as meta information (that is, labeled) in relation to the topic data. Here, a new topic means a topic that increases the load on the brain, and is added by the administrator of the estimation system 10 or the estimation device 12 or the like.

Furthermore, when the current estimation result indicates that the difficulty perceived by the subject is low, the estimation device 12 changes the topic based on the history of the estimation result. In this first embodiment, the estimating device 12 selects a topic different from the current topic and recorded (or labeled) as meta-information indicating that the difficulty level is high as an estimation result. select. If there is no other topic recorded as meta information indicating a high degree of difficulty, a new topic is selected. However, the topic is changed even when the robot 20 has uttered all the contents of the current topic. In this case, for example, the robot 20 requests transmission of the topic.

FIG. 6 is a diagram showing an example of a memory map 300 of the RAM 34 incorporated in the estimation device 12 shown in FIG. As shown in FIG. 6, RAM 34 includes program storage area 302 and data storage area 304 . The program storage area 302 stores an information processing program, which includes a cerebral blood flow acquisition program 302a, a variance calculation program 302b, a differential entropy calculation program 302c, a construction program 302d, an estimation program 302e, an utterance control program 302f, and the like. include.

The cerebral blood flow acquisition program 302a is a program for acquiring cerebral blood flow data corresponding to the cerebral blood flow signal output from the brain activity measuring device 18 and amplified and digitally converted by the A/D converter 16, CPU 30 stores the acquired cerebral blood flow signal in data storage area 304 .

The variance calculation program 302b is a program for calculating the variance for each subsegment of electroencephalogram blood flow data according to Equation (2). The differential entropy calculation program 302c is a program for calculating the differential entropy for each segment of the cerebral blood flow data according to Equation 1 using the variance for each subsegment calculated according to the variance calculation program 302b.

The construction program 302d is a program for constructing a cluster space by clustering the differential entropy vectors for each segment calculated according to the differential entropy calculation program 302c for all segments included in the entire period in the construction step, using the K-means method. is. In this first embodiment, the vector of differential entropy is classified into class C1 or class C2, as described above.

The estimation program 302e determines whether the differential entropy vector based on the cerebral blood flow signal of the subject in the estimation step calculated according to the differential entropy calculation program 302c belongs to class C1 or class C2, and performs a predetermined number of estimations. Based on the results, this is a program for estimating the degree of cognitive load on the current topic.

The speech control program 302f is a program for determining or changing a topic according to a predetermined rule or based on the estimation results according to the estimation program 302e, and transmitting voice data about the determined or changed topic to the robot 20. .

Although illustration is omitted, the program storage area 302 also stores other programs necessary for controlling the operation of the estimation device 12 as a computer.

The data storage area 304 stores topic data 304a, cerebral blood flow data 304b, differential entropy data 304c, cluster data 304d, and estimation result data 304e.

The topic data 304a is speech data for each of a plurality of topics presented to the subject through the robot 20, and includes a plurality of speech data corresponding to the utterance content for each topic. However, for topics whose cognitive load has been estimated in the past, the estimated cognitive load is recorded (labeled) as meta-information.

The cerebral blood flow data 304b is data corresponding to a preprocessed cerebral blood flow signal input from the brain activity measuring device 18 via the A/D converter 16 . The differential entropy data 304c is data on the differential entropy vector for the segment calculated according to Equation 1, and in the construction step, includes data on the differential entropy vector for all segments included in the entire period, The estimation step includes data about the differential entropy vector for a predetermined number of segments.

The cluster data 304d is data on the cluster space constructed according to the construction program 302d, specifically, data on the barycentric position of class C1 and the barycentric position of class C2. The estimation result data 304e is data about the degree of cognitive load on the current topic estimated according to the estimation program 302e.

Although not shown, the data storage area 304 stores other data necessary for executing the information processing program, and is provided with a counter (timer) and flags necessary for executing the information processing program.

FIG. 7 is a flowchart showing an example of construction processing (construction step processing) of the CPU 30 incorporated in the estimation device 12 shown in FIG. As shown in FIG. 7, when the construction process is started, the CPU 30 acquires the cerebral blood flow signal of the subject performing the cognitive task in step S1. Here, the cerebral blood flow signal for the first predetermined time period (5 minutes) is amplified and digitally converted, and preprocessed cerebral blood flow data is obtained.

In the next step S3, the variance for each sub-segment is calculated according to Equation 2 for each segment. Subsequently, in step S5, a differential entropy vector for each segment is calculated. Here, the CPU 30 calculates the differential entropy for each sub-segment according to Equation 1 on a segment-by-segment basis, and calculates the differential entropy vector for each segment.

In the subsequent step S7, all the calculated differential entropy vectors are clustered into two classes C1 and C2 by the K-means method. Then, in step S9, the barycenter positions of classes C1 and C2, that is, the cluster data 304d are stored in the data storage area 304, and the construction process ends.

FIG. 8 is a flow chart showing an example of estimation processing (processing of an estimation step) of the CPU 30 incorporated in the estimation device 12 shown in FIG. The estimation processing will be described below, and the same processing as the construction processing shown in FIG. 6 will be briefly described.

When the robot 20 converses with the subject, after a pause period of 1 minute elapses, the CPU 30 starts the estimation process as shown in FIG. Acquire the blood flow signal. Here, the CPU 30 acquires cerebral blood flow data for the second predetermined time period.

In the next step S23, the variance for each third predetermined time period is calculated, and in step S25, the differential entropy vector of the cerebral blood flow signal for the second predetermined time period is calculated. Subsequently, in step S27, the degree of cognitive load of the current topic is calculated. Here, it is determined whether each of the differential entropy vectors for a predetermined number of times (nine times in this first embodiment) belongs to the class C1 or C2, and the current topic is recognized from the estimation results for the predetermined number of times. The degree of load is determined by majority vote.

Then, in step S29, the estimation result is output, and the estimation process ends. In the first embodiment, the estimation result is output to the speech control processing section of the CPU 30. FIG.

It should be noted that the CPU 30 executes the estimation process a predetermined number of times (nine times in the first embodiment) each time the topic is changed until the robot 20 ends the conversation with the subject.

FIG. 9 is a flowchart of an example of speech control processing of the CPU 30 incorporated in the estimation device 12 shown in FIG. As shown in FIG. 9, when the speech control process is started, the CPU 30 selects a topic according to the predetermined rule described above in step S51. In the next step S51, the topic selected in step S51 is transmitted to the robot 20. FIG. That is, the CPU 30 transmits to the robot 20 the voice data of the utterance content on the selected topic.

In the subsequent step S55, it is determined whether or not a predetermined number of estimation results (9 times in this first embodiment) have been obtained. If "NO" in step S55, that is, if the estimation results for the predetermined number of times have not been acquired, the process returns to step S55. On the other hand, if "YES" in step S55, that is, if estimation results for the predetermined number of times have been acquired, it is determined in step S57 whether or not to change the topic. In this step S57, CPU 30 determines whether or not the result determined based on the estimation results for the predetermined number of times indicates that the degree of cognitive load is low.

The CPU 30 determines not to change the topic when the result determined from the estimation results for the predetermined number of times indicates that the degree of cognitive load is high. On the other hand, the CPU 30 determines that the topic should be changed when the result determined from the predetermined number of estimation results indicates that the degree of recognition is low. However, even if all the contents of the utterance about the current topic have been uttered, the robot 20 requests the transmission of the topic, and it is determined that the topic should be changed.

If "NO" in step S57, that is, if the subject is not changed, the process returns to step S57. On the other hand, if "YES" in step S57, that is, if the topic is to be changed, it is determined in step S59 whether or not the current topic is a new topic. Here, the CPU 30 determines whether information on the degree of cognitive load is labeled for the current topic.

If "YES" in step S59, that is, if the current topic is a new topic, then in step S61, the current topic (that is, the new topic) is labeled with information on the degree of cognitive load estimated this time. , the process proceeds to step S67.

On the other hand, if "NO" in step S59, that is, if the current topic is not a new topic, it is determined in step S63 whether or not the cognitive load level estimated this time matches the cognitive load level in the history.

If "YES" in step S63, that is, if the cognitive load degree estimated this time matches the cognitive load degree in the history, the process proceeds to step S67. On the other hand, if "NO" in step S63, that is, if the currently estimated cognitive load does not match the historical cognitive load, the cognitive load is updated in step S65, and the process proceeds to step S67. That is, in step S65, the CPU 30 rewrites the cognitive load degree of the history to the cognitive load degree estimated this time.

In the first embodiment, when changing the topic, it is determined whether or not the cognitive load level estimated this time matches the cognitive load level in the history, and if they do not match, the cognitive load level is updated. However, even when the topic is not changed, the same determination may be made, and the cognitive load level may be updated according to the determination result.

In step S67, the topic is changed. In the first embodiment, the CPU 30 selects a topic other than the current topic and labeled as having a high degree of cognitive load as information.

In the next step S69, the changed topic is transmitted to the robot 20. FIG. Then, in step S71, it is determined whether or not the processing is finished. Here, it is determined whether or not an end instruction has been input. If "NO" in step S71, that is, if the process is not finished, the process returns to step S55. At this time, a counter that counts the number of obtained estimation results is reset. On the other hand, if "YES" in step S71, that is, if it is finished, the speech control process is finished.

According to this first embodiment, a differential entropy vector is calculated for each predetermined time based on the cerebral blood flow signal of a subject performing a cognitive task, and the calculated differential entropy vector is used for different degrees of cognitive load. Create a cluster space divided into two classes, use this cluster space to acquire the cerebral blood flow signal when the subject talks with the communication target robot, and calculate based on the acquired cerebral blood flow signal. Based on the class to which the differential entropy vector obtained from the analysis belongs to, the degree of difficulty perceived by the subject, that is, the degree of cognitive load, is estimated. can be estimated.

Further, according to the first embodiment, the topic is changed according to the estimated degree of cognitive load of the subject, so that the utterance of the robot can be appropriately controlled.

Furthermore, according to the first embodiment, the degree of cognitive load is estimated based on the cerebral blood flow, so the estimation result of the degree of cognitive load is superior to the case of estimating in consideration of facial expression and/or gaze direction. can be obtained. For example, when the subject looks at the communication target with a serious expression, judging from such facial expressions, it may be assumed that the subject is properly listening to the communication target's utterance. This is because they may not have heard of In addition, even if the communication target person is not looking at the communication target person, there are cases where the communication target person is listening to the utterance or understanding the utterance content.

In the first embodiment, the robot serving as the communication target provides the subject with a topic, and the robot and the subject converse with each other. It may be a virtual character that has been created. In this case, instead of the robot, a display device (14) is arranged so as to face the subject, and the virtual character is displayed on the display device (14) and a speaker (not shown) connected to the estimation device. The voice about the topic is output from. In other words, agents such as robots and virtual characters (that is, conversation agents) can be used as communication targets.

Moreover, in the first embodiment, the cluster space was constructed based on the cerebral blood flow signal when the subject performed the cognitive task for 5 minutes, but it is not necessary to be limited to this. Depending on the application of the estimation system (estimation device), a cluster space may be constructed based on each of the cerebral blood flow signals that a plurality of subjects performed a five-minute cognitive task.

Furthermore, in this first embodiment, the case of using the estimation system and the estimation device for one subject has been described, but there is no need to be limited to this. It can also be used for multiple subjects. In such a case, the labeling of the information on the degree of cognitive load on the topic is performed so that the subject can be identified. This is because the degree of cognitive load on the topic differs for each subject.

Furthermore, in the first embodiment, topic data is stored in the estimating device, and speech data about the selected topic is transmitted to the robot, but the present invention is not limited to this. Topic data may be stored in the robot, the identification information of the selected topic may be transmitted from the estimating device to the robot, and voice data on the topic indicated by the received identification information may be output by the robot.

<Second embodiment>
In the estimation system 10 of the second embodiment, the robot 20 converses with the subject in response to commands from an external computer 22 communicably connected to the estimation device 12, and the degree of cognitive load estimated about the current topic is sent to the external computer 22. This is the same as the first embodiment except that the operator of the external computer 22 who has output and confirmed the degree of cognitive load changes the topic. In the following, the estimation system 10 of the second embodiment will be explained, but the contents overlapping with the contents explained in the first embodiment will be omitted.

As shown in FIG. 10, in the estimation system 10 of the second embodiment, the external computer 22 is connected to the communication circuit 42 of the estimation device 12, and the robot 20 is connected to this external computer 22. As shown in FIG. In the second embodiment, robot 20 converses with a subject in response to commands from external computer 22 . In other words, the topic presented to the subject by the robot 20 is determined by the operator of the external computer 22 .

That is, in the estimation system 10 of the second embodiment, the estimation result is output to the external computer 22 in step S29 in the estimation process. In addition, in the speech control process, the operator of the external computer 22 selects the topic in step S51 and changes the topic in step S67.

However, the speech control process shown in FIG. 9 may be executed by the processor of the external computer 22 to control the speech of the robot 20 without intervention of the operator of the external computer 22 .

Also in the second embodiment, as in the first embodiment, it is possible to estimate the magnitude of the subject's cognitive load during conversation based on brain activity.

Also in the second embodiment, as in the first embodiment, the robot can be appropriately made to converse according to the subject's degree of cognitive load.

Furthermore, in the second embodiment, as in the first embodiment, it is possible to obtain an estimation result of the degree of cognitive load that is superior to the case of estimating in consideration of the facial expression and/or the line-of-sight direction.

In addition, in the second embodiment, it is also possible to present the contents of the speech of the operator of the external computer to the subject through the robot. In such a case, as the robot, it is possible to use not only the above-mentioned Robobee (registered trademark), but also the Telenoid (registered trademark) developed by the institution to which the applicant belongs. However, when using Telenoid (registered trademark), a mobile device such as a smartphone or a mobile phone attached to it is communicatively connected to the estimation device, and voice data is transmitted from the mobile device through a speaker provided in Telenoid. output.

Also, in each of the above-described embodiments, an N-back task was used as a cognitive task to construct a cluster space, but the present invention is not limited to this, and other tasks can also be used. Other tasks include listening span tests, Stroop tasks, and mental arithmetic tasks.

Furthermore, in each of the above-described embodiments, a cluster space containing two classes, a class C1 with a low cognitive load on the N-back task and a class C2 with a high cognitive load, is constructed and used to estimate the cognitive load on the topic. However, a cluster space containing three or more classes may be constructed. In such a case, the stages of cognitive load for the N-back task are set according to the number of classes. Therefore, topics provided by robots and the like are also classified into stages according to stages of cognitive load. Then, when providing a topic, the topic may be changed so as to gradually increase the cognitive load according to the estimated degree of cognitive load.

Further, the specific numerical values shown in each of the above examples are merely examples, and should not be limited, and can be appropriately changed according to the product or the like to be implemented.

DESCRIPTION OF SYMBOLS 10... Estimation system 12... Estimation device 14... Display device 16... A/D converter 18... Brain activity measuring device 20... Robot 30... CPU
32 HDD
34 RAM
40 ... Input I/F
42 ... communication circuit

Claims (6)

An estimating device for estimating the degree of cognitive load perceived by a subject for a topic based on the cerebral blood flow of the subject who has a conversation with a communication target,
Acquisition means for acquiring a cerebral blood flow signal that is a signal about cerebral blood flow of a subject performing a cognitive task;
a calculating means for calculating a differential entropy vector of the cerebral blood flow signal obtained by the obtaining means for each first predetermined time;
A construction means for clustering the differential entropy vector calculated by the calculation means into a predetermined number of classes with different levels of cognitive load for the cognitive task using the K-means method to construct a cluster space;
The acquisition means acquires the cerebral blood flow signal when the subject converses with the communication target in constructing the cluster space, and the calculation means calculates a differential entropy vector of the acquired cerebral blood flow signal. , estimating means for estimating the degree of cognitive load regarding the content of the conversation of the subject by determining one of the predetermined number of classes to which the differential entropy vector belongs in the cluster space ;
The calculation means separates the cerebral blood flow signal for the first predetermined time period into non-overlapping sections for each second predetermined time period, and uses the variance of the prefrontal cortex activity of the subject for each section to obtain a predetermined probability distribution. An estimator for calculating the differential entropy vector for 2. The estimating apparatus according to claim 1, wherein said acquiring means acquires a cerebral blood flow signal in a left side portion of said subject's prefrontal cortex. 3. The estimating apparatus according to claim 1, further comprising information storage means for storing information on the degree of cognitive load estimated by said estimating means in relation to the content of said conversation. the communication target is a conversation agent;
4. The estimation device according to claim 3, further comprising topic selection means for selecting a topic to be provided by said conversation agent based on information stored by said information storage means. An estimation program executed by a computer for estimating the degree of cognitive load perceived by a subject for a topic based on the cerebral blood flow of the subject who has a conversation with a communication target,
to the processor of said computer;
an acquiring step of acquiring a cerebral blood flow signal that is a signal about cerebral blood flow of a subject performing a cognitive task;
a calculation step of calculating a vector of differential entropy of the cerebral blood flow signal obtained in the obtaining step for each first predetermined time;
A construction step of clustering the differential entropy vector calculated in the calculation step into a predetermined number of classes with different levels of cognitive load for the cognitive task using the K-means method to construct a cluster space;
obtaining, in the obtaining step, the cerebral blood flow signal when the subject in the construction of the cluster space converses with a communication target, and calculating a differential entropy vector of the obtained cerebral blood flow signal in the calculating step; executing an estimation step of estimating the degree of cognitive load of the subject's conversational content by determining one of the predetermined number of classes to which the differential entropy vector belongs in the cluster space ;
The calculating step separates the cerebral blood flow signal for the first predetermined time period into non-overlapping sections of the second predetermined time period, and uses the variance of the prefrontal cortex activity of the subject for each section to obtain a predetermined probability distribution. An estimation program that computes the differential entropy vector for . A computer- based estimation method for estimating the degree of cognitive load perceived by a subject for a topic based on the cerebral blood flow of the subject who has a conversation with a communication target,
(a) acquiring a cerebral blood flow signal that is a signal about cerebral blood flow in a subject performing a cognitive task;
(b) calculating a differential entropy vector of the cerebral blood flow signal obtained in step (a) for each first predetermined time;
(c) constructing a cluster space by clustering the differential entropy vector calculated in step (b) into a predetermined number of classes with different levels of cognitive load for the cognitive task using the K-means method;
(d) obtaining the cerebral blood flow signal when the subject converses with the communication target in constructing the cluster space in step (a); Estimate the degree of cognitive load regarding the content of the conversation of the subject by determining one of the predetermined number of classes to which the differential entropy vector belongs in the cluster space calculated in (b) including steps
The step (b) separates the cerebral blood flow signal for the first predetermined time period into non-overlapping sections of the second predetermined time period, and uses the variance of the prefrontal cortex activity of the subject for each section to obtain a predetermined An estimation method , wherein the differential entropy vector for a probability distribution is calculated .

Images