JP2016118575A - Device, system, program and method capable of estimating intracerebral intellectual activity state - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device capable of estimating an intracerebral intellectual activity state of a user who performs activity, using an interface device.SOLUTION: The intracerebral intellectual activity estimation device of the invention is a device coupled to an interface device, in which, intellectual activity information is input and output between the device and a user, and which can process the intellectual activity information. The intracerebral intellectual activity estimation device comprises: brain wave analysis means for, creating a brain wave analysis log in which pieces of brain wave information are recorded in time series, from the pieces of brain wave data acquired from the user; brain wave interpretation means for, based on plural interpretation rules in which, time lapse data of the brain wave information and an interpretation label of the intracerebral intellectual activity are associated in advance, determining an interpretation label candidate from the created brain wave analysis log; activity state grasp means for, based on the processing state of the intellectual activity information on the interface device, determining a state label candidate out of plural intracerebral intellectual activity state labels; and intellectual activity determination means for, determining a label content which is common between the determined interpretation label candidate and the determined state label candidate, to be a user's intracerebral intellectual activity state.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technique for analyzing a brain wave of a user and estimating a state of mental activity of the user.

  The state of mental activity of the user, such as the state of specific emotions or the state of intellectual activity in the brain such as being considered, is better if it is accurately grasped and appropriately reflected in the service content. This is important information that makes it possible to provide such a service to the user.

  As a technique for estimating a user's emotion among such mental activities, for example, in Patent Document 1, an emotion is estimated from a captured facial expression image, and further, an emotion is estimated from biological information obtained by a biosensor. And the technique of calculating an adjustment emotion estimation result by combining these estimation results using a weighting coefficient is disclosed. Here, as biometric information, information on an electroencephalogram or a heart rate is used. In addition, in this technique, it is also possible to adopt a method of estimating an emotion from a user's voice acquired using an acoustic technique with expanded voice recognition, and calculating an adjusted emotion estimation result by further adding this estimation result. .

  Further, Non-Patent Document 1 discloses a technique for adopting a model used in machine learning such as a hidden Markov model and estimating a relatively simple and basic emotion from a measurement result of an electroencephalogram. Specifically, techniques for estimating positive emotions, negative emotions, or corresponding brain wave types such as alpha waves and beta waves have been proposed.

  Furthermore, as an emotion estimation technique, Patent Document 2 calculates an emotion parameter indicating the degree of emotion based on biological information such as heart rate and blood pressure, and converts the emotion parameter into a one-dimensional emotion currency that can be added and subtracted. A technique for classifying items according to items such as time and experience contents and displaying them in a form that is easy to grasp is disclosed.

JP 2012-059107 A JP 2005-124909 A

Wei-Long Zheng, Jia-Yi Zhu, Yong Peng, and Bao-Liang Lu, `` EEG-BASED EMOTION CLASSIFICATION USING DEEP BELIEF NETWORKS '', Proc. Of IEEE International Conference on Multimedia and Expo (ICME) 2014

  However, with the prior art described above, it is possible to estimate the state of intellectual activity in the brain related to more complex emotions and mental processes, for example, “understood” or “under consideration”. It is difficult.

  Actually, in the prior arts described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, emotion estimation is performed using relatively short time acquisition information such as facial expressions, voices, and biological information including brain waves. Is done. In addition, the type of emotion to be estimated is set in advance according to the purpose of use of the emotion estimation result, and it is determined to which emotion type the user's state belongs. For example, in emotion estimation for driving a car, a state where attention is distracted, a sleepy state, a tense state, and the like are to be discriminated, and there is a limit in estimating more complicated emotions than these. As a result, the types of emotions estimated are related to basic emotions, including those related to physical condition, such as emotions, emotions, attention, confusion, some stress, and sleepiness. turn into.

  In this regard, although the technique described in Patent Document 1 attempts to increase estimation accuracy by performing emotion estimation using information acquired from a plurality of information sources, the content of more complicated emotions is still used. It is difficult to estimate the state of intellectual activity in the brain.

  In addition, with the techniques described in Patent Document 2 and Non-Patent Document 1, it is possible to record the time course of emotions. It is not possible to estimate until it is.

  Here, for example, assume an education service providing apparatus used by students for education. With this device, when trying to provide teaching materials and problems suitable for each student's learning status, in addition to the basic emotions of the students, learning such as “understood”, “under consideration”, “not sure” It is very important to understand the state of intellectual activity in the brain. In other words, such intellectual activity states in the brain are often associated with more complex mental processes than basic emotions, and it is only possible to grasp the student's state in more detail by estimating this state. It can be done. For example, according to the current state of intellectual activity of the brain, it is possible to change the question, present another teaching material or problem, or select a hint to give.

  As in this example of the educational service providing device, in the situation where the user is active using some kind of interface device, grasping the state of the intellectual activity in the user's brain is the quality of the provided service. It is very useful from the viewpoint of improvement. However, in the prior art as described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, it is very difficult to grasp such a state.

  Therefore, an object of the present invention is to provide an apparatus, a system, a program, and a method that can estimate an intellectual activity state in a brain of a user who is active using an interface apparatus.

According to the present invention, there is provided an intracerebral intellectual activity estimation device connected to an interface device capable of inputting and / or outputting information related to intellectual activity with a user and processing the information related to the intellectual activity. And
An electroencephalogram analysis means for acquiring the electroencephalogram data of the user and generating an electroencephalogram analysis log in which information relating to the electroencephalogram is recorded in time series from the electroencephalogram data;
An electroencephalogram interpretation means for determining interpretation label candidates from the generated electroencephalogram analysis log based on a plurality of interpretation rules in which temporal data of information relating to the electroencephalogram and interpretation labels of brain intellectual activities are associated in advance; ,
An activity status grasping means for determining status label candidates from a plurality of status labels of intellectual activities in the brain set in advance based on the processing status of information related to the intellectual activity in the interface device;
Intellectual activity in the brain having intelligent activity determination means for determining the content of the label common between the determined interpretation label candidate and the determined state label candidate as the brain's intellectual activity state of the user An estimation device is provided.

As one embodiment of the brain intellectual activity estimation apparatus according to the present invention, the electroencephalogram analysis means provides a time series of electroencephalogram records in which an electroencephalogram type is associated with an electroencephalogram level of the type for each time related to electroencephalogram generation. Generate an EEG analysis log arranged in
The electroencephalogram interpretation means extracts the output pattern represented by the time-lapse data with the state that outputs the electroencephalogram of a certain type and level as a node and the transition between the states as a branch from the electroencephalogram analysis log. It is also preferable to determine an interpretation label associated in advance as an interpretation label candidate.

Further, the interface device can output the task or proposal information to be processed by the user, and can input reaction or response information by the user with respect to the task or proposal information,
The activity status grasping means uses an activity status grasping table in which a status label is associated in advance for each possible combination of the output status of the subject or proposal information and the input status of the response or response information. It is also preferable to determine.

  In addition, each of the interpretation label and status label described above includes a label indicating that the consideration is in progress, a label indicating that the consideration has been completed, a label indicating that the understanding is completed, and a label indicating that the understanding is not achieved. It is also preferred that at least one or all of them are employed.

  As another embodiment of the brain intellectual activity estimation device of the present invention, when the intellectual activity determination means determines the content of the label indicating that the user has understood the intellectual activity state in the brain, the interface The device is instructed to output a task or proposal information corresponding to the next step of the output task or proposal information, while the intellectual activity determining means understands the intellectual activity state in the user's brain. If it is determined that the content of the label indicates that it is not, content instruction means for instructing the interface device to output information for allowing the interface device to understand the content of the output problem or proposal information may be further provided. preferable.

Further, regarding the electroencephalogram interpretation in the brain intellectual activity estimation apparatus of the present invention, at least one of the plurality of interpretation rules is based on a state in which a high gamma wave or a low gamma wave having a level equal to or higher than a predetermined threshold is output. Corresponding temporal data including transition to a state that outputs a low beta wave having a level and an interpretation label indicating that it is understood,
The electroencephalogram interpretation means extracts at least an interpretation label indicating the understanding when the output pattern expressed by the temporal data including the transition to the state of outputting the above-described beta wave is extracted from the electroencephalogram analysis log. It is also preferable to determine to.

According to the present invention, the brain intellectual activity estimation apparatus described above and at least one interface capable of outputting problem information related to education and inputting response information by the user for the problem information An educational system including a device and an electroencephalograph attached to each user using the interface device,
The above brain intellectual activity estimation device is configured to determine the state of intellectual activity of the user's brain determined by the intellectual activity determination means corresponding to the output problem information and / or the input answer information. And / or an educational system characterized by further comprising learning progress management means for recording and managing the information in association with the answer information.

According to the present invention, there is further provided a computer mounted on a device connected to an interface device capable of inputting and / or outputting information relating to intellectual activities with a user and processing the information relating to the intellectual activities. A program for estimating intellectual activity in the brain,
An electroencephalogram analysis means for acquiring the electroencephalogram data of the user and generating an electroencephalogram analysis log in which information relating to the electroencephalogram is recorded in time series from the electroencephalogram data;
An electroencephalogram interpretation means for determining interpretation label candidates from the generated electroencephalogram analysis log based on a plurality of interpretation rules in which temporal data of information relating to the electroencephalogram and interpretation labels of brain intellectual activities are associated in advance; ,
An activity status grasping means for determining status label candidates from a plurality of status labels of intellectual activities in the brain set in advance based on the processing status of information related to the intellectual activity in the interface device;
Brain intelligence that causes the computer to function as an intellectual activity determination means that determines the content of the label common between the determined interpretation label candidate and the determined state label candidate as the intellectual activity state of the user's brain. Activity estimation program is provided.

According to the present invention, furthermore, using the interface device capable of inputting and / or outputting information related to the intellectual activity with the user and processing the information related to the intellectual activity, the brain knowledge of the user can be obtained. A method of estimating social activity,
A first step of acquiring electroencephalogram data of the user and generating an electroencephalogram analysis log in which information relating to the electroencephalogram is recorded in time series from the electroencephalogram data;
Second step of determining interpretation label candidates from the generated electroencephalogram analysis log based on a plurality of interpretation rules in which temporal data of information relating to the electroencephalogram and interpretation labels of brain intellectual activities are associated in advance When,
A third step of determining a state label candidate from a plurality of preset state labels of intellectual activities in the brain, based on the processing status of information related to the intellectual activity in the interface device;
Intracerebral intellectual activity estimation comprising: a fourth step of determining the content of a label common between the determined interpretation label candidate and the determined state label candidate as the intellectual activity state of the user in the brain A method is provided.

  According to the brain intellectual activity estimation apparatus, system, program, and method of the present invention, it is possible to estimate the brain intellectual activity state of a user who is active using an interface device.

It is a mimetic diagram showing one embodiment of an education service providing device containing a brain intellectual activity estimating device by the present invention. It is a schematic diagram which shows one Embodiment of the education system containing the intellectual activity estimation apparatus in a brain by this invention. It is a functional block diagram which shows the function structure in one Embodiment of the brain intellectual activity estimation apparatus by this invention, and an interface apparatus. It is a table which shows the classification of the electroencephalogram measured. It is a table which shows one Embodiment of the produced | generated electroencephalogram analysis log. It is a schematic diagram which shows an example of the time-lapse graph used for interpretation. It is a table which shows an example of an electroencephalogram interpretation rule. It is a table which shows an example of the activity condition grasping table used for state grasping. It is a schematic diagram which shows one Example of the estimation process of the intellectual activity state in a brain in an intellectual activity determination part.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Education service provision device]
FIG. 1 is a schematic diagram showing an embodiment of an education service providing apparatus 1 including an intracerebral intellectual activity estimation apparatus 11 according to the present invention.

In the present embodiment, the education service providing apparatus 1 presents lesson contents and exercises in an E-learning format, for example, to the user, receives the answers, scores them, and proceeds with learning based on the results. It is a device that provides such educational services. Education service providing device 1
(A) an interface device 12 having an input / output interface capable of inputting / outputting information relating to intellectual activity (learning in the present embodiment) with a user, and capable of processing information relating to the intellectual activity;
(B) a brain intellectual activity estimation device 11 connected to the interface device 12 and capable of estimating the state of intellectual activity in the brain of a user who uses the interface device 12 (education service providing device 1); doing. The educational service providing apparatus 1 may be a personal computer (PC), a smartphone, a tablet computer, or the like that is equipped with an application program for providing educational services.

  The input / output interface of the interface device 12 includes a display for displaying lesson / teaching material contents, exercises, etc., a speaker that conveys information about them by voice, a keyboard and mouse that accepts input of answers by the user, and voice Can be at least one or all of the microphones that accept the answer. Alternatively, it may be a touch panel display that can input and output information related to learning.

  On the other hand, the brain intellectual activity estimation device 11 receives electroencephalogram measurement information including measured electroencephalogram data from an electroencephalograph 13 attached to a user who uses the education service providing device 1 and capable of measuring the electroencephalogram of the user. It has a possible wireless or wired communication interface.

According to the present invention, the brain intellectual activity estimation device 11 includes:
(A-1) Obtaining user's brain wave data, generating an electroencephalogram analysis log in which information related to the brain wave is recorded in time series from the brain wave data,
(A-2) “Interpretation label” from the generated electroencephalogram analysis log based on a plurality of interpretation rules in which “time data” of information relating to electroencephalogram and “interpretation label” of brain intellectual activity are associated in advance "Determine the candidate,
(B) Based on the “processing status” of the information relating to the intellectual activity in the interface device 12, a “status label” candidate is determined from the “status label” of a plurality of brain intellectual activities set in advance,
Moreover,
(C) The content of the label common between the determined “interpretation label” candidate and the determined “state label” candidate is determined as the “intellectual activity state in the brain” of the user ing.

  As described above, the brain intellectual activity estimation device 11 takes into account not only the brain wave data conventionally used for emotion estimation, but also the “processing status” of the intellectual activity information in the interface device 12, and the user's brain. Estimate the internal state. This makes it possible to estimate the “intellectual activity state in the brain” related to a more complex mental process than the basic emotion that has been measured in the past.

  Here, “intellectual activity state in the brain” means, for example, “understood”, “under consideration”, “under consideration”, “don't know”, “excellent” ”Or“ not in real condition ”. Such a state is due to a more complex mental process than basic emotions, and is not estimated simply by discriminating the type of electroencephalogram. For example, even if a comma wave is continuously measured, it cannot be determined from this that the subject user is simply frustrated. If the subject user is learning, this gamma wave can be regarded as evidence that they are concentrated on studying. Furthermore, if the alpha wave is measured after this gamma wave, the test user is probably relaxed. It cannot be determined.

  In response to such a situation, the brain intellectual activity estimation device 11 can also indicate the state of intellectual activity in the brain related to such a complex mental process as the “processing status” in the interface device 12. It can be estimated by taking into account.

  In addition, in a single user's brain, a plurality of different “brain intellectual activity states” often occur simultaneously. For example, a situation such as “good” and “under consideration” can occur sufficiently. In response to such a situation, the brain intellectual activity estimation device 11 uses a plurality of “interpretation” from the generated electroencephalogram analysis log based on a plurality of interpretation rules in which “temporal data” and “interpretation labels” are associated with each other. It is also possible to determine “label” candidates. It is also possible to determine a plurality of “state label” candidates based on the “processing status”. As a result, it is possible to estimate a plurality of “intellectual activity states in the brain” that occur simultaneously as described above.

  Further, the brain intellectual activity estimation device 11 outputs an estimation result of the “brain intellectual activity state” to the interface device 12, and the interface device 12 uses the estimation result to indicate the contents of the lesson / teaching material to be presented next, It is also preferable to determine exercises. Alternatively, the brain intellectual activity estimation device 11 itself determines the next lesson / teaching content and exercise problem to be presented based on the estimation result of the “brain intellectual activity state”, and sends the content instruction to the interface device 12. It may be output. Thereby, the educational service providing apparatus 1 can appropriately provide a high-quality educational service suitable for the state of the user to the user.

[Education system]
FIG. 2 is a schematic diagram showing an embodiment of the education system 2 including the brain intellectual activity estimation device 21 according to the present invention.

According to FIG. 2, the education system 2
(A) at least one (a plurality of in FIG. 2) interface device 22 capable of outputting problem information related to education and inputting response information by the user to the problem information;
(B) an electroencephalograph 23 that is attached to each user who uses the interface device 22 and can transmit electroencephalogram measurement data to the interface device 22;
(C) It has a brain intellectual activity estimation device 21.

  Of these, the interface device 22 has the same information processing function and input / output interface as the interface device 12 shown in FIG. 1, and further communicates with the brain intellectual activity estimation device 21 via the network. Communication interface. Here, not only information relating to the processing status in the apparatus 22 but also electroencephalogram measurement data is transmitted from the interface apparatus 22 to the brain intellectual activity estimation apparatus 21 via this network. The interface device 22 may be a PC, a smartphone, a tablet computer, or the like equipped with an application program for providing educational services.

  In addition, this network is the Internet and various access networks, for example, a fixed network such as an optical fiber network or ADSL (Asymmetric Digital Subscriber Line), or a wireless LAN (Local Area Network) such as Wi-Fi (registered trademark), A wireless network such as WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), or 3G (3rd Generation) can be used. In this case, the brain intellectual activity estimation device 21 may be a server installed in an access network managed by a communication carrier or on the Internet. Alternatively, this network may be a private network provided inside a school or company.

  In addition, the brain intellectual activity estimation device 21 has the same brain intellectual activity state estimation function as the brain intellectual activity estimation device 11 shown in FIG. A communication interface for communicating with the interface device 22 is provided. In addition, the brain intellectual activity estimation device 21 inputs the brain intellectual activity state, which is the estimation result, from the intellectual activity determination unit 217 that executes the process of estimating the brain intellectual activity state. It is also preferable to further include a learning progress management unit 219 for managing.

  Based on the processing status received from the interface device 22, the learning progress management unit 219 determines and inputs the intellectual activity in the brain determined in response to the problem information output from the device 22 and / or the input answer information. The state is recorded and managed in association with the problem information and / or answer information. Thereby, the intellectual activity estimation apparatus 21 in a brain can acquire the information about the degree of understanding, the degree of consideration, etc. of each user of the interface device 22 for each presented problem or each returned answer. As a result, based on the acquired information, it is possible to determine the contents of the lesson / teaching material to be presented next and the exercise problem, and to output the content instruction to the interface device 22.

  For example, when a large number of students (users) do not understand, for example, the contents of lessons in E-learning are changed or many students give up without taking the state of “under consideration”. If you change the exercises to be presented or if you have reached the “understood” state after taking the “under consideration” state for a sufficient amount of time, determine that the presented problem is appropriate for the student. It is also possible to do.

  The estimation process and the management process in the brain intellectual activity estimation apparatus 21 as described above cannot be appropriately executed only by grasping the simple emotional state of the user. In this embodiment, since it is possible to accurately estimate the intellectual activity state in the brain related to a more complex mental process, for example, the degree of understanding and the degree of consideration, it is possible to grasp the progress of learning of each user, Thus, for example, it is possible to provide appropriate feedback to the user.

[Equipment configuration / Brain intellectual activity estimation processing]
FIG. 3 is a functional block diagram showing a functional configuration in one embodiment of the brain intellectual activity estimation device 11 and the interface device 12 according to the present invention.

  According to FIG. 3, the brain intellectual activity estimation device 11 includes a communication interface 111 capable of communicating with the electroencephalograph 13, an electroencephalogram data storage unit 112, an electroencephalogram analysis log management unit 114, and a processor memory. . Here, the processor memory realizes the brain intellectual activity estimation function by executing a program that causes the computer mounted in the education service providing apparatus 1 (intracerebral intellectual activity estimation apparatus 11) to function.

  Further, the processor memory includes an electroencephalogram analysis unit 113, an electroencephalogram interpretation unit 115, an activity status grasping unit 116, an intelligent activity determination unit 117, and a content instruction unit 118 as functional components.

  Similarly, according to FIG. 3, the interface device 12 includes a keyboard 121, a microphone 122, a display 123, a speaker 124, a content management unit 125, and an information processing unit 126. Here, the information processing unit 126 is a computer (processor memory) installed in the education service providing apparatus 1 (interface apparatus 12), and realizes an information processing function by executing a program.

  In the present embodiment, an I / F 141 that is a communication interface with an external network is provided as a component of the education service providing apparatus 1. The brain intellectual activity estimation device 11 and / or the interface device 12 can communicate with an external network via the I / F 141. In addition, according to FIG. 3, the flow of the process which connected each function structure part with the arrow is understood also as one Embodiment of the method of estimating the brain intellectual activity state by this invention.

  The electroencephalograph 13 is equipped with an electrode system that enables measurement of a weak electroencephalogram of, for example, several μV (microvolts) and a circuit dedicated to electroencephalogram measurement, and can measure a user's electroencephalogram. It is a wearable device such as headphones or glasses. The electroencephalogram measuring device 13 further transmits the electroencephalogram measurement data via a wired, near field communication network such as Bluetooth (registered trademark), or via a wireless LAN such as Wi-Fi (registered trademark). It is also preferable to have a communication interface capable of transmitting to the other communication interface 111. 2 has a measurement function similar to that of the electroencephalograph 13, but transmits electroencephalogram measurement data to the interface device 22 by the same communication means.

  The I / F 141 is a communication interface of the educational service providing apparatus 1, but the brain intellectual activity estimation apparatus 11 and the interface apparatus 12 communicate with an external server or terminal via a wireless or wired access network and the Internet. It is a communication interface used to do. For example, the intellectual activity determination unit 117 (internal brain intellectual activity estimation device 11) transmits the estimation result of the intellectual activity state in the brain to the outside, or the learning content or educational program is externally transmitted to the information processing unit 126 (interface). It is also preferred that the I / F 141 is used for downloading to the device 12).

  The electroencephalogram data accumulation unit 112 accumulates the electroencephalogram data received from the electroencephalogram measurement device 13 and outputs the accumulated electroencephalogram data to the electroencephalogram analysis unit 113 as appropriate.

  The electroencephalogram analysis unit 113 generates an electroencephalogram analysis log in which information related to the electroencephalogram is recorded in time series from the input electroencephalogram data of the user. Specifically, the electroencephalogram analysis log may be an electroencephalogram record in which an electroencephalogram type and an electroencephalogram level of the type are associated in time series for each time related to the electroencephalogram generation. The generated electroencephalogram analysis log is accumulated in the electroencephalogram log management unit 114.

  FIG. 4 is a table showing the types of electroencephalograms to be measured.

  According to FIG. 4, the types of electroencephalogram to be measured are delta (δ) wave, theta (θ) wave, low alpha (low-α) wave, high alpha (high-α) wave, low beta (low-β). Wave, high beta (high-β) wave, low gamma (low-γ) wave, and high gamma (high-γ) wave, and the definitions in the present embodiment are frequencies of 0.5 to 4 Hz, 4 to 8 Hz, respectively. The brain waves correspond to 8 to 10 Hz, 10 to 12 Hz, 12 to 20 Hz, 20 to 30 Hz, 30 to 40 Hz, and 40 to 50 Hz (or 40 Hz or more). In this way, the type of electroencephalogram can be identified by analyzing the frequency of the measured electroencephalogram. Here, the sampling frequency can be set to, for example, 10 milliseconds (msec). In the table of FIG. 4, for each type of electroencephalogram, the types of basic emotions and emotions corresponding to the state in which the electroencephalogram is generated are described in association with each other.

  FIG. 5 is a table showing an embodiment of the generated electroencephalogram analysis log.

  According to FIG. 5, in the electroencephalogram analysis log, the measured electroencephalogram type, the level of the electroencephalogram (normalized voltage), and the measured electroencephalogram type at predetermined time intervals such as times t, t + 1, t + 2,. Are recorded in association with each other. As shown in the figure, generally, a plurality of types of brain waves are observed simultaneously. Here, regarding the various types of electroencephalograms, the level is defined as a high level (H) if the measured voltage is equal to or higher than a predetermined threshold, and as a low level (L) if the measured voltage is lower than the predetermined threshold. It is also preferable to record information of “none” for the electroencephalogram type that has not been measured. Furthermore, it is also preferable to record the generation duration of the electroencephalogram for each measured electroencephalogram type.

  Returning to FIG. 3, the electroencephalogram interpretation unit 115 generates an electroencephalogram analysis log generated based on a plurality of electroencephalogram interpretation rules in which temporal data of information relating to electroencephalograms and interpretation labels of intellectual activities in the brain are associated in advance. Interpretation label candidates are determined from the above. Here, as the time-dependent data of information relating to the electroencephalogram, for example, a time-lapse graph in which a state in which an electroencephalogram of a certain type and level is output is a node and a transition between the states is a branch can be employed.

  FIG. 6 is a schematic diagram showing an example of a time-dependent graph used for interpretation.

  In FIG. 6, three nodes indicating a state of outputting an H-level high gamma wave, one node indicating a state of outputting an H-level low beta wave, and one node indicating an end state are sequentially arranged. A time-course graph is shown in which the transition from (node) to state (node) is indicated by an arrow (branch). These states (nodes) indicate states for each time at fixed time intervals. Here, the state between the start state and the end state (representing a random variable) cannot be observed (hidden state), and a signal called an electroencephalogram according to the output probability distribution in the transition destination state at each state transition This time graph is exactly a hidden Markov model (HMM) graph.

  In the time chart shown in FIG. 6, based on the contents of the table shown in FIG. 4, it can be interpreted that a strong inference state continues and then a peaceful state appears, and it is estimated that learning progresses smoothly. You can also As a result, as will be described later, at least an interpretation label “very good” is associated with this time-lapse graph.

  FIG. 7 is a table showing an example of an electroencephalogram interpretation rule.

  FIG. 7 shows an electroencephalogram interpretation rule in which a series of electroencephalogram types and levels of the electroencephalogram generated at each time are recorded in association with interpretation labels. Here, the rule of the first line in which “Any” and “excellent” are associated as interpretation labels is defined for the time-dependent graph shown in FIG. This “Any” indicates that it matches any of the brain intellectual activity states.

  The interpretation label is a label having contents corresponding to the intellectual activity state in the brain that can be estimated later. Specifically, as an interpretation label, a label indicating “under consideration”, a label indicating “consideration has ended”, a label indicating “understanding”, and a label indicating “not understanding” Preferably, at least one or all of these are employed.

  In addition, like the rules in the first to third lines, in the electroencephalogram interpretation rule, a transition from a state of outputting a high gamma wave or a low gamma wave having an H level to a state of outputting a low beta wave having an H level is performed. It is also preferable to set a rule that associates at least an “understood” interpretation label with the time graph including the time graph. In any case, the interpretation label that can be associated with the graph over time can be determined by experimentation or by analysis of statistical data.

  Returning to FIG. 3, the electroencephalogram interpretation unit 115 follows the electroencephalogram interpretation rule as shown in FIG. 7, and outputs the output pattern represented by the time-series graph listed in the electroencephalogram interpretation rule of FIG. 7 using the EMM analysis. When extracted from the analysis log, an interpretation label associated with the temporal graph in advance is determined as an interpretation label candidate. For example, when the output pattern of the temporal graph shown in FIG. 6 is extracted from the electroencephalogram analysis log, the interpretation labels “Any” and “I understand” can be determined as interpretation label candidates.

  Here, it is also preferable that the electroencephalogram interpretation unit 115 acquires electroencephalogram analysis data for each time at regular time intervals from the electroencephalogram log management unit 114 and determines an interpretation label candidate for each acquired electroencephalogram analysis data. In this case, the label candidate corresponding to the current brain state of the user can be determined without causing such a large time lag. Furthermore, it is also preferable to record the determined interpretation label as an HMM interpretation log associated with the time.

  In addition, as described above, since a plurality of types of electroencephalograms generally occur simultaneously in the electroencephalogram analysis log, output patterns of a plurality of or many time-dependent graphs are extracted from the electroencephalogram analysis log, and a plurality of or many interpretations are made. In many cases, label candidates are determined.

  Similarly, according to FIG. 3, the keyboard 121 and the microphone 122 in the interface device 12 are input interfaces that accept manual input and voice input of information related to answers and questions by the user, respectively. The display 123 and the speaker 124 are output interfaces for displaying educational contents, teaching materials, and exercises, respectively, and outputting them to the user. Of course, the input / output interface of the apparatus 12 is not limited to these, and for example, a touch panel display may be provided.

  The content management unit 125 manages various types of content used for user learning and education, and outputs the accumulated content to the information processing unit 126 in response to an instruction from the information processing unit 126. The content acquired through F141 is stored.

  The information processing unit 126 processes information related to the user's learning / education, and outputs educational content, teaching materials, and exercises to the user via the output interface, and information related to answers, questions, etc. by the user. Or receive via the input interface. In addition, the activity status grasping unit 116 described later is notified of the processing status of information related to user learning / education, in other words, information related to the user's intellectual activity.

  Similarly, as shown in FIG. 3, the activity status grasping unit 116 also sets a plurality of brain intellectual activities that are set in advance based on the processing status of information related to the intellectual activity input from the information processing unit 126 of the interface device 12. State label candidates are determined from the state labels. Specifically, the interface device 12 can output a task or proposal information to be processed by the user, and input reaction or response information by the user with respect to the task or proposal information. On the other hand, the activity situation grasping unit 116 uses an activity situation grasping table in which a state label is associated in advance for each possible combination of the output situation of the task or proposal information and the input situation of the reaction or response information. It is also preferable to determine the state label candidate.

  Note that the interface device 12 outputs information related to the processing status to the activity status grasping unit 116 at regular time intervals, for example, every second, and the activity status grasping unit 116 displays the status label at each time. It is also preferable to determine candidates. For example, it is also preferable to record the determined state label candidate as an action grasping log in association with the time.

  FIG. 8 is a table showing an example of an activity status grasping table used for grasping the state.

  The activity status grasping table shown in FIG. 8 records a state label in association with each combination of the state of the interface device 12 and the user's reaction / action. Here, the state of the interface device 12 is an output state of assignments or proposal information to the user such as “questioning questions” and “answer explanation after questioning”. In addition, the user's reaction / action is the input status of the user's reaction or response information such as “whether there is a response”, “answer is being created”, “answer creation complete”, “question creation” for the question that has been asked. Yes. Note that “Any” in the activity status grasping table of FIG. 8 indicates that the state, reaction, and action are applicable in any case.

  Further, the status label includes a label indicating that “consideration is in progress”, a label indicating that “consideration has been completed (consideration completed)”, a label indicating that “understanding”, and “not understanding” It is preferable that at least one or all of the labels indicating “ununderstood)” are adopted. In the activity status grasping table of FIG. 8, in addition to these status labels, a status label indicating “very good” is also used. The association with the status label in the activity status grasping table can be performed by a prior experiment or analysis of statistical data.

  As described above, the activity status grasping unit 116 (FIG. 3) prepares the activity status assessment table as shown in FIG. 8, and the operation state on the interface device 12 side and the reaction / operation on the user side using the device 12 are prepared. On the other hand, candidates of mental processes that may occur at that time and are more complicated than basic emotions are set as state labels. As a result, as will be described later, it is possible to determine an intellectual activity state in the brain that is difficult to grasp only by the HMM analysis of the electroencephalogram.

  Returning to FIG. 3, the intellectual activity determination unit 117 determines the content of the label common between the determined interpretation label candidate and the determined state label candidate as the intellectual activity state in the brain of the user. . Here, when the determined interpretation label candidate and the determined state label candidate are collated, a common one is selected from those in which the times corresponding to each match or within a predetermined time range. It is also preferable. It is also preferable to record the determined brain intellectual activity state as a brain state log associated with the interpretation label candidate or the time associated with the state label candidate.

  FIG. 9 is a schematic diagram showing an embodiment of the brain intellectual activity state estimation process in the intellectual activity determination unit 117.

According to FIG.
(A) Based on the electroencephalogram interpretation rules, the electroencephalogram interpretation unit 115 determines “Any” and “excellent”, and “solved the problem” and “understood” as interpretation label candidates from the electroencephalogram interpretation log,
(B) Based on the activity status table, the activity status grasping unit 116 determines “perfect”, “consideration completed”, and “understood” as status label candidates.
The determinations in (a) and (b) are both made based on information acquired at time t, for example. That is, processing is performed so as to determine label candidates for the brain wave and processing status at the same time.

  Next, the intellectual activity determination unit 117 extracts the label contents that are common between the determined interpretation label candidate and the determined state label candidate, that is, “excellent” and “understood”, and extracts them. Determine the state of intellectual activity in the brain.

  Here, the same classification term group is used in advance for the interpretation label and the state label, and the label of the term that matches between the determined interpretation label candidate and the state label candidate can be used as a common label. Alternatively, a similar range of meaning is set in advance for terms used for the interpretation label and the state label, and the labels of the terms having the same or similar meaning between the determined interpretation label candidate and the state label candidate are shared. It can also be a label. Furthermore, numerical parameters that serve as indicators such as comprehension levels are assigned as interpretation labels and state labels, and labels in which the numerical parameter values are within a predetermined range between the determined interpretation label candidates and state label candidates. May be a common label.

  As described above, the intelligent activity determination unit 117 considers not only the estimation result derived from the electroencephalogram data but also the estimation result derived from the processing status in the interface device 12 including the operation status of the user. It is possible to reliably determine the brain's intellectual activity state, which has been difficult to estimate in the past, with a relatively small number of processing steps.

Returning to FIG. 3, the content instruction unit 118
(A) When the intellectual activity determination unit 117 determines the intellectual activity state of the user's brain as the content of the label indicating “understanding”, the output of the problem or proposal information output to the interface device 12 Instruct to output the assignment or proposal information corresponding to the next step,
(B) On the other hand, when the intellectual activity determination unit 117 determines the intellectual activity state in the user's brain as the content of the label indicating “I do not understand”, the task output to the interface device 12 Alternatively, an instruction to output information for understanding the content of the proposal information is given.

  Specifically, the content instruction unit 118 responds, for example, as a content instruction with a command “skip to the next item” according to an estimation result “understood” or according to an estimation result “not understood”. Then, a command “request another explanation method” may be output to the interface device 12. In addition, in a processing situation where one solution to an exercise problem such as mathematics or physics was presented to the user, the state of intellectual activity in the brain of the user was continuously determined to be “unresolvable (not understood)” In this case, a command “explain another solution” may be output to the interface device 12 as the content instruction. Furthermore, a content instruction may be given to change the difficulty level of the problem to be presented in accordance with the estimation result of “I understand” or “I don't understand”.

  Further, as shown in FIG. 2, the learning progress management unit 219 in the brain intellectual activity estimation device 21 of the education system 2 is the user's brain intellectual activity state determined by the intellectual activity determination unit 217 in the device 21. Is recorded and managed in association with problem information and / or answer information processed by the interface device 22 at the same time. This management information can also be used, for example, to revise educational items or re-edit educational content on the educational service provider side.

  As described above, the intellectual activity state in the brain estimated by the intellectual activity determination unit can be used in various forms in order to improve the content of the educational service provided to the user. .

  As described above in detail, in the present invention, not only the electroencephalogram data conventionally used for emotion estimation but also the “processing status” of the intellectual activity information in the interface device, the user's brain state is considered. presume. Thereby, it becomes possible to estimate the “intellectual activity state in the brain” related to a more complicated mental process in a user who is active using the interface device.

  The estimation of the intellectual activity state in the brain according to the present invention is naturally not limited to the application to the education field. For example, it can be adopted in a game application to produce a game development or adjust the progress of the game according to the state of the user's brain. It can also be applied to evaluation of content such as music and movies by users. Furthermore, it can be applied to matching systems such as matchmaking. It can also be used as a group interview system or an automated questionnaire machine for marketing.

  For the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and the viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Education service provision apparatus 11, 21 Intelligent activity estimation apparatus in brain 111 Communication interface 112 EEG data storage part 113 EEG analysis part 114 EEG log management part 115 EEG interpretation part 116 Activity condition grasping part 117, 217 Intelligent activity determination part 118 Content instruction unit 12 Interface device 121 Keyboard 122 Microphone 123 Display 124 Speaker 125 Content management unit 126 Information processing unit 13, 23 EEG measuring device 141 I / F
2 Education System 219 Learning Progress Management Unit 22 PC (Interface Device)

Claims (9)

A brain intellectual activity estimation device connected to an interface device capable of inputting and / or outputting information related to intellectual activity with a user and processing the information related to the intellectual activity,
An electroencephalogram analysis means for acquiring the electroencephalogram data of the user and generating an electroencephalogram analysis log in which information relating to the electroencephalogram is recorded in time series from the electroencephalogram data;
An electroencephalogram interpretation means for determining interpretation label candidates from the generated electroencephalogram analysis log based on a plurality of interpretation rules in which temporal data of information relating to the electroencephalogram and interpretation labels of brain intellectual activities are associated in advance; ,
Based on the processing status of information related to the intellectual activity in the interface device, an activity status grasping means for determining status label candidates from a plurality of preset status labels of intellectual activities in the brain;
An intelligent activity determining means for determining a content of a label common between the determined interpretation label candidate and the determined state label candidate as an intellectual activity state in the brain of the user; Brain intellectual activity estimation device. The electroencephalogram analysis means generates an electroencephalogram analysis log in which an electroencephalogram record in which an electroencephalogram type and an electroencephalogram level of the type are associated with each other is generated in time series for each time related to electroencephalogram generation,
When the electroencephalogram interpretation means extracts an output pattern represented by time-lapse data with a state that outputs an electroencephalogram of a certain type and level as a node and a transition between the states as a branch, when the output pattern is extracted from the electroencephalogram analysis log, 2. The brain intellectual activity estimation apparatus according to claim 1, wherein an interpretation label associated with data in advance is determined as an interpretation label candidate. The interface device is capable of outputting a task or proposal information to be processed by the user, and inputting reaction or response information by the user with respect to the task or proposal information,
The activity status grasping means uses an activity status grasping table in which state labels are associated in advance for each possible combination of the output status of the subject or proposal information and the input status of the reaction or response information. The brain intellectual activity estimation apparatus according to claim 1, wherein a candidate is determined.   As each of the interpretation label and the state label, at least one of a label indicating that the consideration is being performed, a label indicating that the consideration has been completed, a label indicating that the understanding has been performed, and a label indicating that the interpretation has not been understood 4. The brain intellectual activity estimation apparatus according to claim 3, wherein one or all of them are employed.   When the intellectual activity determining means determines the content of the label indicating that the user has understood the intellectual activity state in the brain of the user, to the interface device, to the next step of the output problem or proposal information Instructing the corresponding task or proposal information to be output, while the intellectual activity determining means determines the content of the label indicating that the user does not understand the intellectual activity state in the brain, 5. The brain intellectual activity according to claim 3, further comprising content instruction means for instructing the interface device to output information for allowing the interface device to understand the contents of the output problem or proposal information. Estimating device. At least one of the plurality of interpretation rules includes temporal data including a transition from a state of outputting a high gamma wave or a low gamma wave having a level equal to or higher than a predetermined threshold to a state of outputting a low beta wave having a level equal to or higher than the predetermined threshold. Is associated with an interpretation label that indicates
The electroencephalogram interpretation means extracts at least an interpretation label indicating the understanding when the output pattern expressed by the temporal data including the transition to the state of outputting the low beta wave is extracted from the electroencephalogram analysis log. The brain intellectual activity estimation apparatus according to any one of claims 1 to 5, characterized in that: The brain intellectual activity estimation device according to any one of claims 1 to 6, and at least one capable of outputting problem information related to education and inputting answer information by the user with respect to the problem information An education system including an interface device and an electroencephalograph attached to each user who uses the interface device,
The brain intellectual activity estimation device is configured to determine the state of intellectual activity of the user's brain determined by the intellectual activity determination means corresponding to the output problem information and / or the input answer information. And / or further comprising learning progress management means for recording and managing the information in association with the answer information. Intellectual activity estimation in the brain that allows a computer mounted on a device connected to an interface device capable of processing information related to the intellectual activity to input and / or output information related to the intellectual activity with the user. A program,
An electroencephalogram analysis means for acquiring the electroencephalogram data of the user and generating an electroencephalogram analysis log in which information relating to the electroencephalogram is recorded in time series from the electroencephalogram data;
An electroencephalogram interpretation means for determining interpretation label candidates from the generated electroencephalogram analysis log based on a plurality of interpretation rules in which temporal data of information relating to the electroencephalogram and interpretation labels of brain intellectual activities are associated in advance; ,
Based on the processing status of information related to the intellectual activity in the interface device, an activity status grasping means for determining status label candidates from a plurality of preset status labels of intellectual activities in the brain;
The computer is caused to function as an intellectual activity determination means for determining the content of the label common between the determined interpretation label candidate and the determined state label candidate as the intellectual activity state in the brain of the user. Brain intellectual activity estimation program. A method for estimating intellectual activity in a user's brain using an interface device capable of inputting and / or outputting information related to intellectual activity with a user and processing the information related to the intellectual activity. ,
A first step of acquiring electroencephalogram data of the user and generating an electroencephalogram analysis log in which information relating to the electroencephalogram is recorded in time series from the electroencephalogram data;
Second step of determining interpretation label candidates from the generated electroencephalogram analysis log based on a plurality of interpretation rules in which temporal data of information relating to the electroencephalogram and interpretation labels of brain intellectual activities are associated in advance When,
A third step of determining a state label candidate from a plurality of state labels of intellectual activities in the brain set in advance based on the processing status of information related to the intellectual activity in the interface device;
A brain comprising: a fourth step of determining a content of a label common between the determined interpretation label candidate and the determined state label candidate as an intellectual activity state in the brain of the user. Internal intellectual activity estimation method.

Images