JP2011150408A - Human interface based on measurement of living body and brain function, and human error detecting and preventing method using the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To estimate or forecast an internal state of an individual such as caution and memory from living body and brain function measurement signals, detect a state that a human error tends to occur, and issue a warning to take a precaution against a human error or to prevent the occurrence of a human error. <P>SOLUTION: In a workshop in which machines are used, machine learning algorithm is applied to estimate an internal state (visual caution, memory on operations, a degree of skill or the like) which is obtained from an operator's living body measurement signals. Then, a state that a human error tends to occur is forecasted and detected, and if a state that a risk of making mistakes is high is estimated, the states are fed back to the operator by means of the sense of sight, the sense of hearing, the sense of touch, or a combination of them to prevent the mistakes before they occur. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a human interface based on living body and brain function measurement and a human error detecting / preventing method using the human interface, and in particular, predicting / detecting a state where human error is likely to occur based on living body and brain function measurement, The present invention relates to a human interface that provides feedback to a person and a human error detection / prevention method using the human interface.

  While the reliability of equipment has improved and the number of accidents caused by the machine side has decreased, the proportion of accidents (human errors) caused by people in the total number of accidents has increased. In the field of human factor engineering for the purpose of preventing human errors, emphasis is placed on the development of a foolproof system that reduces mistakes on the human side by preparing an external environment (Non-patent Document 1). Examples include road guides that have been improved in format to prevent mistakes, and plastic covers on operating buttons that do not operate with a slight contact.

  Patent Document 1 proposes a device that estimates an operator's consciousness level using a brain activity signal obtained from an electroencephalogram spectral analysis, and presents information that seems appropriate according to the estimated consciousness level. . Further, Patent Document 2 discloses a biological optical measurement device that detects a signal of a living body and a brain function based on blood circulation / hemodynamics and hemoglobin change in a brain of a living body as an optical topography signal (hereinafter referred to as an optical toppo) using a photodetector. Examples are disclosed.

  In addition, Non-Patent Document 3 shows an example of a change detection task. In Non-Patent Document 4, it is reported that in patients with epilepsy, signs of seizures can be found from brain activity of several hours to seven hours when epilepsy occurs.

Japanese Patent Publication No. 6-87211 JP-A-9-98972

Okada Yusaku, Introduction to Human Factors: Aiming at harmony between humans and machines, 2005 P17-18: Keio University Press, Tokyo Lindsey / Norman, "Introduction to Information Processing Psychology II Attention and Memory" P31-33, P53-54, Science Beck, Rees, Frith, Lavie (2001) Neural correlates of change detection and change blindness.Nature Neuroscience 4 (6): 645-650. Litt et al. (2001) Epileptic seizures may begin hours in advance of clinical onset: a report of five patients. Neuron 30: 51-64.

  However, no matter how much the foolproof system is in place, if the operator is not paying attention to the external environment or the judgment ability is low due to low arousal level, oversight, misunderstanding, misunderstanding, Accidents caused by human error such as forgetting to happen can occur. Moreover, it has been clarified in the recent cognitive neuroscience field that there is an upper limit to the number of visual stimuli visible at one time and the number of terms that can be retained in working memory (Non-patent Document 2). Therefore, there are human factors that cannot be prevented by techniques in the field of human factor engineering such as the maintenance of the external environment. In order to prevent and detect negligence due to this human factor, an approach from human science is needed to predict and detect conditions prone to error by the operator.

  Moreover, while the technique disclosed in Patent Document 1 is suitable for estimating an overall rough state such as the degree of arousal, it can identify an active brain region because of its low spatial resolution. Therefore, it was not possible to clarify what kind of brain function is in what kind of state. In addition, since EEG uses a general five-level indicator of consciousness level, it does not consider the estimation of the brain state according to the characteristics of each operator.

  The problem to be solved by the present invention is to prevent or prevent human errors by estimating or predicting internal states such as individual attention and memory from living body and brain function measurement signals, and detecting and warning states where human errors are likely to occur. It is to be.

  A representative example of the present invention is as follows. The human interface of the present invention obtains a set of information on the internal state of the subject and the biological measurement information of the subject that is temporally related to the information on the internal state and removes the biological noise. Personal database generation means for generating a database of the above, classification algorithm optimization means for optimizing a classification algorithm using the personal database as a training set, and new biometric information obtained by measuring the biosignal of the subject, A behavior estimation unit that estimates the subject's internal state by applying the optimized classification algorithm and an information presentation unit that presents information based on the estimation result are provided.

  According to the present invention, the operator's internal state is estimated or predicted based on the internal state of each operator estimated from the living body and brain function measurement, and a state in which human error is likely to occur is detected and warned. It is possible to call attention and prevent human error.

The conceptual diagram of the human interface provided with the human error prevention function by internal state estimation which becomes Example 1 of this invention. FIG. 3 is a diagram illustrating an example of a unique personal database according to the first embodiment. 5 is a flowchart of calculation processing for internal state estimation in the first embodiment. The figure which shows the mode of an optical toppo measurement. The flowchart which shows the detail of database creation of FIG. The figure which shows an example of the schematic diagram of a change detection task. The figure which shows an example of the schematic diagram of a change detection task. Conceptual diagram of support vector machine. Conceptual diagram of support vector machine. The figure which shows the example of the classification | category probability when changing the number of channels. The figure which shows the time change of a classification probability. The figure which shows the experimental data of 4 test subjects. The figure which shows the brain part related to attention. The figure which shows the confirmation result of reproducibility. The flowchart of optimal task discrimination | determination used as Example 2 of this invention. The flowchart of optimal task discrimination | determination used as Example 2 of this invention.

  The present invention estimates human internal states using machine learning algorithms from living body and brain function measurements, detects pre- or post-faulty states, and uses the information in a database for human use. It is about trying to prevent errors.

That is, the present invention estimates an internal state (visual attention, working memory, skill proficiency, etc.) from an operator's biological measurement signal by applying a machine learning algorithm in a workplace using a machine, and human error Predicting and detecting the state where the risk is likely to occur, and if it is estimated that there is a high risk of negligence, feedback is made to the operator in advance by either visual, auditory, tactile, or a combination thereof. The present invention relates to an apparatus and a method for preventing it.
Hereinafter, embodiments of the present invention will be described in detail.

  The human interface according to the first embodiment of the present invention will be described. In this example, a human interface having a function of monitoring and predicting an error state from a brain activity signal will be mainly described. In order to realize the first embodiment, first, the following brain function measuring apparatus is required.

"Brain function measuring device"
It has a spatial resolution that can measure a specific brain function, and a temporal resolution that can distinguish between error monitoring and prediction. In consideration of application at the actual work site, it is desirable that the device is relatively portable and does not require equipment such as a magnetic shield. An example is an optical topography method, but the present invention is not limited to this.

In the following, the apparatus will be described assuming optical topography as a brain function measurement method.
FIG. 1A is a conceptual diagram of a first embodiment of the human interface of the present invention. In FIG. 1A, the human interface of the present embodiment includes a device main body 100 including a biological measurement device 110, a signal analysis device 120, and a problem presentation device 130, data of biological measurement signals and internal state signals, a database of brain activity, and a reference. An external storage device 140 that holds data and the like and an external device 150 that is operated by an operator, that is, a subject 160, are provided, and these are connected to each other via a communication network. The apparatus main body 100 including the biological measurement apparatus 110, the signal analysis apparatus 120, and the problem presentation apparatus 130, the external storage device 140, and the external device 150 have information on the internal state of the subject and temporal information related to the internal state information. Personal database generation means for acquiring the biometric information of the subject as a set and generating a personal database is configured. Further, the apparatus main body 100 including the signal analysis apparatus 120 and the external storage device 140 are obtained by the classification algorithm optimization means for optimizing the classification algorithm using the personal database as a training set, and the measurement of the biological signal of the subject. Action estimation means for estimating the internal state of the subject by applying the optimized classification algorithm to new biometric information is configured. Further, the external device 150 functions as an information presentation unit that presents information based on the estimation result.

  In other words, the living body measuring device 110 is a device that detects living body information inside the living body (subject), for example, an optical toppo, which irradiates a living body with near-infrared light, and that light is transmitted and scattered inside the living body. Is used to measure the optical properties in the living body continuously in time and observe the state in the living body. However, the number of measurement points is not particularly limited. Alternatively, instead of the optical topo, a device that measures the nerve activity as biological information by measuring the potential of the electroencephalogram at a plurality of parts of the brain may be used. Furthermore, as biological information, functions for performing physiological measurements such as blood pressure, pulse, respiration / thermography, skin blood flow, electrocardiogram, and the like are used as appropriate.

  In addition, the task presentation device 130 presents the optimal information in terms of audiovisual and tactile senses such as tasks (tasks) and analysis results for the subject. The external device 150 has functions such as selecting a task (task) from a subject and receiving a response. In parallel with the measurement of the biological information by the biological measurement device 110, the subject 160 receives a stimulus from the task presentation device 130 and returns a response via the operation button, the microphone, or the like of the external device 150. The response result of the operator is recorded in the storage device 140 such as a hard disk as information on the internal state of the operator constituting a part of the personal database.

  The signal analysis device 120 has a signal processing function for removing noise derived from a living body from a living body measurement signal detected by a photodetector, data of the processed living body measurement signal, and a temporal relationship with this (as opposed to A function that records the internal state signal data regarding the internal state of the operator (visual attention, working memory, skill proficiency level, etc.) as a personal database, and optimizes the machine learning algorithm using this personal database as a training set And a function for estimating an internal state from measurement of a biological signal in an actual work site using the optimized machine learning algorithm. Then, predicting / detecting a state in which human error is likely to occur and, when it is estimated that the risk of negligence is high, presenting it on the display screen of the external device 150, etc. Combine them and feed back to the operator.

  The processing function of the apparatus main body 100 is realized by executing various programs loaded in a memory in a host computer or a personal computer connected to the network by the CPU. The problem presentation device 130 and the external device 150 may be configured to serve as one personal computer. As the storage device 140 which is a storage device for a personal database, a local hard disk or a server using the Internet is used.

  FIG. 1B shows an example of a personal database unique to the operator A based on a response result of a certain operator A acquired in advance and internal biological information. The database 140A is a time series pattern of correct answer probabilities, and the database 140B is data of a determination boundary pattern that provides a determination criterion for the likelihood of human error. The database 140A shows that the correct answer rate of the operator A is high, in other words, the state determination section is between t2 and t3 after the task starts. On the other hand, the database 140B indicates that an area of “state A” in which the state of the operator A is good and an area of “state B” in which the operator A is not good are partitioned by “determination boundaries”. The database generation method and configuration differ depending on the type of task (issue) and its application. Details will be described later.

  Next, the outline of the operation of the first embodiment will be described with reference to FIG. FIG. 2 is a human error detection / prevention flowchart based on the premise of constant measurement of optical topo.

  In the human interface of the present embodiment, the biological measuring device 110 is mounted on the operator's head in advance (S20), and brain function measurement is performed while giving a predetermined task according to the task selected by the operator (S21). . At this time, the acquired information on the internal state of the operator (attention, working memory, learning fixed degree, etc.) and signals of biological signal measurement (optical topography, brain wave, pulse, respiration, blood pressure) are acquired. For example, when the operator selects “attention task”, the optical topography is used as the whole body part, or mainly the occipital parietal and dorsal frontal lobes, as a biometric measurement, in order to detect sensory mistakes such as oversight, mishearing, and misunderstanding. Are measured simultaneously at a plurality of points. Then, signal processing for removing biological noise from these measurement signals is performed, and these data are made into a database as a “brain activity / behavior database” for a training set, that is, an individual, and recorded in the external storage device 140 ( S22). A similar database may be created using an electroencephalogram instead of the optical toppo. Alternatively, the optical toppo or brain wave and at least one of pulse, respiration, and blood pressure may be measured at the same time, and a database may be created from a biological measurement signal obtained by combining these and a signal of the internal state of the operator. In addition, a frequency band pass filter, polynomial baseline correction, principal component analysis, independent component analysis, or the like is used for preprocessing of biological signal measurement.

“Brain Activity / Behavior Database”
In order to estimate the state of increased human error from brain activity, it is necessary to optimize the machine learning classification algorithm, so the actions of the operator are recorded, and errors occur and do not occur It is necessary to create a database of brain activities. That is, a database of actions for each operator and the corresponding brain activity is generated. This may be stored on the local hard disk exclusively for one operator. Alternatively, there is a method of storing data from a plurality of operators who perform the same operation task as a database on the Internet and storing individual characteristics and group characteristics.

  In FIG. 2, the machine learning algorithm is further optimized using the personal database as a training set (S23).

"Classification algorithm optimization"
The classification algorithm is optimized using the database obtained in S22 as training data.

  As the classification algorithm, a support vector machine based on the margin maximization principle or a relevance vector machine based on Bayesian learning can be used. The classification algorithm is trained from the data obtained in S22 and S23, and the optimum parameter is determined. In the case of a support vector machine, in order to prevent overlearning, parameters are learned using a cross-validation method so that the generalization performance is maximized.

  Further, based on the optimized machine learning algorithm, the operator is assigned a task or starts a practical work (no task) such as a personal computer operation in the field instead of the task (S24), In this state, the biological signal is measured (S25). Then, the internal state of the operator is estimated from the measurement result of the biological signal and the information in the personal database acquired in advance (S26). By comparing this internal state with a preset threshold value, a state in which a human error is likely to occur is predicted and detected (S27).

"Behavior classification from brain activity signals obtained in actual workplaces"
The classification algorithm optimized in S23 is applied to the brain activity measured at the actual work site to classify whether or not negligence has occurred, and calculate the probability of negligence (fault probability).

  As this threshold value, a “determination boundary” in FIG. 6B described later is adopted. For example, as shown in FIG. 1B, the values obtained by processing the measurement results of the biological measurement signals of the operator A in the state determination section from the time t2 to the time t3 after the start of the task for the operator A are indicated by circles. For example, since it is in the “state A” above the “determination boundary”, the apparatus main body 100 determines that the current state of the operator A is a good state of mind and body. On the contrary, if it is in “state B” below the “determination boundary”, it is determined that the current state of the operator A is not good. In the interval from time t2 to t3, measurement results are obtained at predetermined time intervals, for example, every second. Therefore, an average value of a plurality of measurement results in this state determination interval is calculated. Determine which side of the boundary is.

  Then, in accordance with this result, the optimum audiovisual and tactile information is presented (S28, S29). For example, when the operator is prone to negligence, a display for alerting the user or the user is performed.

"Awareness-presenting device to prevent negligence"
In other words, a visual / auditory or tactile presentation device that alerts the operator and supervisor is prepared in accordance with the negligence probability obtained in S26 and S27.

  An example is the size adjustment of the alerting screen according to the negligence probability. The purpose is to prevent oversight while minimizing obstacles to work by the alert screen. When the negligence probability is high (p> 0.7), there is a high possibility that an oversight will occur, so that a warning screen is displayed large even at the expense of failure in work. On the other hand, when the negligence probability is small (p <= 0.7), the alert screen is displayed small.

"Updating the Database"
In addition, the brain activity data and action data obtained with the task in S24 are added to the brain activity / behavior database, and the time series pattern of the correct answer probability and the judgment boundary pattern data are updated. The quality of the database is improved (S30).

  Next, the details of the “brain activity / behavior database creation” process (S20 to S22 in FIG. 2) will be described based on the results of experiments conducted by the inventors. That is, in order to show that it is technically possible to estimate the internal state of the operator using the living body and brain function signals according to the present invention, a support vector machine is provided on the optical toppo that the inventor performed for a plurality of subjects. An example of an experiment in which the visual error state is estimated by applying the method and the result will be described below.

  First, FIG. 3 shows a state of measuring optical topo brain activity using a change detection task. During the task presentation period, the subject responds using the keyboard of the external device (or notebook PC) according to the visual stimulus presented on the notebook PC screen. As a result, data on the internal state of the subject was obtained. At the same time, the brain activity at that time was measured for all heads using 88 channels of optical topo. As a result, a biological measurement signal of the subject having a temporal relationship with the acquisition of the internal state data was obtained. In order to experimentally verify the visual error, we presented the change detection task used in Non-Patent Document 3 to 6 subjects and measured.

  In creating the database, the operator is based on the procedure shown in FIG. FIG. 4 is a flowchart for creating a behavior data and brain activity data, in other words, creating a “brain activity / behavior database”.

  The operator wears the optical topography measuring device (S40).

  The operator registers the ID, name, and personal information, and creates a new database (S41).

  From the pull-down menu, a human error factor that the user wants to detect, for example, items such as “attention”, “working memory”, and “skill learning” are selected (S42).

Next, the operator performs a psychophysical task corresponding to the item selected in (S42). That is,
When an attention task is selected (S43), a change detection task is performed (S44). In this case, the behavior data is either the correct answer (reported a visual stimulus change correctly) or an incorrect answer (missed a visual stimulus change) in each trial. 5A and 5B are schematic diagrams of a change detection task used in the experiment. This schematic diagram will be described later.

  When the working memory task is selected (S45), the hiragana presented on the screen is stored for several seconds to several tens of seconds, and then whether or not the stored one is the same as the one presented is answered (S46). In this case, the behavior data is either correct answer (remembered and matched) or incorrect answer (forgotten and failed to match) in each trial.

  When a skill learning task is selected (S47), an operation of following a target on the screen is learned using a joystick whose position is converted into a speed (S48). In this case, the behavior data is a position error from the target.

  Each task is performed about 100 times, and brain activity and behavior data are recorded (S49).

That is, the i-th trial brain activity data (subject's biological measurement signal) measured in parallel with the presentation of the psychophysical task is x _i and the action data (subject's internal state data) is y _i . Make them a pair (x _i , y _i ). These data pairs are recorded on the hard disk over all trials (x _i , y _i ) (i = 1,..., N).

  If necessary, the personal data recorded in (S49) is registered via the Internet, and a database for each subject is created (S50). At this time, take care not to identify individuals.

  In the “change detection task” in FIG. 5A, one trial consists of the following (S51) to (S55), and the subject is presented with a “character task” and a “face task” described below.

  (S51) The subject stares at the fixed point (+) on the liquid crystal display placed 30 cm in front of the face.

  (S52) In the meantime, three characters such as XBV and GVZ alphabets on the top and bottom of the fixation point, and a screen consisting of two faces on the left and right are presented for 500 milliseconds. Whether the upper and lower character strings include a specific alphabet (in this case, “X”) is immediately reported on the keyboard (character task).

  (S53) A gray blank screen is presented for 500 milliseconds.

  (S54) The above steps (S52) and (S53) are repeated four times. At this time, either the left or right face stimulation may be switched (see FIG. 5B).

  (S55) It is reported with the keyboard whether or not the face stimulation on the screen has been changed (face task).

  One session consists of 30 trials, and the same face is repeatedly displayed in 1/3 of the total trials, whereas in the 2/3 trial, either the left or right face is replaced with another face. In the field of cognitive psychology, it has been shown that there is an upper limit to information that can be processed by a person at one time (Non-Patent Document 2). Here, by setting a dual task that imposes a character task and a face task at the same time, it is expected that the resource allocation of attention directed to the face change will be reduced and the face change will be missed.

  Actually, the average correct answer rate of 6 subjects was 55.6%. Among trials with face changes, trials reported by the subject as having changed are referred to as change detection trials, and trials reported by the subject as having no changes are referred to as change oversight enforcement. It is necessary to examine whether the trial that noticed the change and the trial that missed it can be classified from the brain activity measured in the above change detection task.

Next, the details of the “classification algorithm optimization” process (S23 in FIG. 2) will be described based on the results of experiments performed by the inventors.
In the following, a method of classifying a change detection trial and a change oversight trial from the optical topo signal will be sequentially described in four steps.

(S1) “Pre-processing for removing noise contained in optical topo signal”
The optical topo signal includes signals derived from the whole body such as heartbeat and respiration, noise peculiar to the living body, and noise caused by fluctuation of the measuring instrument laser. In order to remove these and extract signals derived from brain activity, smoothing by moving average (3 seconds) and baseline removal by polynomial (third order) fitting were performed.

(S2) “Channel exclusion”
Channels with an optical toppo amplitude of 0.5 or more or whose frequency spectrum cannot be statistically distinguished from white noise were judged to have a strong noise contribution and were excluded from the following analysis.

(S3) “Classification of change detection / change oversight trials by classification algorithm using multiple channel signals”
When multiple channels are used in the classification algorithm, it is necessary to determine how many channels should be used. In general, when the number of channels is increased, the classification probability for training data is improved, but generalization for test data is rather deteriorated. Therefore, the case of 1, 2 and 3 channels was examined. Assuming that the total number of channels not excluded in step (2) is N, N, N (N-1) / 2 !, N (N-1) (N-2) / 3! A classification algorithm will be applied.

  A support vector machine was used as a classification algorithm. When calculating classification correct answer probability from optical topo signal labeled with change detection / change oversight, in order to evaluate generalization performance, test data is randomly divided into training data and test data to evaluate generalization performance N -fold cross validation was used. Since the optical toppo signal is given as a time-series signal of 10 Hz, the above-mentioned calculation is calculated in increments of 0.1 seconds from -5 to 12 seconds with each trial start time being 0, and the time variation of the classification correct answer probability is calculated. followed.

  The support vector machine is a method of a classification algorithm in machine learning, and determines an optimal determination boundary based on a criterion of maximizing a classification data margin from a “determination boundary”. 6A and 6B are conceptual diagrams of the support vector machine. In the training, as shown in FIG. 6A, when the two-dimensional data (white dot) in the state A and the two-dimensional data (black dot) in the state B are given, the “determination boundary is determined so as to maximize the margin. Data surrounded by black circles are called support vectors that play an important role in determining decision boundaries, and maximizing margins minimizes the risk of misclassifying test data.

  Next, as shown in FIG. 6B, the generalization performance to the test data can be calculated using the judgment boundary calculated from the training data. That is, using the determination boundary determined in FIG. 6A, it is determined that the newly obtained data x is in state A if (Expression 1) and in state B if (Expression 2).

  Cross validation is a method for evaluating the performance of statistical analysis using only a given training data set. First, the training data set is artificially divided for training and testing. The former is optimized for statistical analysis, and the latter is used to evaluate the performance of the statistical analysis. In N-fold cross validation, more accurate performance is evaluated by repeating random training data N times.

(S4) Feature extraction of classification correct answer probability using unsupervised clustering algorithm In order to examine what time change pattern the classification correct answer probability obtained from the combination of all the plurality of channels calculated in (S3) has, An unsupervised clustering algorithm was used. The number of clusters can be determined automatically, but here, the number considered to be appropriate was designated by visually observing the obtained time series pattern. A representative k-means clustering method was used as a clustering algorithm.

The following four points were obtained as analysis results of the above experiment.
(A) Improvement of classification probability by multiple channel specification:
FIG. 7 shows the classification probability when the total number N of channels is changed. Here, the top 20 sequences with large classification probability and large amplitude when 1 channel (A), 2 channel (B), and 3 channel (C) are selected from the left and the support vector machine is applied are shown. As is apparent from FIG. 7, the number of channels used in the classification algorithm is not sufficient, and a plurality of channels must be used. In addition, there is not much difference in classification probability between 2 and 3, and it seems that 2 is sufficient for practical use to shorten the calculation time.

  For example, when “attention” is selected as the internal state, one channel from each brain region of the subject's occipital parietal lobe and dorsal frontal lobe is used as a biometric measurement in order to detect sensory errors such as oversight, misunderstanding, and misunderstanding. A sufficient effect can be obtained even if one pair is selected to measure the optical toppo. In addition, when “working memory” is selected as the internal state, one channel is selected from each brain region of the subject's dorsal frontal lobe as a living body measurement in order to detect forgetfulness and apraxia such as forgetting / forgetting. A pair of pairs may be formed to measure the optical toppo. In addition, when “fixed skill learning” is selected as the internal state, in order to detect errors caused by lack of skills, one channel is selected from each brain region of the subject's frontal lobe and motor cortex as a biometric measurement. One pair may be configured to measure the optical toppo.

(B) Time series pattern of classification probability:
FIG. 8 shows a typical example of a time series pattern of classification probabilities indicating whether or not the subject has noticed or overlooked a change detection task in brain function measurement. The example of (A) shows that the classification probability has increased after the “detection task period” of the change detection task has been presented, and that the mechanism of attention that the subject turns to the outside world is not good. On the other hand, the example of (B) has shown that the classification probability has risen before the end of the task, and the subject's attention mechanism toward the outside is in a good state.

FIG. 9 shows experimental data of four subjects regarding a time series pattern of classification probabilities together with a pair of measurement channels. (Here, measurements are made for all channels.)
As described above, there are two types of time-series patterns of classification correct answer probabilities. One shows a classification probability of a chance level until the end of the task, but starts increasing immediately after the end of the task and takes a maximum value around 5 seconds after the end of the task (FIG. 8A and FIG. 9). (A), (B)). The other is a classification probability that is already greater than or equal to the chance level before the task execution, takes a maximum value before and after the task starts, and then gradually returns to the chance level (FIG. 8B and FIG. 9). (C)). Depending on the subject, both patterns may be included ((D) of FIG. 9).

(C) Change detection monitor (monitoring error):
As shown in FIG. 8 (A), it is considered that the one whose classification probability has the maximum value after completion of the task classifies the introspection of whether or not the subject notices the change. In other words, by applying the procedure here to this optical topo multi-channel signal, it becomes possible to monitor whether or not a change in the external environment that has just occurred has been noticed. This signal is mainly derived from the parietal lobe of the left hemisphere and is thought to represent the subject's top-down attention. FIG. 11 shows brain regions related to top-down attention and bottom-up attention.

(D) Change detection prediction (error prediction):
As shown in FIG. 8 (B), it is considered that a subject whose classification probability has the maximum value before the start of the task predicts the performance immediately after whether or not the subject is likely to notice the change. In other words, by applying this procedure to this optical topo multi-channel signal, it is possible to predict whether or not it is likely to notice a change in the external environment that occurs immediately after. This signal is mainly derived from the parietal lobe of the right hemisphere and is considered to represent the subject's bottom-up attention.

Check reproducibility:
We repeated the change detection task for two subjects at a later date and applied the same analysis to confirm that a time-series pattern of classification probabilities was obtained in the same way. The results are shown in FIGS. 10 (A) and 10 (B). That is, (1) the time series pattern of classification probability varies from subject to subject, and (2) if the subject is the same, the time series pattern of classification probability obtained by a predetermined number of repetitions is always the same. It was confirmed.

  From the above experimental results, we have confirmed the possibility of classifying multiple channels of optical topo signals with the machine learning classification algorithm and estimating the internal state of the operator. From this, it is understood that the operator's internal state is estimated using a time with a high correct answer rate for each subject as a state determination section, in other words, a determination database as shown in FIG. 1B may be created.

  Visual errors, such as oversight and misrepresentation, are cited as a factor in human error, and can lead to gross negligence such as signal oversight and email mistransmission. Economically important. In the change detection task in the prior brain function measurement, whether the subject has noticed or overlooked the change can be reproduced with a probability of about 70% -90% from the whole-body optical topography signal at a chance level of 50%. It was. That is, according to the present invention, it is possible to predict and detect a state where a human error is likely to occur with a high probability, and when it is estimated that the risk of negligence is high, it can be fed back to the operator or the like.

  The monitoring and prediction of the error state from the brain activity signal shown in the first embodiment can be expected to be generalizable to other types of human errors.

  As a second embodiment of the present invention, a human interface having a function of preventing forgetting and apraxia by estimating a working memory state will be described.

  Here, an embodiment will be described in which human error factors such as forgetfulness (forgotten) and apraxia (forgotten) are predicted and monitored from brain function measurement. Forgetfulness and expiration requires working memory, which is short-term memory to be used for a particular task, to function correctly. Short-term memory is broadly divided into spatiality and linguisticity, and it is known that the short-term memory is processed by 45 and 46 areas on the outer frontal lobe, respectively.

FIG. 12A is a flowchart for creating an optimal task setting database, and FIG. 12B is a flowchart for an optimal task determination method.
In FIG. 12A, the biological measurement device 110 is mounted on the operator's head in advance (S120), and the brain function of the operator is measured for a predetermined time in a resting state (no task) (S121). Next, for example, every morning at the workplace, the operator starts practical work (no task) such as operation of a personal computer at the workplace (S122), and continues the work without measuring the brain function of the operator (S123). Then, the operator performs a subjective evaluation based on the work contents and work results of the day, and inputs the results to the apparatus main body 100 via an external device every day (S124). The apparatus main body 100 generates and updates a database for estimating the internal state of the operator from the measurement result of the biological signal of the operator in the resting state and the information of subjective evaluation every time an input is received ( S125). This process is repeated every day until a sufficient number of data is obtained (S126, S127). Using such biological signals and data based on subjective evaluation, the machine learning classification algorithm is optimized so as to maximize the generalized classification probability. That is, the state A and the state B as shown in FIG. 6A are classified with respect to these data, and a “determination boundary” as shown in FIG. 6B is obtained. In this manner, an optimal task setting database 140B ′ having a determination boundary pattern that provides a determination criterion for the likelihood of human error, similar to that described in FIG. 1B, is created. The other functions of the apparatus main body 100 are the same as those of the apparatus of the first embodiment.

  Next, FIG. 12B illustrates a state in which the above-described optimal task setting database is used in order to prevent the operator from forgetting and failing in daily work. In this case as well, a biological measuring device is mounted on the operator's head (S130), and brain function measurement is performed for a predetermined time in a resting state (no task) (S131). The apparatus main body 100 estimates the internal state of the operator on the current day based on the biometric information measured on the current day and the database 140B 'for optimum task setting, and generates and outputs a determination result and advice (S132).

  Next, the operator starts a work (no task) such as a personal computer operation in the field (S133), and continues the work without measuring the brain function of the operator (S134). Then, the operator performs a subjective evaluation based on the work contents / work results of the day, and inputs the result to the apparatus main body (S135). The apparatus main body 100 updates the database 140B ′ for estimating the internal state of the operator from the measurement result of the biological signal of the operator in the resting state and the subjective evaluation information (S136).

  In this way, using an optimized classification algorithm, the operator is likely to be able to keep working memory properly for the current task (predicting working memory) and keep working memory properly. It is expected that it can be estimated from the measured brain activity (monitoring of working memory). Human error due to forgetfulness or aggression by alerting the operator or manager using a visual, auditory, or tactile presentation device when the probability of negligence in working memory increases Can be prevented.

  Compared with the first embodiment, this embodiment has a drawback that it takes time to create a database for setting optimal tasks, but has an advantage that it can be easily used at the work site.

The present invention can also be applied to the prevention of negligence caused by a lack of skills by evaluating skill fixing.
One of the causes of human error is the lack of skills that do not have enough skills to accomplish the task. As an example, there is a possibility that a serious accident may occur when the user operates the actual work site without sufficiently acquiring the control panel procedure or goes out on the road with inexperienced driving skills. Therefore, quantifying whether or not a specific skill is sufficiently acquired is one criterion for determining whether or not an operator can place the skill on an actual work site.

  According to recent cognitive psychology, (1) it takes more than a certain amount of time and number of exercises to establish a skill, (2) if the practice ends immediately when the performance is saturated, the skill will not become established, (3) It became clear that in order to establish skills, it was necessary to continue practice with the performance saturated (references). However, it is not clear how much practice will take place in a saturated state, and it will become clear only by retesting at a later date whether skill learning has become established. On the other hand, in actual training, for example, training for nuclear power panel operation, after going out to the training center and being placed at each work site, it is not possible to create a retest opportunity. For this reason, the training center has achieved a certain level of performance with respect to a certain skill, but there is a risk of leading to a major accident if the skill comes into practice without becoming established. Therefore, quantifying the degree of skill establishment is essential to prevent human error.

  In Examples 1 and 2, it was necessary to always wear a brain function measuring device. In the third embodiment, for the purpose of evaluating the skill fixing level of the operator, a case where the constant wearing is not essential will be described.

  In the third embodiment, similarly to the first and second embodiments, the skill fixing degree is calculated by applying the classification algorithm to the brain activity at the time of skill learning. For example, the skill fixing degree is calculated and the behavior and brain activity data is made into a database by the same processing as S20 to S22 in FIG. 2, S41 to S42, and S47 to S50 in FIG.

The third embodiment differs from the first and second embodiments in the following two points.
First, in the “creating a database of brain activity / behavior” in S22, action and brain activity data from a plurality of subjects at various skill establishment stages, not a single subject, are databased, and this is used as a training set for classification algorithm To optimize. A preferable application target of the first and second embodiments is cognitive negligence in a relatively short time (several seconds to several tens of seconds) such as visual error and forgetfulness. In this case, the temporal fluctuation is large, and it seems difficult to compare between individual differences. On the other hand, skill fixation is a phenomenon that occurs on a long time scale (tens of minutes to several hours), so it is expected that a temporal average can be easily taken and comparison between individual differences is possible. This saves labor and time for the subject to repeat the same learning task.

  In the third embodiment, the brain function measurement is used only when skill fixation is evaluated (S48 to S49 in FIG. 4), and the first optical topo measurement process (S40) is omitted. Using such behavior and brain activity data from the subject, the machine learning classification algorithm is optimized so as to maximize the generalized classification probability. That is, the state A and the state B as shown in FIG. 6A are classified with respect to these data, and a “determination boundary” as shown in FIG. 6B is obtained. In this way, a database 140B ″ for optimum task setting having a judgment boundary pattern that gives a judgment criterion for the likelihood of human error, similar to that described with reference to FIG. 1B, etc. is created. The other functions are the same as those of the apparatus of the first embodiment.

  As a result, it is possible to eliminate physical restraint and accompanying physical burden / fatigue by brain function measurement.

  Recent functional brain imaging studies show that when skills become established, the activity center of brain activity is transferred even if there is no change in behavior. For example, in the external force field skill learning, the activity moves from the prefrontal cortex to the motor cortex, and in the sequence learning that learns the procedure, the activity shifts from the rostral part to the caudal part of the basal ganglia as the skill is fixed. Therefore, the degree of skill retention can be quantified by measuring brain activity during skill learning and quantifying how the center of the activity has been transferred. By using this method, retesting at a later date becomes unnecessary, and it is prevented that the skills are insufficient and placed on the actual work site.

  The present invention can also be applied to selection of a task that suits an individual's suitability according to a state estimated from a biological signal.

  Examples 1, 2 and 3 above are examples of binary judgment tasks such as “whether or not an error has occurred (whether it has occurred)”, “whether or not it has been forgotten”, or “whether or not the skill has become established”. Explained. In the fourth embodiment, a more general multi-value determination task is proposed, and a work content determination support apparatus according to the state of the day is described as an application example.

  First, according to Non-Patent Document 4, it is reported that in patients with epilepsy, signs of seizures can be found from brain activity for several hours to seven hours when epilepsy occurs. Therefore, it can be expected that the daily performance of the day can be predicted from the morning brain activity data.

  In Example 4, a database of a subject obtained by pairing brain activity data measured enough in the morning with behavioral data that was engaged in what kind of work during the day and evaluated personal performance at the end of the day. Remember as. The subjects set the work contents largely categorized in advance, for example, a memory task, a presentation, a detailed check task, a sentence structure, and an external negotiation. The data for one day is taken as one unit, and a database measured for several months is used as training data, and the optimum parameters of the classification algorithm are determined. Unlike the first, second, and third embodiments, this embodiment is a multi-value determination problem because the purpose is to select one optimal job content from a plurality of job contents on the day. Relevance vector machines based on Bayesian statistics are useful for such problems. Also in the fourth embodiment, an optimal task setting database that provides a criterion for determining the likelihood of human error, corresponding to the database described in FIG. 1B and the like, is generated. In this way, using the optimized relevance vector machine, it is possible to suggest work contents that show the best performance of the day from morning brain activity to the subject. As a result, it is possible to reduce the time spent on unsuitable work contents on that day and to reduce human errors.

[Modification]
In each of the above-described embodiments, optical topography is used as a brain function measuring method. However, other methods such as functional MRI and brain waves can be similarly implemented.
When functional MRI is used as a brain function measurement method, due to its relatively low temporal resolution, it is difficult to directly apply to an application for detecting human error in real time as in the first and second embodiments. Rather, it seems that it is suitable for analyzing a brain activity state on a long time scale as in Example 3 or 4 with a high spatial resolution of functional MRI. Functional MRI can also measure deep brain activity such as the cerebellum and basal ganglia that cannot be observed by optical topography. Since they are known to be related to functions such as sequence learning and movement automation, they can be applied to quantifying their functions by functional MRI.

  DESCRIPTION OF SYMBOLS 100 ... Apparatus main body, 110 ... Living body measurement apparatus, 120 ... Signal analysis apparatus, 130 ... Problem presentation apparatus, 140 ... External storage device, 150 ... External apparatus, 160 ... Operator (subject).

Claims (20)

A personal database for obtaining a personal database by acquiring information on the internal state of the subject and the biological measurement information of the subject that is temporally related to the information on the internal state and from which biological noise is removed. Generating means;
A classification algorithm optimizing means for optimizing a classification algorithm using the personal database as a training set;
Applying the optimized classification algorithm to new biometric information obtained by measuring the biosignal of the subject, behavior estimating means for estimating the internal state of the subject,
A human interface comprising information presenting means for presenting information based on the estimation result. In claim 1,
The internal state of the subject is related to sensory error,
The personal database generation means includes a subject presentation means for the subject, and a biological measurement means for performing biological measurement by optical topography for the subject in relation to the presentation state of the subject,
In order to detect the sensory error, as the new living body measurement, the optical topography is simultaneously measured in at least one set of brain regions mainly in the occipital parietal lobe and dorsal frontal lobe of the subject. Interface. In claim 1,
The internal state of the subject relates to working memory;
The personal database generation means includes a subject presentation means for the subject, and a biological measurement means for performing biological measurement by optical topography for the subject in relation to the presentation state of the subject,
In order to detect the working memory, as the new living body measurement, the optical topography is simultaneously measured in at least one set of brain regions mainly in the subject's dorsal frontal lobe. In claim 1,
The internal state of the subject relates to the fixation of skill learning;
The personal database generation means includes a subject presentation means for the subject, and a biological measurement means for performing biological measurement by optical topography for the subject in relation to the presentation state of the subject,
In order to detect fixation of the skill learning, as the new biometric measurement, the optical topography is simultaneously measured mainly in at least one set of brain regions of the frontal lobe and the motor area of the subject. . In claim 2,
In order to detect the sensory error, the human interface is characterized by measuring the pulse, respiration or blood pressure of the subject together with the optical topography as the new biological measurement. In claim 1,
The human database generating means includes a subject presenting means for the subject, and a living body measuring means for performing living body measurement by means of an electroencephalogram for the subject in relation to the presenting state of the subject. . In claim 1,
The personal database generation means uses a frequency band-pass filter and at least one of polynomial baseline correction, principal component analysis, or independent component analysis in pre-processing of the signal related to the biological measurement information. Human interface. In claim 1,
The human interface, wherein the personal database generating means includes a local hard disk or a server using the Internet as a storage device for the personal database. In claim 1,
The personal database generation means includes a subject presentation means for the subject, and a biological measurement means for performing biological measurement by optical topography for the subject in relation to the presentation state of the subject,
The human interface, wherein the classification algorithm optimizing means uses a multi-channel signal as a machine learning classification algorithm input in the biological measurement or brain function activity measurement. In claim 9,
The human interface, wherein the classification algorithm optimizing unit uses any one of a support vector machine, a relevance vector machine, and bagging as a classification method as the machine learning algorithm. In claim 9,
The human interface, wherein the classification algorithm optimization means uses a support vector machine and cross-validation in a binary classification task such as memory / disappointment in the machine learning algorithm. In claim 9,
The human interface, wherein the classification algorithm optimizing means uses a relevance vector machine in a binary or higher classification task in the machine learning algorithm. In claim 1,
The classification algorithm optimization means includes:
When the biometric signal of the i-th trial measured in parallel with the presentation of the task to the subject is x _i and the internal state data is y _i , the biometric signal and the internal state data are paired. (x _i , y _i ), these data pairs are recorded on the recording device over all trials (x _i , y _i ) (i = 1,..., N), and the classification is made as these training sets. Human interface characterized by optimizing algorithms. In claim 1,
Information on the internal state of the subject and the biometric information acquired in advance are measurement results in a state in which the subject is attached with a biometric device and given a problem,
Obtaining the new biometric information and new information on the internal state in a state in which the subject is wearing a biometric device and giving a problem, and applying the optimized classification algorithm to the information, A human interface characterized by estimating the internal state of a subject. In claim 1,
Information on the internal state of the subject and the biometric information acquired in advance are measurement results in a state in which the subject is attached with a biometric device and given a problem,
The biometric measurement device is attached to the subject, and the new biometric information and the new internal state information are acquired under actual work without giving a problem, and the optimized classification algorithm is applied to these pieces of information. A human interface characterized by estimating an internal state of the subject. In claim 1,
Subjective evaluation based on biological measurement information obtained by wearing a biological measurement device on the subject and performing brain function measurement in a resting state, and actual work contents / work results in a state where the subject does not wear the biological measurement device With the information on the internal state of the subject generated based on the result of performing the above, the database of the optimal task setting is generated as the personal database,
Based on the new internal state of the subject measured in a resting state and the database of the optimal task settings, the internal state of the subject on the day is estimated, and a determination result and advice are generated and output. Interface. Database generation means for creating a database of data on the degree of skill establishment based on actions from a plurality of subjects in various skill establishment stages and data on brain activity;
Classification algorithm optimization means for optimizing a classification algorithm using the data of the database as a training set;
Applying the optimized classification algorithm to new biometric information obtained by measuring the biosignal of the specific subject, action estimating means for evaluating the skill fixing degree of the specific subject,
A human interface comprising information presenting means for presenting information based on the estimation result. In claim 17,
The classification algorithm optimizing means optimizes a machine learning classification algorithm so as to maximize a generalized classification probability using behavior and brain activity data from the plurality of subjects.
The behavior estimation means measures the brain function of the specific subject only when evaluating the skill fixation.
Human interface characterized by that. A human error detection / prevention method using a human interface,
The human interface is
A personal database generation means, a storage device, a classification algorithm optimization means, a behavior estimation means, and an information presentation means,
Information on the internal state of the operator acquired in advance, and information on the biological measurement signal of the subject that has a temporal relationship with the information on the internal state and removes noise derived from the biological body,
Perform signal processing to remove biological noise from information on the biological measurement signal,
The information on the internal state and the information on the biological measurement signal are made into a database for each person as a training set,
Optimize machine learning algorithms using the personal database as a training set,
Obtaining a new biological measurement signal of the subject at the actual work site, based on the biological measurement signal, estimating the internal state of the subject with the optimized machine learning algorithm,
A human error detection / prevention method characterized in that audiovisually and tactilely optimal information is presented to the information presenting means in accordance with the estimation result. In claim 19,
The internal state of the subject is related to sensory error,
In order to detect the sensory error, human error is characterized in that, as the new biometric measurement, optical topography is simultaneously measured mainly in at least one set of brain regions of the occipital and dorsal frontal lobes of the subject. Detection / prevention method.

Images