JP2022045493A - Signal processing apparatus, signal processing method, and signal processing program - Google Patents

Abstract

To provide a signal processing apparatus with high processing accuracy.SOLUTION: An electroencephalograph 2 comprises an electroencephalogram sensor having a plurality of electrodes to be mounted on a human head, an A/D converter for converting an electroencephalogram signal, which is a potential change collected by the electrodes, into a digital signal, or the like. A signal processing apparatus 3 comprises a signal processing unit 33 for processing the measured signal, and comprises a pre-processing unit 31, a signal classification unit 32, and an intention prediction unit (signal processing unit) 33 as functional blocks. The pre-processing unit performs pre-processing, such as normalizing the electroencephalogram signal. The signal classification unit classifies the electroencephalogram signals according to measurement conditions. The intention prediction unit processes the electroencephalogram signals according to the classification result by the signal classification unit to predict the intention of a subject.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for processing a measured signal, for example, a technique for processing an electroencephalogram signal to predict a human intention.

Humans may have difficulty moving and communicating in words due to stroke, severe sequelae of accidents, certain brain abnormalities, and so on. In such a case, the patient cannot communicate with an occupational therapist or other specialist, and rehabilitation cannot be performed efficiently. In addition, it becomes difficult to communicate with family members, and it is easy to fall into a mentally difficult situation.

In order to overcome the above-mentioned problems, the inventors of the present application are conducting research to predict human intention by measuring brain wave signals (Non-Patent Documents 1 and 2). In this study, humans answer Yes / No to questions in their minds, and computers use artificial intelligence models (learned models) to extract features that mean differences in responses from brain wave signals. Predict the intention of. The trained model is generated by machine learning the EEG signals labeled with Yes / No answers. If this technology is applied to clinical practice, patients will be able to communicate their intentions without any special training, regardless of words or body movements.

Shinichi Ito, 2 outsiders, “An Electroencephalogram Analysis Method to Detect Preference Patterns Using Gray Association Degrees and Support Vector Machines”, Advances in Science, Technology and Engineering Systems Journal, Vol. 3, No. 5, September 2018, p. . 105-108 Shinichi Ito, 3 outsiders, "Electroencephalogram Data for Classifying Answers to Questions with Neural Networks and Support Vector Machine", International Journal of Signal Processing Systems, Vol. 7, No. 4, December 2019, p. 118-122

The electroencephalogram signal is affected by measurement conditions such as the presence or absence of noise such as blinking and changes in human physical condition. In Non-Patent Documents 1 and 2, since a trained model in which an electroencephalogram signal is machine-learned without considering the influence of measurement conditions is used, the prediction accuracy is limited.

Similarly, in other technical fields, it is common to perform machine learning without considering the presence or absence of noise, disturbance, and other information contained in the measured signal when generating a trained model for prediction. This was a target, and this hindered the improvement of accuracy.

The present invention has been made to solve the above problems, and an object of the present invention is to provide a signal processing apparatus having high processing accuracy.

In order to solve the above problems, the present invention includes the following aspects.
Item 1.
A signal processing device equipped with a signal processing unit that processes the measured signal.
Further, a signal classification unit for classifying the signals according to the measurement conditions is provided.
The signal processing unit is a signal processing device that performs processing according to the classification result by the signal classification unit.
Item 2.
Item 2. The signal processing device according to Item 1, wherein the signal classification unit classifies the signal using a machine-learned training model for classification.
Item 3.
Item 1 or 2, wherein the signal processing unit performs processing according to the classification result by using a predictive trained model machine-learned using the signal measured under the conditions corresponding to the classification result. Signal processing equipment.
Item 4.
Item 2. The signal processing device according to any one of Items 1 to 3, wherein the signal is an electroencephalogram signal collected from a human.
Item 5.
Item 4. The signal processing device according to Item 4, wherein the signal processing unit processes the brain wave signal to predict the human intention.
Item 6.
Item 4. The signal processing device according to Item 4 or 5, wherein the signal processing unit processes the brain wave signal to perform personal authentication of the human.
Item 7.
Item 2. The signal processing device according to any one of Items 4 to 6, wherein the signal processing unit processes the brain wave signal to detect a specific human emotion.
Item 8.
Item 6. The signal processing apparatus according to any one of Items 4 to 7, further comprising an operation receiving unit that accepts an input operation based on the processing result of the signal processing unit.
Item 9.
The signal processing unit processes the brain wave signal to determine its own motion image imagined by the human.
Item 8. The signal processing device according to Item 8, wherein the operation receiving unit receives the input operation based on the type of the operation image determined by the signal processing unit.
Item 10.
Item 2. The signal processing device according to Item 8 or 9, wherein the input operation is a character input operation.
Item 11.
Item 2. The signal processing apparatus according to any one of Items 4 to 10, wherein the condition includes the presence or absence of noise.
Item 12.
Item 6. The signal processing apparatus according to any one of Items 4 to 11, wherein the condition includes the human condition.
Item 13.
Item 12. The signal processing device according to Item 12, wherein the human condition is at least one of the presence / absence of caffeine intake, the presence / absence of nicotine intake, the physical condition, and the presence / absence of awakening drinking water intake.
Item 14.
A signal processing method including a signal processing step for processing a measured signal.
Further provided with a signal classification step of classifying the signals according to the measurement conditions,
In the signal processing step, a signal processing method that performs processing according to the classification result in the signal classification step.
Item 15.
A signal processing program that causes a computer to function as each part of the signal processing device according to any one of Items 1 to 13.

According to the present invention, it is possible to provide a signal processing device having high processing accuracy.

It is a block diagram which shows the schematic structure of the intention prediction system which concerns on 1st Embodiment of this invention. It is a flowchart which shows the processing procedure which performs the intention prediction of a subject by the intention prediction system. It is a flowchart which shows the procedure which generates the trained model for classification for classifying a signal based on the presence or absence of noise. It is a flowchart which shows the procedure which generates the trained model for first prediction. It is a flowchart which shows the procedure of generating the trained model for the second prediction. It is a flowchart which shows the procedure which generates the trained model for classification for classifying a signal based on the presence or absence of caffeine intake. It is a schematic diagram which shows the electrode position of the electroencephalogram sensor attached to a subject. It is a block diagram which shows the schematic structure of the personal authentication system which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the processing procedure which performs the personal authentication of a subject by a personal authentication system. It is a block diagram which shows the schematic structure of the emotion detection system which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the processing procedure which detects the emotion of a subject by an emotion detection system. It is a flowchart which shows the procedure which generates the trained model for first detection. This is an example of a task presented to a subject. This is an example of a questionnaire. It is a block diagram which shows the schematic structure of the character input system which concerns on 4th Embodiment of this invention. It is a flowchart which shows the processing procedure which a subject performs a character input by a character input system. It is a figure which shows the procedure of obtaining an electroencephalogram and an answer from a subject in chronological order.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[Embodiment 1]
Hereinafter, the first embodiment of the present invention will be described. In this embodiment, a mode of processing an electroencephalogram signal to predict a human intention will be described.

(System configuration)
FIG. 1 is a block diagram showing a schematic configuration of the intention prediction system 1 according to the present embodiment. The intention prediction system 1 is a system that predicts a human intention by measuring brain waves, and includes an electroencephalogram meter 2, a signal processing device 3, and a display device 4.

The electroencephalograph 2 includes an electroencephalogram sensor having a plurality of electrodes mounted on the human head, an A / D converter that converts an electroencephalogram signal, which is a potential change collected by the electrodes, into a digital signal. The electroencephalogram signal is input to the signal processing device 3 via wired, wireless communication or a recording medium.

The signal processing device 3 can be configured by, for example, a general-purpose computer. The signal processing device 3 includes a processor (not shown) such as a CPU and a GPU, a main storage device (memory, not shown), an auxiliary storage device (not shown), and the like as a hardware configuration. Various data such as a trained model and a signal processing program, which will be described later, are stored in the auxiliary storage device.

Further, the signal processing device 3 includes a pre-processing unit 31, a signal classification unit 32, and an intention prediction unit (signal processing unit) 33 as functional blocks. These functional blocks can be realized by software by the processor of the signal processing device 3 reading the signal processing program into the memory and executing it. That is, the signal processing program causes the computer to function as the pre-processing unit 31, the signal classification unit 32, and the intention prediction unit 33 of the signal processing device 3, whereby the signal processing device 3 uses the signal processing method according to the present embodiment. Execute.

The signal processing program may be downloaded to the signal processing device 3 via a communication network, or may be downloaded to a computer-readable non-temporary recording medium such as a CD-ROM on which the program code of the signal processing program is recorded. The signal processing program may be supplied to the signal processing device 3. The function of the signal processing device 3 will be described later.

The display device 4 is connected to the signal processing device 3. The display device 4 can be composed of a liquid crystal display, an organic EL display, or the like.

(Processing procedure)
Subsequently, the function of the signal processing device 3 will be described. FIG. 2 is a flowchart showing a processing procedure for predicting the intention of a subject (human) by the intention prediction system 1.

A brain wave sensor is attached to the subject, and a question is presented to the subject in a state where the brain wave can be measured (step S1). The presentation method is not particularly limited, and may be verbal or may be displayed on the screen.

The subject thinks of the answer to the question (YES or NO) in his head. At this time, the electroencephalogram is measured by the electroencephalograph 2 (step S2). The measured electroencephalogram signal is input to the signal processing device 3.

Subsequently, the preprocessing unit 31 of the signal processing device 3 performs preprocessing such as normalization on the electroencephalogram signal and inputs it to the signal classification unit 32 (step S3).

Subsequently, the signal classification unit 32 classifies the electroencephalogram signal according to the measurement conditions (step S4, signal classification step). In the present embodiment, the condition is the presence or absence of noise due to blinking, and the signal classification unit 32 uses a machine-learned learning model MC for classification to obtain an electroencephalogram signal, an electroencephalogram signal with noise, and noise. Classify as no EEG signal. The classification result and the electroencephalogram signal are input to the intention prediction unit 33.

As will be described later, the training model MC for classification measures the brain wave signal multiple times for the subject in advance, labels the condition with / without blinking on the brain wave signal, generates a learning data set, and artificially creates the learning data set. It was generated by inputting into an intelligent model and performing machine learning. Therefore, the trained model MC for classification can extract the feature of presence / absence of noise due to blinking from the brain wave signal of the subject.

The intention prediction unit 33 processes the brain wave signal according to the classification result by the signal classification unit 32 to predict the intention of the subject. Specifically, when the electroencephalogram signal is an electroencephalogram signal with noise (YES in step S5), the intention prediction unit 33 predicts the intention of the subject using the first prediction trained model MP1 (step S6, Signal processing step). On the other hand, when the electroencephalogram signal is a noise-free electroencephalogram signal (NO in step S5), the intention prediction unit 33 predicts the intention of the subject using the second prediction trained model MP2 (step S7, signal processing step). ).

As will be described later, the first predictive trained model MP1 and the second predictive trained model MP2 are machine-learned using brain wave signals measured under conditions corresponding to the classification results. More specifically, the first predictive trained model MP1 measures the brain wave signal multiple times while presenting a question to the subject in advance, and the subject responds to the brain wave signal measured when blinking (YES). Or NO) is labeled to create a learning data set, which is input to an artificial intelligence model to perform machine learning. The second predictive learned model MP2 measures the brain wave signal multiple times while presenting a question to the subject in advance, and the subject's intention (YES or NO) to answer the brain wave signal measured when not blinking. It is generated by labeling to create a learning data set and inputting it into an artificial intelligence model to perform machine learning.

The prediction result (YES or NO) in steps S6 and S7 is displayed on the display device 4 (step S8). The method of outputting the prediction result is not limited to this, and may be output by voice, for example.

(Generation of trained model)
In the present embodiment, the above-mentioned trained model MC for classification, the trained model MP1 for the first prediction and the trained model MP2 for the second prediction are, for example, CNN (convolutional neural network) and SVM (support vector). It can be configured by a machine (support vector machine).

FIG. 3 is a flowchart showing a procedure for generating a trained model MC for classification.

A brain wave sensor is attached to the subject, and a question is presented to the subject in a state where the brain wave can be measured (step S1). The subject thinks of the answer to the question (YES or NO) in his head. At this time, the brain wave of the subject is measured, the subject answers, and the content of the reply and the presence or absence of noise due to blinking are recorded (step S2). If the data is not sufficiently accumulated for machine learning (NO in step S13), steps S11 and S12 are repeated with different questions.

When the data is sufficiently accumulated (YES in step S13), the presence or absence of noise is labeled on the EEG signal data to generate a learning data set (step S14). Subsequently, the CNN is machine-learned by the error back-propagation method using the training data set (step S15). Further, the SVM is machine-learned by inputting the output of the CNN to the SVM and determining the hyperplane with the presence or absence of noise as teacher data (step S16). As a result, the trained model MC for classification is generated (step S17).

When an unknown EEG signal is input to the trained model MC for classification, CNN extracts a feature indicating the presence or absence of noise from the EEG signal, and SVM converts the EEG signal into a noisy EEG signal or a noisy EEG signal based on the feature. Classify into.

FIG. 4 is a flowchart showing a procedure for generating the first predicted trained model MP1. This flow is started from the stage in which the electroencephalogram data is sufficiently accumulated (YES in step S13) in the flow shown in FIG.

In step S21, only the noisy data is extracted from the brain wave data. This is done based on the information indicating the presence or absence of noise recorded in step S2 shown in FIG. Subsequently, the answer recorded in step S2 is labeled with the extracted noisy data to generate a learning data set (step S22). Subsequently, the CNN is machine-learned by the error back-propagation method using the training data set (step S23). Further, the SVM is machine-learned by inputting the output of the CNN to the SVM and determining the hyperplane using the answer as the teacher data (step S24). As a result, the trained model MP1 for the first prediction is generated (step S25).

The trained model MP1 for the first prediction generated in this way outputs the intention of the subject when an unknown brain wave signal is input, but particularly when the brain wave signal with noise is input, the subject has high accuracy. Can output the intention of.

FIG. 5 is a flowchart showing a procedure for generating a second predicted trained model MP2. This flow is started from the stage in which the electroencephalogram data is sufficiently accumulated (YES in step S13) in the flow shown in FIG.

In step S31, only noise-free data is extracted from the brain wave data. This is done based on the information indicating the presence or absence of noise recorded in step S2 shown in FIG. Subsequently, the answer recorded in step S2 is labeled with the extracted noise-free data to generate a learning data set (step S32). Subsequently, the CNN is machine-learned by the error back-propagation method using the training data set (step S33). Further, the SVM is machine-learned by inputting the output of the CNN to the SVM and determining the hyperplane using the answer as the teacher data (step S34). As a result, the trained model MP2 for the second prediction is generated (step S35).

The second predicted trained model MP2 generated in this way outputs the subject's intention when an unknown brain wave signal is input, but especially when a noise-free brain wave signal is input, the subject has high accuracy. Can output the intention of.

The trained model MC for classification, the trained model MP1 for first prediction, and the trained model MP2 for second prediction are used for predicting the intentions of humans other than the subject who acquired the brain wave signals for generating these models. , Do not use in principle.

(Summary)
As described above, in the present embodiment, the electroencephalogram signal for classification is classified into the electroencephalogram signal with noise or the electroencephalogram signal without noise by using the trained model MC for classification, and the electroencephalogram signal with noise is input to the trained model MP1 for first prediction. Then, the subject's intention is predicted, and the noise-free electroencephalogram signal is input to the second prediction trained model MP2 to predict the subject's intention. Since the first predictive trained model MP1 is generated by machine learning the noisy electroencephalogram signal, the subject's intention can be predicted with high accuracy from the noisy electroencephalogram signal. Further, since the second predicted learning model MP2 is generated by machine learning the noise-free electroencephalogram signal, the subject's intention can be predicted with higher accuracy from the noise-free electroencephalogram signal. Therefore, the prediction accuracy can be improved as compared with the conventional technique in which the presence / absence of noise is not classified.

(Additional notes)
In the above embodiment, the condition for measuring the brain wave is the presence or absence of noise due to blinking, but the present invention is not limited to this. For example, the conditions for measuring brain waves include the presence / absence of caffeine intake, the presence / absence of nicotine intake, the condition of physical condition, and the presence / absence of awakening drinking water.

Further, in the above embodiment, the case of processing an electroencephalogram signal collected from a human has been described, but the signal to be processed is not limited to the electroencephalogram signal. For example, when the signal is an image signal, the measurement conditions include the brightness of the shooting environment. When the signal is an audio signal, the measurement conditions include the presence or absence of howling.

Further, in the above embodiment, the signal classification unit 32 classifies the electroencephalogram signal into two types, an electroencephalogram signal with noise and an electroencephalogram signal without noise, but the number of the signals to be classified is not particularly limited. In principle, the number of signal classifications is equal to the square of the conditional number.

(Modification example)
Hereinafter, the case where the condition of the electroencephalogram measurement is the presence or absence of caffeine intake will be described. First, a method of generating a learning model will be described. FIG. 6 is a flowchart showing a procedure for generating a trained model MC for classification.

A brain wave sensor is attached to a subject who has not ingested caffeine, and a task image is presented to the subject in a state where the brain wave can be measured (step S41). The task images are two types of images (large circle and small circle) used in the visual odball task. A large circle is displayed 80 times as a standard stimulus and a small circle is displayed 20 times as a target stimulus per set. Repeat this for 3 sets. Then, the brain wave is measured for each presentation of each image (step S42). The electroencephalogram signal is preferably collected from the electrodes located at O1 and O2 in the sensing positions of the international 10-10 system shown in FIG. If the data is not sufficiently accumulated for machine learning (NO in step S43), steps S41 and S42 are repeated.

When the data is sufficiently accumulated (YES in step S43), the subject is asked to ingest caffeine (step S44), and the brain wave is measured while presenting the task image as in steps S41 and S42 (step S45). And S46). Steps S45 and 26 are also repeated until the data is sufficiently accumulated to allow machine learning to be moderate.

When the data is sufficiently accumulated (YES in step S47), the presence or absence of caffeine intake is labeled on the electroencephalogram signal data to generate a learning data set (step S48). Subsequently, machine learning is performed using the training data set in the same manner as in steps S15 and 16 shown in FIG. 3 (step S49). As a result, a trained model MC for classification for classifying brain wave signals based on the presence or absence of caffeine intake is generated (step S50).

In general, in the Odball task, the apex latency of the event-related potential P300 changes depending on the presence or absence of caffeine intake. By learning the brain waves including such characteristics, it is possible to generate a trained model MC for classification that accurately classifies the presence or absence of caffeine intake.

[Embodiment 2]
Hereinafter, a second embodiment of the present invention will be described. In this embodiment, an embodiment in which an electroencephalogram signal is processed to perform personal authentication will be described. Members having the same functions as those in the above-described first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

(System configuration)
FIG. 8 is a block diagram showing a schematic configuration of the personal authentication system 1a according to the present embodiment. The personal authentication system 1a is a system for personally authenticating a human by measuring an electroencephalogram, and includes an electroencephalogram 2, a signal processing device 3a, and a display device 4. That is, the personal authentication system 1a has a configuration in which the signal processing device 3 is replaced with the signal processing device 3a in the intention prediction system 1 shown in FIG.

The signal processing device 3a has the same hardware configuration as the signal processing device 3, and includes a preprocessing unit 31, a signal classification unit 32, and a personal authentication unit (signal processing unit) 33a as functional blocks. There is. In the present embodiment, the signal classification unit 32 has a function of classifying brain wave signals according to the presence or absence of caffeine intake by using the learning model MC for classification generated according to the flowchart shown in FIG. The personal authentication unit 33a has a function of processing an electroencephalogram signal to perform personal authentication of a human using the first authentication trained model MI1 and the second authentication trained model MI2.

(Processing procedure)
Subsequently, the function of the signal processing device 3a will be described. FIG. 9 is a flowchart showing a processing procedure for personally authenticating a subject (human) by the personal authentication system 1a.

An electroencephalogram sensor is attached to the subject, and a task image for classification is presented to the subject in a state where the electroencephalogram can be measured (step S51). The classification task image is the same as the task image presented to the subject in step S41 shown in FIG. At the same time, the electroencephalogram is measured by the electroencephalogram meter 2, the preprocessing unit 31 performs preprocessing such as normalization on the electroencephalogram signal, and inputs the electroencephalogram signal to the signal classification unit 32 (step S52). The electrodes for collecting the electroencephalogram signal are the electrodes located at O1 and O2 shown in FIG.

Subsequently, the signal classification unit 32 classifies the electroencephalogram signal according to the measurement conditions (step S53, signal classification step). In the present embodiment, the condition is the presence or absence of caffeine ingestion, and the signal classification unit 32 uses the learned model MC for classification to obtain an electroencephalogram signal, an electroencephalogram signal of a subject ingesting caffeine, and a cafe. Classify as EEG signals of subjects who have not ingested ingestion. The classification result is input to the personal authentication unit 33a.

Subsequently, while presenting the authentication task image to the subject (step S54), the electroencephalogram is measured by the electroencephalogram meter 2 and preprocessed by the preprocessing unit 31 (step S55). The authentication task image is an image (animal or the like) that gives a characteristic impression, and is an image used in the process of generating the first authentication trained model MI1 and the second authentication trained model MI2. The electroencephalogram signal preprocessed in step S55 is input to the personal authentication unit 33a.

When it is an electroencephalogram signal of a subject who has not ingested caffeine (NO in step S56), the personal authentication unit 33a performs personal authentication of the subject using the learned model MI1 for the first authentication (step S57). On the other hand, in the case of the brain wave signal of the subject ingesting caffeine (YES in step S56), the personal authentication unit 33a performs personal authentication of the subject using the second authentication trained model MI2 (step S58).

The authentication results in steps S57 and S58 are displayed on the display device 4 (step S59).

The trained model MI1 for the first authentication and the trained model MI2 for the second authentication are machine-learned using the electroencephalogram signals measured under the conditions corresponding to the classification results. More specifically, the first authentication trained model MI1 creates a learning data set by labeling the brain wave signal measured while presenting the authentication task image to the subject who does not ingest caffeine. However, it was generated by inputting this into an artificial intelligence model and performing machine learning. The second learning model MP2 for prediction creates a learning data set by labeling the brain wave signal measured while presenting the task image for authentication to the subject ingesting caffeine, and creates a learning data set, which is artificial intelligence. It is generated by inputting to the model and performing machine learning.

It is preferable that the method of presenting the task image for authentication is the same as that of the visual oddball task. Since there are large individual differences in the amplitude and vertex latency of P300, it is possible to generate a predictive trained model that can be accurately personally authenticated by performing machine learning using the electroencephalogram image at the time of presenting the task image for authentication. ..

The measured brain waves are greatly affected by the content and presentation method of the authentication task image. Therefore, in personal authentication by the personal authentication unit 33a, it is necessary to keep the presentation interval, lighting time, shape, color, etc. of the authentication task image constant, but if such information is leaked, the content of the authentication task image or By changing the presentation method and generating a trained model for prediction, risks such as fraudulent authentication can be avoided.

[Embodiment 3]
Hereinafter, a third embodiment of the present invention will be described. In this embodiment, a mode of processing an electroencephalogram signal to detect human emotions will be described. Members having the same functions as those in the first and second embodiments described above are designated by the same reference numerals, and the description thereof will be omitted.

(System configuration)
FIG. 10 is a block diagram showing a schematic configuration of the emotion detection system 1b according to the present embodiment. The emotion detection system 1b is a system that detects human emotions by measuring brain waves, and includes an electroencephalogram meter 2, a signal processing device 3b, and a display device 4. That is, the emotion detection system 1b has a configuration in which the signal processing device 3 is replaced with the signal processing device 3b in the intention prediction system 1 shown in FIG.

The signal processing device 3b has the same hardware configuration as the signal processing device 3, and includes a preprocessing unit 31, a signal classification unit 32, and an emotion detection unit (signal processing unit) 33b as functional blocks. There is. In the present embodiment, the signal classification unit 32 has a function of classifying brain wave signals according to the presence or absence of caffeine intake by using the learning model MC for classification generated according to the flowchart shown in FIG. The emotion detection unit 33b has a function of processing a brain wave signal to detect a specific human emotion by using the first detection trained model MT1 and the second detection trained model MT2. In the present embodiment, the emotion detection unit 33b detects troublesome emotions of human beings in particular.

(Processing procedure)
Subsequently, the function of the signal processing device 3b will be described. FIG. 11 is a flowchart showing a processing procedure for detecting the emotion of a subject (human) by the emotion detection system 1b.

Steps S61 to S63 are performed in the same manner as steps S51 to S53 shown in FIG. In step S63, the signal classification unit 32 classifies the electroencephalogram signal into the electroencephalogram signal of the subject who is ingesting caffeine and the electroencephalogram signal of the subject who is not ingesting caffeine by using the learned model MC for classification. , The classification result is input to the emotion detection unit 33b.

Subsequently, the electroencephalogram is measured by the electroencephalogram meter 2 and preprocessed by the preprocessing unit 31 (step S64). At this time, the electrodes for collecting the electroencephalogram signal are, for example, the electrodes located at AF3, F7, F3, FC5, T7, T8, FC6, F4, F8 and AF4 shown in FIG. The electroencephalogram signal is input to the emotion detection unit 33b.

When it is an electroencephalogram signal of a subject who has not ingested caffeine (NO in step S65), the emotion detection unit 33b detects the troublesome emotion of the subject using the first detection trained model MT1 (step S66). On the other hand, in the case of the brain wave signal of the subject ingesting caffeine (YES in step S64), the emotion detection unit 33b detects the troublesome emotion of the subject using the second detection trained model MT2 (step S67). ..

The detection results in steps S66 and S67 are displayed on the display device 4 (step S59).

(Generation of trained model)
The first detection trained model MT1 and the second detection trained model MT2 are machine-learned using brain wave signals measured under conditions corresponding to the classification results. FIG. 12 is a flowchart showing a procedure for generating the first detection trained model MT1.

A brain wave sensor is attached to a subject who has not ingested caffeine, and a task is presented to the subject in a state where the brain wave can be measured (step S71). The task in this embodiment is to move four kinds of objects (pinto beans, soybeans, confectionery, and konpeito) in a container to another container with chopsticks. Specifically, the task of moving any of the above four types of objects within the time limit is given to the subject 50 times in total, succeeding 25 times out of 50 times, and suppressing the failure within 5 times. Is imposed.

FIG. 13 shows an example of a task presented to a subject. The subject looks at the content of the task and selects whether or not to execute it (step S72). If the task is not executed, the "Next work" button B1 is selected, and if the task is to be executed, the "Start work" button B2 is selected.

Further, the subject answers the reason why the task can be executed (step S73). FIG. 14 shows an example of a questionnaire for entering an answer. The "selected" column is the answer column when the task is selected to be executed, and the "non-selected" column is the answer column when the task is not executed.

If the subject chooses to perform the task, select a number from 1 to 4 below that applies to the reason and enter it in the corresponding column of the "Selection" column.
1: There is time to spare and you can do it.
2: I have a little time to spare and can do it.
3: I don't have time to spare, but I can.
4: I may not be able to do it because I have no time to spare.

On the other hand, if the subject chooses not to perform the task, he / she selects a number corresponding to the reason from 1 to 4 below and fills in the corresponding column in the "non-selected" column.
1: I can do it, but I don't work.
2: It may be possible, but I will not work.
3: I may not be able to do it, so I will not work.
4: I can't do it, so I don't work.

The larger the number, the stronger the troublesome feelings of the subject. In parallel, the electroencephalogram is measured (step S74). If the subject chooses to perform the task (YES in step S75), the subject performs the task (step S76). Steps S71 to S76 are repeated 50 times while randomly changing the content of the task (object, time limit) (YES in step S77).

After that, the questionnaire result is labeled on the measured brain wave signal to generate a learning data set (step S78), and machine learning is performed using the learning data set (step S79). As a result, the trained model MT1 for the first detection is generated.

The first detection trained model MT1 generated in this way can detect the troublesome emotions of the subject from the unknown electroencephalogram signal, but the troublesome emotions of the subject who does not ingest caffeine are particularly high. It can be detected with accuracy.

Similarly, by performing the above-mentioned steps S71 to S78 for a subject ingesting caffeine, a second detection trained model MT2 is generated. The second detection trained model MT2 can detect the troublesome emotions of the subject from an unknown brain wave signal, and in particular, can detect the troublesome emotions of the subject ingesting caffeine with high accuracy. ..

By applying the emotion detection system 1b according to the present embodiment to the control of the electric assist system that assists the operation of the care-requiring person, the presence or absence of troublesome emotions of the care-requiring person is automatically detected and the assisting operation is performed. It becomes possible to control. This makes it possible to appropriately support the independence of the care recipient.

[Embodiment 4]
Hereinafter, a fourth embodiment of the present invention will be described. In this embodiment, a mode in which a character input operation is performed based on a processing result of an electroencephalogram signal will be described. Members having the same functions as those in the first to third embodiments described above are designated by the same reference numerals, and the description thereof will be omitted.

(System configuration)
FIG. 15 is a block diagram showing a schematic configuration of the character input system 1c according to the present embodiment. The character input system 1c is a system that discriminates one's own motion image imagined by a human by measuring brain waves and realizes a character input operation based on the type of motion image, and is a brain wave meter 2, a signal processing device 3c, and a system. The display device 4 is provided. That is, the character input system 1c has a configuration in which the signal processing device 3 is replaced with the signal processing device 3c in the intention prediction system 1 shown in FIG.

The signal processing device 3c has the same hardware configuration as the signal processing device 3, and has a preprocessing unit 31, a signal classification unit 32, an image discrimination unit (signal processing unit) 33c, and an operation reception unit as functional blocks. It is provided with a unit 34. In the present embodiment, the signal classification unit 32 has a function of classifying the electroencephalogram signal according to the presence or absence of noise by using the learned model MC for classification generated according to the flowchart shown in FIG. The image discrimination unit 33c has a function of processing an electroencephalogram signal to discriminate its own motion image imagined by a human using the discriminant learned model MD. The operation receiving unit 34 has a function of receiving a character input operation based on the type of the operation image determined by the image discrimination unit 33c.

(Processing procedure)
Subsequently, the function of the signal processing device 3c will be described. FIG. 16 is a flowchart showing a processing procedure in which a subject (human) inputs characters by the character input system 1c.

The subject is equipped with an electroencephalogram sensor, and in a state where the electroencephalogram can be measured, the subject imagines his / her own motion image according to the character to be input (step S81). Table 1 shows an example of correspondence between characters and motion images.

Here, the image 1 is an "image of holding one's own dominant hand", and the image 2 is an "image of making a voice". "12" means imagining image 2 after image 1 without blinking, and "21" means imagining image 1 after image 2 without blinking.

For example, when you want to input the character "O", first imagine image 1 to select "A line" in the left table, blink once, and then imagine image 2. When the selection in the left table is completed, one blink is performed in order to shift to the selection in the right table. That is, blinking means a delimiter gesture. Then, in order to select "step" in the table on the right, image 1 and image 2 are continuously imagined without blinking, blinking once, and then image 1 is imagined. If you make a mistake in the selection in the table on the left, you can reselect by blinking twice.

If the subject wants to enter "Good morning", perform images and blink in the following order.
"O": (left table) 1 → blink → 1 (select a row) → blink → (right table) 1 → 2 → blink → 1
"Ha": (left table) 2 → blink → 1 → 2 (row selection) → blink → (right table) 1 → blink → 1
"Yo": (left table) 1 → 2 → blink → 2 (or row selection) → blink → (right table) 1 → 2 → blink → 1
"U": (left table) 1 → blink → 1 (select a row) → blink → (right table) 2 → blink → 1

While the subject is performing the image and blinking in this way, the electroencephalogram is measured by the electroencephalogram meter 2, and the preprocessing unit 31 preprocesses the electroencephalogram signal (step S82).

Subsequently, the signal classification unit 32 classifies the electroencephalogram signal according to the measurement conditions (step S83). In the present embodiment, the condition is the presence or absence of noise due to blinking, and the signal classification unit 32 uses a machine-learned learning model MC for classification to obtain an electroencephalogram signal, an electroencephalogram signal with noise, and noise. Classify as no EEG signal. The classification result and the electroencephalogram signal are input to the image discrimination unit 33c.

When the electroencephalogram signal is a noise-free electroencephalogram signal (NO in step S84), the image discrimination unit 33c discriminates its own motion image imagined by the subject using the discriminant learned model MD (step S85). The discriminant trained model MD is a model generated by machine learning the electroencephalogram signals measured while letting the subject imagine the above images 1 and 2 in advance.

On the other hand, when the electroencephalogram signal is an electroencephalogram signal with noise (YES in step S84), the image discrimination unit 33c determines that the subject blinks without discriminating the motion image.

The determination results of steps S85 and S86 are input to the operation reception unit 34. The operation reception unit 34 accepts a character input operation by specifying the character to be input by the subject by collating the motion image and blink of the subject determined by the image discrimination unit 33c with the rules in Table 1 ( Step S87). The entered characters are displayed on the display device 4.

As described above, the subject can perform a character input operation by imagining and blinking a specific motion image.

[Additional notes]
The present invention is not limited to the above embodiment, and various modifications can be made without departing from the spirit of the present invention. For example, an embodiment obtained by appropriately combining the technical means disclosed in each embodiment also belongs to the technical scope of the present invention.

Hereinafter, examples of the present invention will be described, but the present invention is not limited to the following examples.

In this embodiment, an experiment was conducted to demonstrate that the intention of the subject can be predicted with high accuracy by the intention prediction system 1 according to the first embodiment. The subjects were 9 healthy volunteers from Tokushima University (average age 22.7 years).

(Acquisition of brain wave signal)
First, the electroencephalogram sensor of the electroencephalograph 2 was attached to the head of the subject sitting on the chair, and a plurality of questions were presented to the subject while measuring the electroencephalogram. Specifically, 25 questions randomly selected from the 30 questions shown in Table 2 were presented to the subjects.

The 30 questions consist of 9 clearly answered questions (clear), 16 questions about physical condition (physical condition), and 5 questions about requirements (requests). During the experiment, the EEG signals collected from the electrodes located at AF3, F7, F3, T7, T8, F4, F8 and AF4 in the sensing positions of the international 10-10 system shown in FIG. 7 are signal processing devices 3. Entered in.

The subjects answered YES or NO to each question verbally after thinking of a YES or NO answer in their heads. Specifically, as shown in FIG. 17, the question is presented to the subject, and the subject imagines the answer in his / her head. The signal processing device 3 acquires an electroencephalogram signal for 2 seconds 3 to 5 seconds after the question is presented. Then, the same question is presented to the subject again, and the subject answers the correct answer verbally. This was done for 25 questions, and 25 2-second EEG signals were obtained from one subject.

In addition, 10 segmented data were generated from each of the 2 second EEG signals. Specifically, 10 data were generated by setting a data frame for 1 second from the start position of the EEG signal and extracting the waveform in the frame while shifting the data frame by 1/10 second. As a result, a total of 10 × 25 = 250 data was acquired from one subject.

The timing of the subject's blinking was also recorded. The subjects were not careful about blinking so that they would blink unconsciously.

Furthermore, the same experiment was carried out for a total of 4 days at different dates and times, and about 1000 (250 × 4) data were obtained from one subject. Table 3 shows the data obtained from the subjects A to I classified by the presence or absence of noise due to blinking and the type of response (YES or NO).

(Generation of trained model)
Subsequently, the acquired data was subjected to preprocessing such as normalization to generate a trained model MC for classification, a trained model MP1 for first prediction, and a trained model MP2 for second prediction. Specifically, based on the information indicating the timing of blinking, each data acquired from the subject is labeled with the condition of presence / absence of blinking to generate a learning data set, and the artificial intelligence model is machine-learned to classify. A trained model MC was created. Based on the responses obtained verbally from the subjects, each data obtained from the subjects was classified into noisy data or noisy data. Further, by labeling the answers to the noisy data to generate a training data set and machine learning the artificial intelligence model, the first prediction trained model MP1 was generated. The training data set was generated by labeling the answers to the noise-free data, and the trained model MP2 for the second prediction was generated by machine learning the artificial intelligence model.

The artificial intelligence model was composed of CNN and SVM, and the CNN used one having an input layer, three hidden layers and an output layer. The size of the input layer was 8 × 128 (number of channels × length of EEG signal). The filter sizes of the three non-display layers were 3, 3, and 2, respectively, and the number of convolutional layers of the three non-display layers was 50, 50, and 20, respectively. The SVM used a linear SVM classifier.

(inspection result)
In the verification experiment, brain wave signals and responses were obtained from each subject by the method shown in FIG. 17, and a test data set was generated. Then, the training trained model MC for classification is used to classify the data with noise or the data without noise, the data with noise is input to the trained model MP1 for the first prediction, and the data without noise is the trained model MP2 for the second prediction. By inputting to, the intention of each subject was predicted and collated with the actual answer. Table 4 shows the sensitivity and accuracy of the intention prediction of the subjects A to I.

On the other hand, as a comparative example, a trained model for prediction is generated for the same subject as described above without considering the noise due to blinking, and the intention of each subject is predicted using the trained model for prediction. I collated with the answer. Table 5 shows the sensitivity and accuracy of the intention prediction of the subjects A to I.

As described above, in this embodiment, the sensitivity and accuracy are improved as compared with the comparative example. Therefore, it was found that the intention prediction system 1 according to the first embodiment of the present invention can predict the intention of the subject with extremely high accuracy.

1 Intention prediction system 1a Personal authentication system 1b Emotion detection system 1c Character input system 2 Brain wave meter 3 Signal processing device 31 Preprocessing unit 32 Signal classification unit 33 Intention prediction unit (signal processing unit)
33a Personal authentication unit (signal processing unit)
33b Emotion detection unit (signal processing unit)
33c Image discrimination unit (signal processing unit)
34 Operation reception unit 4 Display device MC Classified trained model MP1 1st predictive trained model MP2 2nd predictive trained model MI1 1st certified trained model MI2 2nd certified trained model MT1 1st detection Trained model MT2 Trained model for second detection Trained model for MD discrimination

Claims (15)

A signal processing device equipped with a signal processing unit that processes the measured signal.
Further, a signal classification unit for classifying the signals according to the measurement conditions is provided.
The signal processing unit is a signal processing device that performs processing according to the classification result by the signal classification unit. The signal processing device according to claim 1, wherein the signal classification unit classifies the signal using a machine-learned training model for classification. According to claim 1 or 2, the signal processing unit performs processing according to the classification result by using a predictive trained model machine-learned using the signal measured under the conditions corresponding to the classification result. The signal processing device described. The signal processing device according to any one of claims 1 to 3, wherein the signal is an electroencephalogram signal collected from a human. The signal processing device according to claim 4, wherein the signal processing unit processes the brain wave signal to predict the human intention. The signal processing device according to claim 4 or 5, wherein the signal processing unit processes the brain wave signal to authenticate the person personally. The signal processing device according to any one of claims 4 to 6, wherein the signal processing unit processes the brain wave signal to detect a specific human emotion. The signal processing apparatus according to any one of claims 4 to 7, further comprising an operation receiving unit that accepts an input operation based on the processing result of the signal processing unit. The signal processing unit processes the brain wave signal to determine its own motion image imagined by the human.
The signal processing device according to claim 8, wherein the operation receiving unit receives the input operation based on the type of the operation image determined by the signal processing unit. The signal processing device according to claim 8 or 9, wherein the input operation is a character input operation. The signal processing apparatus according to any one of claims 4 to 10, wherein the condition includes the presence or absence of noise. The signal processing apparatus according to any one of claims 4 to 11, wherein the condition includes the human condition. The signal processing device according to claim 12, wherein the human condition is at least one of the presence / absence of caffeine intake, the presence / absence of nicotine intake, the physical condition, and the presence / absence of awakening drinking water intake. A signal processing method including a signal processing step for processing a measured signal.
Further provided with a signal classification step of classifying the signals according to the measurement conditions,
In the signal processing step, a signal processing method that performs processing according to the classification result in the signal classification step. A signal processing program that causes a computer to function as each part of the signal processing device according to any one of claims 1 to 13.

Images