JP7455315B1 - Program, manufacturing method, manufacturing equipment and connection method - Google Patents

Abstract

[Problem] To provide a program, manufacturing method, manufacturing device, and connection method for generating a trained model capable of estimating a subject's intention with a relatively high degree of accuracy based on the subject's electroencephalogram. [Solution] Each of a plurality of training data used to generate a trained model that estimates the subject's intention includes information related to electroencephalograms, and is used for machine learning taking into account at least a portion of a plurality of parameters. The program causes a computer to execute the steps of generating a plurality of trained models by performing machine learning for each of a plurality of combinations of a plurality of parameters, calculating sensitivity and specificity for each of the plurality of trained models, and displaying a plurality of mosaic graphs on a common screen. Each of the plurality of mosaic graphs shows information related to sensitivity for at least a portion of the plurality of combinations. The specificity assumed for the plurality of mosaic graphs is different from one another. [Selected Figure] Figure 1

Description

The present invention relates to a program, a manufacturing method, a manufacturing device, and a connection method.

JP2018-68886A (Patent Document 1) discloses a BMI (Brain Machine Interface) system. This BMI system includes an intracorporeal device and an extracorporeal device. The intracorporeal device has an electrode group and is implanted within the subject's head. The intracorporeal device is configured to accurately acquire the subject's brain wave signals using a group of electrodes placed on the brain surface (subdural). The brain wave signals acquired by the intracorporeal device are transmitted to the extracorporeal device. The extracorporeal device performs an operation according to the received electroencephalogram signal (see Patent Document 1).

JP2018-68886A

The object of the present invention is to provide a program, a manufacturing method, a manufacturing apparatus, and a computer program for generating a trained model capable of estimating a subject's motor intention (hereinafter also simply referred to as "intention") with relatively high accuracy based on the subject's brain waves. The purpose is to provide a connection method.

A program according to an aspect of the present invention is used to generate a trained model that estimates a subject's intention based on the subject's brain waves. The learned model is generated through machine learning using multiple pieces of training data. In machine learning, each of a plurality of parameters is adjusted in advance. Each of the plurality of training data includes information regarding brain waves, and is used for machine learning based on at least some of the plurality of parameters. The program includes a step of generating a plurality of trained models by performing machine learning for each of a plurality of combinations of a plurality of parameters, a step of calculating sensitivity and specificity for each of the plurality of trained models, and a step of calculating sensitivity and specificity for each of the plurality of trained models. A computer is caused to perform the steps of displaying a mosaic graph on a common screen. Each of the plurality of mosaic graphs shows information regarding the sensitivity in at least some of the plurality of combinations. The specificity values assumed for a plurality of mosaic graphs are different from each other.

When this program is executed by a computer, multiple mosaic graphs are displayed on a common screen. Each of the plurality of mosaic graphs shows information regarding sensitivity in at least some of the plurality of combinations, and the values regarding specificity assumed for the plurality of mosaic graphs are different from each other. Therefore, according to this program, the sensitivity and specificity of at least some of the plurality of combinations are visually represented by a plurality of mosaic graphs, so that the user (physician, etc.) can see a good combination of the plurality of parameters on the screen. can be easily recognized. Users (physicians, etc.) can refer to multiple mosaic graphs to determine the parameters to be used and proceed with the evaluation. Easy to generate.

In each of the plurality of mosaic graphs, at least some of the plurality of parameters may be determined as prerequisite values. The computer may further execute a step of updating each of the plurality of mosaic graphs when the mosaic graph is updated.

When this program is executed by a computer, each of the plurality of mosaic graphs is updated if the value of the assumption is changed. Therefore, according to this program, the user can easily recognize the influence that parameter changes have on sensitivity through the screen display.

The information regarding sensitivity may include color information, and the program includes the step of accepting a change in scale indicating the correspondence between sensitivity values and colors, and updating each of the plurality of mosaic graphs when the scale is changed. The computer may further perform the steps.

When this program is executed by a computer, each of the plurality of mosaic graphs is updated when the scale is changed. Therefore, according to this program, the user can more clearly recognize the relationship between each parameter and sensitivity through scale adjustment.

In the above program, the information regarding sensitivity may include numerical information.

According to this program, the user can easily recognize good combinations of a plurality of parameters through the display of numerical information.

The above program may further cause the computer to perform the steps of deriving the optimal combination of multiple parameters based on both sensitivity and specificity, and displaying the numerical values of each parameter in the optimal combination on the screen. , each of the plurality of mosaic graphs may be generated assuming an optimal combination.

When this program is executed by a computer, the numerical values of each parameter in the optimal combination are displayed on the screen, and each of the plurality of mosaic graphs is generated based on the optimal combination. Therefore, according to this program, the user can easily recognize the combination of parameters that maximizes sensitivity and specificity.

The program may further cause the computer to execute a step of highlighting on the screen information indicating the highest sensitivity among the plurality of sensitivity information shown in the plurality of mosaic graphs.

When this program is executed by a computer, information indicating the highest sensitivity among the plurality of sensitivity information is highlighted on the screen. Therefore, according to this program, the user can easily recognize the combination of parameters corresponding to the information indicating the highest sensitivity among the plurality of pieces of sensitivity information.

A manufacturing method according to another aspect of the present invention is a method of manufacturing a trained model that estimates a subject's intention based on the subject's brain waves. The learned model is generated through machine learning using multiple pieces of training data. In machine learning, each of a plurality of parameters is adjusted in advance. Each of the plurality of training data includes information regarding brain waves, and is used for machine learning based on at least some of the plurality of parameters. The manufacturing method includes a step of generating a plurality of trained models by performing machine learning for each of a plurality of combinations of a plurality of parameters, a step of calculating sensitivity and specificity for each of the plurality of trained models, and a step of calculating sensitivity and specificity for each of the plurality of trained models. displaying the mosaic graph on a common screen. Each of the plurality of mosaic graphs shows information regarding the sensitivity in at least some of the plurality of combinations. The specificity values assumed for a plurality of mosaic graphs are different from each other.

In this manufacturing method, a plurality of mosaic graphs are displayed on a common screen. Each of the plurality of mosaic graphs shows information regarding sensitivity in at least some of the plurality of combinations, and the values regarding specificity assumed for the plurality of mosaic graphs are different from each other. Therefore, according to this manufacturing method, the sensitivity and specificity of at least some of the plurality of combinations are visually represented by the plurality of mosaic graphs, so that the user can easily see the good combinations of the plurality of parameters on the screen. It can be recognized. Users (physicians, etc.) can refer to multiple mosaic graphs to determine the parameters to be used and proceed with the evaluation. Easy to generate.

A manufacturing device according to another aspect of the present invention is a learned model manufacturing device that estimates a subject's intention based on the subject's brain waves. The learned model is generated through machine learning using multiple pieces of training data. In machine learning, each of a plurality of parameters is adjusted in advance. Each of the plurality of training data includes information regarding brain waves, and is used for machine learning based on at least some of the plurality of parameters. The manufacturing device includes a processor and a memory that stores a program executed by the processor. The program includes a step of generating a plurality of trained models by performing machine learning on each of the plurality of combinations, a step of calculating sensitivity and specificity for each of the plurality of trained models, and a step of generating a plurality of mosaic graphs. and causing the processor to perform steps that are displayed on a common screen. Each of the plurality of mosaic graphs shows information regarding the sensitivity in at least some of the plurality of combinations. The specificity values assumed for a plurality of mosaic graphs are different from each other.

In this manufacturing apparatus, a plurality of mosaic graphs are displayed on a common screen. Each of the plurality of mosaic graphs shows information regarding sensitivity in at least some of the plurality of combinations, and the values regarding specificity assumed for the plurality of mosaic graphs are different from each other. Therefore, according to this manufacturing device, the sensitivity and specificity of at least some of the plurality of combinations are visually represented by a plurality of mosaic graphs, so that the user can easily see the good combinations of the plurality of parameters on the screen. It can be recognized. Users (physicians, etc.) can refer to multiple mosaic graphs to determine the parameters to be used and proceed with the evaluation. Easy to generate.

A program according to another aspect of the present invention is a program for displaying an electroencephalogram spectrogram generated based on electroencephalograms acquired through a group of electrodes implanted in the skull of a subject. The electrode group includes electrodes arranged in the first region and electrodes arranged in the second region. The brain waves acquired by the electrodes placed in the first area have a higher correlation with the intention of the detection target among the subject's intentions than the brain waves acquired by the electrodes placed in the second area. Each of the plurality of electrodes included in the electrode group is electrically connected to one of the plurality of amplification elements included in the amplifier chip. Among the plurality of amplification elements, an amplification element that does not meet a predetermined criterion is electrically connected to an electrode arranged in the second region. The program causes the computer to execute the steps of generating a plurality of electroencephalogram spectrograms based on the outputs of each of the plurality of amplification elements, and displaying a screen including the plurality of generated electroencephalogram spectrograms. On the screen, each of the plurality of brain wave spectrograms is placed at a position equivalent to the position of the corresponding electrode in the electrode group.

When this program is executed by a computer, a screen is displayed in which each of the plurality of electroencephalogram spectrograms is placed at a position equivalent to the position of the corresponding electrode in the electrode group. Therefore, according to this program, each electroencephalogram spectrogram is displayed at the same position as the corresponding electrode, so the user can intuitively see an electroencephalogram spectrogram that reflects the subject's unique brain activity area and brain wave intensity by frequency band. This makes it easier to understand the subject's unique brain activity, so it can be expected that a trained model will be produced that can estimate the subject's intentions with relatively high accuracy.

A connection method according to another aspect of the present invention is a method of connecting an electrode group and an amplifier chip for detecting brain waves of a subject. The amplifier chip includes multiple amplification elements. The electrode group includes electrodes arranged in the first region and electrodes arranged in the second region. The brain waves acquired by the electrodes placed in the first area have a higher correlation with the intention of the detection target among the subject's intentions than the brain waves acquired by the electrodes placed in the second area. The connection method includes the steps of determining whether or not the plurality of amplification elements includes an amplification element that does not meet the predetermined criteria; and connecting an amplification element that does not meet a predetermined criterion to an electrode located in the second region.

In this connection method, an amplifying element that does not meet a predetermined criterion is connected to an electrode that acquires brain waves that do not have a high correlation with the subject's intention to be detected. Therefore, according to this connection method, it is possible to connect electrodes placed on brain regions important for intention detection to an amplification element that meets predetermined standards, so that brain waves with a high S/N ratio can be obtained. can do. As a result, it can be expected that a trained model that can estimate the subject's intentions with relatively high accuracy will be produced.

According to the present invention, it is possible to provide a program, a manufacturing method, a manufacturing device, and a connection method that generate a learned model that can estimate a subject's intention with relatively high accuracy based on the subject's brain waves.

FIG. 1 is a diagram schematically showing the configuration of a communication system. FIG. 2 is a diagram schematically showing the electrical configuration of an intracorporeal device. 1 is a diagram schematically showing the configuration of an electroencephalogram decoding device. It is a flowchart showing the operation procedure in the electroencephalogram decoding device. FIG. 1 is a diagram schematically showing the configuration of a learning system. FIG. 1 is a diagram schematically showing the configuration of a trained model generation device. It is a flowchart which shows the collection procedure of the data for learning. FIG. 3 is a diagram for explaining division of learning data in 5-fold cross validation. FIG. 2 is a diagram for explaining a procedure for determining an appropriate combination of multiple parameters through 5-fold cross validation. 3 is a flowchart showing a procedure for generating a trained model. FIG. 3 is a diagram schematically showing an example of an image showing an optimal combination of parameters. FIG. 3 is a diagram showing an example of an image in which numerical information indicating sensitivity is displayed. 13 is an enlarged view of mosaic graph G2 in FIG. 12. FIG. FIG. 3 is a diagram for explaining connection rules between each electrode included in an electrode group and an amplification element. FIG. 3 is a diagram for explaining the priority order of electrodes. 2 is a flowchart showing a procedure for displaying a spectrogram of brain waves detected by each electrode. It is a figure which shows typically an example of the screen displayed on a display part.

Embodiments according to one aspect of the present invention will be described in detail below with reference to the drawings. In addition, the same reference numerals are attached to the same or corresponding parts in the drawings, and the description thereof will not be repeated. Further, each drawing is schematically drawn with objects omitted or exaggerated as appropriate for ease of understanding.

[1. Communication system]
<1-1. Configuration>
FIG. 1 is a diagram schematically showing the configuration of an intention communication system 10. As shown in FIG. The subject who uses the communication system 10 is, for example, a patient with amyotrophic lateral sclerosis (ALS). The subject's brain emits brain waves according to the subject's intentions. In the intention communication system 10, the subject's brain activity is detected based on the subject's brain waves, and the operated device 400 is operated based on the subject's brain activity. Brain activity is detected, for example, based on brain waves emitted from each part of the subject's brain.

In the intention communication system 10, for example, when a predetermined brain activity of the subject is detected, a predetermined input signal (for example, a signal for pressing a "YES" button) is transmitted to the operated device 400. Examples of predetermined brain activity include brain activity when the subject imagines holding hands, brain activity when the subject imagines bending the arm, and brain activity when the subject imagines raising the arm. One example is brain activity. A subject using the intention communication system 10 can operate the operated device 400, for example, by imagining an intention corresponding to a predetermined brain activity.

As shown in FIG. 1, the intention communication system 10 includes an in-body device 100, a receiving device 200, an electroencephalogram decoding device 300, and an operated device 400. The intracorporeal device 100 is implanted within the skull of a subject and detects brain wave signals of the subject. In-vivo device 100 transmits the detected brain wave signal to receiving device 200 . Receiving device 200 performs frequency analysis of the received brain wave signal, and transmits the received brain wave signal and the result of the frequency analysis (hereinafter also referred to as “brain wave signal etc.”) to brain wave decoding device 300.

The brain wave decoding device 300 determines whether or not the subject has imagined a predetermined intention based on the brain wave signal received from the receiving device 200. When it is determined that a predetermined intention has been imagined, the brain wave decoding device 300 transmits a predetermined input signal to the operated device 400. As a result, the operated device 400 is operated.

FIG. 2 is a diagram schematically showing the electrical configuration of the intracorporeal device 100. As mentioned above, intracorporeal device 100 is implanted within the subject's skull. As shown in FIG. 2, the intracorporeal device 100 includes an electrode group 110, an amplifier chip 120, a control section 130, and a communication section 140.

Electrode group 110 includes a plurality of electrodes 111. The electrode group 110 is formed, for example, on one sheet that is substantially rectangular in plan view. The sheet on which the electrode group 110 is formed is placed on the brain surface (subdural) of the subject, and each electrode 111 detects the subject's brain waves.

Amplifier chip 120 includes a plurality of amplification elements 121. Amplifier chip 120 includes, for example, multiple layers. A plurality of amplification elements 121 are formed in each layer. For example, when the amplifier chip 120 has a two-layer configuration and includes 64 amplifying elements 121, 32 amplifying elements 121 may be formed in each layer.

Each of the plurality of amplification elements 121 is associated with the electrode 111. Each amplifying element 121 is electrically connected to the associated electrode 111 through, for example, a platinum wire. Note that a feedthrough may be provided between the electrode group 110 and the amplifier chip 120. The method of connecting the amplifying element 121 and the electrode 111 will be explained in detail later. Each amplification element 121 amplifies the electroencephalogram signal detected by the associated electrode 111. The brain wave signals amplified by each amplification element 121 are output to the control section 130.

The control unit (processor) 130 includes, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The control unit 130 is configured to control each component within the intracorporeal device 100 according to information processing. For example, the control unit 130 controls the communication unit 140 to transmit each brain wave signal output by the amplifier chip 120 to the receiving device 200. Note that each brain wave signal may be directly transmitted from the in-body device 100 to the brain wave decoding device 300 without going through the receiving device 200.

FIG. 3 is a diagram schematically showing the configuration of the electroencephalogram decoding device 300. The brain wave decoding device 300 is realized by, for example, a general-purpose computer. As shown in FIG. 3, the electroencephalogram decoding device 300 includes a control section 310, a communication section 330, an input section 340, a display section 350, and a storage section 320, and each component is electrically connected via a bus. It is connected.

The control unit 310 includes a CPU 312, a RAM 314, a ROM 316, and the like. The control unit 310 is configured to control each component within the electroencephalogram decoding device 300 according to information processing.

The communication unit 330 is configured to communicate with each of the receiving device 200 and the operated device 400 by wired communication or wireless communication. The communication unit 330 is configured to, for example, receive an electroencephalogram signal from the receiving device 200 and transmit an input signal to the operated device 400.

The input unit 340 is configured to receive instructions from a user (operator such as a doctor). The input unit 340 includes, for example, a touch panel, a keyboard, a mouse, and a portion or all of a microphone. Display unit 350 is configured to display images. The display unit 350 is configured with a monitor such as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor. The display unit 350 displays, for example, a spectrogram of the subject's brain waves (hereinafter also referred to as “brain wave spectrogram”) received through the communication unit 330.

The storage unit (memory) 320 is composed of, for example, an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 320 is configured to store, for example, a control program 321 and a learned model 322.

By executing the control program 321 by the control unit 310, various operations in the electroencephalogram decoding device 300 are realized. When the control unit 310 executes the control program 321, the control program 321 is loaded into the RAM 314. The control unit 310 controls each component by having the CPU 312 interpret and execute the control program 321 loaded in the RAM 314.

The trained model 322 is configured to output the likelihood that the subject has imagined a predetermined intention, for example, when brain wave signals etc. continuously received through the communication unit 330 are input. The control unit 310 determines, for example, based on the output of the learned model 322, whether the subject has imagined a predetermined intention. The method for generating the trained model 322 will be explained in detail later.

<1-2. Operation＞
FIG. 4 is a flowchart showing the operation procedure in the electroencephalogram decoding device 300. The processing shown in this flowchart is repeatedly executed by the control unit 310 when electroencephalogram signals and the like are continuously received through the communication unit 330.

Referring to FIG. 4, control unit 310 inputs electroencephalogram signals and the like received through communication unit 330 to learned model 322 (step S100). The control unit 310 determines whether the subject has imagined a predetermined intention based on the output of the learned model 322 (step S110). For example, if the likelihood that the subject has imagined a predetermined plan is greater than or equal to a predetermined value, the control unit 310 determines that the test subject has imagined a predetermined plan.

If it is determined that the subject is not imagining the predetermined intention (NO in step S110), control unit 310 executes the process of step S100 again. On the other hand, if it is determined that the subject imagined a predetermined intention (YES in step S110), the control unit 310 causes the communication unit 330 to send an input signal associated with the predetermined intention to the operated device 400. (step S120). Thereby, the operated device 400 is operated according to the subject's intention.

[2. Learning system】
<2-1. Configuration>
FIG. 5 is a diagram schematically showing the configuration of the learning system 20. The learning system 20 is configured, for example, to generate the trained model 322 described above.

As shown in FIG. 5, the learning system 20 includes an in-body device 100, a receiving device 200, and a learned model generating device 500. The intracorporeal device 100 and the receiving device 200 have, for example, the same configuration as the intracorporeal device 100 and the receiving device 200 included in the intention communication system 10 (FIG. 1), respectively. The learned model generation device 500 generates the learned model 322 through machine learning using received brain wave signals and the like. As machine learning, various known methods such as support vector machine, deep learning, neural network, decision tree learning, association rule learning, and Bayesian network can be applied.

FIG. 6 is a diagram schematically showing the configuration of the trained model generation device 500. The trained model generation device 500 is realized by, for example, a general-purpose computer. As shown in FIG. 6, the learned model generation device 500 includes a control section 510, a communication section 530, an input section 540, a display section 550, and a storage section 520, and each component is connected to electricity via a bus. connected.

The control unit (processor) 510 includes a CPU 512, a RAM 514, a ROM 516, and the like. The control unit 510 is configured to control each component within the trained model generation device 500 according to information processing.

The communication unit 530 is configured to communicate with the receiving device 200 by wired communication or wireless communication. The communication unit 530 is configured to receive, for example, an electroencephalogram signal from the receiving device 200. The input unit 540 is configured to receive instructions from a user (operator such as a doctor). The input unit 540 includes, for example, a touch panel, a keyboard, a mouse, and a portion or all of a microphone. Display unit 550 is configured to display images. An example of an image displayed on the display unit 550 will be described in detail later. The display unit 550 is configured with a monitor such as a liquid crystal monitor or an organic EL monitor, for example.

The storage unit (memory) 520 is composed of, for example, an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 520 is configured to store a control program 521, for example. By executing the control program 521 by the control unit 510, various operations in the learned model generation device 500 are realized. When the control unit 510 executes the control program 521, the control program 521 is loaded into the RAM 514. The control unit 510 controls each component by having the CPU 512 interpret and execute the control program 521 loaded in the RAM 514.

<2-2. Trained model generation procedure>
In generating the learned model 322, a plurality of brain wave signals and the like are used as learning data. The learning data is generated based on brain wave signals emitted when the subject imagines a predetermined intention. Below, first, a procedure for collecting electroencephalogram signals used to generate learning data will be explained.

FIG. 7 is a flowchart showing a procedure for collecting electroencephalogram signals used to generate learning data. The processing shown in this flowchart is repeatedly executed by the control unit 510 when electroencephalogram signals and the like are continuously received through the communication unit 530. Note that during the period in which the processing shown in this flowchart is being executed, the subject is, for example, visually recognizing the display unit 550.

Referring to FIG. 7, control unit 510 controls display unit 550 to display a screen instructing the subject to visualize a predetermined intention (step S200). The control unit 510 causes the storage unit 520 to store the brain wave signals and the like received through the communication unit 530 during a predetermined period that correlates with the display timing of the screen as brain wave signals and the like for generating learning data (step S210). By repeating the process shown in this flowchart, brain wave signals and the like necessary for generating learning data are collected. Note that the learning data may be generated based on either the received electroencephalogram signal or the result of frequency analysis.

The learned model 322 is a regression model. In machine learning for generating the learned model 322, each of a plurality of parameters is adjusted in advance. Examples of multiple parameters include a parameter (Window Size) that indicates the time interval used for learning data among electroencephalogram signals having a certain time interval, the timing at which the subject is instructed to imagine a predetermined intention, and the learning data. Examples include a parameter (Window Offset) that indicates the amount of time deviation from the interval of the electroencephalogram signal used for the data used in the data, and a parameter (PLS Comps) that indicates the dimension used in partial least squares regression. For example, the learning data is used for machine learning in consideration of each of Window Size and Window Offset.

In generating the trained model 322, cross-validation is performed in advance to determine an appropriate combination of multiple parameters. Thereafter, a learned model 322 is generated by performing machine learning using all the collected learning data and the determined combination of the plurality of parameters. Here, we will explain an example where 5-fold cross validation is performed.

FIG. 8 is a diagram for explaining division of learning data in 5-fold cross validation. As shown in FIG. 8, in 5-fold cross validation, the learning data is divided into five groups (first data group to fifth data group). For example, if 600 pieces of training data are collected, the 600 pieces of training data are divided into 5 groups each containing 120 pieces of training data.

FIG. 9 is a diagram for explaining a procedure for determining an appropriate combination of multiple parameters through 5-fold cross validation. Referring to FIG. 9, in 5-fold cross validation, four data groups among the first to fifth data groups are classified as teacher data, and one data group is classified as test data. The classification patterns include patterns 1 to 5.

In pattern 1, the second to fifth data groups are classified as teacher data, and the first data group is classified as test data. In pattern 2, the first data group, third data group to fifth data group are classified as teacher data, and the second data group is classified as test data. In pattern 3, the first data group to the second data group and the fourth data group to the fifth data group are classified as teacher data, and the third data group is classified as test data. In pattern 4, the first to third data groups and the fifth data group are classified as teacher data, and the fourth data group is classified as test data. In pattern 5, the first to fourth data groups are classified as teacher data, and the fifth data group is classified as test data.

In each pattern, machine learning using training data is performed for each combination of multiple parameters, and a regression model for the number of combinations of multiple parameters is generated. That is, five regression models (regression model 1 to regression model 5) are generated for each combination of a plurality of parameters. Note that the plurality of parameters include a specificity threshold of the generated regression model. In machine learning, a regression model whose specificity is greater than or equal to a threshold is generated. In this example, four types of threshold values, 0.90, 0.95, 0.97, and 0.99, can be set as the specificity threshold value. Further, the specificity of the generated regression model may be specified by a parameter, and in that case, a regression model having the specified specificity is generated through machine learning.

For each combination of multiple parameters, the sensitivity and specificity of each of the five regression models are calculated by using the test data. Sensitivity refers to the probability that a positive result is correctly determined to be positive, and specificity refers to the probability that a negative result is correctly determined to be negative. In other words, in these regression models, sensitivity refers to the probability that the subject will judge that the subject has imagined the predetermined plan when the subject imagines the predetermined plan, and specificity refers to the probability that the subject has imagined the predetermined plan. This refers to the probability that the subject will not judge that he or she imagined the specified intention if the subject does not imagine the given intention.

The average sensitivity and average specificity in the five regression models are calculated for each combination of multiple parameters. The combination of multiple parameters with the highest average sensitivity and highest average specificity is considered the optimal parameter combination. Appropriate parameters to be used for generating the learned model 322 are determined by adjusting the parameters based on the optimal parameter combination. The parameter adjustment method will be explained in detail later.

After that, a trained model 322 is generated by performing machine learning using all the collected learning data (first data group to fifth data group) and an appropriate combination of the determined multiple parameters. be done. Note that in this machine learning, all data included in the first to fifth data groups are used as teacher data.

FIG. 10 is a flowchart showing the procedure for generating the trained model 322. The processing shown in this flowchart is executed by the control unit 510 while the learning data is collected.

Referring to FIG. 10, control unit 510 generates multiple regression models through cross-validation for each combination of multiple parameters using the collected learning data, and determines the sensitivity and specificity of each regression model. is calculated (step S300). The control unit 510 calculates the average sensitivity and average specificity of the regression model for each combination of multiple parameters (step S310). The control unit 510 regards the combination of multiple parameters with the highest average sensitivity and highest average specificity as the optimal parameter combination, and controls the display unit 550 to display an image showing the optimal parameter combination (step S320 ). Each parameter can be adjusted while an image showing the optimal combination of parameters is displayed on the display unit 550.

FIG. 11 is a diagram schematically showing an example of an image IM1 showing an optimal parameter combination. Referring to FIG. 11, image IM1 is displayed on display section 550. Image IM1 includes mosaic graphs G1, G2, G3, G4, scale adjustment section S1, areas T1, T2, and buttons B1, B3, B4, B5, B6. In other words, these can be said to be displayed on a common screen.

Each of the mosaic graphs G1, G2, G3, and G4 shows information regarding the sensitivity of at least some of the plurality of combinations of parameters. In this example, the horizontal axis of each graph is Window Offset (parameter), and the vertical axis of each graph is Window Size (parameter). Information regarding sensitivity is, for example, color information. Sensitivity and color are associated, and sensitivity is indicated by color for each combination of a plurality of parameters.

Regarding the mosaic graphs G1, G2, G3, and G4, the prerequisite specificity thresholds are different from each other. For example, the specificity threshold assumed in the mosaic graph G1 is 0.90, and the specificity threshold assumed in the mosaic graph G2 is 0.95. Further, the specificity threshold assumed in the mosaic graph G3 is 0.97, and the specificity threshold assumed in the mosaic graph G4 is 0.99.

The scale adjustment unit S1 is used to adjust the correspondence between sensitivity and color (hereinafter also referred to as "scale"). For example, the scale is adjusted by changing each of the upper and lower limit values shown in the scale adjustment section S1. When the scale adjustment section S1 makes the adjustment, each of the mosaic graphs G1, G2, G3, and G4 is updated. That is, each of the mosaic graphs G1, G2, G3, and G4 is generated based on the adjusted scale.

In area T1, an optimal combination of multiple parameters is displayed. The optimal combination of multiple parameters displayed in area T1 is the optimal combination obtained through the above-mentioned cross-validation. In area T2, currently selected combinations of a plurality of parameters are displayed. By operating buttons B5 and B6, the values of the corresponding parameters can be adjusted, and each of the mosaic graphs G1, G2, G3, and G4 is updated to one generated based on the combination of adjusted parameters. be done. For example, the parameters that can be adjusted here are the parameters that are assumed in each of the mosaic graphs G1, G2, G3, and G4.

When button B3 is operated, a trained model is generated based on the optimal parameter combination. When button B4 is operated, a trained model is generated based on the currently selected parameter combination. When the button B1 is operated, numerical information indicating the sensitivity for each combination of parameters is displayed in each of the mosaic graphs G1, G2, G3, and G4.

FIG. 12 is a diagram showing an example of an image IM2 in which numerical information indicating sensitivity is displayed. Referring to FIG. 12, image IM2 is displayed on display section 550, for example, when button B1 is operated with image IM1 displayed on display section 550. In image IM2, numerical information indicating sensitivity is displayed for each combination of parameters in each of mosaic graphs G1, G2, G3, and G4.

FIG. 13 is an enlarged view of mosaic graph G3 in FIG. 12. As shown in FIG. 13, in the mosaic graph G3, the area exhibiting the highest sensitivity is highlighted and surrounded by a frame C1. For example, the frame C1 may have a color that stands out compared to the background color. This allows the user to easily recognize the combination of parameters corresponding to the region exhibiting the highest sensitivity. Note that the frame C1 may be displayed in each of the mosaic graphs G1, G2, G3, and G4, or may be displayed only in the area that is the best parameter combination among the currently selected parameter combinations.

Referring again to FIG. 10, when the optimal parameter combination is displayed on display unit 550 in step S320, control unit 510 determines whether the user has finished adjusting the parameters (step S330). Specifically, the control unit 510 determines whether the user has selected either button B3 or B4. Even if the combination of parameters is considered to be optimal, for example, a combination of parameters where even a small change in some parameters causes a large decrease in sensitivity is not necessarily optimal. In such a case, the user adjusts the parameters.

If it is determined that the parameter adjustment is not completed (NO in step S330), the control unit 510 waits until the parameter adjustment is completed. On the other hand, if it is determined that the adjustment of the parameters has been completed (YES in step S330), the control unit 510 generates the learned model 322 through machine learning using the combination of the adjusted parameters (step S340).

In this way, in the trained model generation device 500, the mosaic graphs G1, G2, G3, and G4 are displayed on a common screen. Each mosaic graph shows information regarding the sensitivity of at least a portion of the plurality of combinations of the plurality of parameters, and the values regarding the specificity assumed for the plurality of mosaic graphs are different from each other. Therefore, according to the trained model generation device 500, the sensitivity and specificity regarding at least some combinations of the plurality of parameters are visually represented by the mosaic graphs G1, G2, G3, and G4. A user (such as a doctor) can easily recognize the combination by displaying the combination on the screen. The user (physician, etc.) can refer to the mosaic graphs G1, G2, G3, and G4 to determine the parameters to be used and proceed with the evaluation, so it is possible to estimate the subject's intentions with relatively high accuracy based on the subject's brain waves. This makes it easy to generate a trained model.

Furthermore, when the values of parameters on which the mosaic graphs G1, G2, G3, and G4 are based are changed, each of the mosaic graphs G1, G2, G3, and G4 is updated. Therefore, according to the trained model generation device 500, the user can easily recognize the influence that parameter changes have on sensitivity through screen display.

Furthermore, when the scale is changed through the scale adjustment section S1, each of the mosaic graphs G1, G2, G3, and G4 is updated. Therefore, according to the trained model generation device 500, the user can more clearly recognize the relationship between each parameter and sensitivity through scale adjustment.

Further, in each of the mosaic graphs G1, G2, G3, and G4, numerical information indicating sensitivity can be displayed. According to the trained model generation device 500, a user can easily recognize a good combination of a plurality of parameters through display of numerical information.

Furthermore, in the trained model generation device 500, the numerical values of each parameter in the optimal combination are displayed on the screen, and each of the mosaic graphs G1, G2, G3, and G4 is generated based on the optimal combination of parameters. Therefore, the trained model generation device 500 allows the user to easily recognize the combination of parameters that maximizes sensitivity and specificity.

[3. In-body device]
<3-1. Connection rules between electrode group and amplifier chip>
As described above, each electrode 111 included in the electrode group 110 is electrically connected to the amplification element 121 (FIG. 2) included in the amplifier chip 120 through a platinum wire. The quality of each amplifying element 121 included in the amplifier chip 120 is not necessarily constant. Among the plurality of amplification elements 121 included in the amplifier chip 120, some of the amplification elements 121 may not have a predetermined quality.

The brain waves detected by some of the electrodes 111 included in the electrode group 110 have a strong correlation with the subject's intention. Moreover, the brain waves detected by some of the electrodes 111 included in the electrode group 110 have almost no correlation with the subject's intention. For example, when detecting a subject's intention to move a hand, brain waves measured at an electrode located far from the sensorimotor cortex region of the subject's hand have little correlation with the subject's intention. Other examples of intentions include intentions regarding movement of the subject's face, intentions regarding movement of the subject's mouth, and intentions regarding movement of the subject's legs. The arrangement position of the electrode group 110 on the subject's brain surface is adjusted depending on the intended detection target. The arrangement position of the electrode group 110 may be determined, for example, by referring to Penfield's homunculus, depending on the intended detection target.

In this embodiment, an electrode 111 that detects brain waves that have a strong correlation with the subject's intentions is connected to an amplification element 121 that satisfies a predetermined quality, and an electrode that detects brain waves that have little correlation with the subject's intentions. 111 is connected to an amplification element 121 that does not meet a predetermined quality. That is, in the present embodiment, a portion of the electrode located on a brain region of the subject's brain that is remote from a brain region that is important for intention detection is connected to the amplification element 121 that does not meet the predetermined quality. The electrode is predetermined as the electrode. This makes it possible to detect brain waves that have a strong correlation with the subject's intentions with higher precision.

FIG. 14 is a diagram for explaining the connection rules between each electrode 111 included in the electrode group 110 and the amplification element 121. Referring to FIG. 14, each amplification element 121 included in amplifier chip 120 is electrically connected to an electrode pad 126 included in electrode pad group 125. By connecting electrode pad 126 and electrode 111, amplifying element 121 and electrode 111 are electrically connected. Note that the electrode 111 included in area A of the electrode group 110 is connected to the electrode pad 126 included in area A of the electrode pad group 125, and the electrode 111 included in area B of the electrode group 110 is connected to area B of the electrode pad group 125. It is connected to the electrode pad 126 included in the.

For example, when detecting a test subject's intention regarding hand movement, the brain waves measured by the electrode 111 placed near the sensorimotor cortex of the hand in the test subject's brain have a high correlation with the test subject's intention. On the other hand, the brain waves measured by the electrodes 111 placed in a region of the subject's brain remote from the sensorimotor cortex of the hand have almost no correlation with the subject's intentions. For example, the electroencephalogram signals obtained by each electrode 111 arranged in the first region ET1 are compared with the electroencephalogram signals obtained by each electrode 111 arranged in the second region ET2. has a high correlation with

In this embodiment, before connecting the electrode 111 and the electrode pad 126, a test is performed to determine whether each amplification element 121 satisfies a predetermined quality. In this inspection, for example, the input equivalent noise of each amplifying element 121 is examined, and it is determined that the amplifying element 121 whose input equivalent noise is equal to or greater than a predetermined value does not satisfy the predetermined quality. Furthermore, the amplification elements 121 that satisfy a predetermined quality are also ranked in descending order of input equivalent noise.

Some of the electrodes 111 located near both ends in the width direction of the electrode group 110 among the electrodes located on the brain region distant from the brain region that is important for intention detection (the electrodes 111 arranged in the second region ET2) ) are predetermined as electrodes to be connected to the amplification element 121 that does not meet a predetermined quality. In this example, the electrodes 111 associated with numerical values 1 to 8 in FIG. 14 are predetermined as electrodes to be connected to the amplifying element 121 that does not meet the predetermined quality. This value indicates the order in which amplifying elements 121 that do not meet the predetermined quality are preferentially connected.

Furthermore, in the present embodiment, the priority order of the electrodes 111 to which higher quality amplification elements 121 are connected is determined in advance. That is, the priority order of the electrodes 111 is determined in advance so that the amplification element 121 of higher quality is connected to the electrode 111 that detects brain waves having a higher correlation with the subject's intentions.

FIG. 15 is a diagram for explaining the priority order of the electrodes 111. Referring to FIG. 15, the numerical values associated with each electrode 111 in the figure indicate the priority order of each electrode 111. The priority order of each electrode 111 is determined in each of the first region and the second region of the electrode group 110. Amplifying elements 121 of higher quality are connected in descending order of the associated numerical value. This makes it possible to detect brain waves that have a strong correlation with the subject's intentions with higher precision.

Note that an electrode map showing the connection relationship between the electrode 111 and the amplification element 121 is stored in the storage unit 520 of the trained model generation device 500, for example. The electrode map includes information indicating the connection relationship between the electrode 111 and the amplifying element 121, and information indicating whether the amplifying element 121 satisfies a predetermined quality. For example, by referring to the electrode map, the control unit 510 can determine which electrode 111 the output of each amplifying element 121 is generated based on the brain waves detected by.

<3-2. Display of information regarding brain waves detected by each electrode>
FIG. 16 is a flowchart showing a procedure for displaying a spectrogram of brain waves detected by each electrode 111. The process shown in this flowchart is repeatedly executed by the control unit 510 of the trained model generation device 500.

The control unit 510 acquires the output, etc. (brain wave signal, etc.) of each amplification element 121 through the communication unit 330 (step S400). The control unit 510 generates a plurality of brain wave spectrograms based on the acquired brain wave signals and the like (step S410). The control unit 510 generates a screen including a plurality of electroencephalogram spectrograms with reference to the electrode map stored in the storage unit 520 (step S420). In this screen, each of the plurality of electroencephalogram spectrograms is arranged at a position equivalent to the position of the corresponding electrode 111 in the electrode group 110. The control unit 510 controls the display unit 350 to display the generated screen (step S430).

FIG. 17 is a diagram schematically showing an example of a screen displayed on the display section 350. Referring to FIG. 17, on this screen, each of the plurality of brain wave spectrograms is arranged at the same position as the corresponding electrode 111 in the electrode group 110. For example, an electroencephalogram spectrogram 111A is displayed for an electrode 111 connected to an amplification element 121 that satisfies a predetermined quality, and a blacked-out image 111B is displayed for an electrode 111 connected to an amplification element 121 that does not meet a predetermined quality. Good too.

In this way, in the trained model generation device 500, a screen is displayed in which each of the plurality of brain wave spectrograms is arranged at the same position as the corresponding electrode 111 in the electrode group 110. Therefore, according to the trained model generation device 500, each electroencephalogram spectrogram is displayed at the same position as the corresponding electrode 111, so that the electroencephalogram spectrogram reflects the brain activity region specific to the subject and the intensity of brain waves for each frequency band. This allows the user to intuitively recognize the subject's specific brain activity, making it easier to understand the subject's specific brain activity, so we can expect to produce a trained model that can estimate the subject's intentions with relatively high accuracy. .

Furthermore, the subject's intention has a strong correlation with the brain waves detected by the electrode 111 (electrode 111 placed in the first region ET1) located on the brain region important for intention detection. In the present embodiment, an amplifying element 121 that does not meet a predetermined criterion is located on an electrode 111 (second region ET2) located on a brain region that is remote from a brain region that is important for intentional detection among the plurality of electrodes 111. The electrode 111) is connected to the disposed electrode 111). Therefore, according to the present embodiment, it is possible to connect the electrode 111 located on the brain region important for intention detection to the amplification element 121 that satisfies the predetermined criteria, and as a result, the subject's intention can be compared. It can be expected that a trained model that can be estimated with high accuracy will be produced.

[4. Features]
As described above, in the trained model generation device 500, the mosaic graphs G1, G2, G3, and G4 are displayed on a common screen. Each mosaic graph shows information regarding the sensitivity of at least a portion of the plurality of combinations of the plurality of parameters, and the values regarding the specificity assumed for the plurality of mosaic graphs are different from each other. Therefore, according to the trained model generation device 500, the sensitivity and specificity of at least some of the combinations of the parameters are visually represented by the mosaic graphs G1, G2, G3, and G4. A user (such as a doctor) can easily recognize a good combination by displaying it on the screen. Users (physicians, etc.) can refer to multiple mosaic graphs to determine the parameters to be used and proceed with the evaluation. Easy to generate.

Further, in the trained model generation device 500, a screen is displayed in which each of the plurality of brain wave spectrograms is arranged at a position equivalent to the position of the corresponding electrode 111 in the electrode group 110. Therefore, according to the trained model generation device 500, each electroencephalogram spectrogram is displayed at the same position as the corresponding electrode 111, so that the electroencephalogram spectrogram reflects the brain activity region specific to the subject and the intensity of brain waves for each frequency band. This allows the user to intuitively recognize the subject's specific brain activity, making it easier to understand the subject's specific brain activity, so we can expect to produce a trained model that can estimate the subject's intentions with relatively high accuracy. .

In addition, in this embodiment, the amplification element 121 that does not meet the predetermined criteria is located on the electrode 111 (second region It is connected to the electrode 111) located in ET2. Therefore, according to the present embodiment, the electrode 111 located on the brain region important for intention detection (the electrode 111 arranged in the first region ET1) can be connected to the amplification element 121 that satisfies the predetermined criteria. As a result, it can be expected that a trained model that can estimate the subject's intentions with relatively high accuracy will be produced.

[5. Other embodiments]
The idea of the above embodiments is not limited to the embodiments described above. Hereinafter, an example of another embodiment to which the idea of the above embodiment can be applied will be described.

In the embodiment described above, the electroencephalogram decoding device 300 and the trained model generation device 500 are realized by different devices. However, the electroencephalogram decoding device 300 and the trained model generation device 500 do not necessarily need to be implemented as different devices. The electroencephalogram decoding device 300 and the trained model generation device 500 may be realized by one common device.

Furthermore, in the above embodiment, the trained model 322 is configured to output the likelihood that the subject imagined a predetermined intention when the amplified electroencephalogram signal is input. Here, the predetermined plan does not necessarily have to be one type of plan. For example, multiple types of plans may be determined in advance as the predetermined plan. In this case, the trained model 322 may be a classifier model that outputs the likelihood that the subject imagined each intention when the amplified electroencephalogram signal is input. Even in such a case, it is sufficient if a plurality of mosaic graphs are displayed on a common screen.

The embodiments of the present invention have been exemplarily described above. That is, the detailed description and accompanying drawings have been disclosed for purposes of illustration. Therefore, some of the components described in the detailed description and the attached drawings may not be essential for solving the problem. Therefore, just because non-essential components are described in the detailed description and accompanying drawings, such non-essential components should not be immediately identified as essential.

Furthermore, the above embodiments are merely illustrative of the present invention in all respects. Various improvements and changes can be made to the above embodiments within the scope of the present invention. That is, in implementing the present invention, specific configurations can be adopted as appropriate depending on the embodiment.

10 intention communication system, 20 learning system, 100 intracorporeal device, 110 electrode group, 111 electrode, 120 amplifier chip, 121 amplification element, 125 electrode pad group, 126 electrode pad, 130, 310, 510 control unit, 140, 330, 530 Communication department, 200 Receiving device, 300 Brain wave decoding device, 312,512 CPU, 314,514 RAM, 316,516 ROM, 320,520 Storage section, 321,521 Control program, 322 Learned model, 340,540 Input section, 350,550 display unit, 400 operated device, 500 trained model generation device, B1, B3, B4, B5, B6 button, C1 frame, ET1 first area, ET2 second area, G1, G2, G3, G4 mosaic Graph, IM1, IM2 image, S1 scale adjustment section, T1, T2 area.

Claims (1)

A program for displaying an electroencephalogram spectrogram generated based on electroencephalograms acquired through a group of electrodes implanted in the skull of a subject,
The electrode group includes an electrode arranged in a first region and an electrode arranged in a second region,
The brain waves acquired by the electrodes placed in the first region have a higher correlation with the intentions of the subject to be detected than the brain waves acquired by the electrodes placed in the second area. death,
Each of the plurality of electrodes included in the electrode group is electrically connected to one of the plurality of amplification elements included in the amplifier chip,
Among the plurality of amplification elements, an amplification element that does not meet a predetermined criterion is electrically connected to an electrode arranged in the second region,
The program is executed by a computer,
The computer includes a memory;
the memory stores an electrode map;
The electrode map indicates a connection relationship between the plurality of electrodes and the plurality of amplification elements,
The program is
generating a plurality of brain wave spectrograms based on the output of each of the plurality of amplification elements;
generating a screen including the plurality of generated electroencephalogram spectrograms by referring to the electrode map;
causing the computer to execute the step of displaying the generated screen ;
In the screen, each of the plurality of electroencephalogram spectrograms is arranged at a position equivalent to a position of a corresponding electrode in the electrode group.