JP2024056674A - PROGRAM, MANUFACTURING METHOD, MANUFACTURING APPARATUS, AND CONNECTION METHOD - Google Patents

Abstract

[Problem] To provide a program, manufacturing method, manufacturing device, and connection method for generating a trained model capable of estimating a subject's intention with a relatively high degree of accuracy based on the subject's electroencephalogram. [Solution] Each of a plurality of training data used to generate a trained model that estimates the subject's intention includes information related to electroencephalograms, and is used for machine learning taking into account at least a portion of a plurality of parameters. The program causes a computer to execute the steps of generating a plurality of trained models by performing machine learning for each of a plurality of combinations of a plurality of parameters, calculating sensitivity and specificity for each of the plurality of trained models, and displaying a plurality of mosaic graphs on a common screen. Each of the plurality of mosaic graphs shows information related to sensitivity for at least a portion of the plurality of combinations. The specificity assumed for the plurality of mosaic graphs is different from one another. [Selected Figure] Figure 1

Description

The present invention relates to a program, a manufacturing method, a manufacturing device, and a connection method.

JP 2018-68886 A (Patent Document 1) discloses a BMI (brain-machine interface) system. This BMI system includes an internal device and an external device. The internal device has a group of electrodes and is embedded in the subject's head. The internal device is configured to accurately acquire the subject's electroencephalogram signal using a group of electrodes placed on the brain surface (subdural area). The electroencephalogram signal acquired by the internal device is transmitted to the external device. The external device performs an operation according to the received electroencephalogram signal (see Patent Document 1).

JP 2018-68886 A

The object of the present invention is to provide a program, a manufacturing method, a manufacturing device, and a connection method for generating a trained model that can estimate a subject's motor intention (hereinafter, simply referred to as "intention") with a relatively high degree of accuracy based on the subject's electroencephalogram.

A program according to an aspect of the present invention is used to generate a trained model that estimates a subject's intention based on the subject's electroencephalogram. The trained model is generated through machine learning using a plurality of training data. In the machine learning, each of the plurality of parameters is adjusted in advance. Each of the plurality of training data includes information on the electroencephalogram and is used for machine learning based on at least a portion of the plurality of parameters. The program causes a computer to execute the steps of generating a plurality of trained models by performing machine learning on each of a plurality of combinations of a plurality of parameters, calculating sensitivity and specificity for each of the plurality of trained models, and displaying a plurality of mosaic graphs on a common screen. Each of the plurality of mosaic graphs shows information on sensitivity for at least a portion of the plurality of combinations. The specificity values assumed for the plurality of mosaic graphs are different from one another.

When this program is executed by a computer, multiple mosaic graphs are displayed on a common screen. Each of the multiple mosaic graphs shows information regarding the sensitivity of at least some of the multiple combinations, and the values regarding the specificity that are the premise for the multiple mosaic graphs are different from each other. Therefore, according to this program, the sensitivity and specificity of at least some of the multiple combinations are visually represented by multiple mosaic graphs, so that a user (doctor, etc.) can easily recognize a good combination of multiple parameters by the screen display. Since a user (doctor, etc.) can refer to the multiple mosaic graphs to determine the parameters to use and proceed with the evaluation, it becomes easy to generate a trained model that can estimate the subject's intention with a relatively high accuracy based on the subject's brain waves.

In each of the plurality of mosaic graphs, at least a portion of the plurality of parameters may be determined as premise values, and the program may further cause the computer to execute a step of accepting a change in the premise value, and a step of updating each of the plurality of mosaic graphs when the premise value is changed.

When this program is executed by a computer, each of the multiple mosaic graphs is updated when the premise value is changed. Therefore, this program allows the user to easily recognize the effect of parameter changes on sensitivity through the screen display.

The information relating to sensitivity may include color information, and the program may further cause the computer to execute a step of accepting a change in a scale indicating the correspondence between sensitivity values and colors, and a step of updating each of the multiple mosaic graphs when the scale is changed.

When this program is executed by a computer, each of the multiple mosaic graphs is updated when the scale is changed. Therefore, this program allows the user to more clearly recognize the relationship between each parameter and sensitivity through adjusting the scale.

In the above program, the information regarding sensitivity may include numerical information.

This program allows users to easily recognize good combinations of multiple parameters through the display of numerical information.

The above program may further cause the computer to execute a step of deriving an optimal combination of multiple parameters based on both sensitivity and specificity, and a step of displaying on a screen the numerical values of each parameter in the optimal combination, and each of the multiple mosaic graphs may be generated based on the optimal combination.

When this program is executed by a computer, the numerical values of each parameter in the optimal combination are displayed on the screen, and multiple mosaic graphs are generated based on the optimal combination. Therefore, this program allows the user to easily recognize the parameter combination that maximizes sensitivity and specificity.

The program may further cause the computer to execute a step of highlighting on the screen the information showing the highest sensitivity among the information on the multiple sensitivities shown in the multiple mosaic graphs.

When this program is executed by a computer, the information indicating the highest sensitivity among the multiple pieces of information regarding sensitivities is highlighted on the screen. Therefore, this program allows the user to easily recognize the combination of parameters that corresponds to the information indicating the highest sensitivity among the multiple pieces of information regarding sensitivities.

A manufacturing method according to another aspect of the present invention is a manufacturing method for a trained model that estimates a subject's intention based on the subject's electroencephalogram. The trained model is generated through machine learning using a plurality of training data. In the machine learning, each of the plurality of parameters is adjusted in advance. Each of the plurality of training data includes information on the electroencephalogram and is used for the machine learning based on at least a portion of the plurality of parameters. The manufacturing method includes a step of generating a plurality of trained models by performing machine learning for each of a plurality of combinations of a plurality of parameters, a step of calculating sensitivity and specificity for each of the plurality of trained models, and a step of displaying a plurality of mosaic graphs on a common screen. Each of the plurality of mosaic graphs shows information on sensitivity for at least a portion of the plurality of combinations. The values related to specificity that are assumed for the plurality of mosaic graphs are different from each other.

In this manufacturing method, multiple mosaic graphs are displayed on a common screen. Each of the multiple mosaic graphs shows information about the sensitivity in at least some of the multiple combinations, and the values related to the specificity that are the premise for the multiple mosaic graphs are different from each other. Therefore, according to this manufacturing method, the sensitivity and specificity in at least some of the multiple combinations are visually represented by the multiple mosaic graphs, so that the user can easily recognize a good combination of multiple parameters by the screen display. Since the user (doctor, etc.) can refer to the multiple mosaic graphs to determine the parameters to be used and proceed with the evaluation, it becomes easy to generate a trained model that can estimate the subject's intention with a relatively high accuracy based on the subject's brain waves.

A manufacturing apparatus according to another aspect of the present invention is an apparatus for manufacturing a trained model that estimates a subject's intention based on the subject's electroencephalogram. The trained model is generated through machine learning using a plurality of training data. In the machine learning, each of a plurality of parameters is adjusted in advance. Each of the plurality of training data includes information on electroencephalograms and is used for machine learning based on at least a portion of the plurality of parameters. The manufacturing apparatus includes a processor and a memory that stores a program executed by the processor. The program causes the processor to execute the steps of generating a plurality of trained models by performing machine learning on each of the plurality of combinations, calculating sensitivity and specificity for each of the plurality of trained models, and displaying a plurality of mosaic graphs on a common screen. Each of the plurality of mosaic graphs shows information on sensitivity for at least a portion of the plurality of combinations. The values related to specificity that are assumed for the plurality of mosaic graphs are different from each other.

In this manufacturing device, multiple mosaic graphs are displayed on a common screen. Each of the multiple mosaic graphs shows information about the sensitivity of at least some of the multiple combinations, and the specificity values that are the premise for the multiple mosaic graphs are different from one another. Therefore, according to this manufacturing device, the sensitivity and specificity of at least some of the multiple combinations are visually represented by multiple mosaic graphs, so that the user can easily recognize good combinations of multiple parameters through the screen display. Since the user (doctor, etc.) can refer to the multiple mosaic graphs to determine the parameters to be used and proceed with the evaluation, it becomes easy to generate a trained model that can estimate the subject's intentions with a relatively high degree of accuracy based on the subject's brain waves.

A program according to another aspect of the present invention is a program for displaying an electroencephalogram spectrogram generated based on electroencephalograms acquired through an electrode group implanted in the skull of a subject. The electrode group includes an electrode arranged in a first region and an electrode arranged in a second region. The electroencephalogram acquired by the electrode arranged in the first region has a high correlation with the intention of the subject to be detected, compared with the electroencephalogram acquired by the electrode arranged in the second region. Each of the multiple electrodes included in the electrode group is electrically connected to one of the multiple amplification elements included in the amplifier chip. An amplification element among the multiple amplification elements that does not satisfy a predetermined criterion is electrically connected to an electrode arranged in the second region. The program causes a computer to execute a step of generating multiple electroencephalograms based on the output of each of the multiple amplification elements, and a step of displaying a screen including the generated multiple electroencephalograms. On the screen, each of the multiple electroencephalograms is arranged at a position equivalent to the position of the corresponding electrode in the electrode group.

When this program is executed by a computer, a screen is displayed in which each of the multiple EEG spectrograms is positioned at a position equivalent to the position of the corresponding electrode in the electrode group. Therefore, according to this program, since each EEG spectrogram is displayed at a position equivalent to the corresponding electrode, the user can intuitively recognize the EEG spectrogram that reflects the subject-specific brain activity areas and the EEG intensity by frequency band, making it easier to grasp the subject-specific brain activity, and it is expected that a trained model that can estimate the subject's intentions with relatively high accuracy will be produced.

A connection method according to another aspect of the present invention is a method for connecting an electrode group for detecting brain waves of a subject to an amplifier chip. The amplifier chip includes a plurality of amplification elements. The electrode group includes an electrode arranged in a first region and an electrode arranged in a second region. The brain waves acquired by the electrodes arranged in the first region have a high correlation with the intention of the subject to be detected, compared with the brain waves acquired by the electrodes arranged in the second region. The connection method includes a step of determining whether the plurality of amplification elements includes an amplification element that does not satisfy a predetermined criterion, and a step of connecting the amplification element that does not satisfy the predetermined criterion to the electrode arranged in the second region when the plurality of amplification elements includes an amplification element that does not satisfy the predetermined criterion.

In this connection method, an amplifier element that does not meet a predetermined criterion is connected to an electrode that acquires electroencephalograms that do not have a high correlation with the subject's intention to be detected. Therefore, according to this connection method, electrodes placed on brain regions that are important for intention detection can be connected to amplifier elements that meet the predetermined criterion, making it possible to acquire electroencephalograms with a high S/N ratio. As a result, it is expected that a trained model that can estimate the subject's intention with a relatively high degree of accuracy can be produced.

The present invention provides a program, manufacturing method, manufacturing device, and connection method for generating a trained model that can estimate a subject's intention with a relatively high degree of accuracy based on the subject's brain waves.

FIG. 1 is a diagram illustrating a configuration of a communication system. FIG. 2 is a diagram showing a schematic diagram of the electrical configuration of the intracorporeal device. FIG. 1 is a diagram illustrating a schematic configuration of an electroencephalogram decoding device. 13 is a flowchart showing an operation procedure of the electroencephalogram decoding device. FIG. 1 is a diagram illustrating a configuration of a learning system. FIG. 1 is a diagram illustrating a configuration of a trained model generation device. 13 is a flowchart showing a procedure for collecting learning data. FIG. 13 is a diagram for explaining division of learning data in 5-fold cross validation. FIG. 1 is a diagram for explaining a procedure for determining an appropriate combination of multiple parameters through 5-fold cross validation. 1 is a flowchart showing the procedure for generating a trained model. FIG. 10 is a diagram illustrating an example of an image showing an optimal parameter combination. FIG. 13 is a diagram showing an example of an image in which numerical information indicating sensitivity is displayed. FIG. 13 is an enlarged view of the mosaic graph G2 in FIG. 12. 4 is a diagram for explaining the connection rules between each electrode included in the electrode group and an amplifying element. FIG. FIG. 13 is a diagram for explaining the priority order of electrodes. 10 is a flowchart showing a procedure for displaying a spectrogram of an electroencephalogram detected by each electrode. FIG. 4 is a diagram illustrating an example of a screen displayed on a display unit.

Below, an embodiment of one aspect of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the drawings are given the same reference numerals and their description will not be repeated. Also, each drawing is drawn in a schematic manner with objects appropriately omitted or exaggerated for ease of understanding.

[1. Communication System]
<1-1. Configuration>
FIG. 1 is a diagram showing a schematic configuration of a communication system 10. A subject using the communication system 10 is, for example, a patient with amyotrophic lateral sclerosis (ALS). Electroencephalograms corresponding to the subject's intentions are emitted from the subject's brain. In the communication system 10, the subject's brain activity is detected based on the subject's brain waves, and the operated device 400 is operated based on the subject's brain activity. The brain activity is detected based on, for example, brain waves emitted from each part of the subject's brain.

In the intention communication system 10, for example, when a predetermined brain activity of the subject is detected, a predetermined input signal (e.g., a signal to press a "YES" button) is sent to the operated device 400. Examples of the predetermined brain activity include brain activity when the subject imagines clenching his/her hand, brain activity when the subject imagines bending his/her arm, and brain activity when the subject imagines raising his/her arm. A subject using the intention communication system 10 can operate the operated device 400, for example, by imagining an intention that corresponds to the predetermined brain activity.

As shown in FIG. 1, the communication system 10 includes an internal device 100, a receiving device 200, an EEG decoding device 300, and an operated device 400. The internal device 100 is embedded in the subject's skull and detects the subject's EEG signal. The internal device 100 transmits the detected EEG signal to the receiving device 200. The receiving device 200 performs frequency analysis of the received EEG signal, and transmits the received EEG signal and the results of the frequency analysis (hereinafter also referred to as "EEG signal, etc.") to the EEG decoding device 300.

The electroencephalogram decoding device 300 determines whether or not a specific intention has been imagined by the subject based on the electroencephalogram signal received from the receiving device 200. If it is determined that a specific intention has been imagined, the electroencephalogram decoding device 300 transmits a specific input signal to the operated device 400. This causes the operated device 400 to be operated.

Figure 2 is a diagram showing a schematic diagram of the electrical configuration of the intracorporeal device 100. As described above, the intracorporeal device 100 is implanted in the subject's skull. As shown in Figure 2, the intracorporeal device 100 includes an electrode group 110, an amplifier chip 120, a control unit 130, and a communication unit 140.

The electrode group 110 includes a plurality of electrodes 111. The electrode group 110 is formed, for example, on a sheet that is generally rectangular in plan view. The sheet on which the electrode group 110 is formed is placed on the brain surface (subdural area) of the subject, and each electrode 111 detects the subject's brain waves.

The amplifier chip 120 includes a plurality of amplifying elements 121. The amplifier chip 120 includes, for example, a plurality of layers. A plurality of amplifying elements 121 are formed in each layer. For example, when the amplifier chip 120 has a two-layer structure and includes 64 amplifying elements 121, 32 amplifying elements 121 may be formed in each layer.

Each of the multiple amplification elements 121 is associated with an electrode 111. Each amplification element 121 is electrically connected to the associated electrode 111, for example, via a platinum wire. A feedthrough may be provided between the electrode group 110 and the amplifier chip 120. The method of connecting the amplification elements 121 to the electrodes 111 will be described in detail later. Each amplification element 121 amplifies the brainwave signal detected by the associated electrode 111. The brainwave signal amplified by each amplification element 121 is output to the control unit 130.

The control unit (processor) 130 includes, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The control unit 130 is configured to control each component in the intracorporeal device 100 in accordance with information processing. The control unit 130 controls the communication unit 140 to transmit each brainwave signal output by the amplifier chip 120 to the receiving device 200, for example. Note that each brainwave signal may be transmitted directly from the intracorporeal device 100 to the brainwave decoding device 300 without passing through the receiving device 200.

Figure 3 is a diagram showing a schematic configuration of the brainwave decoding device 300. The brainwave decoding device 300 is realized, for example, by a general-purpose computer. As shown in Figure 3, the brainwave decoding device 300 includes a control unit 310, a communication unit 330, an input unit 340, a display unit 350, and a memory unit 320, and each component is electrically connected via a bus.

The control unit 310 includes a CPU 312, a RAM 314, and a ROM 316. The control unit 310 is configured to control each component within the brainwave decoding device 300 in accordance with information processing.

The communication unit 330 is configured to communicate with each of the receiving device 200 and the operated device 400 by wired communication or wireless communication. The communication unit 330 is configured to receive, for example, an electroencephalogram signal from the receiving device 200 and to transmit an input signal to the operated device 400.

The input unit 340 is configured to receive instructions from a user (an operator such as a doctor). The input unit 340 is configured to include, for example, some or all of a touch panel, a keyboard, a mouse, and a microphone. The display unit 350 is configured to display images. The display unit 350 is configured to include, for example, a monitor such as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor. The display unit 350 displays, for example, a spectrogram of the subject's brain waves (hereinafter also referred to as an "brain wave spectrogram") received via the communication unit 330.

The storage unit (memory) 320 is configured, for example, as an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 320 is configured to store, for example, a control program 321 and a trained model 322.

The control program 321 is executed by the control unit 310 to realize various operations in the brainwave decoding device 300. When the control unit 310 executes the control program 321, the control program 321 is deployed in the RAM 314. The control unit 310 then controls each component by interpreting and executing the control program 321 deployed in the RAM 314 by the CPU 312.

The trained model 322 is configured to output the likelihood that the subject has imagined a specific intention when, for example, an electroencephalogram signal or the like continuously received through the communication unit 330 is input. The control unit 310 determines, for example, based on the output of the trained model 322, whether or not the subject has imagined a specific intention. A method for generating the trained model 322 will be described in detail later.

<1-2. Operation>
4 is a flowchart showing the operation procedure of the electroencephalogram decoding device 300. The process shown in this flowchart is repeatedly executed by the control unit 310 when electroencephalogram signals and the like are continuously received through the communication unit 330. do.

Referring to FIG. 4, the control unit 310 inputs the EEG signal, etc. received through the communication unit 330 to the trained model 322 (step S100). Based on the output of the trained model 322, the control unit 310 determines whether or not the subject has imagined a predetermined intention (step S110). For example, when the likelihood that the subject has imagined a predetermined intention is equal to or greater than a predetermined value, the control unit 310 determines that the subject has imagined a predetermined intention.

If it is determined that the subject has not imagined a specific intention (NO in step S110), the control unit 310 executes the process of step S100 again. On the other hand, if it is determined that the subject has imagined a specific intention (YES in step S110), the control unit 310 controls the communication unit 330 to transmit an input signal associated with the specific intention to the operated device 400 (step S120). This allows the operated device 400 to be operated according to the subject's intention.

[2. Learning System]
<2-1. Configuration>
5 is a diagram illustrating a schematic configuration of the learning system 20. The learning system 20 is configured to generate, for example, the trained model 322 described above.

As shown in FIG. 5, the learning system 20 includes an intracorporeal device 100, a receiving device 200, and a trained model generating device 500. The intracorporeal device 100 and the receiving device 200 have the same configurations as the intracorporeal device 100 and the receiving device 200 included in the intention communication system 10 (FIG. 1), for example. The trained model generating device 500 generates a trained model 322 through machine learning using the received electroencephalogram signal, etc. As machine learning, various known methods such as support vector machines, deep learning, neural networks, decision tree learning, association rule learning, and Bayesian networks can be applied.

Figure 6 is a diagram showing a schematic configuration of a trained model generation device 500. The trained model generation device 500 is realized, for example, by a general-purpose computer. As shown in Figure 6, the trained model generation device 500 includes a control unit 510, a communication unit 530, an input unit 540, a display unit 550, and a memory unit 520, and each component is electrically connected via a bus.

The control unit (processor) 510 includes a CPU 512, a RAM 514, a ROM 516, etc. The control unit 510 is configured to control each component within the trained model generation device 500 in accordance with information processing.

The communication unit 530 is configured to communicate with the receiving device 200 by wired communication or wireless communication. The communication unit 530 is configured to receive, for example, an electroencephalogram signal or the like from the receiving device 200. The input unit 540 is configured to receive instructions from a user (an operator such as a doctor). The input unit 540 is configured to include, for example, some or all of a touch panel, a keyboard, a mouse, and a microphone. The display unit 550 is configured to display an image. An example of an image displayed on the display unit 550 will be described in detail later. The display unit 550 is configured to include, for example, a monitor such as a liquid crystal monitor or an organic EL monitor.

The storage unit (memory) 520 is configured, for example, with an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 520 is configured, for example, to store a control program 521. The control program 521 is executed by the control unit 510 to realize various operations in the trained model generation device 500. When the control unit 510 executes the control program 521, the control program 521 is deployed in the RAM 514. The control unit 510 then controls each component by interpreting and executing the control program 521 deployed in the RAM 514 with the CPU 512.

<2-2. Procedure for generating trained model>
In generating the trained model 322, a plurality of electroencephalogram signals and the like are used as training data. The training data is generated based on the electroencephalogram signals emitted when the subject imagines a predetermined intention. First, the procedure for collecting the electroencephalogram signals used to generate the training data will be described below.

Figure 7 is a flowchart showing the procedure for collecting EEG signals used to generate learning data. The process shown in this flowchart is repeatedly executed by the control unit 510 when EEG signals, etc. are continuously received via the communication unit 530. Note that during the period in which the process shown in this flowchart is being executed, the subject is, for example, viewing the display unit 550.

Referring to FIG. 7, the control unit 510 controls the display unit 550 to display a screen instructing the subject to imagine a predetermined intention (step S200). The control unit 510 stores the EEG signals, etc. received through the communication unit 530 during a predetermined period that correlates with the timing of displaying the screen, in the storage unit 520 as EEG signals, etc. for generating learning data (step S210). By repeating the process shown in this flowchart, the EEG signals, etc. required for generating learning data are collected. Note that the learning data may be generated based on either the received EEG signals or the results of the frequency analysis.

The trained model 322 is a regression model. In machine learning for generating the trained model 322, each of a plurality of parameters is adjusted in advance. Examples of the plurality of parameters include a parameter (Window Size) indicating a time interval used for training data from EEG signals having a fixed time interval, a parameter (Window Offset) indicating the time lag between the timing at which the subject is instructed to imagine a specific intention and the interval of the EEG signals used for training data, and a parameter (PLS Comps) indicating a dimension used in partial least squares regression. For example, the training data is used for machine learning taking into account each of the Window Size and Window Offset.

When generating the trained model 322, cross-validation is performed in advance to determine an appropriate combination of multiple parameters. Then, machine learning is performed using all of the collected training data and the determined combination of multiple parameters to generate the trained model 322. Here, an example in which 5-fold cross validation is performed is described.

Figure 8 is a diagram for explaining the division of training data in 5-fold cross validation. As shown in Figure 8, in 5-fold cross validation, training data is divided into five groups (first data group to fifth data group). For example, when 600 pieces of training data are collected, the 600 pieces of training data are divided into five groups each including 120 pieces of training data.

Figure 9 is a diagram for explaining the procedure for determining an appropriate combination of multiple parameters through 5-fold cross validation. With reference to Figure 9, in 5-fold cross validation, of the first to fifth data groups, four data groups are classified as training data, and one data group is classified as test data. There are pattern 1 to pattern 5 as classification patterns.

In pattern 1, the second to fifth data groups are classified as teacher data, and the first data group is classified as test data. In pattern 2, the first data group and the third to fifth data groups are classified as teacher data, and the second data group is classified as test data. In pattern 3, the first to second data groups and the fourth to fifth data groups are classified as teacher data, and the third data group is classified as test data. In pattern 4, the first to third data groups and the fifth data group are classified as teacher data, and the fourth data group is classified as test data. In pattern 5, the first to fourth data groups are classified as teacher data, and the fifth data group is classified as test data.

For each pattern, machine learning using training data is performed for each combination of multiple parameters, and regression models are generated for the number of combinations of multiple parameters. That is, five regression models (regression model 1 to regression model 5) are generated for each combination of multiple parameters. The multiple parameters include a specificity threshold for the generated regression model. In machine learning, a regression model with a specificity equal to or greater than the threshold is generated. In this example, four types of thresholds, 0.90, 0.95, 0.97, and 0.99, can be set as the specificity threshold. The specificity of the generated regression model may also be specified by a parameter, in which case a regression model having the specified specificity is generated through machine learning.

For each combination of multiple parameters, the sensitivity and specificity of each of the five regression models are calculated using test data. Sensitivity refers to the probability of correctly determining a positive as positive, and specificity refers to the probability of correctly determining a negative as negative. In other words, in these regression models, sensitivity refers to the probability of determining that a subject has imagined a specific intention when the subject has imagined the specific intention, and specificity refers to the probability of not determining that a subject has imagined the specific intention when the subject has not imagined the specific intention.

The average sensitivity and average specificity of the five regression models are calculated for each combination of multiple parameters. The combination of multiple parameters with the highest average sensitivity and average specificity is considered to be the optimal parameter combination. The parameters are adjusted based on the optimal parameter combination, and appropriate parameters to be used in generating the trained model 322 are determined. The parameter adjustment method will be explained in detail later.

After that, machine learning is performed using all of the collected learning data (first data group to fifth data group) and an appropriate combination of the determined multiple parameters to generate a trained model 322. In this machine learning, all of the data included in the first data group to the fifth data group is used as training data.

Figure 10 is a flowchart showing the procedure for generating the trained model 322. The process shown in this flowchart is executed by the control unit 510 after training data has been collected.

Referring to FIG. 10, the control unit 510 uses the collected learning data to generate multiple regression models through cross-validation for each combination of multiple parameters, and calculates the sensitivity and specificity of each regression model (step S300). The control unit 510 calculates the average sensitivity and average specificity of the regression model for each combination of multiple parameters (step S310). The control unit 510 considers the combination of multiple parameters with the highest average sensitivity and average specificity to be the optimal parameter combination, and controls the display unit 550 to display an image showing the optimal parameter combination (step S320). With the image showing the optimal parameter combination displayed on the display unit 550, each parameter can be adjusted.

Figure 11 is a diagram showing a schematic example of an image IM1 showing an optimal parameter combination. Referring to Figure 11, image IM1 is displayed on display unit 550. Image IM1 includes mosaic graphs G1, G2, G3, and G4, scale adjustment unit S1, regions T1 and T2, and buttons B1, B3, B4, B5, and B6. In other words, these can be said to be displayed on a common screen.

Each of the mosaic graphs G1, G2, G3, and G4 shows information about sensitivity for at least some of the multiple combinations of parameters. In this example, the horizontal axis of each graph is Window Offset (parameter), and the vertical axis of each graph is Window Size (parameter). The information about sensitivity is, for example, color information. Sensitivity and color are associated, and sensitivity is indicated by color for each combination of multiple parameters.

The specificity thresholds assumed for the mosaic graphs G1, G2, G3, and G4 are different from one another. For example, the specificity threshold assumed for the mosaic graph G1 is 0.90, and the specificity threshold assumed for the mosaic graph G2 is 0.95. Furthermore, the specificity threshold assumed for the mosaic graph G3 is 0.97, and the specificity threshold assumed for the mosaic graph G4 is 0.99.

The scale adjustment unit S1 is used to adjust the correspondence between sensitivity and color (hereinafter also referred to as "scale"). For example, the scale is adjusted by changing the upper and lower limit values indicated in the scale adjustment unit S1. When an adjustment is made in the scale adjustment unit S1, each of the mosaic graphs G1, G2, G3, and G4 is updated. In other words, each of the mosaic graphs G1, G2, G3, and G4 is generated based on the adjusted scale.

Area T1 displays the optimal combination of multiple parameters. The optimal combination of multiple parameters displayed in area T1 is the optimal combination obtained through the above-mentioned cross-validation. Area T2 displays the currently selected combination of multiple parameters. By operating buttons B5 and B6, the value of the corresponding parameter can be adjusted, and each of the mosaic graphs G1, G2, G3, and G4 is updated to one generated based on the adjusted parameter combination. For example, the adjustable parameters here are the parameters that are assumed in each of the mosaic graphs G1, G2, G3, and G4.

When button B3 is operated, a trained model is generated based on the optimal parameter combination. When button B4 is operated, a trained model is generated based on the currently selected parameter combination. When button B1 is operated, numerical information indicating the sensitivity for each parameter combination is displayed in each of the mosaic graphs G1, G2, G3, and G4.

Fig. 12 is a diagram showing an example of image IM2 in which numerical information indicating sensitivity is displayed. Referring to Fig. 12, image IM2 is displayed on display unit 550, for example, when button B1 is operated in a state in which image IM1 is displayed on display unit 550. In image IM2, numerical information indicating sensitivity for each combination of parameters is displayed in each of mosaic graphs G1, G2, G3, and G4.

Figure 13 is an enlarged view of the mosaic graph G3 in Figure 12. As shown in Figure 13, in the mosaic graph G3, the area showing the highest sensitivity is highlighted by being surrounded by a frame C1. For example, the frame C1 may be a color that stands out compared to the background color. This allows the user to easily recognize the parameter combination corresponding to the area showing the highest sensitivity. Note that the frame C1 may be displayed in each of the mosaic graphs G1, G2, G3, and G4, or may be displayed only in the area that is the best parameter combination among the currently selected parameter combinations.

Referring again to FIG. 10, when the optimal parameter combination is displayed on display unit 550 in step S320, control unit 510 determines whether the user has finished adjusting the parameters (step S330). Specifically, control unit 510 determines whether the user has selected either button B3 or B4. Even if an optimal parameter combination is determined to be optimal, for example, a parameter combination in which a small change in some parameters significantly reduces sensitivity is not necessarily optimal. In such a case, the user adjusts the parameters.

If it is determined that the parameter adjustment is not complete (NO in step S330), the control unit 510 waits until the parameter adjustment is complete. On the other hand, if it is determined that the parameter adjustment is complete (YES in step S330), the control unit 510 generates a trained model 322 through machine learning using the adjusted parameter combination (step S340).

In this way, in the trained model generating device 500, the mosaic graphs G1, G2, G3, and G4 are displayed on a common screen. Each mosaic graph shows information about the sensitivity of at least some of the combinations of multiple parameters, and the values related to the specificity that are the premise for the multiple mosaic graphs are different from each other. Therefore, according to the trained model generating device 500, the sensitivity and specificity for at least some combinations of multiple parameters are visually represented by the mosaic graphs G1, G2, G3, and G4, so that the user (doctor, etc.) can easily recognize a good combination of multiple parameters by the screen display. Since the user (doctor, etc.) can refer to the mosaic graphs G1, G2, G3, and G4 to determine the parameters to be used and proceed with the evaluation, it becomes easy to generate a trained model that can estimate the subject's intention with a relatively high accuracy based on the subject's brain waves.

In addition, when the values of the parameters that are the premise of the mosaic graphs G1, G2, G3, and G4 are changed, the mosaic graphs G1, G2, G3, and G4 are updated. Therefore, the trained model generation device 500 allows the user to easily recognize the effect of parameter changes on sensitivity through the screen display.

In addition, when the scale is changed through the scale adjustment unit S1, each of the mosaic graphs G1, G2, G3, and G4 is updated. Therefore, the trained model generation device 500 allows the user to more clearly recognize the relationship between each parameter and the sensitivity through the scale adjustment.

In addition, each of the mosaic graphs G1, G2, G3, and G4 can display numerical information indicating sensitivity. The trained model generation device 500 allows the user to easily recognize good combinations of multiple parameters through the display of numerical information.

In addition, in the trained model generation device 500, the numerical values of each parameter in the optimal combination are displayed on the screen, and each of the mosaic graphs G1, G2, G3, and G4 is generated on the premise of the optimal parameter combination. Therefore, the trained model generation device 500 allows the user to easily recognize the parameter combination that maximizes the sensitivity and specificity.

[3. Internal Device
<3-1. Rules for connecting electrodes and amplifier chips>
As described above, each electrode 111 included in the electrode group 110 is electrically connected to an amplifying element 121 (FIG. 2) included in the amplifier chip 120 via a platinum wire. The quality of the amplifier elements 121 is not necessarily constant. Some of the amplifier elements 121 included in the amplifier chip 120 may not meet a predetermined quality standard.

The brain waves detected by some of the electrodes 111 included in the electrode group 110 have a strong correlation with the subject's intention. In addition, the brain waves detected by some of the electrodes 111 included in the electrode group 110 have almost no correlation with the subject's intention. For example, when detecting an intention related to the subject's hand movement, the brain waves measured at electrodes located away from the sensorimotor cortex area of the subject's hand have almost no correlation with the subject's intention. Other examples of intention include intention related to the subject's face movement, intention related to the subject's mouth movement, and intention related to the subject's foot movement. The position of the electrode group 110 on the subject's brain surface is adjusted according to the intention to be detected. The position of the electrode group 110 may be determined by referring to Penfield's homunculus according to the intention to be detected, for example.

In this embodiment, electrodes 111 that detect brain waves that are strongly correlated with the subject's intentions are connected to amplifier elements 121 that meet a predetermined quality, and electrodes 111 that detect brain waves that are barely correlated with the subject's intentions are connected to amplifier elements 121 that do not meet the predetermined quality. That is, in this embodiment, some of the electrodes located in brain regions of the subject's brain that are distant from brain regions that are important for intention detection are predetermined as electrodes to be connected to amplifier elements 121 that do not meet the predetermined quality. This makes it possible to detect brain waves that are strongly correlated with the subject's intentions with higher accuracy.

Figure 14 is a diagram for explaining the connection rules between each electrode 111 included in the electrode group 110 and the amplifier element 121. Referring to Figure 14, each amplifier element 121 included in the amplifier chip 120 is electrically connected to an electrode pad 126 included in the electrode pad group 125. By connecting the electrode pad 126 and the electrode 111, the amplifier element 121 and the electrode 111 are electrically connected. Note that the electrode 111 included in region A of the electrode group 110 is connected to the electrode pad 126 included in region A of the electrode pad group 125, and the electrode 111 included in region B of the electrode group 110 is connected to the electrode pad 126 included in region B of the electrode pad group 125.

For example, when detecting an intention regarding a subject's hand movement, the electroencephalogram measured by the electrodes 111 placed near the hand sensory-motor area in the subject's brain has a high correlation with the subject's intention. On the other hand, the electroencephalogram measured by the electrodes 111 placed in an area of the subject's brain away from the hand sensory-motor area has almost no correlation with the subject's intention. For example, the electroencephalogram signals acquired by the electrodes 111 placed in the first area ET1 have a high correlation with the subject's intention regarding the hand movement compared to the electroencephalogram signals acquired by the electrodes 111 placed in the second area ET2.

In this embodiment, before connecting the electrode 111 and the electrode pad 126, an inspection is performed to check whether each amplifying element 121 meets a predetermined quality. In this inspection, for example, the input-equivalent noise of each amplifying element 121 is checked, and an amplifying element 121 whose input-equivalent noise is equal to or greater than a predetermined value is determined to not meet the predetermined quality. Furthermore, even among the amplifying elements 121 that meet the predetermined quality, they are ranked in order of decreasing input-equivalent noise.

Of the electrodes located in brain regions away from brain regions important for intention detection, some of the electrodes 111 located toward both ends in the width direction of the electrode group 110 (electrodes 111 arranged in the second region ET2) are predetermined as electrodes to be connected to amplification elements 121 that do not meet a specified quality. In this example, the electrodes 111 associated with numbers 1 to 8 in FIG. 14 are predetermined as electrodes to be connected to amplification elements 121 that do not meet a specified quality. This number indicates the order in which amplification elements 121 that do not meet a specified quality are preferentially connected.

In addition, in this embodiment, the priority order of the electrodes 111 to which the higher quality amplifier elements 121 are connected is determined in advance. In other words, the priority order of the electrodes 111 is determined in advance so that the higher quality amplifier elements 121 are connected to the electrodes 111 that detect brain waves that have a higher correlation with the subject's intentions.

Figure 15 is a diagram for explaining the priority of the electrodes 111. Referring to Figure 15, the numerical values associated with each electrode 111 in the figure indicate the priority of each electrode 111. The priority of each electrode 111 is determined for each of the first and second regions of the electrode group 110. Amplification elements 121 of higher quality are connected in ascending order of the associated numerical values. This makes it possible to detect brain waves that have a strong correlation with the subject's intentions with higher accuracy.

An electrode map indicating the connection relationship between the electrodes 111 and the amplification elements 121 is stored, for example, in the memory unit 520 of the trained model generation device 500. The electrode map includes information indicating the connection relationship between the electrodes 111 and the amplification elements 121, and information indicating whether the amplification elements 121 satisfy a predetermined quality. For example, by referring to the electrode map, the control unit 510 can determine which electrode 111 detected the brainwaves that generated the output of each amplification element 121.

<3-2. Display of information on brain waves detected by each electrode>
16 is a flowchart showing a procedure for displaying a spectrogram of an electroencephalogram detected by each electrode 111. The process shown in this flowchart is repeatedly executed by the control unit 510 of the trained model generation device 500.

The control unit 510 acquires the output (brainwave signal, etc.) of each amplification element 121 through the communication unit 330 (step S400). The control unit 510 generates a plurality of brainwave spectrograms based on the acquired brainwave signals, etc. (step S410). The control unit 510 generates a screen including a plurality of brainwave spectrograms by referring to the electrode map stored in the memory unit 520 (step S420). In this screen, each of the plurality of brainwave spectrograms is positioned at a position equivalent to the position of the corresponding electrode 111 in the electrode group 110. The control unit 510 controls the display unit 350 to display the generated screen (step S430).

Figure 17 is a diagram that shows a schematic example of a screen displayed on the display unit 350. Referring to Figure 17, in this screen, each of the multiple EEG spectrograms is arranged at a position equivalent to the position of the corresponding electrode 111 in the electrode group 110. For example, an EEG spectrogram 111A may be displayed for an electrode 111 connected to an amplifier element 121 that satisfies a predetermined quality, and a blackened image 111B may be displayed for an electrode 111 connected to an amplifier element 121 that does not satisfy the predetermined quality.

In this way, in the trained model generation device 500, a screen is displayed in which each of the multiple EEG spectrograms is positioned at a position equivalent to the position of the corresponding electrode 111 in the electrode group 110. Therefore, according to the trained model generation device 500, since each EEG spectrogram is displayed at a position equivalent to the corresponding electrode 111, the user can intuitively recognize the EEG spectrogram that reflects the subject-specific brain activity areas and the EEG intensity by frequency band, making it easier to grasp the subject-specific brain activity, and it is expected that a trained model that can estimate the subject's intentions with relatively high accuracy will be produced.

In addition, the subject's intention has a strong correlation with the brain waves detected by the electrode 111 (electrode 111 arranged in the first region ET1) located in a brain region important for intention detection. In this embodiment, an amplifier element 121 that does not satisfy a predetermined criterion is connected to an electrode 111 (electrode 111 arranged in the second region ET2) located in a brain region away from the brain region important for intention detection among the multiple electrodes 111. Therefore, according to this embodiment, the electrode 111 located in the brain region important for intention detection can be connected to an amplifier element 121 that satisfies the predetermined criterion, and as a result, it can be expected that a trained model capable of estimating the subject's intention with a relatively high degree of accuracy will be produced.

[4. Features]
As described above, in the trained model generating device 500, the mosaic graphs G1, G2, G3, and G4 are displayed on a common screen. Each mosaic graph shows information about sensitivity in at least some of the combinations of a plurality of parameters, and the values related to the specificity that are the premise for the plurality of mosaic graphs are different from each other. Therefore, according to the trained model generating device 500, the sensitivity and specificity in at least some of the combinations of a plurality of parameters are visually represented by the mosaic graphs G1, G2, G3, and G4, so that a user (doctor, etc.) can easily recognize a good combination of a plurality of parameters by the screen display. Since the user (doctor, etc.) can refer to the plurality of mosaic graphs to determine the parameters to be used and proceed with the evaluation, it becomes easy to generate a trained model that can estimate the subject's intention with a relatively high accuracy based on the subject's electroencephalogram.

In addition, in the trained model generation device 500, a screen is displayed in which each of the multiple EEG spectrograms is positioned at a position equivalent to the position of the corresponding electrode 111 in the electrode group 110. Therefore, according to the trained model generation device 500, since each EEG spectrogram is displayed at a position equivalent to the corresponding electrode 111, the user can intuitively recognize the EEG spectrogram that reflects the subject-specific brain activity areas and the EEG intensity by frequency band, making it easier to grasp the subject-specific brain activity, and it is expected that a trained model that can estimate the subject's intentions with relatively high accuracy will be produced.

In addition, in this embodiment, an amplifier element 121 that does not satisfy a predetermined criterion is connected to an electrode 111 (electrode 111 arranged in the second region ET2) that is located in a brain region away from the brain region that is important for intention detection among the multiple electrodes 111. Therefore, according to this embodiment, an electrode 111 (electrode 111 arranged in the first region ET1) located in a brain region that is important for intention detection can be connected to an amplifier element 121 that satisfies the predetermined criterion, and as a result, it can be expected that a trained model that can estimate the subject's intention with a relatively high degree of accuracy will be produced.

5. Other embodiments
The concept of the above embodiment is not limited to the embodiment described above. Hereinafter, an example of another embodiment to which the concept of the above embodiment can be applied will be described.

In the above embodiment, the EEG decoding device 300 and the trained model generating device 500 are realized as different devices. However, the EEG decoding device 300 and the trained model generating device 500 do not necessarily have to be realized as different devices. The EEG decoding device 300 and the trained model generating device 500 may be realized as a single common device.

In the above embodiment, the trained model 322 outputs the likelihood that the subject imagined a specific intention when an amplified EEG signal is input. Here, the specific intention does not necessarily have to be one type of intention. For example, multiple types of intentions may be predefined as the specific intention. In this case, the trained model 322 may be a classifier model that outputs the likelihood that the subject imagined each intention when an amplified EEG signal is input. Even in such a case, it is sufficient that multiple mosaic graphs are displayed on a common screen.

The above is an illustrative description of the embodiment of the present invention. In other words, the detailed description and the accompanying drawings are disclosed for the purpose of illustrative description. Therefore, the components described in the detailed description and the accompanying drawings may include components that are not essential for solving the problem. Therefore, just because these non-essential components are described in the detailed description and the accompanying drawings, it should not be immediately determined that these non-essential components are essential.

The above-described embodiment is merely an example of the present invention in every respect. Various improvements and modifications of the above-described embodiment are possible within the scope of the present invention. In other words, when implementing the present invention, specific configurations can be appropriately adopted according to the embodiment.

10 Intention communication system, 20 Learning system, 100 Intracorporeal device, 110 Electrode group, 111 Electrode, 120 Amplifier chip, 121 Amplification element, 125 Electrode pad group, 126 Electrode pad, 130, 310, 510 Control unit, 140, 330, 530 Communication unit, 200 Receiving device, 300 EEG decoding device, 312, 512 CPU, 314, 514 RAM, 316, 516 ROM, 320, 520 Storage unit, 321, 521 Control program, 322 Learned model, 340, 540 Input unit, 350, 550 Display unit, 400 Operated device, 500 Learned model generation device, B1, B3, B4, B5, B6 Button, C1 Frame, ET1 First area, ET2 Second area, G1, G2, G3, G4 mosaic graph, IM1, IM2 image, S1 scale adjustment section, T1, T2 area.

Claims (10)

A program used to generate a trained model that estimates a subject's intention based on the subject's electroencephalogram,
The trained model is generated through machine learning using a plurality of training data,
In the machine learning, each of a plurality of parameters is adjusted in advance,
Each of the plurality of teacher data includes information regarding the electroencephalogram and is used for the machine learning based on at least a part of the plurality of parameters;
The program is
generating a plurality of trained models by performing the machine learning for each of a plurality of combinations of the plurality of parameters;
Calculating sensitivity and specificity for each of the plurality of trained models;
and displaying the plurality of mosaic graphs on a common screen;
each of the plurality of mosaic graphs indicates information regarding the sensitivity in at least a portion of the plurality of combinations;
The program, wherein the values related to the specificity assumed for the multiple mosaic graphs are different from each other. In each of the plurality of mosaic graphs, at least a part of the plurality of parameters is determined as a premise value;
accepting changes to the values of the assumptions;
and updating each of the plurality of mosaic graphs if the value of the premise is changed. the sensitivity information includes color information;
accepting a change to a scale indicating a correspondence between the sensitivity value and a color;
3. The program according to claim 1, further causing the computer to execute a step of updating each of the plurality of mosaic graphs when the scale is changed. The program according to claim 1 or claim 2, wherein the information about the sensitivity includes numerical information. deriving an optimal combination of the plurality of parameters based on both the sensitivity and the specificity;
and displaying, on the screen, the numerical values of the parameters in the optimal combination.
The computer-readable medium according to claim 1 , wherein each of the plurality of mosaic graphs is generated on the basis of the optimal combination. The program according to claim 1 or 2, further causing the computer to execute a step of highlighting on the screen the information showing the highest sensitivity among the multiple pieces of information on the sensitivities shown in the multiple mosaic graphs. A method for manufacturing a trained model that estimates a subject's intention based on the subject's electroencephalogram,
The trained model is generated through machine learning using a plurality of training data,
In the machine learning, each of a plurality of parameters is adjusted in advance,
Each of the plurality of teacher data includes information regarding the electroencephalogram and is used for the machine learning based on at least a part of the plurality of parameters;
The manufacturing method includes:
generating a plurality of trained models by performing the machine learning for each of a plurality of combinations of the plurality of parameters;
Calculating sensitivity and specificity for each of the plurality of trained models;
displaying a plurality of mosaic graphs on a common screen;
each of the plurality of mosaic graphs indicates information regarding the sensitivity in at least a portion of the plurality of combinations;
A manufacturing method in which the specificity values assumed for the multiple mosaic graphs are different from each other. An apparatus for producing a trained model that estimates a subject's intention based on the subject's electroencephalogram,
The trained model is generated through machine learning using a plurality of training data,
In the machine learning, each of a plurality of parameters is adjusted in advance,
Each of the plurality of teacher data includes information regarding the electroencephalogram and is used for the machine learning based on at least a part of the plurality of parameters;
The manufacturing apparatus includes:
A processor;
a memory for storing a program to be executed by the processor;
The program is
generating a plurality of trained models by performing the machine learning for each of a plurality of combinations of the plurality of parameters;
Calculating sensitivity and specificity for each of the plurality of trained models;
and displaying the plurality of mosaic graphs on a common screen;
each of the plurality of mosaic graphs indicates information regarding the sensitivity in at least a portion of the plurality of combinations;
A manufacturing apparatus, wherein the values relating to the specificity assumed for the multiple mosaic graphs are different from each other. A program for displaying an electroencephalogram generated based on electroencephalograms acquired through a group of electrodes implanted in the skull of a subject,
the electrode group includes an electrode disposed in a first region and an electrode disposed in a second region,
The electroencephalogram acquired by the electrodes arranged in the first region has a higher correlation with a detection target intention among the intentions of the subject, compared with the electroencephalogram acquired by the electrodes arranged in the second region;
Each of the plurality of electrodes included in the electrode group is electrically connected to one of the plurality of amplifying elements included in the amplifier chip;
Among the plurality of amplifying elements, an amplifying element that does not satisfy a predetermined criterion is electrically connected to an electrode disposed in the second region,
The program is
generating a plurality of electroencephalogram spectrograms based on outputs of the plurality of amplifying elements;
and displaying a screen including the generated plurality of electroencephalograms.
A program in which, on the screen, each of the multiple electroencephalogram spectrograms is placed at a position equivalent to the position of a corresponding electrode in the group of electrodes. A method for connecting an electrode group and an amplifier chip for detecting brain waves of a subject, comprising:
The amplifier chip includes a plurality of amplification elements,
the electrode group includes an electrode disposed in a first region and an electrode disposed in a second region,
The electroencephalogram acquired by the electrodes arranged in the first region has a higher correlation with a detection target intention among the intentions of the subject, compared with the electroencephalogram acquired by the electrodes arranged in the second region;
The connection method includes:
determining whether the plurality of amplifying elements includes an amplifying element that does not satisfy a predetermined criterion;
A connection method comprising: when the plurality of amplification elements includes an amplification element that does not satisfy the specified criterion, connecting the amplification element that does not satisfy the specified criterion to an electrode arranged in the second region.