JP2014092561A - Intention transmission support device, intention transmission support method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an intention transmission support device, an intention transmission support method, and a program which accurately and stably support the transmission of information.SOLUTION: The intention transmission support device includes a computing unit for outputting information specifying visual stimulation sensuously received or imaged by a subject from a result calculated based on information including a feature quantity due to a brain surface signal of the region and a feature quantity due to an intra-brain signal of the region extracted respectively and repeatedly acquired according to the predefined test period in the region including the cerebra visual cortex or the cerebra association cortex of the brain of the subject.

Description

  The present invention is a communication support device, a communication support method, and a program.

  The communication support device reads information on what the subject (subject) has thought of in his / her head, and outputs the read information on behalf of the test subject to support the subject's communication. In the communication support device, it is desired to support the communication of the subject's intention by accurately and stably outputting what the subject has imagined. Examples of the subject of the communication support apparatus include a severe patient who has a disorder of ability to express intention. The disorder of ability to express intention may occur due to diseases such as stroke (cerebrovascular disorder) and amyotrophic lateral sclerosis (ALS disease). Some severe patients who have developed such a cause are not able to move their limbs or speak words even if they can see and understand letters and pictures. It may not be possible to communicate well.

As a technology related to the communication support device, a method of supporting communication using brain signals is disclosed.
The first method uses an electroencephalogram that can be detected non-invasively (see, for example, Patent Document 1). According to Patent Document 1, intention transmission is supported by using a message display device that uses an electroencephalogram switch control method that uses an electroencephalogram as a control signal for selecting a condition.
The second method uses invasive detection means known as the Utah large system (see, for example, Patent Document 2). The Utah large-scale detection means detects the spike activity (action potential) of a nerve cell by directly inserting into the brain using a sword mountain type microelectrode.
In both the first method and the second method, a plurality of selection candidates, a cursor, and the like are displayed on the display unit, and selection is made based on brain signals (information) obtained from the subject (subject). While moving the cursor position to the display position of the selection candidate, the selection candidate (displayed character etc.) indicated by the cursor position is selected.

Japanese Patent Laid-Open No. 7-20774 US Patent Application Publication No. 2007/46486

However, according to Patent Document 1, it is shown that even if a person with severe muscular dysfunction such as ALS disease loses information transmission means for communication by means of conversation or muscular strength, it is possible to communicate. However, it merely uses information on the presence or absence of an electroencephalogram, and does not obtain detailed information from the detected electroencephalogram. In addition, because the motor plan is identified from the local neural activity of the cerebral motor cortex, the target is limited to rare diseases such as ALS and confinement syndrome, and the motor cortex itself due to cerebrovascular disorders is damaged. There is a problem that it cannot be applied to a large patient.
Further, according to Patent Document 2, since the spike activity (action potential) of a nerve cell is recorded, it is necessary to arrange electrodes so that the potential of the nerve cell can be detected. It is difficult to obtain information stably.
Furthermore, according to the detection methods in Patent Documents 1 and 2, since an indirect method of selecting characters displayed on the screen is used, the time required until the information recognized by the subject (subject) can be transmitted. Longer and burdens the subject.
As described above, each of the above methods uses different detection methods, but has not been able to provide sufficient support. In addition, each of the methods has a problem in that if the number of types of information (images) to be transmitted is further increased, the distinguishability is deteriorated and the information cannot be accurately transmitted.

  The present invention has been made to solve the above problems, and an object of the present invention is to provide a communication support apparatus, a communication support method, and a program that support information transmission accurately and stably.

  The present invention has been made to solve the above-described problem, and one embodiment of the present invention is repeatedly performed in a region including the cerebral visual cortex or cerebral association cortex in the subject's brain according to a predetermined trial period. From the results calculated based on the information including the feature amount derived from the brain surface signal of the region and the feature amount derived from the brain signal of the region, respectively extracted from the acquired brain surface signal and brain signal A communication support apparatus comprising an arithmetic processing unit that outputs information for identifying a visual stimulus that a subject is receiving or sensing.

  Further, according to one aspect of the present invention, in the above invention, the brain signal measured by the first measuring unit in the brain of the region and the brain surface measured by the second measuring unit in the brain table of the region. A multipoint electric potential measurement unit for obtaining electrical characteristics of the brain at the measured position from a plurality of types of brain signals detected according to the trial period, including the signal, and the obtained electrical brain power And a feature amount extraction unit that extracts a feature amount according to the target characteristic.

  One embodiment of the present invention is characterized in that, in the above-mentioned invention, the multipoint potential measuring unit uses a local field potential (LFP) method as the first measuring means.

  Further, one embodiment of the present invention is characterized in that, in the above invention, the multipoint potential measurement unit uses a cortical electroencephalogram (ECoG) method as the second measurement means.

  Further, according to one aspect of the present invention, in the above invention, a second electrode for detecting the brain surface signal by the ECoG method is provided in the brain, the first electrode portion for detecting the brain signal by the LFP method being disposed in the brain. A part is arranged on the brain surface.

  Further, according to one aspect of the present invention, in the above invention, the multipoint potential measurement unit is provided on the same surface as the first electrode unit that detects the plurality of brain signals, and the plurality of brain surface signals. And the second electrode part for detecting the first electrode part, wherein the first electrode part is provided so as to protrude from one surface on which the second electrode part is provided.

  One embodiment of the present invention is the above invention, wherein the multipoint potential measurement unit is at least a part of a first polygon formed by positions of a plurality of terminals provided in the first electrode unit. And at least a part of the second polygon formed by the positions of the plurality of terminals provided in the second electrode portion overlap with each other in plan view. It is arranged so that a part exists.

  In one embodiment of the present invention, in the above invention, the region including the cerebral visual cortex or the cerebral association cortex is a region centering on the higher visual association cortex, and at least the ventral part of the temporal lobe or the occipital lobe. Any one of the visual related areas is included.

  According to another aspect of the present invention, in the above invention, the feature amount extraction unit determines a feature amount corresponding to the obtained electrical characteristics of the brain as a frequency component included in the brain signal and a predetermined predetermined amount. And the time from when the response of the brain signal is detected to the time when the response is detected.

  According to another aspect of the present invention, in the above invention, the feature amount extraction unit defines a detection frequency range so as to include at least a main frequency component included in the brain signal, and a part of the determined detection frequency range is determined. A plurality of frequency ranges are defined to include a detection time range so as to include at least a period from a predetermined time before detecting a response of the brain signal until a response of the brain signal is detected, and the determined A plurality of time ranges are defined so as to include a part of the detection time range, and the signal intensity of the brain signal included in a range limited by a combination of any of the determined frequency ranges and any of the determined time ranges A representative value is included in the feature amount.

  Further, according to one aspect of the present invention, in the above invention, the arithmetic processing unit includes information further including a feature amount derived from a correlation between a plurality of signals selected from the brain surface signal and the brain signal. Information specifying the visual stimulus is output from the result calculated based on the result.

  Further, according to one aspect of the present invention, in the above invention, the arithmetic processing unit calculates the result based on information calculated based on information further including a feature amount derived from a correlation between the brain surface signal and the intracerebral signal. It outputs the information specifying the visual stimulus.

  According to another aspect of the present invention, in the above invention, the feature amount extraction unit correlates the feature amount according to the obtained electrical characteristics of the brain based on the brain surface signal and the intracerebral signal. Extracted from the processing result, the arithmetic processing unit identifies the visual stimulus from a result calculated based on information further including a feature amount derived from a correlation between the brain surface signal and the signal in the brain It is characterized by outputting information.

  Further, according to one aspect of the present invention, in the above invention, the feature amount extraction unit divides the detection frequency range into a predetermined number, and the plurality of frequency ranges are arranged so that the ranges do not overlap each other. It is defined.

  In addition, according to another aspect of the present invention, in the above invention, the feature amount extraction unit includes the brain signal included in a range limited by a combination of any of the defined frequency ranges and any of the defined time ranges. An average value of signal intensity is included in the feature quantity as a representative value of signal intensity of the brain signal.

  Further, according to one aspect of the present invention, in the above invention, the arithmetic processing unit is associated from the extracted feature amount based on the plurality of types of brain signals from a plurality of predetermined images. An image is selected, and information on the selected image is output as the calculated information.

  Further, according to one aspect of the present invention, in the above invention, the arithmetic processing unit is detected along with display of the image in the process of selecting the associated image from the plurality of predetermined images. The variable of the process is optimized by the process of associating the characteristic quantity of the electrical characteristics of the brain at the measurement point of the brain with the image.

  In addition, according to one aspect of the present invention, in the above invention, the multipoint potential measurement unit is configured such that, as the electrical characteristics, a potential (visual evoked potential) that changes with a visual stimulus at each measurement point or a visual stimulus. It is characterized by using a change in a specific frequency band component of the changing potential.

  Further, according to one aspect of the present invention, in the region including the cerebral visual cortex or cerebral association cortex in the subject's brain, each of the brain surface signals and brain signals that are repeatedly acquired according to a predetermined trial period, respectively, is extracted. Vision that the subject senses or images from the result calculated based on information including the feature amount derived from the brain surface signal of the region and the feature amount derived from the brain signal of the region It is a communication support method characterized by including the step of outputting the information which specifies a stimulus.

  In one embodiment of the present invention, a computer included in the communication support device includes a brain surface signal that is repeatedly acquired in a region including a cerebral visual cortex or a cerebral association cortex in a subject's brain according to a predetermined trial period. The subject receives sensory perception from a result calculated based on information including a feature amount derived from the brain surface signal of the region and a feature amount derived from the brain signal of the region, respectively extracted from the brain signal. This is a program for executing a step of outputting information for identifying a visual stimulus that is being or imaged.

  According to the present invention, it is possible to provide an intention transmission support apparatus, an intention transmission support method, and a program that support transmission of information accurately and stably.

It is a functional block diagram which shows the intention communication assistance apparatus by the visual image in 1st Embodiment. It is a figure which shows the multipoint electric potential measurement part which measures ECoG of a brain surface, and LFP in a brain simultaneously, and its arrangement | positioning. It is a figure which shows the electrode part 31 in a multipoint electric potential measurement part. It is a figure which shows the electrode part 32 in this embodiment. It is a figure which shows the position of the electrode part 31 and the electrode part 32 in this embodiment. It is a figure which shows the process with respect to the cortical electroencephalogram detected by the electrode array. It is a figure which shows an example of the result of the process shown in FIG. It is a figure explaining the process which extracts the feature-value from the spatiotemporal frequency component converted by the spatiotemporal frequency conversion process shown in FIG. It is a flowchart which shows the process in this embodiment. It is a figure which shows the determination result by the intention communication assistance apparatus shown to this embodiment. It is a functional block diagram which shows the intention communication assistance apparatus by the visual image in 2nd Embodiment. It is a flowchart which shows the process in this embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. In the following description, the same components are denoted by the same reference numerals, and the description is simplified. In the illustrated configuration, a part of the configuration may be omitted or a detailed configuration may be added.

(First embodiment)
With reference to FIG. 1, the outline | summary of the communication support apparatus by the visual image by this embodiment is demonstrated. FIG. 1 is a functional block diagram showing a visual communication image communication support device according to this embodiment.

  First, an outline of the intention communication support apparatus 100 (Brain Machine Interface (BMI)) according to the present embodiment will be described. In the communication support apparatus 100, the visual information S1 based on the image stimulus (v) when a human (subject) sees a certain object (obj), for example, the letter “X”, is visually transmitted from the visual receptor 11 (retina). It reaches the brain via the nervous system and is converted into a response signal S2 specific to the form (visual information S1) by the visual association area 12 through the primary visual cortex (V1 (FIG. 2)) in the brain. When a character “Y” different from the character “X” is seen, a different response signal (S2) is obtained. The brain response signal S2 itself due to the image stimulus (v) is measured by the multipoint potential measuring unit 3 by a method such as a local field potential (LFP) method or an ECoG method. The electrical characteristics of the brain associated with the image stimulus (v) can be extracted from a plurality of potential measurement values obtained by repeated measurement in a trial period including a common period in the multipoint potential measurement unit 3. Here, the electrical characteristic refers to a change in potential accompanying the image stimulus (v) (that is, visual evoked potential) and will be described below. Note that a change in the configuration of a specific frequency band component (for example, a γ frequency band of 30 to 100 Hz) of the potential accompanying the image stimulus (v) can be the above-described electrical characteristics.

The multipoint potential measurement unit 3 supplies a set of electrical characteristics observed at each measurement point with image stimulation (v) to the feature quantity extraction unit 4 as neural activity (r).
The feature amount extraction unit 4 extracts a feature amount of information indicated by the neural activity (r) based on the supplied neural activity (r) (electrical characteristics observed at each measurement point). The feature quantity extraction unit 4 supplies the extracted feature quantity to the arithmetic processing unit 5.

Here, a decoding process for identifying detailed information will be described.
The communication support device 100 in this embodiment identifies detailed information efficiently without increasing the load of decoding processing based on the collected brain signals.
The communication support apparatus 100 according to the present embodiment uses visual image information such as characters and icons that a subject (a subject such as a human or a monkey) envisions in his / her head, and the neural activity of the brain, that is, cortical electroencephalogram (ECoG). And real-time decoding based on the combination of multipoint measurement values of the brain and local field potential (LFP).
The information read and understood by the communication support apparatus 100 includes visual stimuli that the subject senses or images. The communication support apparatus 100 reads such visual stimuli as visual stimuli encoded in the brain.
The intention transmission support apparatus 100 enables real-time decoding by machine learning processing divided into two stages: (1) machine learning phase and (2) intention transmission phase.

  The machine learning phase in the first stage will be described. In the first stage of machine learning phase, the communication support device 100 first detects a multipoint neural response to presentation of visual stimuli such as individual characters, and features of the multipoint neural response corresponding to the presented visual stimuli. Is represented as a vector in an n-dimensional space. The communication support device 100 (the arithmetic processing unit 5) uses a feature amount indicated by a vector in an n-dimensional space to obtain an optimal hyperplane solution that separates a response to an individual stimulus from a response to another stimulus in advance of machine learning. Find according to the algorithm of education. The intention transmission support device 100 (arithmetic processing unit 5), as a response by each of the above stimuli, uses information relating the spatio-temporal domain and the frequency domain derived from the brain multi-point LFP recorded simultaneously and the ECoG of the brain surface. The feature quantity to be extracted is used. Note that a known method such as a support vector machine (SVM) is used for the machine learning process. For example, the SVM uses a machine learning processing algorithm that uses a pre-education type machine learning method that uses a hyperplane calculated as an optimal solution.

Next, the second stage communication phase will be described. In the second stage communication phase, the communication support device 100 is detected according to the neural activity being observed in accordance with the same process as the machine learning process algorithm such as SVM in the first stage machine learning phase. The feature amount is extracted, and the character or icon type information that the subject wants to select next is decoded (decoded). The intention transmission support device 100 (the arithmetic processing unit 5) displays the decoded icon and character information on the display device 9.
Note that the intention transmission support apparatus 100 (arithmetic processing unit 5) may supply the decoded icon or character information to a control device, a computer, or the like, and cause the control device, the computer, or the like to process the information. By processing the decoded icon and character information in the control device, computer, etc., the control device, computer, etc., functions effectively as a communication support device for spelling characters and selecting objects.

  The above is the outline of the intention transmission support apparatus shown in the present embodiment.

  As described above, when an object (such as a character) is to be specified as an image image, the object being imaged, for example, a character is specified based on the feature amount derived from the neural activity (r) observed in the brain. The identification accuracy (resolution, identification rate) of identification can be increased.

  By the way, as a technique related to the present embodiment, there is a technique that uses a cortical electroencephalogram (ECoG) as a detection target (see, for example, JP 2010-257343 A). According to the technique described in Japanese Patent Application Laid-Open No. 2010-257343, responsiveness is improved by directly selecting visual images such as characters and icons based on information obtained from a subject. Also in the present embodiment, a method of directly selecting a visual image such as a character or an icon based on information obtained from a subject is used, as in the above technique.

However, in the technique described in Japanese Patent Application Laid-Open No. 2010-257343, a cortical electroencephalogram (ECoG) is used alone, and the cortical electroencephalogram (ECoG) is not combined with other types of brain signals, but one kind of brain. The signal is used. Further, a determination method based on the peak potential of cortical electroencephalogram (ECoG) is used. In this way, in the determination method based on the peak potential of one type of brain signal (cortical EEG), the type and accuracy of information that can be read by the cortical EEG alone is limited, so the resolution for identifying the information to be conveyed is increased. May be difficult. In short, even if it is possible to read the rough visual category (for example, whether the displayed image is a character or a face) from brain information (cortical EEG), detailed differences There may be a case where it is difficult to increase the resolution until a point (for example, each factor included in the above classification such as a character difference or individual identification based on a face image) is identified.
Therefore, in the present embodiment, a technique for supporting information transmission accurately and stably by increasing the resolution to discriminate from the related technology described above. explain.

(Multi-point potential measurement unit that simultaneously measures ECoG on the brain surface and LFP in the brain)
A multipoint potential measurement unit that measures ECoG on the brain surface and LFP in the brain during the same trial period (hereinafter sometimes referred to as “simultaneously”) will be described with reference to FIGS. The multipoint potential measurement unit described here can increase the density of information obtained from brain signals by simultaneously measuring ECoG on the brain surface and LFP in the brain.
FIG. 2 is a diagram showing a multipoint potential measurement unit that simultaneously measures ECoG on the brain surface and LFP in the brain, and an arrangement thereof. FIG. 3 is a diagram illustrating the electrode unit 31 in the multipoint potential measurement unit.

FIG. 2A is a longitudinal sectional view (front view) of the brain showing an arrangement example of the multipoint potential measurement unit. FIG. 2A shows the position of the multipoint potential measurement unit in the image of the head imaged by MRI.
The central bright portion indicates the position of the brain, and the tip of the multipoint potential measuring unit 3 (microelectrode 31T side) is arranged in contact with the brain (TE field). A coordinate system indicated by an A axis, an L axis, and a D axis indicates the direction of the brain, the A axis indicates the front direction of the brain, the L axis indicates the left direction of the brain, and the D axis indicates the parietal direction of the brain.
A coordinate system based on the orientation of the multipoint potential measuring unit 3 is shown as a coordinate system indicated by the x-axis, y-axis, and z-axis. In the coordinate system indicated by the x-axis, y-axis, and z-axis, the z-axis is perpendicular to the brain surface when the electrode unit 31 (first electrode unit) of the multipoint potential measuring unit 3 is mounted on the head. Indicates the direction in which the electrode portion 31 (first electrode portion) is advanced and retracted, and the direction in which the x-axis and the y-axis are orthogonal to the z-axis.
Chamber portions 37 and 38 are provided on the head so that the electrode portion 31 (first electrode portion) is in contact with and perpendicular to the brain surface. The electrode portion 31 is formed by the chamber portions 37 and 38. The measurement is performed in the held state. The brain signal detected in the electrode part 31 can be taken out from the connection terminal part 31TB. The other electrode part 32 (second electrode part) is disposed between the electrode part 31 and the brain.

FIG. 2B shows a side view (temporal head) side of the brain showing an arrangement example of the multipoint potential measurement unit as an overhead view.
As shown in FIG. 2 (B), for example, the multipoint potential measuring unit 3 is arranged from the lower temporal lobe related to the visual acuity to the ventral side of the occipital lobe from the upper temporal groove L related to vision. It is arranged in the area Za of the visual association area. For example, the multipoint potential measurement unit 3 arranges the TE field in the visual association field region so as to be included in the detection range. Further, for example, the area Zb including the visual association area is an area centered on the TE area, and at least one of the visual related areas in the temporal lobe or the ventral side of the occipital lobe may be included. As described above, the multipoint potential measuring unit 3 can detect the visual stimulus of the subject (subject) by being provided in the cerebral visual cortex including the visual association area or the cerebral association area. The cerebral visual cortex includes a primary visual cortex (V1 field) and a visual association cortex. The cerebral association area indicates an area including the cerebral cortex other than the primary motor area and the primary sensory area (including the primary visual area).
In FIG. 2B, TE, TEO, V1, V2, and V4 are TE field, TEO field, primary visual field (V1 field), secondary visual field (V2 field), and fourth visual field, respectively. (V4 field) is shown, and it is made to correspond to the general name which shows each part of the brain. The cerebral visual association area is shaded.

As described above, the multipoint potential measuring unit 3 in the present embodiment includes the electrode unit 31 (first electrode unit) that detects LFP and unit action potential in the brain, and the electrode unit 32 (second electrode) that detects ECoG on the brain surface. 2 types of electrode portions).
The multipoint potential measuring unit 3 shown in FIG. 2 (B) shows the positions where the respective units are arranged in an expanded manner. In order from the brain side, the electrode unit 32, the chamber unit 38, the chamber unit 37, and the electrode unit 31 are sequentially displayed. It shows that it is arranged. The electrode part 32 includes a head part 32H provided with an electrode 32T and a node part 32N from which wiring is drawn out from the head part 32H, and is formed on the same film-like substrate part 32B. In short, the head part 32H of the electrode part 32 is located in the head because it is provided in contact with the brain. Therefore, a node portion 32N is provided in a gap provided between the chamber portion 37 and the chamber portion 38, and brain signals are taken out of the head via the node portion 32N.
Moreover, the brain signal detected in the electrode part 31 can be taken out from the connection terminal part 31TB.
In the present embodiment, the measurement is performed by the multipoint potential measuring unit 3 provided on the head as described above.

As shown in FIG. 3, the electrode unit 31 detects LFP and unit action potential in the brain.
The electrode portion 31 includes a cylindrical protection portion that holds the microelectrode 31T at a predetermined position, and the end surface on the (+ Z) axis side (a surface along the (xy) plane in the drawing) of the protection portion is Z. A plurality of microelectrodes 31T protruding in the axial direction are arranged. A connection terminal portion 31TB is provided on the end surface of the protective portion in the electrode portion 31 on the (−Z) axis side, and signals from the microelectrode 31T can be respectively taken out.
The position where the electrode part 31 is arranged in the brain corresponds to the same position as the electrode part 32 arranged in the brain. The electrode part 32 is placed over the brain so as to contact the brain, and the electrode part 31 is provided on the electrode part 32. Here, since the (+ Z) axis direction is the direction of the brain to be detected, the microelectrode 31T is provided so as to protrude in the brain direction. In short, the microelectrode 31T of the electrode unit 31 is provided in a state of being inserted into the brain.

FIG. 4 is a diagram showing the electrode part 32 in the present embodiment.
The electrode part 32 shown by this FIG. 4 is formed of the board | substrate part 32B which has comprised the film form. The substrate part 32B includes a plurality of rectangular electrodes 32T arranged along one surface of the substrate part 32B. For example, the size of the electrode 32T is 0.05 to 1 mm (millimeters) square (x = y = 0.05 to 1), and the plurality of electrodes 32T are arranged in the y-axis direction in the x-axis direction. The interval is 0.5 to 5 mm (millimeter) (dx = 0.5 to 5), and the interval at which the electrodes 32T in the row 32C are arranged in the y-axis direction is 0.5 to 5 mm (millimeter) (dy = 0) 5 to 5). The above dimensions are merely examples, and can be adjusted to a suitable size according to the size of nerve cells in the brain of the subject and the type of signal to be detected.
In order to record stably with the electrode part 32 left in place for a long period of time, the size of the electrode 32T (electrode surface area) is about 0.1 to 1 mm (millimeter), and the distance between the electrodes 32T is 1 To about 2.5 mm is desirable. Further, the larger the number of poles, the higher the spatial density for detecting signals, so it is desirable that the number is large. However, there are physical limitations (limitations such as securing connectors, amplifiers, wireless communications, and power sources for driving them), and there is a limit to the range that can easily be multipolarized. For example, in order to efficiently collect information under the above conditions, the number of poles should be limited to about 256 channels (channels).

FIG. 5 is a diagram showing the positions of the electrode part 31 and the electrode part 32 in the present embodiment.
FIG. 5A is a photograph taken from the electrode part 32 side while maintaining a state in which the electrode part 31 and the electrode part 32 are combined as in the measurement. FIG. 5B is an enlarged photograph of part of FIG. FIG. 5C is a diagram schematically showing the part of the photograph shown in FIGS. 5A and 5B.
Each figure shows a state in which the electrode part 32 is viewed from the brain. In short, the electrode 32T provided on the surface of the substrate part 32B of the electrode part 32 and the microelectrode 31T (electrode part 31) protruding from the surface of the substrate part 32B can be confirmed.
Note that the distance dx in the x-axis direction of the row 32C is determined so as not to interfere with the position of the microelectrode 31T of the electrode portion 31 and to contact each other. The wiring of the electrode portion 31 is provided within a predetermined width of the row 32C, and a comb-like space 32S is provided between the rows 32C. In this space 32S, the microelectrode 31T of the electrode part 31 is provided.

Thus, the microelectrode 31T of the electrode part 31 is provided so as to protrude from one surface of the substrate part 32B on which the electrode part 32 is provided. In the present embodiment, the electrode portion 31 has the main body portion of the electrode portion 31 disposed on the back side of one surface of the substrate portion 32B on which the electrode portion 32 is provided, and the microelectrode 31T of the electrode portion 31 is provided. Is provided so as to penetrate the one surface from the back side of the one surface of the substrate portion 32B and protrude beyond the one surface of the substrate portion 32B.
Therefore, in the multipoint potential measurement unit 3, the electrode unit 31 for detecting the first brain signal (for example, LFP) by the LFP method is arranged at the first measurement point (in the brain), and the second brain signal by the ECoG method ( For example, the electrode part 32 for detecting ECoG) can be arranged at the second measurement point (brain surface).

Thus, by separating the electrode part 31 and the electrode part 32, the multipoint potential measurement part 3 can detect the brain signal and the brain surface signal independently.
One surface of the film-like substrate portion 32B on which the electrode portion 32 is provided is in contact with the brain. Further, in the electrode part 32, the electrode 32T is a flexible thin film electrode (about 20 microns), the biocompatibility of the material of the electrode part 32 is high, and a gap or surface is formed on the surface of the substrate part 32B. By providing a through-hole, it is more easily fitted to the brain surface. Therefore, the electrode part 32 is difficult to shift with respect to the brain. By providing the electrode of the electrode part 31 so as to penetrate the substrate part 32B (film), the electrode part 31 is not easily displaced with respect to the substrate part 32B (film) on which the electrode part 32 is provided. With this configuration, the electrode portion 31 is less likely to be displaced with respect to the brain position, and can be detected stably.

  As shown in FIG. 5, a plurality of terminals (microelectrodes 31T) provided in the electrode portion 31 are arranged according to a lattice. A unit cell is shown with reference numeral 31SQ. A plurality of terminals (electrodes 32T) provided in the electrode part 32 are arranged according to a lattice. A unit cell is indicated by reference numeral 32SQ. As shown in FIG. 5, there is a region that partially overlaps the region of the unit cell 31SQ and the region of the unit cell 32SQ.

  In short, a unit lattice SQ31 (at least a part of polygons) of a lattice (first polygon) formed by the positions of a plurality of terminals (microelectrodes 31T) provided in the electrode portion 31 and the electrode portion 32 are provided. The unit lattice SQ32 (at least a part of the polygons) of the lattice (second polygon) formed by the positions of the plurality of terminals (electrodes 32T) is a position where one surface of the substrate portion 32B is viewed in plan view. It is arranged so that it can overlap. By arranging the mutual electrode portions (the microelectrode 31T and the electrode 32T) in this way, brain signals in the same region can be detected in multiple ways by the respective electrode portions.

Next, returning to FIG. 1, a configuration example of the intention transmission support apparatus 100 according to the present embodiment will be described in detail.
The above-mentioned region extracted from the brain surface signal and the intracerebral signal that are acquired repeatedly according to a predetermined trial period in the region including the cerebral visual cortex or the cerebral association cortex in the subject's brain The visual stimuli that the subject senses or perceives is identified from the results calculated based on the information that includes the features derived from the brain surface signals and the features derived from the brain signals in the above region Output information. For example, the range of the region is determined so as to include the visual association area among the cerebral visual cortex or the cerebral association area.

  The intention transmission support device 100 shown in FIG. 1 includes a multipoint potential measurement unit 3, a feature amount extraction unit 4, an arithmetic processing unit 5, a storage unit 6, and a display device 9.

The multipoint potential measurement unit 3 in the present embodiment obtains the electrical characteristics of the brain at the plurality of measurement points from a plurality of types of brain signals corresponding to the plurality of measurement points in the region including the visual association area of the brain.
The multipoint potential measurement unit 3 includes a first brain signal measured by a first measurement unit corresponding to the first measurement point among a plurality of measurement points in a region including the visual association area of the brain, and the plurality of measurement points. The second brain signal measured by the second measuring means corresponding to the second measurement point is detected by being included in the plurality of types of brain signals corresponding to the plurality of measurement points.
As a more specific example, the multipoint potential measurement unit 3 includes a second signal in the brain signal measured by the first measuring means in the brain of the region including the visual association area of the brain, and the second in the brain surface of the region. The electrical characteristics of the brain at the measured position are obtained from a plurality of types of brain signals detected during the trial period, including the brain surface signal measured by the measuring means.
For example, the multipoint potential measuring unit 3 uses a local field potential (LFP) method as the first measuring means. The local field potential (LFP) method can be measured more stably than the UNIT method that measures a single cell potential.
For example, the multipoint potential measurement unit 3 uses a cortical electroencephalogram (ECoG) method as the second measurement means. This cortical electroencephalogram (ECoG) method can be stably measured over a long period of time.

The multipoint potential measurement unit 3 in the present embodiment includes an electrode unit 31, an electrode unit 32, a signal conversion unit 33, and a signal conversion unit 34.
The electrode unit 31 detects a plurality of the first brain signals. For example, the electrode unit 31 detects a local field potential (LFP). The electrode unit 32 detects the plurality of second brain signals. For example, the electrode unit 32 detects cortical brain waves (ECoG). The signal conversion unit 33 performs signal conversion for enabling output of information corresponding to the electrical characteristics indicated by the local field potential (LFP) detected by the electrode unit 31. For example, the signal conversion unit 33 includes a low-pass filter that passes a predetermined frequency band, an amplifier, an analog / digital conversion unit 33d, and the like. The amplifier amplifies the signal intensity of the brain signal (local field potential (LFP)) that has passed through the low-pass filter, and the analog / digital conversion unit 33d converts the amplified signal into discrete-time digital information. Desirably, the signal converter 33 can process the signals of the electrodes in parallel.

The signal conversion unit 34 performs signal conversion for enabling output of information corresponding to the electrical characteristics indicated by the cortical brain wave (ECoG) detected by the electrode unit 32. For example, the signal converter 34 includes a low-pass filter that passes a predetermined frequency band, an amplifier, an analog / digital converter 34d, and the like. The amplifier amplifies the signal intensity of the brain signal (cortical brain wave (ECoG)) that has passed through the low-pass filter, and the analog / digital conversion unit 33d converts the amplified signal into digital information in discrete time. Desirably, the signal conversion part 34 shall process the signal of each electrode in parallel.
Note that the signal conversion unit 33 and the signal conversion unit 34 cause the analog / digital conversion unit 33d and the analog / digital conversion unit 34d to convert a signal that has been sampled in a period determined as the same trial period. The signals converted by the analog / digital conversion unit 33d and the analog / digital conversion unit 34d are signals detected at the same time by the corresponding electrodes, and are time-series signals for the respective electrodes. The sampling timings in the signal converter 33 and the signal converter 34 may be synchronized.

  For example, the multipoint potential measurement unit 3 uses a change in potential (visual evoked potential) associated with visual stimulation at each measurement point as the electrical characteristics of the brain at a plurality of measurement points.

The feature quantity extraction unit 4 in the present embodiment extracts a feature quantity according to the electrical characteristics of the brain obtained by the multipoint potential measurement unit 3.
The feature amount extraction unit 4 includes a frequency conversion unit 41, a frequency conversion unit 42, and a feature amount generation unit 43. The frequency conversion unit 41 converts a time-series signal obtained from a local field potential (LFP) detected by the electrode unit 31 into a frequency domain signal. The frequency conversion unit 42 converts a time-series signal obtained from a cortical brain wave (ECoG) detected by the electrode unit 32 into a frequency domain signal.

The feature quantity generation unit 43 extracts a feature quantity derived from the local field potential (LFP) and a feature quantity derived from the cortical brain wave (ECoG).
When extracting the above-described feature amount, the feature amount generation unit 43 extracts a local field potential (LFP) time series signal, a frequency domain signal converted from the simultaneous series signal, a cortical brain wave (ECoG) time series signal, The feature amount is extracted based on the frequency domain signal converted from the simultaneous sequence signal.
For example, the feature quantity generation unit 43 in the present embodiment converts a feature quantity derived from the local field potential (LFP) into a time series signal of the local field potential (LFP) and a frequency domain signal converted from the simultaneous series signal. Extract based on In addition, the feature value generation unit 43 in the present embodiment is based on a feature value derived from a cortical brain wave (ECoG) based on a time series signal of the cortical brain wave (ECoG) and a frequency domain signal converted from the simultaneous signal. To extract.
The feature value generation unit 43 of the feature value extraction unit 4 uses a feature value corresponding to the electrical characteristics of the brain obtained by the multipoint potential measurement unit 3 as a frequency component included in a brain signal (for example, LFP and ECoG). And the time from the predetermined time to the time when the response of the brain signal is detected. Note that the latency may be used as the time from the predetermined time to the time when the response of the brain signal is detected.

The feature amount extraction unit 4 determines a detection frequency range so as to include at least a main frequency component included in the brain signal, and determines a plurality of frequency ranges so as to include a part of the determined detection frequency range. For example, the plurality of frequency ranges determined to include a part of the detection frequency range may be set according to the frequency component of the detected signal, or may be calculated arithmetically so as to be a predetermined interval. Also good.
Further, the feature amount extraction unit 4 determines a detection time range so as to include at least a period from a predetermined time before detecting the response of the brain signal until the response of the brain signal is detected. A plurality of time ranges are defined so as to include a part of the detection time range. For example, the plurality of time ranges determined to include a part of the detection time range may be set according to the time until a brain response is detected, or may be set at a predetermined interval. Arithmetic may be obtained.

The feature amount extraction unit 4 refers to a range limited by a combination of any of the frequency ranges determined as described above and any of the time ranges determined as described above, and the brain signals included in the range A representative value of the signal intensity is included in the feature amount. Note that a combination of any of the frequency ranges determined as described above and any of the time ranges determined as described above is determined in advance.
The feature amount extraction unit 4 divides the detection frequency range into a predetermined number, and the plurality of frequency ranges are determined so that the divided ranges do not overlap each other. In this way, by defining a plurality of frequency ranges so that the ranges do not overlap with each other, confusion between different frequency components is reduced, so that the detection sensitivity of specific components is increased.
In addition, the feature amount extraction unit 4 has a time range determined according to a determined time width in the above detection time range, for example. The above time range is a so-called sliding time window that is known as a spatio-temporal filtering method such as moving average (MA), and is adjacent to each other in the order in which they are arranged according to the passage of time. The two matching time ranges are arranged such that the time ranges overlap each other.
As described above, by overlapping the time regions by the sliding time window technique, it is possible to reduce variation in detection sensitivity and increase the signal body noise ratio (S / N ratio).
The detection time ranges may be divided into a predetermined number, and the plurality of time ranges may be determined so that the ranges do not overlap each other.

The feature amount extraction unit 4 refers to a range limited by a combination of any of the frequency ranges determined as described above and any of the time ranges determined as described above, and the brain signals included in the range Is calculated as a representative value of the signal strength of the brain signal.
As described above, the feature amount extraction unit 4 determines the above calculation result as a feature amount corresponding to the same range, and outputs it included in the feature amount used in the subsequent processing.

The arithmetic processing unit 5 extracts the features extracted from the brain surface signal and the intracerebral signal that are repeatedly acquired according to a predetermined trial period in a region including the cerebral visual cortex or cerebral association cortex in the subject's (subject) brain. Process according to the amount. The extracted feature value includes a feature value derived from the brain surface signal of the region and a feature value derived from the brain signal of the region. The arithmetic processing unit 5 outputs information that identifies the visual stimulus that the subject senses or images from the result calculated based on the information including these feature amounts.
The arithmetic processing unit 5 includes a correspondence learning unit 51 and a real-time intention transmission unit 52.
The correspondence learning unit 51 of the arithmetic processing unit 5 selects, in the process of selecting the associated image from the plurality of predetermined images, the brain at the measurement point of the brain detected along with the display of the image. The processing variables are optimized by associating the characteristic values of the electrical characteristics with the images.
The real-time intention transmission unit 52 of the arithmetic processing unit 5 performs a process of selecting an image associated with the extracted feature amount from a plurality of predetermined images.

The storage unit 6 stores a variable used for processing by the arithmetic processing unit 5 that selects the associated image from among a plurality of images in association with identification information indicating the plurality of images.
The display device 9 includes a display unit that can display image information, for example. The display device 9 displays image information corresponding to the information supplied from the arithmetic processing unit 5 on the display unit provided. The display device 9 may display image information generated based on information supplied from the arithmetic processing unit 5. Examples of the image information to be generated include characters and icons. More specifically, the display device 9 may have a function such as an editor (a word processor, a mail terminal, a personal computer, etc.), or may be a communication device terminal using icons. The display device 9 may be configured separately from the main body of the intention transmission support apparatus 100, or may be an external device of the intention transmission support apparatus 100.
The above is an example of the configuration of the intention transmission support apparatus 100 in the present embodiment.

Next, the operation principle of the intention transmission support apparatus 100 in this embodiment will be described.
The case where the intention transmission support apparatus 100 of this embodiment decodes visual image information, such as a character and an icon which a test subject (human) envisions in the head, in real time based on the brain signal by the nerve activity of a brain is illustrated.
The processing performed by the intention transmission support device 100 according to the present embodiment can be broadly divided into the following two stages.
The first stage process is a “machine learning phase” process. The second stage process is a “real-time communication phase” process.

First, the first stage process, that is, the “machine learning phase” process will be described.
In this machine learning phase, for example, while the display device 9 visually presents image information selected from image information indicating several tens of types of characters or icons one by one, the multipoint potential measuring unit 3 is 1 Responses of cortical electroencephalogram (ECoG) and brain local collective potential (LFP) are simultaneously measured at a sampling frequency of about ˜2 kHz (kilohertz). The multipoint potential measuring unit 3 amplifies the intensity (voltage) of the signal measured at the recording point (recording channel) corresponding to each electrode. The analog / digital converter 34d generates digitized time-series potential data from the amplified signal. The multipoint potential measurement unit 3 supplies the generated time series potential data to the feature amount extraction unit 4. The feature amount extraction unit 4 generates a spectrum of event-related potentials from time series potential data by a technique such as Fourier transform. The feature quantity extraction unit 4 divides the frequency component of the spectrum of the event-related potential and the above-described time-series potential data according to a predetermined division rule, and extracts the feature quantity of the spatiotemporal domain.

For example, when the time series potential data of the spectrum of the event-related potential is divided into five stages and the frequency is divided into seven stages, 35 (5 × 7 = 35) feature amounts can be obtained for one electrode (one channel). An entire feature amount derived from a plurality (large number) of recording channels included in the occipital lobe to the temporal lobe, which is a higher visual association area of the cerebrum, is defined as one vector (feature vector). For example, if the total number of recording channels is 128 channels, a feature quantity vector of 4480 dimensions (35 (pieces) × 128 (channels) = 4480 dimensions) is generated. This feature quantity vector is set as the feature quantity generated by the feature quantity extraction unit 4.
In the case of the above example, the feature amount generated by the feature amount extraction unit 4 can represent a response to each stimulus, that is, a stimulus corresponding to each presented image information, using a 4480-dimensional feature amount vector. it can.

  As described above, the method according to this embodiment is optimized so that the response to a specific stimulus can be separated from the response to another stimulus among the responses indicated by the high-order feature quantity vector. A hyperplane capable of separating a response to a specific stimulus and a response to another stimulus is defined in a feature vector space, and an optimized hyperplane is obtained using a machine learning algorithm. A known method such as a support vector machine is used as the machine learning algorithm.

Next, the process of the second stage, that is, the process of “intention transmission phase” will be described.
In this communication phase, according to the decision algorithm identified based on the hyperplane optimized in the above-mentioned “machine learning phase”, the character or icon that the subject wants to select next is selected from the feature vector of the neural activity being observed. Decodes the type in real time.
The arithmetic processing unit 5 supplies the decoded character and icon information to the display device 9 and causes the display device 9 to display the supplied information.
For example, when the display device 9 has a function such as an editor (a word processor, a mail terminal, a personal computer, etc.) and the decoded information is a character, the arithmetic processing unit 5 displays the type of the decoded character on the display device 9. To supply. The display device 9 displays information corresponding to the decoded character type.
Alternatively, the decoded information may be other than characters. For example, in this case, the display device 9 is a communication device terminal that uses icons instead of supplying character types. The display device 9 displays an icon at a predetermined position and displays an icon selected according to information supplied by the arithmetic processing unit 5. The subject confirms the information displayed on the display device 9 and recognizes that the selected icon matches the intended one. The intention transmission support apparatus 100 selects the one of the plurality of (for example, three or more) icons displayed by the display device 9 from the state of the visual stimulus imaged by the subject at that time. It may be determined whether or not.
Further, the intention transmission support apparatus 100 (the arithmetic processing unit 5) transmits the decoded icon information to another machine or computer by transmitting it to the other machine or computer. Processing according to the icon can be instructed. In addition, by supplying the decoded character information to another machine or computer, the subject can instruct the other machine or computer to write the character he / she wants to spell, and the subject's intention Helps communicate.
The display device 9 may be an external device separated from the intention transmission support device 100.
The above is the operation principle of the intention transmission support apparatus 100.

Next, the signal detection process in this embodiment will be described with reference to FIGS. Here, the cortical electroencephalogram detected by the electrode array 10 as a target of the signal detection process will be described.
FIG. 6 is a diagram showing processing for cortical electroencephalograms detected by the electrode array.
FIG. 6 (a) is a waveform showing a change in signal intensity of a brain signal (for example, cortical brain wave) after the start of examination for presenting a visual stimulus to a subject, and each waveform is arranged in the order of numbers for identifying electrodes. According to. In short, the vertical axis indicates the signal intensity (x) of a brain signal (for example, cortical electroencephalogram) detected by each electrode. The horizontal axis shows the passage of time. Further, the fk (and f (k + 1)) axis extending in the depth direction shows the waveform W _i and the waveform W _{(i + 1)} of each electrode arranged in the order of numbers (... i, (i + 1)...) For identifying the electrodes. ing. The signal strength of electrode i at time k is x _i (k), and the signal strength of electrode (i + 1) is x _{(i + 1)} (k). In addition, when the starting point of the time axis in FIG. 6A is the time when an image is presented to the subject and a visual stimulus is given, the time (T _i , T _{(i + 1)} ) until a response appears in the brain signal is obtained. Latency. The latency value differs depending on the position to be detected.

  FIG. 6B shows a time window for extracting a frequency component in the vicinity of a predetermined time from the waveform of each electrode shown in FIG. The vertical axis represents time window characteristics (gain characteristics: G), and the horizontal axis represents the passage of time. The time window shown in the figure shows the waveform of a time window known as a Hanning window as an example. For example, the time width Tw of the time window is 50 ms (milliseconds). A desired signal component can be detected by determining the length of the time window according to the characteristics of the signal to be analyzed. In this way, a signal near a predetermined time is calculated by multiplying the amplitude of the waveform of each electrode shown in FIG. 6A by the characteristics of the time window.

Next, FIG. 6 (c), the result of the signal represented by the waveform W _i of FIG. 6 (a), based on a result of the limited by the time window shown in FIG. 6 (b), and extracts a frequency component Indicates. The vertical axis indicates the signal intensity corresponding to the frequency component of the cortical electroencephalogram detected by each electrode. An axis extending in the depth direction (f-axis) indicates a frequency. For example, frequency components at a predetermined time in the discrete time information indicated by the waveform shown in FIG. 6A are shown according to each time.
For example, (X _ik (0), X _ik (1),...) Is obtained in order from the lowest frequency at time k. In addition, (X _{i (k + 1)} (0), X _{i (k + 1)} (1),...) Are obtained in order from the lowest frequency at time (k + 1).

Here, with reference to FIG. 7, an example of the result of the process shown in FIG. 6 will be shown, and the description of feature quantity extraction will be supplemented.
FIG. 7 is a diagram illustrating an example of a result of the process illustrated in FIG.
In the coordinate system shown in FIG. 7, the horizontal axis represents the time domain (for example, 0 to 600 ms (milliseconds)), and the vertical axis represents the frequency domain (for example, 4 to 100 Hz). The shade in the first quadrant of this coordinate system indicates the signal intensity at that time, and the darker the color, the stronger the signal intensity. For example, from 100 ms to 200 ms in the time domain, there is a region that is particularly dark in the vicinity of 10 Hz to 20 Hz in the frequency domain. ing. In short, it can be said that almost no event was detected in the region exceeding 30 Hz at the same time.

Thus, the distribution of the concentration appearing in a specific region shows a response based on the visual stimulus that the subject perceived at that time. However, when such a density distribution is processed as it is, it is equivalent to processing so-called image information as image data. Therefore, detailed information can be handled, but more information than necessary is handled.
Therefore, in the present embodiment, as shown below, the time domain and the frequency domain are divided within a range in which an event can be identified, and feature amounts are handled according to information in the area.

With reference to FIG. 8, a description will be given of a process of extracting the frequency domain feature value converted by the conversion process to the frequency domain shown in FIG.
FIG. 8 is a diagram illustrating a process of extracting feature amounts from frequency domain components converted by the frequency domain conversion process shown in FIG. 6.

Figure 8 shows a process for the signal waveform _{W i} in FIG. 6 (a). The horizontal axis indicates the passage of time, and the vertical axis indicates the signal intensity. An axis extending in the depth direction (f-axis) indicates a frequency.
(T0, T1, T2, T3,...) Shown on the horizontal axis indicate threshold values that divide the time range on the time axis. Further, (FT0, FT1, FT2, FT3,...) Shown on the f-axis respectively indicate threshold values for dividing the frequency range on the frequency axis. In the description of the time domain in FIG. 8, it is assumed that the time ranges do not overlap. However, as described above, the time ranges may be determined to overlap, and moving average processing with a predetermined time width may be performed. Good.

  A representative value in the range from time T0 to T1 and a frequency in the range from FT0 to FT1 is indicated by P11, a representative value in the range from FT1 to FT2 is indicated by P12, and the frequency is in the range from FT2 to FT3. Is represented by P13.

  A typical value in the range from time T1 to T2 and a frequency in the range from FT0 to FT1 is indicated by P21, a typical value in the range from FT1 to FT2 is indicated by P22, and the frequency is in the range from FT2 to FT3. Is represented by P23.

  A representative value in the range from time T2 to T3 and a frequency in the range from FT0 to FT1 is indicated by P31, a representative value in the range from FT1 to FT2 is indicated by P32, and the frequency is in the range from FT2 to FT3. Is represented by P33.

  For example, the representative value indicated by P11 is an average value (or total value) of signal components of spatio-temporal frequencies that exist in the range from time T0 to T1 and in the frequency range from FT0 to FT1. The range from time T0 to T1 includes times (k-1), k, and (k + 1), and the signal intensity calculated as the frequency component at each time (see FIG. 6C). Calculate based on

  The representative value of each range defined by the time range and the frequency range can be used as the feature amount of each range. In other words, the feature quantities from FT0 to FT3 are collectively shown from time T0 to T3 shown in this figure. In the case shown in this figure, (P11, P12, P13, P21, P22, P23, P31, P32, P33) is the feature amount in the signal obtained from the electrode i.

  In this way, the feature amount of each range determined by the time range and the frequency range is set to the average value of the signal intensity of each signal detected in the range, thereby reducing the influence of random fluctuation due to noise. Can be reduced. In addition, since fluctuations due to interference between signals can also be suppressed, the reproducibility of the representative value is increased compared to the case of simply using the peak value. The accuracy of discrimination can be increased.

  The feature amounts in the signals obtained from the other electrodes are calculated by the same method as that shown in FIGS. 6 to 8, and the feature amounts from the respective electrodes provided on the brain surface are collected. The feature amount is derived from.

  6 to 8, the feature amount derived from the brain surface signal has been described. However, the feature amount derived from the brain signal can be extracted in the same procedure. In the following explanation, in order to distinguish and explain the feature quantity derived from the brain surface signal and the feature quantity derived from the brain signal, the feature quantity derived from the brain signal is referred to as “feature quantity A”, The derived feature amount is referred to as “feature amount B”.

(Machine learning processing based on feature quantities derived from brain signals (feature quantity A) and feature quantities derived from brain surface signals (feature quantities B))
Next, the machine learning process in this embodiment will be described.
The machine learning process in the present embodiment will be described by taking as an example the case where a support vector machine (SVM), which is known as one method of machine learning, is applied.
The SVM uses the acquired sample data as a learning sample, and identifies desired data in the sample data and sample data other than the desired data. The SVM generates and learns a boundary for identifying desired data from the sample data based on the sample data in advance. In addition, detailed description of SVM itself is omitted here.

Hereinafter, in the present embodiment, the extracted feature amounts (feature amount A and feature amount B) are used as SVM sample data (input vector).
For example, the feature quantity A extracted by the above-described extraction process from the brain signal detected by a specific electrode (i) is shown as a column vector AV _i as shown in Expression (E1). The feature amount PA is

AV _i = [PA ₁₁ ... PA _1N PA ₂₁ ... PA _2N ... PA _M1 ... PA _MN ] ^T
... Formula (E1)

This equation (E1) shows the column vector AV _i in the form of a transposed matrix. The subscript “T” indicates a transposed matrix. In this formula (E1), PA ₁₁ to PA _1N , PA ₂₁ to PA _2N ,..., PA _M1 to PA _MN are elements of the feature amount A, M represents the number of divisions on the time axis, and N represents the frequency Indicates the number of axis divisions. In this way, feature quantities including a large number of elements can be represented by vectors for each electrode. The other electrodes are also represented by vectors in the same manner as described above.

  As shown in the equation (E1), the brain signal obtained from one electrode is the feature quantity A extracted from the brain signal detected by each electrode as a column vector GAV as shown in the equation (E2). Show.

GAV = [AV ₁ AV ₂ ... AV _N ] ^T ... Formula (E2)

This equation (E2) shows the column vector GAV in the form of a transposed matrix. In this equation (E2), each of AV ₁ to AV _N represents a vector of the feature amount A of each electrode, and N represents the number of brain signals to be processed. In this way, feature quantities extracted from brain signals obtained from a plurality of electrodes can be collectively shown as a vector.

  Further, in the case of the brain surface signal, the feature amounts extracted from the brain signals obtained from the plurality of electrodes are summarized as shown in Expression (E3), as in the case of the intracerebral signal.

GBV = [BV ₁ BV ₂ ... BV _N ] ^T ... Formula (E3)

  In the formula (E3), N represents the number of brain surface signals to be processed. Here, the feature amount A and the feature amount B shown in the equations (E2) and (E3), respectively, are combined into one vector G1V as shown in the equation (E4).

G1V = [GAV GBV] ^T ... Formula (E4)

  The G1V is used as SVM sample data (input vector) to optimize the parameters of the SVM identification function. The calculated parameters (variables) are stored in the storage unit 6 and the learning process is finished.

Subsequently, the real-time identification processing phase will be described.
The brain response (electrical characteristic) at the timing of identification is detected in the same manner as in the machine learning phase. Based on the detected electrical characteristics, the feature quantity of the timing is extracted.
Based on the variables learned in the machine learning phase, it is calculated as which input vector the feature quantity extracted at each timing is classified. Information on the input vector corresponding to the extracted feature amount is output according to the calculation result.
In this way, even when the number of input vectors increases for detailed identification, the processing can be simplified by combining them into one.

With reference to FIG. 9, the process in this embodiment is demonstrated.
FIG. 9 is a flowchart showing processing in the present embodiment.
Hereinafter, a case where SVM is applied to the machine learning process will be exemplified.

First, the intention transmission support apparatus 100 performs machine learning processing in the SVM according to the following procedure (steps Sa11 to Sa16).
The multipoint potential measuring unit 3 simultaneously measures the brain response indicated by the LFP and ECoG (step Sa11). The feature amount extraction unit 4 calculates the frequency spectrum of LFP and ECoG (step Sa12). The feature quantity extraction unit 4 extracts a feature quantity derived from LFP (step Sa13), and extracts a feature quantity derived from ECoG (step Sa14). The arithmetic processing unit 5 optimizes the variable for setting the hyperplane of the SVM based on the feature amount derived from LFP and the feature amount derived from ECoG, and stores the optimized variable in the storage unit 6 (step Sa16). ).

Next, the intention transmission support apparatus 100 performs a real-time identification process using the result of the machine learning process in the SVM according to the following procedure (steps Sa21 to Sa28).
The multipoint potential measuring unit 3 simultaneously measures brain responses indicated by LFP and ECoG (step Sa21). The feature amount extraction unit 4 calculates the frequency spectrum of LFP and ECoG (step Sa22). The feature quantity extraction unit 4 extracts a feature quantity derived from LFP (step Sa23), and extracts a feature quantity derived from ECoG (step Sa24). The arithmetic processing unit 5 uses the variables stored in the storage unit 6 to perform identification processing based on the feature quantity derived from LFP and the feature quantity derived from ECoG (step Sa26). The arithmetic processing unit 5 outputs the identification result (step Sa27).

  The arithmetic processing unit 5 determines whether to perform the learning process again based on the operation input information (step Sa28). If it is determined in step Sa28 that the learning process is not performed again (step Sa28: No), the process proceeds to step Sa21, and a series of real-time identification processes are repeated. On the other hand, when it is determined in step Sa28 that the learning process is performed again (step Sa28: Yes), the real-time identification process is terminated.

  In the above processing, the example in which the machine learning processing and the real-time identification processing are separately performed has been shown. However, the quality of the identification result of the real-time identification processing is sequentially stored in the storage unit 6 as history information. When the identification rate falls below a predetermined identification rate, the machine learning process may be performed again. Alternatively, it may be performed at a predetermined timing.

With reference to FIG. 10, the result determined by the intention transmission support apparatus shown in this embodiment will be described.
FIG. 10 is a diagram showing a determination result by the intention transmission support apparatus shown in the present embodiment. The determination result shown in this figure shows the determination result of an experiment that uses a monkey (monkey) instead of the above-mentioned subject as an object and supports monkey communication. The type of monkey used as the subject in this experiment was macaque monkey, and one male and one female were targeted.
In FIG. 10, A to E, which are sequentially shown in the direction on the horizontal axis, indicate six groups that are grouped according to the characteristics of the image to be detected. The vertical axis indicates the probability that the result of certifying the image is recognized as correct. In short, the larger the value of the graph, the higher the probability of correct recognition. Each group shown as A to F is “being a face” (A), “being a body part (eg, hands or feet)” (B), “being inanimate (inanimate, creatures) Nothing) ”(C),“ Animal species of the face (whether it is a human face or a monkey face) ”(D),“ Direction of the face (for example, front, diagonal, etc.) ”(E) It shows in order.

Further, seven bar graphs g1 to g7 are shown side by side in each group. Each bar graph shows different conditions in the experiment and compares the differences.
The first three (g1, g2, g3) shown side by side show the results of experiments using only one type of signal, and the next four (g4, g5, g6, g7) The results of experiments using different types of signals are shown.
In the results of experiments using only one type of signal, g1 is “when only the ECoG signal is used as a detection signal”, g2 is “when only the unit signal is used as a detection signal”, and g3 is “LFP signal”. "Only the detection signal is used as a detection signal". Here, “ECOG signal” indicates a detection signal that detects the aforementioned ECoG, “unit signal” indicates a detection signal that detects the aforementioned unit action potential, and “LFP signal” indicates a detection signal that detects the aforementioned LFP. The unit action potential indicates the potential of a single cell. In the present embodiment, the unit action potential and LFP are detected as brain signals.
In the results of experiments using a plurality of types of signals, g4 is “when the unit signal and the LFP signal are used as detection signals” and g5 is “when the unit signal and the ECoG signal are used as detection signals” in order. , G6 indicates “when the LFP signal and the ECoG signal are detection signals” and g7 indicates “when the unit signal, the LFP signal, and the ECoG signal are detection signals”.

The following trends can be cited as common trends in each group.
(Part 1) The result of an experiment based on a plurality of types of signals tends to have a higher accuracy rate than the result of an experiment based on one type of signal. In particular, the results under the conditions of g5, g6, and g7 that combine the signals in the brain and the brain surface tend to have a higher accuracy rate than the results of experiments based on one type of signal.

(Part 2) In an experiment based on a plurality of types of signals, g4 is “when the unit signal and the LFP signal are detection signals”, g6 is “when the LFP signal and the ECoG signal are detection signals”, g7 As shown in “When a unit signal, an LFP signal, and an ECoG signal are used as detection signals”, when an “LFP signal” is included in a brain signal, the accuracy rate tends to increase stably. .

(Part 3) In an experiment based on one type of signal, g1: “when only the ECoG signal is used as a detection signal”, g2: “when only the unit signal is used as a detection signal”, and g3 “only the LFP signal” There is a tendency for the correct answer rate to increase in the order of “when the detection signal is used”. In the A group and the B group, g2: “When only the unit signal is used as a detection signal” has the lowest accuracy rate.

(No. 4) In the experiment based on one type of signal, the correct answer rate varies greatly depending on the target, but in the experiment based on a plurality of types of signals, the change in the correct answer rate due to the target is relatively small. small. From this, it can be presumed that the region and amount of reaction differ depending on the type of visual stimulus.

As described above, the intention transmission support apparatus 100 according to the present embodiment can support correct communication by accurately and stably reading a brain surface signal and a brain signal.
In addition, since it is extremely difficult to stably measure unit signals derived from the same nerve cell for several weeks or more with the current level of brain science, it is practically necessary to use LFP, ECoG and a combination thereof to support communication. This is the most promising condition for developing a device (method).

(Second Embodiment)
With reference to FIG. 1 to FIG. 12, a description will be given of a communication support device using visual images according to the present embodiment.
FIG. 11 is a functional block diagram illustrating the communication support apparatus based on visual images according to the present embodiment.
The intention transmission support device 100A shown in FIG. 11 includes a multipoint potential measurement unit 3, a feature amount extraction unit 4A, an arithmetic processing unit 5A, and a storage unit 6. The feature amount extraction unit 4A and the calculation processing unit 5A have the following differences from the feature amount extraction unit 4 and the calculation processing unit 5 of the intention transmission support apparatus 100 shown in FIG.

The feature amount extraction unit 4A includes a frequency conversion unit 41, a frequency conversion unit 42, and a feature amount generation unit 43A.
The feature value generation unit 43A includes a feature value in the electrical feature value of the local field potential (LFP), a feature value in the electrical feature value of the cortical brain wave (ECoG), and an electrical feature value in the local field potential (LFP). A feature amount is extracted as a result of the correlation processing based on the cortical electroencephalogram (ECoG) and the electrical feature amount.

The arithmetic processing unit 5A extracts each of the brain surface signal and the intracerebral signal, which are repeatedly acquired according to a predetermined trial period, in a region including the cerebral visual cortex or cerebral association cortex in the subject's (subject) brain. From the result of calculation based on information including the feature amount derived from the brain surface signal of the region and the feature amount derived from the intracerebral signal of the region, the subject (subject) senses or images Outputs information that identifies the visual stimulus that is present.
The arithmetic processing unit 5A includes a correspondence learning unit 51A and a real-time intention transmission unit 52A.
The correspondence learning unit 51A of the arithmetic processing unit 5A further obtains a feature amount derived from the correlation between signals of the brain surface signal and the intracerebral signal with respect to information targeted by the correspondence learning unit 51 of the arithmetic processing unit 5. Based on the included information, the process variables are optimized by the process of associating the characteristic amount of the electrical characteristics of the brain with the image.
The real-time intention transmitting unit 52A of the arithmetic processing unit 5A correlates a plurality of signals selected from the brain surface signal and the intracerebral signal with respect to information targeted by the real-time intention transmitting unit 52 of the arithmetic processing unit 5 Based on the information further including the feature quantity derived from the above, a process of selecting an image associated with the extracted feature quantity from among a plurality of predetermined images is performed. In the following description, the real-time intention transmission unit 52A of the arithmetic processing unit 5A is a signal of the brain surface signal and the brain signal with respect to information targeted by the real-time intention transmission unit 52 of the arithmetic processing unit 5. Based on information that further includes a feature amount derived from the correlation between the two, a case where processing for selecting an image associated with the extracted feature amount from a plurality of predetermined images is described as an example. To do.

Next, the feature amount extraction processing in the feature amount extraction unit 4A will be described.
The feature amount extraction process according to the present embodiment is different from the feature amount extraction process shown in FIGS. 6 to 8 in that a feature amount derived from the correlation between the brain signal and the brain surface signal is further extracted. In short, in the feature amount extraction processing in the present embodiment, the feature amount derived from the brain signal, the feature amount derived from the brain surface signal, and the feature amount derived from the correlation between the brain signal and the brain surface signal are calculated. Extract.
As described above, the feature amount derived from the intracerebral signal is described as “feature amount A”, the feature amount derived from the brain surface signal is described as “feature amount B”, and the brain newly added in the present embodiment The feature amount derived from the correlation between the internal signal and the brain surface signal will be described as “feature amount C”.

  First, a predetermined time range (for example, 0 to 300 ms (milliseconds)) after presentation of the visual stimulus in the LFP signal and the ECoG signal of one trial is selected as a target range, and the feature amount extraction unit 4A Reads out the time-series information based on the LFP signal and the ECoG signal from the storage unit 6 during this period.

  Next, among all the electrodes, a pair of a specific electrode 32T (ECoG electrode that records ECoG) in the electrode unit 32 and a specific microelectrode 31T (microelectrode that records LFP) in the electrode unit 31 is paired. Extract electrode combinations.

  Next, based on the time-series information of the LFP signal corresponding to the pair of electrodes y and the time-series information of the ECoG signal from the combinations extracted as the pair of electrodes, A correlation coefficient of time series information is calculated.

In the above case, for example, if the sampling frequency of the time series information is 1 kHz (kilohertz), the number of pieces of information in the period in 300 ms (milliseconds) is 300 (n = 300). One correlation coefficient is obtained by calculating the time series information of the LFP signal selected by 300, the time series information of the ECoG signal, and the correlation coefficient.
Furthermore, it implements about all the combinations of the said electrode combination used as a pair, and obtains the same correlation coefficient as the number of pairs.
Identification information for identifying the pair (a specific electrode 32T in the electrode part 32 and a specific microelectrode 31T in the electrode part 31) is defined. The correlation coefficient derived from the pair is associated with identification information for identifying the pair. The correlation coefficient and the identification information for identifying the pair are used as feature quantities derived from the correlation between the brain signal and the brain surface signal.

(The feature quantity derived from the intracerebral signal (feature quantity A), the feature quantity derived from the brain surface signal (feature quantity B), and the feature quantity derived from the correlation between the brain signal and the brain surface signal are referred to as “feature quantity C”. Machine learning process based on
Next, the machine learning process in this embodiment will be described.
In 2nd Embodiment, the process after Formula (E4) shown in 1st Embodiment is different. Subsequent processing will be described with reference to equations (E1) to (E3).

As described above, in addition to the feature quantity derived from the brain signal (referred to as feature quantity A) and the feature quantity derived from the brain surface signal (feature quantity B), the correlation between the signal of the brain surface signal and the brain signal is considered. There is a derived feature amount (feature amount C). The feature amount C is calculated as a correlation coefficient between the brain surface signal and the brain signal.
In the present embodiment, the feature amount derived from the brain signal (referred to as feature amount A), the feature amount derived from the brain surface signal (feature amount B), and the correlation between the brain surface signal and the brain signal. The feature amount (feature amount C) derived from is acquired. The acquired feature amount (feature amount A, feature amount B, and feature amount C) is used as sample data (input vector) of SVM.

  In the present embodiment, a column vector GCV having a feature amount derived from the correlation between the brain signal and the brain surface signal as an element is determined. For example, the element of the column vector GCV is a correlation coefficient derived from a pair that is a feature amount. The correlation coefficient is associated with identification information for identifying a pair, and is arranged according to the order of the identification information.

  Here, the feature amount A, the feature amount B, and the feature amount C shown in the equations (E2) and (E3) are combined into one vector G2V as shown in the equation (E5).

G2V = [GAV GBV GCV] ^T ... Formula (E5)

  The G1V is used as SVM sample data (input vector) to optimize the parameters of the SVM identification function. The calculated parameters (variables) are stored in the storage unit, and the learning process is finished.

Subsequently, the real-time identification processing phase will be described.
The brain response (electrical characteristic) at the timing of identification is detected in the same manner as in the machine learning phase. Based on the detected electrical characteristics, the feature quantity of the timing is extracted.
Based on the variables learned in the machine learning phase, it is calculated as which input vector the feature quantity extracted at each timing is classified. Information on the input vector corresponding to the extracted feature amount is output according to the calculation result.
Thus, even if the number of input vectors increases for detailed identification, the processing can be simplified by combining them into one.

With reference to FIG. 12, the processing in the present embodiment will be described.
FIG. 12 is a flowchart showing processing in the present embodiment.
Hereinafter, the case where the SVM is applied to the machine learning process will be described as an example, and the process in the intention transmission support device 100A will be described.

First, the intention transmission support device 100A performs machine learning processing in the SVM according to the following procedure (steps Sb11 to Sb16).
The multipoint potential measurement unit 3 simultaneously measures the responses of LFP and ECoG (step Sb11). The feature amount extraction unit 4A calculates the frequency spectrum of LFP and ECoG (step Sb12). The feature amount extraction unit 4A extracts a feature amount derived from LFP (step Sb13), extracts a feature amount derived from ECoG (step Sb14), and extracts a feature amount derived from the correlation between LFP and ECoG (step Sb14). Step Sb15).
The arithmetic processing unit 5A optimizes the variable for setting the hyperplane of the SVM based on the feature quantity derived from LFP, the feature quantity derived from ECoG, and the feature quantity derived from the correlation between LFP and ECoG. The converted variable is stored in the storage unit 6 (step Sb16).

Next, the intention transmission support apparatus 100 performs a real-time identification process using the result of the machine learning process in the SVM according to the following procedure (steps Sb21 to Sb28).
The multipoint potential measuring unit 3 simultaneously measures the responses of LFP and ECoG (step Sb21). The feature amount extraction unit 4A calculates the frequency spectrum of LFP and ECoG (step Sb22). The feature quantity extraction unit 4A extracts a feature quantity derived from LFP (step Sb23), extracts a feature quantity derived from ECoG (step Sb24), and extracts a feature quantity derived from the correlation between LFP and ECoG (step Sb24). Step Sb25). The arithmetic processing unit 5A uses the variables stored in the storage unit 6, based on the feature amount derived from LFP, the feature amount derived from ECoG, and the feature amount derived from the correlation between LFP and ECoG. Identification processing is performed (step Sb26). The arithmetic processing unit 5A outputs the identification result (step Sb27).

  The arithmetic processing unit 5A determines whether or not to perform the learning process again based on the operation input information (step Sb28). If it is determined in step Sb28 that the learning process is not performed again (step Sb28: No), the process proceeds to step Sb21, and a series of real-time identification processes is repeated. On the other hand, when it is determined by the determination in step Sb28 that the learning process is performed again (step Sb28: Yes), the real-time identification process is terminated.

  As described above, the communication support apparatus 100A of the present embodiment can support the communication of communication by accurately and stably reading by combining the brain surface signal and the brain signal.

  In the above processing, the case where the feature amount derived from the correlation between the brain surface signal and the intracerebral signal is used for calculating the feature amount derived from the correlation is exemplified. It is not limited to use a feature amount derived from the correlation of a plurality of signals selected from the brain signals. For example, a plurality of signals to be processed may be a plurality of signals selected from brain surface signals or a plurality of signals selected from brain signals. It can be selected appropriately.

  Further, the real-time intention transmitting unit 52A has a feature amount derived from the correlation of a plurality of signals selected from the brain surface signals with respect to information targeted by the real-time intention transmitting unit 52 of the arithmetic processing unit 5, and the brain The processing may be performed based on information each including a feature amount derived from the correlation of a plurality of signals selected from the internal signals.

  Further, the real-time intention transmitting unit 52A has a feature amount derived from the correlation of a plurality of signals selected from the brain surface signals with respect to information targeted by the real-time intention transmitting unit 52 of the arithmetic processing unit 5, and the brain The processing may be performed based on information each including a feature amount derived from the correlation of a plurality of signals selected from the internal signals.

Further, the real-time intention transmitting unit 52A may select and process at least one of the brain surface signal and the brain signal. For example, the real-time communication unit 52A extracts either one of the feature amount derived from the brain surface signal and the feature amount derived from the brain signal extracted from the selected one signal, and the selected one The processing may be performed based on information including a feature amount derived from the correlation of a plurality of signals selected from among the signals.
Further, the real-time intention transmitting unit 52A may switch between the case where one of the brain surface signal and the brain signal is used and the case where the brain surface signal and the brain signal are used as described above. Good. The switching in the above case may be switched based on predetermined setting information, or may be switched by selecting the case where the identification rate is highest according to the value of the identification rate in each case. .

  In the above processing, the example in which the machine learning processing and the real-time identification processing are separately performed has been shown. However, the quality of the identification result of the real-time identification processing is sequentially stored in the storage unit 6 as history information. When the identification rate falls below a predetermined identification rate, the machine learning process may be performed again. Alternatively, it may be performed at a predetermined timing.

As shown in the above-described embodiment, according to the communication support device 100 (100A), the multi-channel local field potential measured by the brain microelectrode and the multi-channel cortical EEG signal measured by the cortical electroencephalogram electrode on the brain surface. Both of them are input, and using a machine learning algorithm such as a support vector machine, it is possible to read information in the brain of characters and visual images with higher accuracy than before.
Both cortical electroencephalogram and local field potential can be recorded stably for a long period of time compared to the spike activity (unit action potential) of nerve cells. In addition, compared to the case of using spike activity alone and cortical EEG information alone, the accuracy of brain information decoding is improved by using both the local field potential and cortical EEG signals measured simultaneously.

According to the communication support apparatus 100 (100A) shown in the present embodiment, a signal in the brain (LFP) and a signal on the brain surface (ECoG) in the same region of the brain are simultaneously detected and used as an input signal. As a result, the problem that the stability of the operation for decoding the motion plan, which is seen in the intention communication support device based on the spike activity signal in the brain, is not sustained or the accuracy is insufficient is solved. The transmission support apparatus 100 eliminates the effect that the transmission of information can be supported accurately and stably.
In addition, the communication support device 100 can significantly improve the estimation accuracy for discriminating the visual object, and supports accurate and stable information transmission compared to the communication support device using only the brain surface signal. There is an effect that can be done.

  According to the communication support device 100 (100A) shown in the present embodiment, characters and icons are visually displayed for severely ill patients whose sensation / understanding is maintained but their motor / language expression ability is reduced. Supports communication with images. Severe patients who have sensation / understanding but have reduced ability to exercise and speak can develop not only amyotrophic lateral sclerosis (ALS) but also stroke (cerebrovascular disorder) It has been known. The intention transmission support device 100 (100A) can be used in various industrial fields such as medical care, welfare, and education as a medical care device that supports such a patient's intention display. The intention transmission support device 100 (100A) can provide an opportunity to perform intention display by supporting such a patient's intention display.

Note that the requirements of the above-described embodiment can be changed as appropriate without departing from the technical characteristics of the invention. Some components may not be used.
For example, the multipoint potential measurement unit 3 may use, as the electrical characteristics of the brain at a plurality of measurement points, a change in a specific frequency band component of a potential that changes with a visual stimulus at each measurement point. In that case, instead of the process of converting to the frequency component based on the time series information performed in the feature quantity extraction unit 4 in the subsequent stage, the process of converting to the time series information based on the frequency component (frequency spectrum) is performed.
In the future, if the level of brain science is improved, and the unit signal can be stably recorded for several months or more, the multipoint potential measuring unit 3 uses the cortical electroencephalogram (ECoG) method of the second measuring means. Alternatively, or as a third means, the UNIT method may be used.

  In addition, each part with which the communication support apparatus 100 (100A) in the above embodiment is provided may be realized by dedicated hardware, or may be realized by a memory and a microprocessor. .

  Each unit included in the intention transmission support device 100 (100A) includes a memory and a CPU (arithmetic processing unit), and a program for realizing the functions of each unit included in the intention transmission support device 100 (100A) is stored in the memory. The function may be realized by loading and executing.

  In addition, a program for realizing the function of each unit included in the communication support device 100 (100A) is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by a computer system and executed. By doing so, you may perform the process by each part. Here, the “computer system” includes an OS and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, it also includes those that hold a program for a certain period of time, such as a volatile memory inside a computer system serving as a server or client in that case. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

100, 100A communication support device 3, 3A multi-point potential measurement unit,
4, 4A feature quantity extraction unit,
5, 5A arithmetic processing unit 6 storage unit

Claims (20)

Derived from the brain surface signals of the region extracted from the brain surface signals and brain signals repeatedly acquired according to a predetermined trial period in the region including the cerebral visual cortex or cerebral association cortex in the subject's brain An operation for outputting information specifying the visual stimulus that the subject senses or images from the result calculated based on the information including the feature amount and the feature amount derived from the brain signal of the region A communication support device comprising a processing unit. Detecting in accordance with the trial period, including the brain signal measured by the first measuring means in the brain of the region and the brain surface signal measured by the second measuring means in the brain surface of the region A multi-point potential measurement unit for obtaining electrical characteristics of the brain at the measured position from a plurality of types of brain signals that are measured;
A feature amount extraction unit for extracting a feature amount according to the obtained electrical characteristics of the brain;
The communication support apparatus according to claim 1, comprising: The multipoint potential measurement unit
The intention transmission support apparatus according to claim 2, wherein a local field potential (LFP) method is used as the first measurement unit. The multipoint potential measurement unit
The cortex electroencephalogram (ECoG) method is used as the second measuring means. 4. The communication support apparatus according to claim 2 or 3, wherein: A first electrode portion for detecting the brain signal by the LFP method is disposed in the brain, and a second electrode portion for detecting the brain surface signal by the ECoG method is disposed on the brain surface. The communication support apparatus according to claim 4. The multipoint potential measurement unit
A plurality of the first electrode portions for detecting the brain signals;
The second electrode unit that is provided on one surface and detects a plurality of the brain surface signals;
With
The intention transmission support apparatus according to claim 5, wherein the first electrode part is provided so as to protrude from one surface on which the second electrode part is provided. The multipoint potential measurement unit
According to at least some polygons of the first polygon formed by the positions of the plurality of terminals provided in the first electrode part and the positions of the plurality of terminals provided in the second electrode part. The at least some polygons of the 2nd polygon formed are distribute | arranged so that the part which overlaps by planarly viewing the said 1 surface may exist. Communication support device. The region including the cerebral visual cortex or the cerebral association cortex is a region centering on the higher visual association cortex, and includes at least one of the visual-related regions in the temporal lobe or the ventral side of the occipital lobe. The communication support apparatus according to any one of claims 2 to 6. The feature amount extraction unit includes:
The characteristic amount according to the obtained electrical characteristic of the brain is based on the frequency component included in the brain signal and the time from a predetermined time to when the response of the brain signal is detected. The communication support device according to any one of claims 2 to 8, wherein the support device is extracted. The feature amount extraction unit includes:
A detection frequency range is defined to include at least a main frequency component included in the brain signal, and a plurality of frequency ranges are defined to include a part of the determined detection frequency range;
A detection time range is defined so as to include at least a period from a predetermined time before detecting the response of the brain signal until the response of the brain signal is detected, and includes a part of the determined detection time range. Define multiple time ranges,
The representative value of the signal intensity of the brain signal included in a range limited by a combination of any of the predetermined frequency ranges and any of the predetermined time ranges is included in the feature amount. The communication support apparatus described. The arithmetic processing unit includes:
Outputting information identifying the visual stimulus from a result calculated based on information further including a feature amount derived from correlation between a plurality of signals selected from the brain surface signal and the intracerebral signal. The intention transmission support device according to any one of claims 2 to 10, wherein the communication device is a communication support device. The arithmetic processing unit includes:
The information for specifying the visual stimulus is output from a result calculated based on information further including a feature amount derived from a correlation between signals of the brain surface signal and the intracerebral signal. The communication support apparatus according to any one of 11. The feature amount extraction unit includes:
Extracting the characteristic amount according to the obtained electrical characteristics of the brain from the result of the correlation processing based on the brain surface signal and the intracerebral signal,
The arithmetic processing unit includes:
12. The information for specifying the visual stimulus is output from a result calculated based on information further including a feature amount derived from a correlation between the signal of the brain surface signal and the signal in the brain. The communication support apparatus according to any one of the above. The feature amount extraction unit includes:
11. The communication support apparatus according to claim 10, wherein the plurality of frequency ranges are determined such that the detection frequency ranges are divided into a predetermined number and the ranges do not overlap each other. The feature amount extraction unit includes:
An average value of the signal strength of the brain signal included in a range limited by a combination of any of the defined frequency ranges and any of the defined time ranges is used as the feature value as a representative value of the signal strength of the brain signals. The communication support apparatus according to claim 14, comprising: The arithmetic processing unit includes:
From among a plurality of predetermined images, an image associated with the extracted feature amount is selected based on the plurality of types of brain signals, and information on the selected image is output as the calculated information. The communication support apparatus according to any one of claims 2 to 14, characterized in that The arithmetic processing unit includes:
In the process of selecting the associated image from the plurality of predetermined images, the feature value of the electrical characteristic of the brain at the measurement point of the brain detected with the display of the image and the image The communication support apparatus according to claim 16, wherein a variable of the process is optimized by a process of associating. The multipoint potential measurement unit
The electrical characteristic is characterized by using a potential (visual evoked potential) that changes with a visual stimulus at each measurement point or a change in a specific frequency band component of a potential that changes with a visual stimulus. 18. The communication support device according to any one of items 17 to 17. In the region of the subject's brain including the cerebral visual cortex or cerebral association cortex, the brain surface signal derived from the brain surface signal of the region extracted from the brain surface signal and the intracerebral signal repeatedly acquired according to a predetermined trial period, respectively. Outputting information specifying a visual stimulus that the subject senses or images from a result calculated based on information including a feature amount and a feature amount derived from a brain signal of the region A communication support method characterized by including: In the computer provided in the communication support device,
In the region of the subject's brain including the cerebral visual cortex or cerebral association cortex, the brain surface signal derived from the brain surface signal of the region extracted from the brain surface signal and the intracerebral signal repeatedly acquired according to a predetermined trial period, respectively. Outputting information specifying a visual stimulus that the subject senses or images from a result calculated based on information including a feature amount and a feature amount derived from a brain signal of the region A program for running