WO2019235458A1 - Dispostif et procédé d'estimation d'image de souvenir, programme de commande et support d'enregistrement - Google Patents

Dispostif et procédé d'estimation d'image de souvenir, programme de commande et support d'enregistrement Download PDF

Info

Publication number
WO2019235458A1
WO2019235458A1 PCT/JP2019/022113 JP2019022113W WO2019235458A1 WO 2019235458 A1 WO2019235458 A1 WO 2019235458A1 JP 2019022113 W JP2019022113 W JP 2019022113W WO 2019235458 A1 WO2019235458 A1 WO 2019235458A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
subject
decoder
decoding information
brain
Prior art date
Application number
PCT/JP2019/022113
Other languages
English (en)
Japanese (ja)
Inventor
琢史 ▲柳▼澤
良平 福間
晴彦 貴島
伸志 西本
Original Assignee
国立大学法人大阪大学
国立研究開発法人情報通信研究機構
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 国立大学法人大阪大学, 国立研究開発法人情報通信研究機構 filed Critical 国立大学法人大阪大学
Priority to JP2020523110A priority Critical patent/JP7352914B2/ja
Publication of WO2019235458A1 publication Critical patent/WO2019235458A1/fr

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/05—Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves 
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer

Definitions

  • the present invention relates to estimation of a recall image, and more particularly to a recall image estimation device that supports presentation of an arbitrary recalled image.
  • BMI brain-machine-interface
  • BMI measures the action potential of a patient's cranial nerve cells or cortical electroencephalograms to interpret motor intentions, controls the operation of machines such as robot arms, and controls to select and input characters intended by the patient.
  • Patent Document 1 the presented image and the electrical characteristics measured at a plurality of measurement points in the region including the visual association area of the brain when the image is presented are measured in association with each other.
  • a communication support apparatus that supports communication by specifying an image to be transmitted based on the electrical characteristics is disclosed.
  • Non-Patent Document 1 describes a method in which the firing activity of a nerve cell recorded from the hippocampus of a subject is measured, and the subject considers one of the images by overlapping the two images. A technique capable of strongly displaying the image is disclosed.
  • Patent Literature 1 since the communication support apparatus described in Patent Literature 1 determines an image to be displayed based on the electrical characteristics associated with the presented image, the displayable image is limited to the presented image. , Can not display any recalled image.
  • Non-Patent Document 1 a subject viewing a state where two images overlap each other causes the image on the side toward which the consciousness is directed to be strongly displayed by directing consciousness to one of the images. Although it can, it does not display any recalled image.
  • An object of one aspect of the present invention is to realize a recall image estimation apparatus and a recall image estimation method that accurately estimate a target image recalled by a subject.
  • a recall image estimation device is a multipoint potential that measures electrical characteristics of a subject's brain at a plurality of measurement points in a brain region including a visual association area. Estimated by the measurement unit, a decoder for estimating decoding information indicating the content of the target image recalled by the subject from the electrical characteristics measured while the subject visually recognizes the candidate image, and the decoder An image determining unit that determines a candidate image to be visually recognized by the subject based on the decoded information.
  • the recall image estimation method is measured at a plurality of measurement points in the brain region including the visual association area while the subject visually recognizes the candidate image in order to solve the above problem. From the electrical characteristics of the brain, an estimation step for estimating decoding information indicating the content of the target image recalled by the subject, and a candidate image to be visually recognized by the subject based on the decoding information estimated in the estimation step An image determining step for determining.
  • a target image recalled by a subject can be accurately estimated.
  • (A) is a flowchart which shows an example of the method of producing
  • (b) is a flowchart which shows the preparation process of the decoding information which shows the image for learning and the content of each image.
  • the recall image estimation device 10 is a device that estimates the decoding information indicating the content of the target image of the target image recalled by the subject, and determines a candidate image to be visually recognized by the subject based on the estimated decoding information. .
  • the recall image estimation device 10 does not determine a candidate image based on a one-to-one correspondence between the image visually recognized by the subject and the electrical characteristics of the subject's brain B when the image is viewed. Absent. Therefore, the recall image estimation apparatus 10 can determine an arbitrary candidate image that is not an image visually recognized by the subject in advance as a candidate image to be visually recognized by the subject.
  • the recall image estimation device 10 is a device that supports the subject to be able to present any images and images that the subject desires to present outside.
  • the “candidate image” is intended to be an image visually recognized by the subject in order to measure the electrical characteristics of the brain B, and the “target image” is recalled while the subject visually recognizes the candidate image. Is intended (ie, the image that the subject wants to present).
  • FIG. 1 is a functional block diagram showing a schematic configuration example of a recall image estimation apparatus 10 according to an embodiment of the present invention.
  • the recall image estimation device 10 includes the display unit 5
  • the present invention is not limited to this.
  • a configuration in which an external display device is applied instead of the display unit 5 may be used.
  • the recall image estimation device 10 includes a multipoint potential measurement unit 1, a decoder 2, an image determination unit 3, a display control unit 4, a display unit 5, and a storage unit 6.
  • the multipoint potential measuring unit 1 measures the electrical characteristics of the subject's brain B at a plurality of measurement points in the region of the brain B including the visual association area. More specifically, the multipoint potential measurement unit 1 includes a plurality of electrodes E, and measures a cortical electroencephalogram (Electro-Cortico-Graphy: ECoG) of the brain B (low invasive configuration).
  • the electrode E is an ECoG electrode placed under the dura mater.
  • the electrode E is an electrode for detecting the cortical potential generated in the brain B of the subject who is viewing the image.
  • Electrode E can be placed on the surface of the brain B cerebral cortex that contains the visual association area and on the surface of the sulcus.
  • the number of the electrodes E should just be plural (for example, 100), and is not specifically limited.
  • the multipoint potential measuring unit 1 is not limited to the configuration for measuring the cortical potential.
  • the multipoint potential measuring unit 1 ⁇ Configuration to measure action potential (Multi-unit Activity: MUA) of nerve cell using electrode inserted into brain B as electrode E (invasive configuration) ⁇ Structure for measuring electroencephalogram (stereotactic Electro-Graphy: stereotactic EEG) using an insertion electrode in brain B as electrode E (invasive structure) ⁇ Scalp Electro-Encephalo-Graphy (scalp EEG) measurement using electrode E placed on scalp (non-invasive configuration) ⁇ Configuration to measure intravascular electro-encephalogram (intravascular EEG) using electrode E placed in cerebral blood vessel (minimally invasive configuration) ⁇ Either a configuration (non-invasive configuration) for measuring a magnetic field generated by an electrical activity of the brain B using a magnetoencephalogram (Magneto-Encephalo-Graphy: MEG) sensor as the electrode E Good.
  • the sensitivity of the electrical characteristics of the brain B to be measured is generally in the order of scalp EEG ⁇ MEG ⁇ intravasual EEG ⁇ stereotactic EEG ⁇ ECoG ⁇ MUA.
  • MEG and ECoG are desirable as the multipoint potential measuring unit 1.
  • an alpha wave (8 to 13 Hz), a beta wave (13 to 30 Hz), a low frequency gamma wave (30 to 80 Hz), and a high frequency gamma wave an alpha wave (8 to 13 Hz), a beta wave (13 to 30 Hz), a low frequency gamma wave (30 to 80 Hz), and a high frequency gamma wave
  • an electroencephalogram in each band 80 to 150 Hz
  • the decoder 2 estimates decoding information indicating the contents of the target image recalled by the subject from the electrical characteristics measured while visually recognizing the candidate image.
  • “decoding information” is information indicating the content and meaning of an image. More specifically, “decoding information” is information representing the content and meaning of an image as a vector in a semantic space (which may be expressed as a “word vector space”). A method of expressing the content of an image as a vector in a semantic space will be described later with a specific example.
  • the decoder 2 may be a learned neural network.
  • the learning for creating the decoder 2 is generated in advance using a predetermined candidate image and a word vector corresponding to one or more words included in one or more explanatory sentences explaining the contents of the predetermined candidate image.
  • Teacher decoding information is used.
  • the decoder 2 includes an input layer and an output layer, and when the electrical characteristics of the brain B measured while viewing the predetermined candidate image are input to the input layer, the predetermined candidate Learning is performed so that the teacher decoding information associated with the image is output from the output layer.
  • a process of generating the decoder 2 by learning will be described later with a specific example.
  • the image determination unit 3 determines a candidate image to be visually recognized by the subject based on the decoding information estimated by the decoder 2. More specifically, the image determination unit 3 causes the candidate image associated with the same or similar decoding information as the decoding information estimated by the decoder 2 to be viewed following the candidate image that is being viewed by the subject. Determine as a candidate image.
  • the display control unit 4 controls the display unit 5 to display the candidate image determined by the image determination unit 3. Further, the display control unit 4 controls the display unit 5 to display a predetermined candidate image prepared for learning in the process of generating the decoder 2 by learning.
  • the display unit 5 is a display that displays an image. The subject recalls an arbitrary target image while visually recognizing the image displayed on the display unit 5.
  • the storage unit 6 stores candidate images to be displayed on the display unit 5. Each candidate image is associated with decoding information indicating the contents of each candidate image.
  • the recall image estimation device 10 also has a function of performing machine learning (supervised learning) of the decoder 2
  • the storage unit 6 corresponds to each learning image (predetermined candidate image) and each learning image.
  • the attached decoding information (teacher decoding information) is stored.
  • the decoding information which shows the content of the image which the said test subject is recalling is estimated from the electrical property of the brain B of the test subject who is visually recognizing the candidate image, and based on the estimated decoding information The subject is made to visually recognize the determined image.
  • a closed-loop control mechanism can be configured in which the subject image is visually recognized by the subject, the decoding information is estimated, and the next candidate image is determined based on the estimated decoding information.
  • the closed-loop control mechanism is a “closed control mechanism” in which a candidate image to be visually recognized by the subject is determined from the electrical characteristics of the brain B measured when the subject is visually recognizing the candidate image. Is intended.
  • the subject repeats the process of visually recognizing the candidate image while recalling the desired target image. Therefore, the top-down control of the brain activity by the subject himself / herself is input to the visual cortex of the brain B, and the electrical characteristics of the brain B when this top-down control is input can be measured. Therefore, the target image recalled by the subject can be accurately estimated.
  • top-down control is one of the forms of neural information control when the brain B processes visual information, as in the bottom-up control.
  • Top-down control is control for selecting a target stimulus by actively biasing neural information when there is prior knowledge about a stimulus to be selected from visual information.
  • bottom-up control is a control that passively pays attention to a prominent stimulus, such as when a stimulus significantly different from the surrounding stimulus is included among the multiple stimuli included in the visual information. It is.
  • FIG. 2 is a flowchart illustrating an example of a process flow of the recall image estimation apparatus 10.
  • the decoder 2 is generated by machine learning. Specifically, when an electrical characteristic of the brain B measured while viewing a predetermined candidate image is input, so as to output teacher decoding information associated with the predetermined candidate image, The decoder 2 is learned (step S1: decoder generation step). In the recall image estimation apparatus 10 as shown in FIG. 1, the learned decoder 2 is applied.
  • the display control unit 4 controls the display unit 5 so that candidate images to be visually recognized by the subject are displayed (step S2: candidate image display step).
  • the image may be a moving image including a plurality of images. It does not matter if there is sound.
  • the candidate image that is first visually recognized by the subject is not particularly limited. For example, an arbitrary image such as a screen for notifying that the estimation process by the recall image estimation device 10 has started may be displayed.
  • the decoder 2 estimates decoding information from the electrical characteristics of the brain B of the subject viewing the displayed candidate image measured by the multipoint potential measuring unit 1 (step S3: estimation step).
  • the image determination unit 3 determines a candidate image to be visually recognized by the subject based on the decoding information estimated by the decoder 2 (step S4: image determination step).
  • the display control unit 4 controls the display unit 5 so that the image determined by the image determination unit 3 is displayed following the subject.
  • a closed-loop control mechanism is configured in which the subject visually recognizes the candidate image while recalling the desired target image.
  • FIG. 3 is a functional block diagram illustrating an example of a schematic configuration of the recall image estimation apparatus 10a that performs machine learning for creating the decoder 2.
  • the recall image estimation device 10a may have the same function and the same configuration as the recall image estimation device 10 shown in FIG. 1 (for example, the image determination unit 3 not related to the learning of the decoder 2).
  • the recall image estimation device 10a includes a decoded information comparison unit 7 and a weight coefficient correction unit 8.
  • the decoding information comparison unit 7 uses the decoding information estimated by the decoder 2 before learning (or during learning) from the electrical characteristics of the brain B measured from the brain B of the subject viewing the learning image, and the learning The teacher decoding information associated with the image for use is compared.
  • the weighting factor correction unit 8 corrects the weighting factor of the decoder 2 based on the comparison result by the decoding information comparison unit 7. Specifically, the weight coefficient correction unit 8 is associated with the learning image when the electrical characteristics of the brain B measured from the brain B of the subject who is viewing the learning image are input. The current weighting factor of the decoder 2 is corrected so as to output the teacher decoding information.
  • the decoder 2 can estimate with high accuracy decoding information indicating the content of the target image from the electrical characteristics of the brain B of the subject recalling the target image. Can be created.
  • FIG. 4A is a flowchart illustrating an example of a method for generating a decoder by machine learning
  • FIG. 4B is a flowchart illustrating a preparation process of a learning image and decoding information indicating the contents of each image. It is.
  • FIG. 5 is a diagram illustrating an example of a learning image for creating the decoder 2 and an explanatory text explaining the content of the learning image.
  • step S11 learning image preparation step
  • steps S113 to S115 can be performed using a general personal computer.
  • Step S111 Step for preparing learning images used for machine learning
  • Step S112 Step for preparing an explanatory text (caption or annotation) explaining the content and meaning of the learning image for each learning image (step S112).
  • the explanatory text may be a single sentence or may include a plurality of sentences.
  • the explanatory text is preferably a text that simply and accurately describes the content of the image and the impression received when the image is viewed.
  • the explanatory note may be created by showing an image to one or a plurality of people, or may be created artificially using artificial intelligence having an image recognition function.
  • the learning image for creating the decoder 2 and the explanatory text explaining the content of the learning image will be described later with a specific example.
  • a step of extracting words included in the explanatory text (step S113).
  • a known morphological analysis engine can be applied to this step. Examples of such a known morphological analysis engine include “MeCab”, “Chasen”, “KyTea”, and the like. This process is a process that is necessary when the explanatory text is written in Japanese. If the explanatory text is written in a language in which each word is separated (for example, a space exists between words), such as English, this step is omitted.
  • a step of generating a word vector for each extracted word (step S114).
  • a known tool for example, artificial intelligence
  • Examples of such known tools include “Word2vec”, “GloVe”, “fastText”, “Doc2Vec”, and “WordNet”.
  • “Word2vec” learned using many existing sentences means a predetermined dimension (for example, 1000 dimensions) for each word extracted from the explanatory text.
  • the word vector in the space can be output with high accuracy.
  • the word vector is preferably a vector in a linear space in which linear operations can be performed, but may be a word vector in a non-linear space. Note that this step can be performed in the same manner regardless of the type of language used in the description. For example, when the description is written in English, Word2vec or the like may be learned using an English version of Wikipedia or the like, and a word vector may be output using the learned Word2vec.
  • a step of generating teacher decoding information associated with the learning image as an average of word vectors For words extracted from the explanatory text explaining the content of the learning image, the vector average of the word vectors generated in step S114 is obtained, and teacher decoding information indicating the content of the explanatory text is generated.
  • the teacher decoding information is generated by averaging vectors in the meaning space of words extracted from sentences explaining the contents of each learning image. Note that decoding information is also generated for each of the candidate images provided to the recall image estimation apparatus 10 according to the present embodiment by the processes of S111 to S115.
  • the multipoint potential measuring unit 1 measures the electrical characteristics measured in the brain B of the subject who visually recognizes the learning image (step S12: measurement step). In this step, it is desirable that the subject merely visually recognizes the learning image without recalling the target image.
  • the decoder 2 is trained using the measured electrical characteristics as an input signal and the teacher decoding information indicating the contents of the currently viewed learning image as a teacher signal.
  • the decoding information comparison unit 7 is estimated by the decoder 2 before learning (or during learning) from the electrical characteristics of the brain B measured from the brain B of the subject viewing the learning image.
  • the decoded information is compared with the teacher decoded information associated with the learning image.
  • the weighting coefficient correction unit 8 performs the teacher decoding associated with the learning image.
  • the current weighting factor of the decoder 2 is corrected so as to output information.
  • steps S11 to S13 shown in FIG. 4 (a) do not have to be performed continuously, and can be performed individually.
  • the process of step S11 may be performed before step S12 is performed, or may be performed after step S12 is performed.
  • the configuration may be such that step S12 is performed, data in which the measured electrical characteristics are associated with the image visually recognized by the subject is stored, and the data is used for the learning of the decoder 2.
  • Example of learning image The image shown in FIG. 5 is an example of a learning image.
  • the image shown in FIG. 5 is an example of a learning image.
  • a plurality of explanatory texts may be created for one learning image (and candidate image). For example, for the learning image shown in FIG. 5, “It seems that three families are taking pictures of people wearing spacesuits with a camera. They seem to have fun and experience wearing spacesuits. "You can see a child wearing a space suit. Dad is taking a picture. I'm glad if you can have this experience.”
  • FIG. 6 is an image diagram illustrating an example of a procedure for generating the decoder 2 using the learning image.
  • the case where the electrical characteristic of the brain B of the subject is a cortical electroencephalogram will be described as an example.
  • the cortical electroencephalogram of the brain B of the subject who is viewing the learning image is measured by the multipoint potential measuring unit 1.
  • the measured cortical electroencephalogram is frequency-analyzed to determine the power of each band of the alpha wave, the beta wave, and the gamma wave, and these are used as a feature matrix that is input to the decoder 2.
  • a word is extracted from the explanatory text for each image viewed by the subject, and decoding information is generated from the explanatory text.
  • the explanatory text shown in FIG. 6 “The top of the mountain with snow. The sky with clear blue and white clouds, the snowy ground and the exposed waterside mountains.
  • words such as “snow”, “mountain”, “top”, “mode” are extracted.
  • decoding information averaged for each element (for example, 1000 dimensions) of the extracted word vector is determined as teacher decoding information.
  • a word vector for each extracted word is generated as a 1000-dimensional word vector using learned Word2vec.
  • the weight matrix is corrected so that the teacher decoding information of each image can be output with a desired accuracy when the power of each band of the alpha wave, the beta wave, and the gamma wave is used as an input signal.
  • FIG. 6 shows an example of learning to output decoding information for 3600 images using regression processing such as ridge-regulation.
  • regression processing such as ridge-regulation.
  • analysis methods such as deep learning and Sparse Logistic Regression (SLR).
  • the configuration may be such that the candidate images to be visually recognized by the subject are not determined from the images stored in the storage unit 6 but are acquired from an arbitrary information group to be searched.
  • the recall image estimation device 10a uses a wide variety of images as candidate images by searching for images from the information group to be searched. First, the recall image estimation device 10a will be described with reference to FIG.
  • FIG. 9 is a functional block diagram illustrating a schematic configuration example of the recall image estimation apparatus 10a according to the embodiment of the present invention.
  • the recall image estimation device 10a shown in FIG. 9 includes an image search unit 3a (image determination unit) instead of the image determination unit 3.
  • the image search unit 3a generates a search query using the same or similar decoding information as the decoding information estimated by the decoder 2.
  • the image search unit 3a uses the generated search query to search for an image associated with the same or similar decoding information as the decoding information from the information group to be searched.
  • the information group to be searched may be an arbitrary information group. For example, as shown in FIG. 9, a website A 60a and a website B 60b existing on the Internet may be included.
  • the image search unit 3a determines an image acquired as a search result as a candidate image.
  • the image search part 3a determines the image acquired as a search result as a candidate image which makes a subject visually recognize a candidate image.
  • the decoder 2 estimates decoding information indicating the contents of the target image recalled by the subject from the electrical characteristics measured while viewing the candidate image.
  • the decoder 2 can estimate one or more words close to the estimated decoding information (for example, a vector in the semantic space).
  • the decoder 2 selects several words in order of increasing distance between a vector in the semantic space of the estimated decoded information and a vector in the semantic space of each word close to the decoded information.
  • the image search unit 3a selects several verbs and adjectives from the words estimated by the decoder 2 and uses them for a known image search (for example, Google (registered trademark) image search). Generate a search query.
  • the image search unit 3a can search the web for an image associated with the word estimated by the decoder 2 using the generated search query.
  • the image search unit 3 a determines an image listed at the top in the search result as a candidate image to be displayed on the display unit 5.
  • a variety of images of a search target information group including a website existing on the Internet can be used as a candidate image to be presented to the subject. it can.
  • the subject repeats the process of visually recognizing the candidate image while recalling the desired target image. Therefore, the subject searches for an arbitrary image from a search target information group including a website existing on the Internet by changing an electrical characteristic measured while viewing the candidate image. Can do.
  • the recall image estimation device 10 a shown in FIG. 9 does not include the storage unit 6 that stores candidate images to be displayed on the display unit 5. However, this is only an example, and the recall image estimation device 10a may be configured to include the storage unit 6 as in the recall image estimation device 10 illustrated in FIG.
  • the image search unit 3a acquires the image associated with the same or similar decoding information as the decoding information estimated by the decoder 2 from the storage unit 6, the website A 60a, the website B 60b, and the like.
  • control blocks (particularly the decoder 2, the image determination unit 3, and the display control unit 4) of the recall image estimation device 10 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like. It may be realized by software.
  • the recall image estimation apparatus 10 includes a computer that executes instructions of a program that is software for realizing each function.
  • the computer includes, for example, one or more processors and a computer-readable recording medium storing the program.
  • the processor reads the program from the recording medium and executes the program, thereby achieving the object of the present invention.
  • a CPU Central Processing Unit
  • the recording medium a “non-temporary tangible medium” such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used.
  • a RAM Random Access Memory
  • the program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program.
  • an arbitrary transmission medium such as a communication network or a broadcast wave
  • one embodiment of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.
  • the recall image estimation apparatus includes a multipoint potential measurement unit that measures electrical characteristics of a subject's brain at a plurality of measurement points in a brain region including a visual association area, and the subject is a candidate image. From the electrical characteristics measured while visually recognizing, based on the decoding information estimated by the decoder, the decoder for estimating the decoding information indicating the content of the target image recalled by the subject, An image determining unit that determines candidate images to be visually recognized by the subject.
  • the decoding information indicating the content of the image recalled by the subject is estimated from the electrical characteristics of the brain of the subject viewing the candidate image, and is determined based on the estimated decoding information.
  • the subject is made to visually recognize the image.
  • a closed-loop control mechanism can be configured in which the subject image is visually recognized by the subject, the decoding content is estimated, and the next candidate image is determined based on the estimated decoding information.
  • the subject repeats the process of visually recognizing the candidate image while recalling the desired target image. Therefore, the top-down control of brain activity by the subject himself / herself is input to the visual cortex of the brain, and the electrical characteristics of the brain when this top-down control is input can be measured. Therefore, the target image recalled by the subject can be accurately estimated.
  • the recall image estimation device is the recall image estimation apparatus according to aspect 1, in which the image determination unit determines an image associated with the decoding information that is the same as or similar to the decoding information estimated by the decoder, You may determine as a candidate image made to visually recognize after the said candidate image.
  • the image determination unit generates and generates a search query using the decoding information that is the same as or similar to the decoding information estimated by the decoder.
  • the search query is used to search the information group to be searched for an image associated with the decoding information that is the same as or similar to the decoding information, and an image acquired as a search result is used as the candidate image. May be determined as
  • the image determination unit determines the image acquired as the search result as a candidate image to be visually recognized following the candidate image. Also good.
  • search target information group may include websites on the Internet.
  • the recall image estimation device is the recall image estimation device according to any one of the aspects 1 to 4, wherein a word vector corresponding to one or more words included in one or more explanatory texts describing the contents of a predetermined candidate image.
  • the teacher decoding information generated in advance and the predetermined candidate image are associated with each other, and the decoder receives the electrical characteristics of the brain measured while viewing the predetermined candidate image. In such a case, the learning may be performed so that the teacher decoding information associated with the predetermined candidate image is output.
  • a decoder capable of estimating the decoding information indicating the content of the target image with high accuracy from the electrical characteristics of the brain of the subject recalling the target image is generated. be able to.
  • the recall image estimation apparatus is the recall image estimation apparatus according to any one of the aspects 1 to 5, wherein the decoder measures the cortical potential of the brain and the electrical brain, which are measured while viewing the candidate image.
  • the decoding information indicating the contents of the candidate image may be estimated using at least one of the magnetic fields generated by the active activity.
  • the recall image estimation method provides a plurality of measurement points in a brain region including a visual association area while a subject visually recognizes a candidate image. Based on the measured electrical characteristics of the brain, the estimation step of estimating the decoding information indicating the content of the target image recalled by the subject, and the subject is made to visually recognize based on the decoding information estimated in the estimation step And an image determining step for determining a candidate image.
  • a control program for causing a computer to function as the recall image estimation device according to any one of the above aspects 1 to 6, the control program for causing the computer to function as the decoder and the image determining unit, and A computer-readable recording medium recording the control program is also included in the technical scope of the present invention.
  • the cortical electroencephalogram of the subject's brain B was measured by the multipoint potential measuring unit 1 while allowing the subject to visually recognize a 60-minute moving image including various types of meaning content.
  • the videos to be viewed by the subjects were prepared by connecting the edited videos by dividing the introduction video of the movie into short segments. In a 60-minute video, various videos including the same video appear several times in random order. The subject was instructed to view the video without fixing the viewpoint.
  • the moving image visually recognized by the subject was converted into a still image (scene) every second.
  • scene For each scene, explanations explaining the contents of the scene were created by a plurality of people.
  • the power of each band of an alpha wave, a beta wave, and a gamma wave was analyzed about the cortical electroencephalogram measured in the same 1 second.
  • a word was extracted from the description for each scene using MeCab. For each extracted word, a 1000-dimensional word vector was generated using Word2vec learned using Wikipedia. Each scene was associated with decoding information generated as an average of word vectors for words extracted from the explanatory text.
  • the solid black line in FIG. 7 shows the frequency distribution of the correlation coefficient between the decoding information estimated from the cortical brain waves of the brain B of the subject viewing the scene and the decoding information (that is, the correct answer) associated with the scene. Is shown.
  • the gray line in FIG. 7 shows the correlation between the shuffled label of the decoding information associated with each scene and the decoding information estimated from the cortical electroencephalogram of the brain B of the subject viewing the scene. The frequency distribution of numbers is shown. According to FIG. 7, it was demonstrated that the decoding information associated with the scene can be estimated with significantly high accuracy from the cortical electroencephalogram of the brain B of the subject viewing the scene.
  • time 0 indicates the timing when the subject is instructed to recall the image (“character”, “landscape”, etc.).
  • the black line in FIG. 8 indicates the trial average of the correlation coefficient normalized with respect to the decoding information associated with the image including the content instructed to the subject and the decoding information estimated from the cortical brain wave of the brain B of the subject. (* P ⁇ 0.05, Student's t-test).
  • the gray line in FIG. 8 shows the trial average of the decoded information associated with the image that does not include the recalled image, the decoded information estimated from the cortical EEG of the subject's brain B, and the normalized correlation coefficient. Show. According to FIG. 8, it was demonstrated that the image recalled by the subject can be estimated with significantly high accuracy.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Surgery (AREA)
  • Public Health (AREA)
  • Pathology (AREA)
  • Theoretical Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Veterinary Medicine (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Radiology & Medical Imaging (AREA)
  • Psychiatry (AREA)
  • Psychology (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente invention permet d'estimer, avec une précision élevée, une image cible dont une personne se souvient. L'invention concerne donc un dispositif d'estimation d'image de souvenir (10) comprenant : une unité de mesure de potentiel électrique multipoint (1) qui mesure des caractéristiques électriques du cerveau d'une personne en de multiples points de mesure dans une région cérébrale comprenant un cortex d'association visuelle ; un décodeur (2) qui, à partir des caractéristiques électriques mesurées pendant que le sujet reconnaît visuellement des images candidates, estime des informations de décodage indiquant le contenu d'une image cible dont la personne se souvient ; et une unité de détermination d'image (3) qui, sur la base des informations de décodage estimées, détermine une image candidate pour amener la personne à effectuer une reconnaissance visuelle.
PCT/JP2019/022113 2018-06-04 2019-06-04 Dispostif et procédé d'estimation d'image de souvenir, programme de commande et support d'enregistrement WO2019235458A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2020523110A JP7352914B2 (ja) 2018-06-04 2019-06-04 想起画像推定装置、想起画像推定方法、制御プログラム、記録媒体

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018107185 2018-06-04
JP2018-107185 2018-06-04

Publications (1)

Publication Number Publication Date
WO2019235458A1 true WO2019235458A1 (fr) 2019-12-12

Family

ID=68769873

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/022113 WO2019235458A1 (fr) 2018-06-04 2019-06-04 Dispostif et procédé d'estimation d'image de souvenir, programme de commande et support d'enregistrement

Country Status (2)

Country Link
JP (1) JP7352914B2 (fr)
WO (1) WO2019235458A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111782853A (zh) * 2020-06-23 2020-10-16 西安电子科技大学 基于注意力机制的语义图像检索方法
WO2023007293A1 (fr) * 2021-07-29 2023-02-02 Ofer Moshe Procédés et systèmes de rendu et d'injection d'informations non sensorielles
WO2024100859A1 (fr) * 2022-11-10 2024-05-16 日本電信電話株式会社 Procédé de génération d'image, dispositif de génération d'image et programme de génération d'image

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS6354618A (ja) * 1986-08-25 1988-03-09 Canon Inc 入力装置
JPH07204168A (ja) * 1994-01-12 1995-08-08 Nou Kinou Kenkyusho:Kk 生体情報自動識別装置
JP2010257343A (ja) * 2009-04-27 2010-11-11 Niigata Univ 意思伝達支援装置
JP2016067922A (ja) * 2014-09-25 2016-05-09 エスエヌユー アールアンドディービー ファウンデーション ブレイン−マシンインタフェース装置および方法
JP2016513319A (ja) * 2013-03-15 2016-05-12 インテル コーポレイション 収集された生物物理的信号の時間的パターンおよび空間的パターンに基づく脳‐コンピューターインターフェース（ｂｃｉ）システム
WO2017022228A1 (fr) * 2015-08-05 2017-02-09 セイコーエプソン株式会社 Dispositif de lecture d'image mentale

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS6354618A (ja) * 1986-08-25 1988-03-09 Canon Inc 入力装置
JPH07204168A (ja) * 1994-01-12 1995-08-08 Nou Kinou Kenkyusho:Kk 生体情報自動識別装置
JP2010257343A (ja) * 2009-04-27 2010-11-11 Niigata Univ 意思伝達支援装置
JP2016513319A (ja) * 2013-03-15 2016-05-12 インテル コーポレイション 収集された生物物理的信号の時間的パターンおよび空間的パターンに基づく脳‐コンピューターインターフェース（ｂｃｉ）システム
JP2016067922A (ja) * 2014-09-25 2016-05-09 エスエヌユー アールアンドディービー ファウンデーション ブレイン−マシンインタフェース装置および方法
WO2017022228A1 (fr) * 2015-08-05 2017-02-09 セイコーエプソン株式会社 Dispositif de lecture d'image mentale

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111782853A (zh) * 2020-06-23 2020-10-16 西安电子科技大学 基于注意力机制的语义图像检索方法
CN111782853B (zh) * 2020-06-23 2022-12-02 西安电子科技大学 基于注意力机制的语义图像检索方法
WO2023007293A1 (fr) * 2021-07-29 2023-02-02 Ofer Moshe Procédés et systèmes de rendu et d'injection d'informations non sensorielles
WO2024100859A1 (fr) * 2022-11-10 2024-05-16 日本電信電話株式会社 Procédé de génération d'image, dispositif de génération d'image et programme de génération d'image

Also Published As

Publication number Publication date
JPWO2019235458A1 (ja) 2021-07-29
JP7352914B2 (ja) 2023-09-29

Similar Documents

Publication Publication Date Title
Abdelrahman et al. Cognitive heat: exploring the usage of thermal imaging to unobtrusively estimate cognitive load
Astolfi et al. Imaging the social brain by simultaneous hyperscanning during subject interaction
Heisz et al. More efficient scanning for familiar faces
US11301775B2 (en) Data annotation method and apparatus for enhanced machine learning
Phelps et al. Emotion facilitates perception and potentiates the perceptual benefits of attention
Reed et al. The body-inversion effect
Levelt et al. An MEG study of picture naming
Miellet et al. Local Jekyll and global Hyde: the dual identity of face identification
WO2019235458A1 (fr) Dispostif et procédé d'estimation d'image de souvenir, programme de commande et support d'enregistrement
Moon et al. Implicit analysis of perceptual multimedia experience based on physiological response: A review
Moon et al. Perceptual experience analysis for tone-mapped HDR videos based on EEG and peripheral physiological signals
WO2016163556A1 (fr) Procédé d'estimation de contenu sémantique de perception par l'analyse de l'activité cérébrale
KR102085693B1 (ko) 사용자의 집중력 레벨 결정 방법 및 장치
Sakaki et al. Association learning for emotional harbinger cues: When do previous emotional associations impair and when do they facilitate subsequent learning of new associations?
JP2016195716A5 (fr)
Hahn et al. Thatcherization impacts the processing of own-race faces more so than other-race faces: An ERP study
Kim et al. An analysis of smartphone overuse recognition in terms of emotions using brainwaves and deep learning
Wache The secret language of our body: Affect and personality recognition using physiological signals
Dzafic et al. Dynamic emotion perception and prior expectancy
Đorđević Čegar et al. Modelling effects of S3D visual discomfort in human emotional state using data mining techniques
CN116312971B (zh) 认知训练素材生成方法、认知训练方法、装置和介质
CN112613364A (zh) 目标物体的确定方法、系统、存储介质和电子设备
Nussbaum et al. Analysis of viewer EEG data to determine categorization of short video clip
Fukuma et al. Decoding visual stimulus in semantic space from electrocorticography signals
Raz et al. A robust neural fingerprint of cinematic shot-scale

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19815429

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020523110

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19815429

Country of ref document: EP

Kind code of ref document: A1