WO2024100859A1 - Procédé de génération d'image, dispositif de génération d'image et programme de génération d'image - Google Patents

Procédé de génération d'image, dispositif de génération d'image et programme de génération d'image Download PDF

Info

Publication number
WO2024100859A1
WO2024100859A1 PCT/JP2022/041996 JP2022041996W WO2024100859A1 WO 2024100859 A1 WO2024100859 A1 WO 2024100859A1 JP 2022041996 W JP2022041996 W JP 2022041996W WO 2024100859 A1 WO2024100859 A1 WO 2024100859A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
information
category
user
category information
Prior art date
Application number
PCT/JP2022/041996
Other languages
English (en)
Japanese (ja)
Inventor
慶人 村岡
愛 中根
信哉 志水
高雄 中村
Original Assignee
日本電信電話株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電信電話株式会社 filed Critical 日本電信電話株式会社
Priority to PCT/JP2022/041996 priority Critical patent/WO2024100859A1/fr
Publication of WO2024100859A1 publication Critical patent/WO2024100859A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/80—Creating or modifying a manually drawn or painted image using a manual input device, e.g. mouse, light pen, direction keys on keyboard

Definitions

  • the present invention relates to an image generation method, an image generation device, and an image generation program.
  • Non-Patent Documents 1 and 2 In recent years, advances in image generation technology have made it easier to generate high-quality images. Furthermore, in the field of research into decoding perceptions and imaginations using brain information as input, progress is being made in the development of technology to generate images that are seen and what is imagined (see Non-Patent Documents 1 and 2).
  • the present invention has been made in consideration of the above, and aims to efficiently generate both perceived images and recalled images from brain waves.
  • the image generation method of the present invention is an image generation method executed by an image generation device, and is characterized by including an acquisition step of acquiring brainwave information at least when a user perceives or imagines an image, category information representing a classification category of the image perceived by the user, and at least one of category information of the image imagined by the user, and a generation step of generating an image corresponding to the category information using the acquired brainwave information and the category information of each image.
  • the present invention makes it possible to efficiently generate both perceived and recalled images from brain waves.
  • FIG. 1 is a diagram for explaining an overview of an image generating apparatus according to this embodiment.
  • FIG. 2 is a schematic diagram illustrating a schematic configuration of the image generating apparatus according to the present embodiment.
  • FIG. 3 is a diagram for explaining the process of the generation unit.
  • FIG. 4 is a diagram for explaining the processing of the learning unit.
  • FIG. 5 is a flowchart showing the procedure of the image generation process.
  • FIG. 6 is a diagram illustrating an example of a computer that executes an image generating program.
  • Fig. 1 is a diagram for explaining an overview of the image generating device of this embodiment.
  • the image generating device acquires electroencephalogram information of a user (S1), and acquires category information indicating a classification category of each image for a perceived image perceived by the user and an imagined image imagined by the user (S2).
  • the image generating device generates images to reconstruct the perceived image and the imagined image using the brainwave information and the category information of each image (S3).
  • the image generating device learns a model using the brain wave information and perceptual category information when the user perceives an image, and the brain wave information and imaginative category information when the user imagines an image, as learning data.
  • the image generating device uses the learned model to generate an image from the brain wave information, perceptual category information, and imaginative category information.
  • the image generating device may estimate perceptual category information or imagined category information from electroencephalogram information (S21) using the learned category estimation model.
  • the image generating device learns the category estimation model using the electroencephalogram information and perceptual category information when the user perceives an image, and the electroencephalogram information and imagined category information when the user imagines an image, as learning data.
  • the image generating device then presents the generated image to the user by outputting it to an output unit such as a display.
  • an output unit such as a display.
  • the image generating device can easily generate both perceptual images and imaginary images from brain waves by separately using the perceptual category information and the imaginary category information.
  • FIG. 2 is a schematic diagram illustrating a schematic configuration of an image generating device according to the present embodiment.
  • an image generating device 10 according to the present embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.
  • the input unit 11 is realized using input devices such as a keyboard, mouse, and electroencephalograph, and inputs various instruction information such as starting processing to the control unit 15 in response to input operations by an operator.
  • the output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, etc. For example, the output unit 12 displays a facial image with an expression generated in the image generation process described below.
  • the communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and external devices via telecommunication lines such as a LAN (Local Area Network) or the Internet.
  • the communication control unit 13 controls communication between the control unit 15 and a management device that manages various types of information.
  • the storage unit 14 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.
  • the storage unit 14 stores in advance the processing program that operates the image generating device 10 and data used during execution of the processing program, or stores them temporarily each time processing is performed.
  • the storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.
  • the storage unit 14 stores a model 14a and a category estimation model 14b used in the image generation process described below.
  • the control unit 15 is realized using a CPU (Central Processing Unit) or the like, and executes a processing program stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, a generation unit 15b, a learning unit 15c, and a presentation unit 15d, as illustrated in FIG. 2, to execute image generation processing. Note that each of these functional units, or some of them, may be implemented in different hardware. For example, the learning unit 15c may be implemented in hardware different from the other functional units.
  • the control unit 15 may also include other functional units.
  • the acquisition unit 15a acquires brainwave information at least when the user perceived or imagined an image, and at least one of category information indicating a classification category of the image perceived by the user and category information of the image imagined by the user. For example, the acquisition unit 15a acquires, via the input unit 11 or from a user terminal or the like via the communication control unit 13, brainwave information at the time the user to be processed perceived or imagined an image, and at least one of perceived category information of the perceived image perceived by the user or imagined category information of the imagined image imagined by the user.
  • the acquisition unit 15a acquires time-series data of electroencephalogram and magnetoencephalogram information measured by an electroencephalograph, such as a high-precision electroencephalograph that acquires electroencephalograms from the entire head or a simple electroencephalograph that uses a small number of electrodes.
  • the acquisition unit 15a then performs pre-processing on the time-series data to reduce noise contained therein and improve the signal-to-noise ratio, and treats the data as electroencephalogram information.
  • the acquisition unit 15a at least either extracts information of the target frequency band or applies a denoising method to remove components other than brain waves contained in the time series data.
  • the acquisition unit 15a extracts information to include one or more of delta waves of 1-4 Hz, theta waves of 4-7 Hz, alpha waves of 8-13 Hz, beta waves of 14-30 Hz, and gamma waves of 30 Hz or higher.
  • independent component analysis is used as a denoising method to remove biological noise other than brain waves. This allows the acquisition unit 15a to acquire brain wave information from which noise components other than brain waves have been removed.
  • the acquisition unit 15a also acquires category information that represents a classification category of a perceived image or an imaginary image. For example, the acquisition unit 15a accepts a user's input of linguistic information that represents the user's perception or imagination content, and converts it into perceived category information or imaginary category information.
  • the acquisition unit 15a may acquire category information using the trained category estimation model 14b. That is, the acquisition unit 15a may acquire category information estimated by the category estimation model 14b from the acquired brain wave information using the category estimation model 14b trained with brain wave information at least when the user perceived or imagined an image and at least one of category information representing a classification category of the image perceived by the user and category information of the image imagined by the user as training data.
  • the learning unit 15c described later trains the category estimation model 14b.
  • the acquisition unit 15a may store the acquired brainwave information and category information in the storage unit 14, or may transfer the information to the following functional units without storing it in the storage unit 14.
  • the generating unit 15b uses the acquired brainwave information and the category information of each image to generate an image that corresponds to the category information.
  • the generating unit 15b uses the model 14a trained with brainwave information at least when the user perceived or imagined an image, and category information representing the classification category of the image perceived by the user and/or category information of the image imagined by the user as training data, to generate an image corresponding to the category information from the acquired brainwave information and category information.
  • FIG. 3 is a diagram for explaining the processing of the generation unit.
  • model 14a is specifically composed of a feature conversion mechanism and a decoder.
  • the feature conversion mechanism converts EEG information and perceptual category information into perceptual features, and converts EEG information and imaginary category information into imaginary features.
  • the decoder converts perceptual features into perceptual images, and converts imaginary features into imaginary images.
  • the decoder that converts perceptual features into perceptual images and the decoder that converts imaginary features into imaginary images may be the same decoder, or may be different decoders specialized for each. Using the same decoder reduces calculation costs. On the other hand, using different decoders improves the accuracy with which each of the perceptual images and imaginary images is generated.
  • the learning unit 15c learns the model 14a. For example, the learning unit 15c learns the model 14a using previously acquired electroencephalogram information and correct perceptual category information and imagination category information.
  • the electroencephalogram information may be information converted into electroencephalogram features such as power spectral density and instantaneous phase. In that case, the generation unit 15b generates an image using similar electroencephalogram features as electroencephalogram information.
  • FIG. 4 is a diagram for explaining the processing of the learning unit.
  • the learning unit 15c acquires, as learning data of electroencephalogram information, (a) electroencephalogram information when the user perceives an image, (b) electroencephalogram information when the user imagines an image, and (c) electroencephalogram information when the perceived image and the imagined image differ.
  • the learning unit 15c acquires, as learning data of category information, perceptual category information of the perceived images or imagined category information of the imagined images of (a) to (c).
  • the learning unit 15c learns the model 14a so as to minimize the difference between the generated perceived image and the correct image presented for perception, and the difference between the generated imaginary image and the correct image for imagination.
  • the learning unit 15c adjusts parameters only in the feature conversion mechanism, and does not change the parameters of the decoder.
  • the learning unit 15c adjusts parameters only in the feature conversion mechanism, and does not change the parameters of the decoder.
  • a decoder for this image generation that has been trained in advance to be able to generate images from a latent space, it is possible to reduce the learning cost.
  • the learning unit 15c may change the parameters of the decoder. In that case, the number of parameters to be learned increases, which increases the calculation cost, but improves the accuracy of the generated image.
  • the learning unit 15c may also learn the category estimation model 14b as described above. In that case, similar to the case of learning the model 14a shown in FIG. 4, the learning unit 15c acquires, as learning data for the electroencephalogram information, (a) electroencephalogram information when the user perceives an image, (b) electroencephalogram information when the user imagines an image, and (c) electroencephalogram information when the perceived image and the imagined image differ. In this case, too, the electroencephalogram information may be information converted into electroencephalogram feature quantities such as power spectral density and instantaneous phase.
  • the learning unit 15c also acquires perceptual category information of the perceptual images (a) to (c) or imaginary category information of the imaginary images as correct answer data.
  • the learning unit 15c then learns a category estimation model 14b that estimates category information from electroencephalogram information.
  • the estimated category information is then acquired by the acquisition unit 15a and supplied to the generation unit 15b. This makes it possible to generate images more efficiently simply by acquiring electroencephalogram information.
  • the presentation unit 15d presents the generated perceptual image and imaginary image to the user via the output unit 12.
  • FIG. 5 is a flowchart showing the procedure of the image generation process.
  • the flowchart in Fig. 5 is started, for example, when a user performs an operation input to instruct the start of the process.
  • the acquisition unit 15a acquires electroencephalogram information at least when the user perceives or imagines an image (step S1).
  • the acquisition unit 15a also acquires category information at least of either perceptual category information of the perceived image perceived by the user or imaginative category information of the imaginative image imagined by the user (step S2).
  • the acquisition unit 15a may acquire category information estimated by the category estimation model 14b from the acquired brainwave information, using the category estimation model 14b trained on brainwave information at least when the user perceived or imagined an image, and at least one of the category information of the perceived category information of the perceived image perceived by the user and the imaginary category information of the imaginary image imagined by the user, as learning data.
  • the generating unit 15b uses the acquired brainwave information and at least one of the category information of the perceptual category information of the perceptual image and the imaginative category information of the imaginative image to generate a perceptual image or an imaginative image corresponding to the category information (step S3).
  • the generating unit 15b uses the model 14a trained on brainwave information at least when the user perceived or imagined an image, and at least one of the category information of the perceived category information of the perceived image perceived by the user and the imaginary category information of the imaginary image imagined by the user as learning data, to generate a perceived image or an imaginary image corresponding to the category information from the acquired brainwave information and category information.
  • the presentation unit 15d presents the generated perceptual image and imaginary image to the user via the output unit 12 (step S4). This completes the series of image generation processes.
  • the acquiring unit 15a acquires electroencephalogram information at least when the user perceived or imagined an image, and at least one of category information of perceptual category information indicating a classification category of the perceived image perceived by the user and imaginary category information of the imaginary image imagined by the user.
  • the generating unit 15b uses the acquired electroencephalogram information and the category information of each image to generate an image corresponding to the category information.
  • the generation unit 15b uses the model 14a trained with brainwave information at least when the user perceived or imagined an image, and imaginary category information representing a classification category of the perceived image perceived by the user, and imaginary category information of the imaginary image imagined by the user as learning data, to generate an image corresponding to the category information from the acquired brainwave information and category information.
  • the image generating device 10 can easily generate both perceptual images and imaginary images from brain waves by separately using perceptual category information and imaginary category information.
  • the learning unit 15c also learns the model 14a. This makes it possible to generate a perceived image and an imaginary image with high accuracy.
  • the acquisition unit 15a may also acquire category information estimated by the category estimation model 14b from the acquired brain wave information, using brain wave information at least when the user perceived or imagined an image, and at least one of perceptual category information indicating a classification category of a perceived image perceived by the user and imaginary category information of an imaginary image imagined by the user as learning data. This makes it possible to generate images corresponding to category information more efficiently, simply by acquiring brain wave information.
  • the learning unit 15c learns the category estimation model 14b. This makes it possible to estimate category information with even higher accuracy.
  • a program in which the process executed by the image generating device 10 according to the above embodiment is written in a language executable by a computer can also be created.
  • the image generating device 10 can be implemented by installing an image generating program that executes the above image generating process as package software or online software on a desired computer.
  • the information processing device can function as the image generating device 10 by executing the above image generating program on an information processing device.
  • the information processing device here includes desktop or notebook personal computers.
  • the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and even slate terminals such as PDAs (Personal Digital Assistants).
  • the functions of the image generating device 10 may be implemented on a cloud server.
  • FIG. 6 is a diagram showing an example of a computer that executes an image generation program.
  • the computer 1000 has, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
  • the memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012.
  • the ROM 1011 stores a boot program such as a BIOS (Basic Input Output System).
  • BIOS Basic Input Output System
  • the hard disk drive interface 1030 is connected to a hard disk drive 1031.
  • the disk drive interface 1040 is connected to a disk drive 1041.
  • a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041.
  • the serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example.
  • the video adapter 1060 is connected to a display 1061, for example.
  • the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. Each piece of information described in the above embodiment is stored, for example, in the hard disk drive 1031 or memory 1010.
  • the image generation program is stored in the hard disk drive 1031, for example, as a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the image generation device 10 described in the above embodiment is written is stored in the hard disk drive 1031.
  • data used for information processing by the image generation program is stored as program data 1094, for example, in the hard disk drive 1031.
  • the CPU 1020 reads the program module 1093 and program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the procedures described above.
  • the program module 1093 and program data 1094 related to the image generation program are not limited to being stored in the hard disk drive 1031, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1041 or the like.
  • the program module 1093 and program data 1094 related to the image generation program may be stored in another computer connected via a network, such as a LAN or WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)

Abstract

Dans un dispositif de génération d'image (10), une unité d'acquisition (15a) acquiert des informations d'électroencéphalogramme dans au moins l'un des cas où l'utilisateur perçoit ou imagine une image, et au moins l'une des informations de catégorie représentant une catégorie de classification de l'image perçue par l'utilisateur ou des informations de catégorie concernant l'image imaginée par l'utilisateur. Une unité de génération (15b) utilise les informations d'électroencéphalogramme acquises et les informations de catégorie concernant chaque image pour générer une image correspondant aux informations de catégorie.
PCT/JP2022/041996 2022-11-10 2022-11-10 Procédé de génération d'image, dispositif de génération d'image et programme de génération d'image WO2024100859A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/041996 WO2024100859A1 (fr) 2022-11-10 2022-11-10 Procédé de génération d'image, dispositif de génération d'image et programme de génération d'image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/041996 WO2024100859A1 (fr) 2022-11-10 2022-11-10 Procédé de génération d'image, dispositif de génération d'image et programme de génération d'image

Publications (1)

Publication Number Publication Date
WO2024100859A1 true WO2024100859A1 (fr) 2024-05-16

Family

ID=91032096

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/041996 WO2024100859A1 (fr) 2022-11-10 2022-11-10 Procédé de génération d'image, dispositif de génération d'image et programme de génération d'image

Country Status (1)

Country Link
WO (1) WO2024100859A1 (fr)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019235458A1 (fr) * 2018-06-04 2019-12-12 国立大学法人大阪大学 Dispostif et procédé d'estimation d'image de souvenir, programme de commande et support d'enregistrement

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019235458A1 (fr) * 2018-06-04 2019-12-12 国立大学法人大阪大学 Dispostif et procédé d'estimation d'image de souvenir, programme de commande et support d'enregistrement

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
RYOHEI FUKUMA; TAKUFUMI YANAGISAWA; SHINJI NISHIMOTO; HIDENORI SUGANO; KENTARO TAMURA; SHOTA YAMAMOTO; YASUSHI IIMURA; YUYA FUJITA: "Voluntary control of semantic neural representations by imagery with conflicting visual stimulation", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 7 November 2021 (2021-11-07), 201 Olin Library Cornell University Ithaca, NY 14853, XP091106092 *
SHEN GUOHUA, HORIKAWA TOMOYASU, MAJIMA KEI, KAMITANI YUKIYASU: "Deep image reconstruction from human brain activity", PLOS COMPUTATIONAL BIOLOGY, vol. 15, no. 1, 14 January 2019 (2019-01-14), pages e1006633, XP093088770, DOI: 10.1371/journal.pcbi.1006633 *

Similar Documents

Publication Publication Date Title
Lakhani et al. Machine learning in radiology: applications beyond image interpretation
Metzger et al. Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis
Cox et al. 'Breaking'position-invariant object recognition
Bautista et al. Neural responses to grammatically and lexically degraded speech
Liu et al. Emotion recognition from single-trial EEG based on kernel Fisher's emotion pattern and imbalanced quasiconformal kernel support vector machine
KR102424085B1 (ko) 기계-보조 대화 시스템 및 의학적 상태 문의 장치 및 방법
von Wangenheim et al. Creating a web infrastructure for the support of clinical protocols and clinical management: an example in teledermatology
Renton et al. Optimising non-invasive brain-computer interface systems for free communication between naïve human participants
Tang et al. Optimizing SSVEP-based BCI system towards practical high-speed spelling
JP6822220B2 (ja) 推論結果可視化装置、推論結果可視化方法、及びプログラム
Pereira Soares et al. Testing potential transfer effects in heritage and adult L2 bilinguals acquiring a mini grammar as an additional language: An ERP approach
Xi et al. UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language
WO2024100859A1 (fr) Procédé de génération d'image, dispositif de génération d'image et programme de génération d'image
Wilson et al. EEG-based BCI dataset of semantic concepts for imagination and perception tasks
Kinder et al. Short article: Sequence learning at optimal stimulus–response mapping: Evidence from a serial reaction time task
CN117078811A (zh) 模型训练方法、图像生成方法、动画生成方法及系统
WO2024100844A1 (fr) Dispositif de génération d'expression faciale, procédé de génération d'expression faciale et programme de génération d'expression faciale
CN113781998B (zh) 基于方言纠正模型的语音识别方法、装置、设备及介质
Chen et al. Mindgpt: Interpreting what you see with non-invasive brain recordings
Curlin What does any of this have to do with being a physician? Kierkegaardian irony and the practice of medicine
Qi et al. Cases: A cognition-aware smart eyewear system for understanding how people read
Asaadi et al. Effects and prediction of cognitive load on encoding model of brain response to auditory and linguistic stimuli in educational multimedia
Komeiji et al. Feasibility of decoding covert speech in ECoG with a Transformer trained on overt speech
Huang et al. On-device edge-learning for cardiac abnormality detection using a bio-inspired and spiking shallow network
Wang et al. Exploring the Potential of Large Language Models in Artistic Creation: Collaboration and Reflection on Creative Programming

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22965183

Country of ref document: EP

Kind code of ref document: A1