CN104598575B - The brain-machine interaction image indexing system being imaged based on real-time functional magnetic resonance - Google Patents
The brain-machine interaction image indexing system being imaged based on real-time functional magnetic resonance Download PDFInfo
- Publication number
- CN104598575B CN104598575B CN201510016541.8A CN201510016541A CN104598575B CN 104598575 B CN104598575 B CN 104598575B CN 201510016541 A CN201510016541 A CN 201510016541A CN 104598575 B CN104598575 B CN 104598575B
- Authority
- CN
- China
- Prior art keywords
- image
- real
- subject
- time
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5838—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Library & Information Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Magnetic Resonance Imaging Apparatus (AREA)
Abstract
The invention discloses a kind of brain-machine interaction image indexing system being imaged based on real-time functional magnetic resonance, overcome in the prior art, the problem of brain-machine interaction image indexing system still needs to improve.The invention, which contains, to have the following steps：1st, by gathering fMRI data, training image semanteme real-time grading model, the real-time assessment models of characteristics of image similitude and vision attention real-time decoding model in advance；2nd, an image to be retrieved is presented as visual stimulus to subject, while gathering subject fMRI data, uses the semantic classes of image application processing Model checking image to be retrieved；3rd, the image and image to be retrieved that represent differentiated semantic classes result are presented to subject together, whether subject is correct by the semantic classes of vision attention feedback image semantic classification Model checking；4th, the result differentiated according to the semantic classification of subject feedback, provides the image retrieved.The present invention is significant to improving fMRI image retrieval accuracy.
Description
Technical field
The invention is related to a kind of brain-machine interaction image indexing system, it is more particularly to a kind of based on real-time functional magnetic resonance into
The brain-machine interaction image indexing system of picture.
Background technology
Functional mri（functional Magnetic Resonance Imaging）Main method be blood oxygen
Horizontal dependency (BOLD, Blood Oxygenation Level Dependent) method, this method is gathered using MRI machine
The information of brain, main reflection is activated in brain area capillary using voxel as the instantaneous change of unit oxygenated blood red blood cell concentration
Change, abbreviation BOLD-fMRI (Blood oxygenation level dependent functional magnetic
resonance imaging, BOLD-fMRI).Compared with other cerebral function imaging modes, BOLD-fMRI be imaged on speed and
With with very big advantage, especially causing the relative complex brain Cognitive Function Research such as visual performance to obtain in terms of spatial resolution
To carry out in a deep going way, therefore, the main flow that brain Cognitive Study has become cerebral function imaging is carried out using it.
Real-time functional magnetic resonance is imaged（rtfMRI, real-time functional Magnetic Resonance
Imaging）By the way of online acquisition and processing data, make data processing suitable with data acquisition speed, can be in scanning
During real-time monitoring data quality and one gather the repetition time（TR, Time of Repetition）Interior completion is simultaneously real
When obtain result, the brain activity state that result reflects is fed back into subject in the way of vision or the sense of hearing, so that shape
Into from subject to fMRI again to the information circuits of subject so that the brain-machine interaction based on rtfMRI is possibly realized.
Existing achievement in research, is mainly based upon fMRI image classification and identification, and these researchs are generally using offline
The mode of data processing.Although these models and method can complete the assessment for image, semantic classification and characteristic similarity,
But do not possess online image retrieval ability also, it is impossible to be individually formed image indexing system.
It is always the Disciplinary Frontiers and hot issue in brain science research, spy for the information decoding of human brain visual performance
It is not what kind of brain response research human brain visual performance related brain areas can produce to the image seen, achieves a series of key
Achievement.2001, Haxby et al. utilized the difference of the class image human brain visual performance fMRI data of multi-voxel proton analysis method research 8,
Give preferable semantic classification result.2008, Kay et al. utilized Gabor wavelet pyramid simulation human brain Visual Neuron
Receptive field, the image similarity assessment models based on brain signal are established, so as to realize accurately identifying for 120 width images.
Naselaris in 2009 et al. proposes image similarity and semantic fusion model based on Bayes frameworks.
LaConte in 2007 constructs the rtfMRI systems of online classification that can carry out first, and the system can be using complete
Brain image is trained, and On-line funchon state classification is carried out using trained listening group, and the system realizes two class mood work(
The classification of energy, demonstrates the feasibility that brain-computer interface is built using cerebral function state classification method.Lee in 2009 et al. is built
RtfMRI systems are classified to right-hand man to the signal of the functional areas of finger, then the control that classification results are converted into mechanical arm refers to
Order, completes control of the brain-computer interface to external equipment.Andersson in 2013 et al. is by SVM classifier to visual attention
Direction is classified, and the rtfMRI brain machine interface systems that tele-robotic is roamed can be manipulated by constructing.There is presently no
RtfMRI brain-computer interfaces technology is applied to image retrieval by research.
The content of the invention
There is provided a kind of performance for the problem of still needing to improve instant invention overcomes brain-machine interaction image indexing system in the prior art
The brain-machine interaction image indexing system being preferably imaged based on real-time functional magnetic resonance.
The present invention technical solution be to provide it is a kind of have steps of based on real-time functional magnetic resonance be imaged
Brain-machine interaction image indexing system：Containing having the following steps：
Step 1, by gathering fMRI data in advance, training image semanteme real-time grading model, characteristics of image similitude are real
When assessment models and vision attention real-time decoding model；
Step 2, to subject an image to be retrieved is presented as visual stimulus, while gathering subject fMRI data, uses
The semantic classes of image application processing Model checking image to be retrieved；
Step 3, the image and image to be retrieved that represent differentiated semantic classes result be presented to subject together, pinged
Whether the semantic classes for crossing vision attention feedback image semantic classification Model checking is correct；
Step 4, the result differentiated according to the semantic classification of subject feedback, fused images semanteme real-time grading model and image
The real-time assessment models of similitude, provide the image retrieved from image library.
The step 1 is concretely comprised the following steps：Subject participates in a brain area registered placement experiment, an image, semantic point in advance
Class training experiment and a vision attention classification based training experiment, gather corresponding fMRI data, are respectively trained based on supporting vector
Realtime graphic semantic classification model, the vision attention real-time decoding model of machine are similar with based on the pyramidal image of Gabor wavelet
Property assessment models.
The step 2 is concretely comprised the following steps：An image to be retrieved is presented as visual stimulus to subject, gathers simultaneously
FMRI data, stimulus duration is 3 TR, and then stimulating image is entered using the image, semantic real-time grading model trained
Row semantic classification, whole data prediction and assorting process are completed in 1 TR.
The step 3 is concretely comprised the following steps：The image and image to be retrieved of differentiated semantic classes result will be represented together
Subject is presented to, image to be retrieved is placed on the left side of visual stimulus screen, and semantic classes result images are placed on visual stimulus
The right side of screen；Subject differentiates whether result is correctly fed back by vision attention to semanteme, if kind judging is correct, notes
Meaning sees category result image, if kind judging mistake, notes seeing image to be retrieved, vision attention interaction continues 2 TR；
The visual attention information of subject then is calculated using vision attention real-time decoding model, so as to obtain semantic classes result of determination
Whether correct information, data prediction and vision attention decoding process are completed in 1 TR；The result of vision decoding is next
Individual TR is presented to subject by visual stimulus screen.
The step 4 is concretely comprised the following steps：Result is differentiated according to the semantic classification of subject feedback, result is correct if differentiating,
Directly use the real-time assessment models of image similarity, using in image library with the width image of image similarity highest 10 to be retrieved as
Retrieval result is exported in visual stimulus screen；If differentiating result mistake, the semantic real-time grading of Bayes methods fused images is used
Model and the real-time assessment models of image similarity, provide 10 maximum width images of posterior probability as retrieval result in visual stimulus
Exported on screen, all calculating of the process and result are shown in completion in 2 TR.
Compared with prior art, the present invention is had based on the brain-machine interaction image indexing system that real-time functional magnetic resonance is imaged
Advantages below：The system innovation proposes to merge on image, semantic real-time grading model and the real-time assessment models of image similarity,
And online image retrieval is realized with reference to rtfMRI technologies, real-time brain-machine interaction method is introduced into image retrieval, to improving fMRI figures
As retrieval accuracy is significant.This image indexing system can be used as and utilize human brain visual information Intelligent treatment mechanism
A platform, can also regard as real-time functional magnetic resonance imaging on the basis of brain-machine interaction Vision information processing example,
For effect and value of the research with uniqueness of human brain Vision information processing mechanism.
Brief description of the drawings
Fig. 1 is the schematic flow sheet for the brain-machine interaction image indexing system that the present invention is imaged based on real-time functional magnetic resonance.
Embodiment
The brain-machine interaction figure being imaged with reference to the accompanying drawings and detailed description to the present invention based on real-time functional magnetic resonance
As searching system is described further：
As illustrated, the present embodiment is broadly divided into model training, semantic classes judges, the interaction of classification judged result and retrieval
As a result four functional modules are exported.The system is by the preparation function of first online image retrieval of Implement of Function Module, mainly
Including collection fMRI data in advance, training image semanteme real-time grading model, the real-time assessment models of image similarity and vision note
Meaning real-time decoding model；Then online retrieving is realized by rear three modules, including：It is in subject by second module
An existing image to be retrieved is as visual stimulus, while subject fMRI data are gathered, first using image application processing Model checking
The semantic classes of image to be retrieved；Then the image of differentiated semantic classes result will be represented and to be retrieved by the 3rd module
Image is presented to subject together, and whether just subject passes through the semantic classes of vision attention feedback image semantic classification Model checking
Really；The result differentiated finally by the 4th module according to the semantic classification of subject feedback, fused images semanteme real-time grading mould
Type and the real-time assessment models of image similarity, provide the image that image indexing system is retrieved.
Comprise the following steps that, step 1：By gathering fMRI data, training image semanteme real-time grading model, image in advance
The real-time assessment models of similitude and vision attention real-time decoding model, specific method is：Subject participates in a brain area registration in advance
Positioning experiment, an image application processing training experiment and a vision attention classification based training experiment, gather corresponding fMRI numbers
According to the realtime graphic semantic classification model based on SVMs, vision attention real-time decoding model is respectively trained and is based on
The pyramidal image similarity assessment model of Gabor wavelet.
Step 2：An image to be retrieved is presented as visual stimulus to subject, while gathering subject fMRI data, uses
The semantic classes of image, semantic real-time grading Model checking image to be retrieved, specific method is：It is to be retrieved to subject presentation one
Image is as visual stimulus, while gathering fMRI data, stimulus duration is 3 TR（6s）.Then utilize the figure trained
The semantic real-time grading model of picture carries out semantic classification to stimulating image, and whole data prediction and assorting process are complete in 1 TR
Into.
Step 3：The image and image to be retrieved that represent differentiated semantic classes result are presented to subject together, pinged
Whether the semantic classes for crossing the semantic real-time grading Model checking of vision attention feedback image is correct, and specific method is：Institute will be represented
Differentiate that the image and image to be retrieved of semantic classes result are presented to subject together, image to be retrieved is placed on visual stimulus screen
Left side, semantic classes result images are placed on the right side of visual stimulus screen.Subject is differentiated by vision attention to semanteme to be tied
Whether fruit is correctly fed back, if kind judging is correct, notes seeing category result image, if kind judging mistake, notes
See image to be retrieved.Vision attention interaction continues 2 TR.Then subject is calculated using vision attention real-time decoding model
Visual attention information, so as to obtain the whether correct information of semantic classes result of determination, data prediction and vision attention solution
Code process is completed in 1 TR.The result of vision decoding is presented to subject in next TR by visual stimulus screen.
Step 4：The result differentiated according to the semantic classification of subject feedback, fused images semanteme real-time grading model and image
The real-time assessment models of similitude, provide the image that image indexing system is retrieved, and specific method is：According to the semanteme of subject feedback
Discriminant classification result, if differentiating, result is correct, directly the use real-time assessment models of image similarity, by image library with it is to be retrieved
The width image of image similarity highest 10 is exported as retrieval result in visual stimulus screen；If differentiating result mistake, use
Bayes methods fused images semanteme real-time grading model and the real-time assessment models of image similarity, provide posterior probability maximum
10 width images are exported as retrieval result in visual stimulus screen, the process it is all calculating and result be shown in it is complete in 2 TR
Into.
Claims (5)
1. a kind of brain-machine interaction image indexing system being imaged based on real-time functional magnetic resonance, it is characterised in that：Contain following step
Suddenly：
Step 1, by gathering fMRI data in advance, training image semanteme real-time grading model, characteristics of image similitude are commented in real time
Estimate model and vision attention real-time decoding model；
Step 2, to subject an image to be retrieved is presented as visual stimulus, while gathering subject fMRI data, uses image
The semantic classes of semantic classification Model checking image to be retrieved；
Step 3, the image and image to be retrieved that represent differentiated semantic classes result be presented to subject together, subject is by regarding
Feel and notice whether the semantic classes of feedback image semantic classification Model checking is correct；
Step 4, the result differentiated according to the semantic classification of subject feedback, fused images semanteme real-time grading model are similar with image
The real-time assessment models of property, provide the image retrieved from image library.
2. the brain-machine interaction image indexing system according to claim 1 being imaged based on real-time functional magnetic resonance, its feature
It is：The step 1 is concretely comprised the following steps：Subject participates in a brain area registered placement experiment, an image application processing in advance
Training experiment and a vision attention classification based training experiment, gather corresponding fMRI data, are respectively trained based on SVMs
Realtime graphic semantic classification model, vision attention real-time decoding model and based on the pyramidal image similarity of Gabor wavelet
Assessment models.
3. the brain-machine interaction image indexing system according to claim 1 being imaged based on real-time functional magnetic resonance, its feature
It is：The step 2 is concretely comprised the following steps：An image to be retrieved is presented as visual stimulus to subject, while gathering fMRI
Data, stimulus duration is 3 TR, then carries out language to stimulating image using the image, semantic real-time grading model trained
Justice classification, whole data prediction and assorting process are completed in 1 TR.
4. the brain-machine interaction image indexing system according to claim 1 being imaged based on real-time functional magnetic resonance, its feature
It is：The step 3 is concretely comprised the following steps：The image and image to be retrieved that represent differentiated semantic classes result are presented together
To subject, image to be retrieved is placed on the left side of visual stimulus screen, and semantic classes result images are placed on visual stimulus screen
Right side；Subject differentiates whether result is correctly fed back by vision attention to semanteme, if kind judging is correct, notes seeing
Category result image, if kind judging mistake, notes seeing image to be retrieved, vision attention interaction continues 2 TR；Then
The visual attention information of subject is calculated using vision attention real-time decoding model, so as to whether obtain semantic classes result of determination
Correct information, data prediction and vision attention decoding process are completed in 1 TR；The result of vision decoding is in next TR
Subject is presented to by visual stimulus screen.
5. the brain-machine interaction image indexing system according to claim 1 being imaged based on real-time functional magnetic resonance, its feature
It is：The step 4 is concretely comprised the following steps：Result is differentiated according to the semantic classification of subject feedback, result is correct if differentiating, directly
Using the real-time assessment models of image similarity, retrieval will be used as with the width image of image similarity highest 10 to be retrieved in image library
As a result exported in visual stimulus screen；If differentiating result mistake, the semantic real-time grading model of Bayes methods fused images is used
With the real-time assessment models of image similarity, 10 maximum width images of posterior probability are provided as retrieval result in visual stimulus screen
Upper output, all calculating of the process and result are shown in completion in 2 TR.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510016541.8A CN104598575B (en) | 2015-01-14 | 2015-01-14 | The brain-machine interaction image indexing system being imaged based on real-time functional magnetic resonance |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510016541.8A CN104598575B (en) | 2015-01-14 | 2015-01-14 | The brain-machine interaction image indexing system being imaged based on real-time functional magnetic resonance |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104598575A CN104598575A (en) | 2015-05-06 |
| CN104598575B true CN104598575B (en) | 2017-09-01 |
Family
ID=53124360
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510016541.8A Active CN104598575B (en) | 2015-01-14 | 2015-01-14 | The brain-machine interaction image indexing system being imaged based on real-time functional magnetic resonance |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104598575B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105426446A (en) * | 2015-11-06 | 2016-03-23 | 西安电子科技大学 | Application of Gabor-Zernike characteristics in medical image retrieval |
| CN106022384B (en) * | 2016-05-27 | 2019-07-12 | 中国人民解放军信息工程大学 | Image attention target semanteme dividing method based on fMRI visual performance data DeconvNet |
| CN106056602B (en) * | 2016-05-27 | 2019-06-28 | 中国人民解放军信息工程大学 | FMRI visual performance datum target extracting method based on CNN |
| CN108121750B (en) * | 2016-11-30 | 2022-07-08 | 西门子公司 | Model processing method and device and machine readable medium |
| CN107092925B (en) * | 2017-03-30 | 2019-09-17 | 中国人民解放军国防科学技术大学 | Cerebral function magnetic resonance imaging blind source separation method based on SIM algorithm in groups |
| CN109567803B (en) * | 2018-12-18 | 2022-07-19 | 中国人民解放军战略支援部队信息工程大学 | Real-time neural feedback technology-based hippocampus self-regulation analysis method |
| CN109685804B (en) * | 2019-01-04 | 2021-04-23 | 清华大学深圳研究生院 | Multi-channel head magnetic resonance imaging tissue segmentation method |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103778240A (en) * | 2014-02-10 | 2014-05-07 | 中国人民解放军信息工程大学 | Image retrieval method based on functional magnetic resonance imaging and image dictionary sparse decomposition |
-
2015
- 2015-01-14 CN CN201510016541.8A patent/CN104598575B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103778240A (en) * | 2014-02-10 | 2014-05-07 | 中国人民解放军信息工程大学 | Image retrieval method based on functional magnetic resonance imaging and image dictionary sparse decomposition |
Non-Patent Citations (2)
| Title |
|---|
| "Identifying natural images from human brain activity";Kendrick N.Kay 等;《Nature》;20080320;第452卷(第20期);全文 * |
| "Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1?";BRUNO A.OLSHAUSEN 等;《Vision Research》;19971231;第37卷(第23期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104598575A (en) | 2015-05-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104598575B (en) | The brain-machine interaction image indexing system being imaged based on real-time functional magnetic resonance | |
| Cichy et al. | AM/EEG-fMRI fusion primer: resolving human brain responses in space and time | |
| US20170356976A1 (en) | System and method for quantifying cell numbers in magnetic resonance imaging (mri) | |
| Chen et al. | A unified model of human semantic knowledge and its disorders | |
| CN109222972B (en) | fMRI whole brain data classification method based on deep learning | |
| Esfahani et al. | Classification of primitive shapes using brain–computer interfaces | |
| Wang et al. | A novel multimodal MRI analysis for Alzheimer's disease based on convolutional neural network | |
| Nichols et al. | The neurobiology of cognition | |
| WO2017177128A1 (en) | Systems and methods for deep reinforcement learning using a brain-artificial intelligence interface | |
| CN103778240B (en) | Image retrieval method based on functional magnetic resonance imaging and image dictionary sparse decomposition | |
| CN104809480A (en) | Retinal vessel segmentation method of fundus image based on classification and regression tree and AdaBoost | |
| CN104799852B (en) | The extracting method of Mental imagery EEG signals feature based on the learning machine own coding that transfinites | |
| CN106127263A (en) | The human brain magnetic resonance image (MRI) classifying identification method extracted based on three-dimensional feature and system | |
| CN105534534A (en) | Emotion recognition method, device and system based on real-time functional magnetic resonance | |
| KR102481817B1 (en) | Method for Estimating Driver's Gaze Zone by Transfer Learning | |
| CN106022384A (en) | Image attention semantic target segmentation method based on fMRI visual function data DeconvNet | |
| CN109987102A (en) | A kind of method and apparatus of the High Precision Automatic identification driver's unsafe behaviors based on convolutional neural networks | |
| Sikander et al. | A novel machine vision-based 3D facial action unit identification for fatigue detection | |
| CN111714118A (en) | Brain cognition model fusion method based on ensemble learning | |
| CN112101424A (en) | Generation method, identification device and equipment of retinopathy identification model | |
| Zhao et al. | Two-stage spatial temporal deep learning framework for functional brain network modeling | |
| Zuo et al. | Deep Learning-based Eye-Tracking Analysis for Diagnosis of Alzheimer's Disease Using 3D Comprehensive Visual Stimuli | |
| CN110604572A (en) | Brain activity state identification method based on human brain characteristic map | |
| Gu et al. | An integrated Gaussian mixture model to estimate vigilance level based on EEG recordings | |
| CN103345754B (en) | Method for detecting image edge based on the response of cortical neuron visual direction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Yan Bin Inventor after: Li Yongli Inventor after: Tong Li Inventor after: Zheng Zaizhou Inventor after: Wang Linyuan Inventor after: Ma Lijia Inventor after: Li Zhonglin Inventor after: Zeng Ying Inventor after: Zhang Chi Inventor before: Yan Bin Inventor before: Li Yongli Inventor before: Tong Li Inventor before: Zheng Zaizhou Inventor before: Wang Linyuan Inventor before: Ma Lijia Inventor before: Li Zhonglin Inventor before: Zeng Ying Inventor before: Zhang Chi |