WO2023173804A1

WO2023173804A1 - Brain-computer information fusion classification method and system for shared subspace learning

Info

Publication number: WO2023173804A1
Application number: PCT/CN2022/134523
Authority: WO
Inventors: 梁继民; 郭开泰; 郑洋; 闫健璞; 胡海虹; 任胜寒; 王梓宇
Original assignee: 西安电子科技大学
Priority date: 2022-03-16
Filing date: 2022-11-26
Publication date: 2023-09-21
Also published as: CN114742092A

Abstract

The present invention relates to the technical field of brain-computer interface technology applications, and disclosed is a brain-computer information fusion classification method and system for shared subspace learning. The brain-computer information fusion classification method comprises a training stage and a reasoning stage. In the training stage, paired images and brain response data are utilized, shared subspace model parameters of the images and brain responses are optimized by means of a contrastive learning policy of positive/negative sample sampling, and an image classifier is trained; and in the reasoning stage, image features are extracted for classification, and an application target of the whole brain-computer information fusion classification system is achieved. The brain-computer information fusion classification system for shared subspace learning of the present invention can train a shared subspace in an end-to-end mode, efficient migration of brain cognitive information is achieved, and the performance of an image classification task in a complex open scenario is improved; by means of an application that "the brain is not in a loop", the efficiency and the stability in the practical application are improved, and the present invention has a wide application prospect under a new normal form of brain-computer information cooperative work.

Description

A brain-computer information fusion classification method and system based on shared subspace learning

Technical field

The invention belongs to the technical field of brain-computer interface technology application, and in particular relates to a brain-computer information fusion classification method and system for shared subspace learning.

Background technique

In recent years, artificial intelligence methods represented by deep learning have developed rapidly, and their performance in image classification tasks has surpassed humans. However, at present, deep learning systems are only used on a large scale in limited specific simple scenarios such as face recognition, speech recognition, and optical character recognition. They are mainly driven by data and need to build appropriate models and use sufficient computing power to fully Mining distribution rules in massive data cannot achieve human-like cognitive abilities. Therefore, when faced with complex open scenarios such as complex targets/backgrounds, occlusions, and counter-interference, such as autonomous driving, remote sensing image interpretation, etc., it is difficult to fully simulate the complete distribution of the physical world despite sufficient data, and it is difficult to establish The target is universally robust representation, resulting in a sharp drop in performance, which is far from reaching human-like strong generalization capabilities.

Currently, in complex open application scenarios with low tolerance for erroneous decision-making, such as military applications, medical diagnosis, and autonomous driving, manual interpretation methods based on visual recognition experts are still the mainstream means of image recognition and decision-making. However, the manual interpretation process of visual experts is a subjective visual cognitive decision-making process. Their behavior may be affected by external environmental factors, fatigue, injuries and other internal factors, resulting in decision-making errors. Compared with machine intelligence, it is difficult for visual experts to To achieve long-term, high-intensity, and large-scale real-time interpretation, visual experts in most fields require long-term, high-investment training to become qualified experts.

In view of the fact that the brain is the material basis and control center for the behavior and cognition of primates represented by humans, from an engineering perspective, brain-computer interface technology is used to build a brain-computer information fusion system to achieve a deep connection between biological intelligence and machine intelligence. Information perception, interaction and integration are expected to form a more advanced intelligent model. This brain-computer information fusion system provides a new processing paradigm for image classification tasks in complex open environments by migrating the brain's high-level cognitive information into machine intelligence models.

Currently, there are two main types of technologies in the industry for building brain-computer information fusion classification systems. Existing technology one: fusion method based on image-brain response complementary information; existing technology two: shared subspace learning based on image-brain response correlation information method. The main theoretical basis of the existing technology one is to use brain response and image information as expressions of image targets from different sources, and use information fusion methods to maximize the complementary information of the two to obtain a more complete joint representation of image target expressions. Its technical characteristics It lies in designing reasonable information fusion methods to maximize the effective information of different modalities. The main representative methods of existing technology one include: "A brain-computer interface for the detection of mine-like objects in sidescan sonar imagery" (IEEE Journal of Oceanic Engineering, 2016, 41(1):123-138) using feature level The joint method fuses the image Haar type features and the subject's EEG features, effectively improving the performance of mine target detection in side-scan sonar images; "An adaptive brain-computer information fusion classification method and system" (Application No.: CN202111017296.4) By constructing a feature reliability learning model of two modalities, learning the feature reliability of images and brain responses, adaptively adjusting the fusion weights of their different modalities, and using adaptive fusion features for classification, this method The complementary information of the two modalities is maximized and the performance of image classification is improved. However, the existing technology requires real-time participation of the brain in the application paradigm. This "brain-in-the-loop" application paradigm is limited by subjective factors such as fatigue and injuries of the subjects, and it is difficult to achieve real-time and high-intensity The fully automated application does not fully utilize the respective advantages of the brain and the machine. The main theory of the second prior art is to construct a shared representation space based on the relevant information between the image and the brain response, thereby achieving the goal of migrating high-level cognitive information in the brain's cognitive decision-making process to the machine learning model. Its technical characteristics It lies in designing efficient associated information learning models. The main representative methods of existing technology 1 include: "Bridging the Semantic Gap via Functional Brain Imaging" (IEEE Transactions on Multimedia: 2012,14(2):314-325) uses PCA method to establish brain magnetic resonance data features and video low-level images The correlation prediction model between features realizes the emotional classification of videos by mapping video features to the brain response representation space. However, with the development of deep learning technology, it has been able to extract high-level semantic information of videos, and its performance is no longer weaker than human recognition. , so similar applications are gradually decreasing; "Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features" (IEEE Transactions on Pattern Analysis and Machine Intelligence: 2020) uses triplet loss to optimize a two-stream network built based on deep learning methods, constraint The high-level semantic space of image features approximates the feature space of EEG to achieve the transfer of brain cognitive information. However, the training triplet loss is difficult to achieve convergence under a large amount of training data. It is limited by the characteristics of the brain response. It is difficult to collect high-quality EEG data, and existing public data cannot support the training of such models; "A brain-computer information fusion classification method and system for brain-out-of-loop applications" (Application No.: CN202111017290.7 ) achieves prediction of image-to-brain response by building a feature domain reconstruction model, learns the reliability of features from different sources by building a feature reliability prediction module, and achieves adaptive information fusion classification for "brain-out-of-the-loop" applications. However, related methods pass through multiple The separate learning process for each stage is cumbersome and troublesome during model deployment. How to build an image-brain response shared subspace learning model when brain response data is scarce, learn the associated information between the two end-to-end, and maximize the transfer of brain cognitive information.

Through the above analysis, the problems and defects existing in the existing technology are:

(1) Existing technology is limited by the "brain-in-the-loop" application paradigm, making it difficult to achieve high-intensity, real-time fully automated processing, and to fully leverage the advantages of fully automated machine intelligence processing.

(2) Existing technology is limited by the difficulty of obtaining brain response data. It is difficult to learn high-quality image-EGG shared subspace end-to-end with limited data, and it is difficult to realize the transfer of all brain cognitive information.

(3) The existing adaptive information fusion classification method for "brain-out-of-the-loop" applications uses multiple stages of separate learning and processing, which makes the process cumbersome and troublesome during model deployment.

Contents of the invention

In view of the problems existing in the existing technology, the present invention provides a brain-computer information fusion classification method and system based on shared subspace learning, and particularly relates to a brain-computer information fusion classification method and system based on shared subspace learning, and its technical characteristics In order to use the contrastive learning method based on positive and negative sample sampling to build an image-brain response shared subspace end-to-end to achieve the transfer of brain cognitive abilities.

The present invention is implemented as follows: a brain-computer information fusion classification method of shared subspace learning. The brain-computer information fusion classification method of shared subspace learning includes a training phase and an inference phase; wherein the training phase uses pairs Image and brain response data, through the contrastive learning strategy of positive and negative sample sampling, optimize the shared subspace model parameters of the image and brain response, and train the image classifier; the inference stage extracts image features for classification, realizing the entire brain-computer information The application goals of the fusion classification system.

Another object of the present invention is to provide a brain-computer information fusion classification system that applies the brain-computer information fusion classification method of shared subspace learning. The brain-computer information fusion classification system includes:

Data loading device, used to load test images and perform preliminary size transformation and format conversion functions to be suitable for the input model;

Feature extraction device, used to store model parameters successfully trained by the contrastive learning method based on positive and negative sample sampling, load input image data and perform forward inference to obtain image features in the shared subspace;

The classifier device is used to store the successfully trained SVM classifier parameters, load image features for SVM classification, and output the classification results.

Another object of the present invention is to provide a computer device. The computer device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the following steps: step:

In the training phase, a dual-stream network is used to map images and brain responses to the same subspace respectively. Paired image and brain response data are used to train the dual-stream network model parameters of the shared subspace. The current batch of image and brain response features are extracted in the shared subspace. ; The positive and negative sample sampling method based on category information obtains the positive and negative feature set of the current sample, uses the InfoNCE loss function to calculate the loss value of the current sample, and extracts the image features of the shared subspace after optimization to train the SVM classifier; the inference phase is performed through load testing Image, extract the image features of the shared subspace and input them into the SVM classifier for classification.

Another object of the present invention is to provide a computer-readable storage medium that stores a computer program. When the computer program is executed by a processor, it causes the processor to perform the following steps:

Another object of the present invention is to provide an information data processing terminal, which is used to implement the brain-computer information fusion classification system.

Combined with the above technical solutions and the technical problems to be solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

First, in view of the technical problems existing in the above-mentioned existing technologies and the difficulty of solving the problems, closely combine the technical solutions to be protected by the present invention and the results and data in the research and development process, etc., to conduct a detailed and profound analysis of how to solve the technical solutions of the present invention. Technical problems, and some creative technical effects brought about by solving the problems. The specific description is as follows:

The present invention uses a contrastive learning method of positive and negative sample sampling based on category information to optimize a dual-stream network model of shared subspace under the constraints of the InfoNCE loss function; the image classification system includes a data loading device, a feature extraction device and a classifier device, By saving the model parameters of the shared subspace, "brain-out-of-the-loop" image classification applications can be realized. Compared with the existing technology, the brain-computer information fusion classification system of shared subspace learning extracted by the present invention can train the shared subspace end-to-end, realize efficient transfer of brain cognitive information, and greatly improve the performance in complex open scenarios. The performance of image classification tasks, the application paradigm of the brain-computer information fusion image classification system proposed by the present invention can naturally avoid the limitations of "brain-in-the-loop" applications. Through "brain-out-of-the-loop" applications, it can be greatly improved. It improves efficiency and stability in real-world applications and has broad application prospects under the new paradigm of brain-computer information collaborative work.

Second, considering the technical solution as a whole or from a product perspective, the technical effects and advantages possessed by the technical solution to be protected by the present invention are specifically described as follows:

The brain-computer information fusion classification method based on shared subspace learning proposed by this invention can learn the correlation information between images and brain response data end-to-end under limited brain response data, and is compared with triple loss Function, the InfoNCE contrast loss function of positive and negative sample sampling proposed by the present invention can quickly converge the dual-stream network and efficiently realize the migration of cognitive information of the brain response to the image model. In addition, the shared subspace learning method proposed by the present invention can directly realize the "brain-out-of-the-loop" application, greatly exerting the advantages of machine intelligence automation applications, and greatly improving the deployment and application of brain-computer information fusion classification systems. The efficiency is of extremely high application significance. The present invention proposes a contrastive learning method based on positive and negative sample sampling, constructs a shared subspace of image-brain response, effectively realizes the migration of brain cognitive information, and can realize "brain-out-of-the-loop" applications with extremely high It improves the performance of image classification in complex open scenes.

Third, as auxiliary evidence of inventive step for the claims of the present invention, it is also reflected in the following important aspects:

(1) The expected income and commercial value after the transformation of the technical solution of the present invention is: after the transformation of the technology of the present invention, it can be used for error detection, such as automatic driving, remote sensing image interpretation, synthetic aperture radar image interpretation, intelligent medical auxiliary recognition detection, etc. Image recognition and classification tasks in complex open application scenarios with low rate tolerance can combine the dual advantages of machine intelligence and human intelligence in the above application scenarios, which can improve the classification accuracy of its application system.

(2) The technical solution of the present invention fills the technical gap in the industry at home and abroad: the technical solution of the present invention fills the gap in applying the contrastive learning method to the field of brain-computer hybrid intelligent computing, and realizes the use of a small amount of data to construct an image-brain response shared subspace the goal of.

(3) Whether the technical solution of the present invention solves the technical problem that people have been eager to solve but have never been successful: The brain-computer hybrid intelligent computing solution based on shared subspace learning proposed by the present invention adopts an end-to-end learning method. Constructing a shared subspace breaks through the technical problems of "brain-in-the-loop" modeling and "brain-out-of-the-loop" applications.

(4) Whether the technical solution of the present invention overcomes technical bias: The technical solution of the present invention confirms the application prospects of introducing the specific brain responses of visual experts into computer vision methods.

Description of the drawings

Figure 1 is a flow chart of a brain-computer information fusion classification method provided by an embodiment of the present invention.

Figure 2 is a process schematic diagram of the training phase and the inference phase provided by the embodiment of the present invention.

Figure 3 is a principle framework diagram based on shared subspace learning provided by an embodiment of the present invention.

Figure 4 is a schematic diagram of positive and negative sample sampling provided by an embodiment of the present invention.

Figure 5 is a computer application system diagram of the brain-computer hybrid intelligent classification system provided by an embodiment of the present invention.

Figure 6 is a schematic diagram of part of the stimulus images used for the classification task provided by the embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with examples. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

In view of the problems existing in the prior art, the present invention provides a brain-computer information fusion classification method and system for shared subspace learning. The present invention will be described in detail below with reference to the accompanying drawings.

1. Explain the embodiment. In order to enable those skilled in the art to fully understand how the present invention is specifically implemented, this section is an illustrative example that expands and explains the technical solutions of the claims.

In view of the problems existing in the existing technology, the present invention provides a brain-computer information fusion classification method and system based on shared subspace learning, which can realize brain visual cognitive information learning from machines under the condition of "brain not in the loop" application. Efficient migration of models improves target recognition performance in complex scenarios.

As shown in Figure 1, the brain-computer information fusion classification method provided by the embodiment of the present invention includes the following steps:

S101, use the ResNet feature extraction structure and the fully connected layer to construct a dual-stream feature extraction network for image and brain response respectively, as a feature extraction model of shared subspace;

S102, load paired stimulus images and brain response data sets, and optimize the parameters of the dual-stream network model of the shared subspace based on the contrastive learning method of positive and negative sampling until the model converges;

S103, use the converged dual-stream network to extract the image feature set of the training set stimulus image in the shared subspace, and use the image feature set to train the SVM classifier;

S104, load the test image and the image branch model in the dual-stream network, and extract the image features of the test image in the shared subspace;

S105: Send the image features to the SVM classifier and output the probability category of the image feature classification.

S101~S103 constitute the training phase, and S104~S105 constitute the inference phase.

The brain-computer information fusion classification method based on shared subspace learning provided by the embodiment of the present invention loads pairs of stimulation images and brain response data; using pairs of brain response data, the training is optimized based on the contrastive learning method of positive and negative sample sampling. Image-brain response dual-stream network model parameters of the shared subspace; extract the image feature set of the shared subspace and train a linear SVM classifier to output the classification results.

As shown in Figure 2, the brain-computer information fusion classification method based on shared subspace learning provided by the embodiment of the present invention specifically includes the following steps:

Step 1. Training phase:

(1) Use the ResNet feature extraction structure and the fully connected layer to construct a dual-stream feature extraction network for image and brain response respectively, as a shared subspace feature extraction model.

1) Use the PyTorch deep learning framework to build the ResNet34 model structure, remove its fully connected layer, and add a fully connected layer with an input size of 512 and an output size of 168 dimensions. Set the model parameter "pretrained=True" and load ImageNet pre-training. model parameters. The above serves as the image feature extraction branch of the dual-stream network.

2) Use the PyTorch deep learning framework to build a three-layer fully connected network, whose input and output sizes are both 168 dimensions, and given random initialization parameters as the brain response feature extraction branch of the dual-stream network.

3) Integrate image and brain response feature extraction module classes into common class modules of the dual-stream network.

(2) Load paired stimulation images and brain response data sets, and optimize the dual-stream network model parameters based on the contrastive learning method of positive and negative sampling until the model converges.

The data loading process mainly includes the loading of image data, the loading of brain response data, and the loading process of paired image-brain response data:

Image data loading process:

1) Use PyTorch’s Dataset toolkit to load stimulus images.

2) Use torchvision's transforms toolkit to transform the image size to 224*224, perform random left and right flipping for data enhancement, and then convert the read image data into tensor format.

The loading process of brain response data:

1) Load the brain response dataset and average the brain responses captured when the same stimulus image is presented multiple times.

2) Select electrodes placed in the inferior temporal lobe (IT) area and extract the brain response signals of the corresponding electrodes.

3) On the brain response signal of each electrode, average the value along the time dimension to remove the influence of the time dimension.

4) Flip the processed brain response into 1*168-dimensional features and convert it into tensor format as the average brain response feature of the stimulation image on each electrode in the IT area.

Paired image-brain response data pair loading process:

1) Construct the dataset public class, index to the stimulus image name information, load the image data, then index to the corresponding brain response data information according to the image name, and load the brain response data.

2) Return paired image-brain response data.

The present invention trains a dual-stream network for shared subspace learning in an end-to-end manner. As shown in Figure 3, an embodiment of the present invention provides a principle framework diagram of a dual-stream network for shared subspace learning. Paired image and brain response features are extracted respectively, and a positive sample set and a negative sample set of the current sample are constructed within the batch through the method of category-based positive and negative sample sampling. As shown in Figure 4, the embodiment of the present invention provides a method based on Schematic diagram of positive and negative sample sampling of categories. After determining the set of positive and negative samples in the current sample batch, the current loss value is calculated through InfoNCE, gradient backpropagation is performed, and the network parameters are optimized. The specific steps are as follows:

1) Use the PyTorch deep learning framework to load pairs of image and brain response data, with the batch size set to 256 and 256 pairs of data loaded each time.

2) Load the dual-stream network model parameters, perform forward reasoning, and obtain the feature set of batch images and brain responses, recorded as <f(v), f(b)>.

3) For any image feature f( _vi ) in the batch, its category is c, and all brain response features of the same category in the batch

are all positive sample pairs of the current image feature, that is, the positive sample pair of the image feature f( _vi ) is

Combine all brain response features in the batch that are different from it

does not belong to category c, it is recorded as the negative sample pair of the current image feature, that is, the negative sample pair of the image feature f( _vi ) is

That is, the positive/negative brain response feature set corresponding to each image feature is obtained accordingly.

4) Use the InfoNCE loss function to calculate the contrast loss _Li corresponding to each image feature f(vi ₎ in the batch:

Among them, m and n respectively represent the number of positive and negative samples of the brain response corresponding to the current image feature f(vi ₎ , and S(.) represents the cosine similarity of the two features;

5) Based on the contrast loss calculated by the above InfoNCE loss function, backpropagate and optimize the model parameters of the dual-stream network until the contrast loss converges stably. When training the dual-stream network, the Adam optimizer is used for backpropagation. When the loss function converges, the model parameters are saved. Among them, the batch size is set to 128, the initial learning rate is set to 0.1, the learning rate decays to 0.1, and the decay occurs every 30 epochs, and a total of 100 epochs are trained.

(3) Use the converged dual-stream network to extract the image feature set of the training set stimulation image in the shared subspace, and train the SVM classifier.

1) Load the dual-stream network image branch model parameters, load the training set image data, perform forward inference, and obtain the feature set of the image in the shared subspace.

2) Use Python's sklearn toolkit to build a linear SVM classifier, use the image features extracted in the above steps to train the classifier parameters, and save the model parameters.

Step 2: Reasoning stage

(1) To load the image branch model parameters of the dual-stream network, you only need to load the test image, and extract the image features in the shared subspace through forward reasoning of the image branch model.

(2) Load the model parameters of the SVM classifier, input the image features extracted in the above steps into the classifier, and obtain the image classification results.

As shown in Figure 5, an embodiment of the present invention provides an application example of a computer image classification system based on a brain-computer fusion system based on shared subspace learning. The system mainly includes a data loading device, a feature extraction device and a classifier device. Each device of the system can store the computer program required for the corresponding module and the successfully trained model parameters to ensure the correct application of the system. The specific information of each device in the system is as follows:

(1) Data loading device: loads the test image and performs preliminary size transformation and format conversion functions to be suitable for the input model.

(2) Feature extraction device: used to store model parameters successfully trained by the contrastive learning method based on positive and negative sample sampling, load input image data, and perform forward inference to obtain image features in the shared subspace.

(3) Classifier device: used to store successfully trained SVM classifier parameters, load image features for SVM classification, and output the classification results.

2. Application examples. In order to prove the creativity and technical value of the technical solution of the present invention, this section is an application example of the claimed technical solution in specific products or related technologies.

The creativity of the technical solution of the present invention lies in proposing a brain-computer information fusion classification method and system based on shared subspace learning, and proposing a contrastive learning method based on positive and negative sample sampling to construct a shared subspace. The application basis of the present invention is to use the contrastive learning strategy based on positive and negative sample sampling proposed by the present invention to train the feature extraction model of the shared subspace on the image-brain response data set, and train the classifier based on the features in the shared subspace. The application implementation of the present invention needs to save the above-mentioned successfully trained shared subspace feature extraction model parameters and classifier parameters into the computer hardware system. Subsequent applications can load the image data to be tested through the software system described in Figure 5 of the embodiment, and then perform feature extraction and classifier inference through the above model parameters to achieve classification result output.

It should be noted that embodiments of the present invention may be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or specially designed hardware. Those of ordinary skill in the art will understand that the above-described apparatus and methods may be implemented using computer-executable instructions and/or included in processor control code, for example on a carrier medium such as a disk, CD or DVD-ROM, such as a read-only memory. Such code is provided on a programmable memory (firmware) or on a data carrier such as an optical or electronic signal carrier. The device and its modules of the present invention may be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., It can also be implemented by software executed by various types of processors, or by a combination of the above-mentioned hardware circuits and software, such as firmware.

3. Evidence of relevant effects of the embodiment. The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages compared with the existing technology. The following content is described in conjunction with the data, charts, etc. of the test process.

1. Experimental conditions:

The hardware conditions for the experiment of this invention are: an ordinary computer, Intel i5 CPU, 8G memory, and an NVIDIA GeForce GTX 1070 graphics card; software platform: Ubuntu 18.04, PyTorch deep learning framework, python 3.6 language; the brain response used in this invention is The stimulus image data set comes from the public data of the Brain-Score platform of the McGovern Institute for Brain Science at MIT.

2. Training data and test data:

The data set used in the present invention includes two parts: stimulation images and brain response data. The stimulus images are composite images of 8 categories of targets and random natural scenes, with a total number of 3200, 400 images of each category. Each stimulus image contains only one target. The target image is generated by changing the posture of the three-dimensional model of the target object, as shown in Figure 6. By changing the target posture and random natural background, this data set can effectively simulate the complex transformation of targets and scenes. Open scene. Brain response data were collected from the ventral stream area of two well-trained adult rhesus monkeys. The brain response of the corresponding brain area was captured through a 168-channel electrode array in the inferotemporal area (IT). During the EEG collection process, each A group of 5 to 10 stimulus images were presented in the center of the monitor in sequence. Each image was displayed for 100 ms, followed by a 100 ms blank. During the entire process, the rhesus monkey was kept focused on the center of the monitor. Each stimulus image was presented multiple times, at least 28 times, and on average 50 times. Among them, the data processing framework disclosed by the Brain-Score (https://brain-score.readthedocs.io/en/latest/index.html) platform can be used to preprocess the brain response and obtain the preprocessed brain response characteristics.

3.Experimental content:

According to the above steps in the training phase, the computer's GPU is used to accelerate the training process of the dual-stream network of the shared subspace. After training, the model converges and the SVM classifier is trained. After the model training is successful, save the model parameters.

The inference process loads the parameters of each model, performs forward reasoning, and obtains the classification results.

4. Analysis of experimental results

This invention uses classification accuracy to describe the performance of classification, and evaluates the classification results of shared subspace learning under different image feature extraction branches, which mainly include four image feature extraction networks: AlexNet, VGG, GoogLeNet and ResNet, and are shown in Table 1 The performance of IT and image single-modality classification was compared with the brain-computer information fusion classification method based on shared subspace learning. It can be seen from the table that the contrastive learning method based on positive and negative sample sampling proposed by the present invention can effectively improve the image classification performance by training the image-brain response shared subspace. Compared with single-modal SVM classification, the average improvement is 7.43%. The performance of optimization by directly using InfoNCE loss is improved by 6.05%, which shows that the contrastive learning method of positive and negative sample sampling based on category information proposed by the present invention can efficiently realize the migration of brain cognitive information and improve image recognition in downstream complex open scenes. performance. In addition, the application paradigm of the present invention can naturally avoid the limitations of "brain-in-the-loop" applications, and greatly improves the efficiency and stability in real-world applications through "brain-out-of-the-loop" applications. Therefore, the present invention has more practical application value and has broad application prospects under the new paradigm of brain-computer information collaborative work.

Table 1 Simulation results

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field shall, within the technical scope disclosed in the present invention, be within the spirit and principles of the present invention. Any modifications, equivalent substitutions and improvements made within the above shall be included in the protection scope of the present invention.

Claims

A brain-computer information fusion classification method for shared subspace learning, characterized in that the brain-computer information fusion classification method for shared subspace learning includes a training phase and a reasoning phase; wherein the training phase utilizes paired images and Brain response data, through the contrastive learning strategy of positive and negative sample sampling, optimizes the shared subspace model parameters of the image and brain response, and trains the image classifier; the inference stage extracts image features for classification, realizing the entire brain-computer information fusion classification system application goals.
The brain-computer information fusion classification method of shared subspace learning according to claim 1, characterized in that the brain-computer information fusion classification method of shared subspace learning includes the following steps:

Step 1, training phase:

(1) Use the ResNet feature extraction structure and the fully connected layer to construct a dual-stream feature extraction network for image and brain response respectively, as a feature extraction model of shared subspace;

(2) Load paired stimulus images and brain response data sets, and optimize the parameters of the dual-stream network model of the shared subspace based on the contrastive learning method of positive and negative sampling until the model converges;

(3) Use the converged dual-stream network to extract the image feature set of the training set stimulation image in the shared subspace, and use the image feature set to train the SVM classifier;

Step 2, reasoning stage:

(1) Load the test image and the image branch model in the dual-stream network, and extract the image features of the test image in the shared subspace;

(2) Send the image features to the SVM classifier and output the probability category of the image feature classification.
The brain-computer information fusion classification method of shared subspace learning according to claim 2, wherein the construction of a shared subspace dual-stream feature extraction model in step one includes:

1) Use the PyTorch deep learning framework to build the ResNet34 model structure, remove the fully connected layer, and add the fully connected layer. The input size is 512 and the output size is 168 dimensions. Set the model parameter "pretrained=True" and load the ImageNet pre-trained model parameters. As the image feature extraction branch of the dual-stream network;

2) Use the PyTorch deep learning framework to build a three-layer fully connected network, with input and output sizes of 168 dimensions, and assign random initialization parameters as the brain response feature extraction branch of the dual-stream network;

3) Integrate image and brain response feature extraction module classes into common class modules of the dual-stream network.
The brain-computer information fusion classification method of shared subspace learning according to claim 2, wherein the loading of paired stimulus images and brain response data sets in step one includes:

1) Image data loading process:

①Use PyTorch’s Dataset toolkit to load stimulus images;

②Use torchvision's transforms toolkit to transform the image size to 224*224, perform random left and right flipping for data enhancement, and then convert the read image data into tensor format;

2) Loading process of brain response data:

① Load the brain response data set and average the brain responses captured when the same stimulus image is presented multiple times;

②Select the electrodes placed in the inferior temporal lobe area and extract the brain response signals of the corresponding electrodes;

③ On the brain response signal of each electrode, average the value along the time dimension to remove the influence of the time dimension;

④ Flip the processed brain response into 1*168-dimensional features and convert it into tensor format as the average brain response feature of the stimulation image on each electrode in the IT area;

3) Paired image-brain response data pair loading process:

① Construct the dataset public class, index to the stimulus image name information, and load the image data; index to the corresponding brain response data information according to the image name, and load the brain response data;

②Return paired image-brain response data.
The brain-computer information fusion classification method of shared subspace learning according to claim 2, characterized in that the step 1 of optimizing the dual-stream network model parameters based on the contrastive learning method based on positive and negative sampling includes:

1) Use the PyTorch deep learning framework to load pairs of image and brain response data, with the batch size set to 256 and 256 pairs of data loaded each time;

2) Load the dual-stream network model parameters, perform forward inference, and obtain the feature set of batch images and brain responses, recorded as <f(v), f(b)>;

3) For any image feature f( vi ) in the batch, the category is c, all brain response features of the same category in the batch
are all positive sample pairs of the current image feature, and the positive sample pair of the image feature f(vi ) is
Combine all brain response features in the batch that are different from it
does not belong to category c, it is recorded as the negative sample pair of the current image feature, and the negative sample pair of the image feature f( vi ) is
Then obtain the positive/negative brain response feature set corresponding to each image feature;

4) Use the InfoNCE loss function to calculate the contrast loss Li corresponding to each image feature f(vi ) in the batch:

Among them, m and n respectively represent the number of positive and negative samples of the brain response corresponding to the current image feature f(vi ) , and S(.) represents the cosine similarity of the two features;

5) Back propagate the contrast loss calculated by the InfoNCE loss function and optimize the model parameters of the dual-stream network until the contrast loss converges stably.
The brain-computer information fusion classification method of shared subspace learning according to claim 2, wherein the step one of using a dual-stream network to extract image features and train the SVM classifier includes:

1) Load the dual-stream network image branch model parameters, load the training set image data, perform forward inference, and obtain the feature set of the image in the shared subspace;

2) Use Python's sklearn toolkit to build a linear SVM classifier, use the extracted image features to train the classifier parameters, and save the model parameters;

The reasoning stage in step two is the application reasoning process of the brain-computer information fusion classification model, including:

1) To load the image branch model parameters of the dual-stream network, you only need to load the test image, and extract the image features in the shared subspace through forward reasoning of the image branch model;

2) Load the model parameters of the SVM classifier, input the extracted image features into the classifier, and obtain the image classification results.
A brain-computer information fusion classification system that implements the brain-computer information fusion classification method of shared subspace learning according to any one of claims 1 to 6, characterized in that the brain-computer information fusion classification system includes:

Data loading device, used to load test images and perform preliminary size transformation and format conversion functions to be suitable for the input model;

Feature extraction device, used to store model parameters successfully trained by the contrastive learning method based on positive and negative sample sampling, load input image data and perform forward inference to obtain image features in the shared subspace;

The classifier device is used to store the successfully trained SVM classifier parameters, load image features for SVM classification, and output the classification results.
A computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, it causes the processor to perform the following steps:

In the training phase, a dual-stream network is used to map images and brain responses to the same subspace respectively. Paired image and brain response data are used to train the dual-stream network model parameters of the shared subspace. The current batch of image and brain response features are extracted in the shared subspace. ; The positive and negative sample sampling method based on category information obtains the positive and negative feature set of the current sample, uses the InfoNCE loss function to calculate the loss value of the current sample, and extracts the image features of the shared subspace after optimization to train the SVM classifier; the inference phase is performed through load testing Image, extract the image features of the shared subspace and input them into the SVM classifier for classification.
A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it causes the processor to perform the following steps:

In the training phase, a dual-stream network is used to map images and brain responses to the same subspace respectively. Paired image and brain response data are used to train the dual-stream network model parameters of the shared subspace. The current batch of image and brain response features are extracted in the shared subspace. ; The positive and negative sample sampling method based on category information obtains the positive and negative feature set of the current sample, uses the InfoNCE loss function to calculate the loss value of the current sample, and extracts the image features of the shared subspace after optimization to train the SVM classifier; the inference phase is performed through load testing Image, extract the image features of the shared subspace and input them into the SVM classifier for classification.
An information data processing terminal, characterized in that the information data processing terminal is used to implement the brain-computer information fusion classification system as claimed in claim 7.