WO2021159230A1

WO2021159230A1 - Brain-computer interface system and method based on sensory transmission

Info

Publication number: WO2021159230A1
Application number: PCT/CN2020/074610
Authority: WO
Inventors: 李晓涛; 王立平
Original assignee: 中国科学院深圳先进技术研究院
Priority date: 2020-02-10
Filing date: 2020-02-10
Publication date: 2021-08-19

Abstract

Provided are a brain-computer interface system and method based on sensory transmission. The system comprises: a brain information terminal decoding unit for deciphering neural information, which has been transmitted via sensory organs, of a target object in a non-intrusive manner so as to obtain psychological and physiological data; a psychological emotion modeling unit for recognizing, on the basis of the collected psychological and physiological data, an emotional cognitive state of the target object by utilizing machine learning; and a physical output signal stimulation unit for generating, on the basis of the emotional cognitive state, human-computer interaction information having a sensory stimulation effect so as to prompt the target object to generate an expected emotional cognitive state. The present invention provides a non-invasive brain-computer interface technique for performing decoding and recognition on the basis of a sensory signal flow, such that craniotomy is not needed, there is also no need to be limited by electroencephalogram signals, and the prediction, prevention and intervention of cognitive brain diseases in a timely manner are facilitated.

Description

Brain-computer combination system and method based on sensory transmission

Technical field

The invention relates to medical equipment and intelligent product technology, in particular to a brain-computer combination system and method based on sensory transmission.

Background technique

Brain-computer interface (BCI) technology is considered to be one of the most important science and technology that will profoundly affect the human world in the future. At present, there are two main brain-computer integration technologies. The first is the implantable brain-computer integration technology, that is, the microcircuit chip is directly implanted into the brain to read and regulate the electrical signals of the brain's neurons. For example, the flexible electrode wires (threads) recently developed by Musk Neuralink can be implanted into the brains of mice and monkeys for neural activity experiments with a success rate of 87%. The second is a non-implantable brain-computer combination technology, which is used to interpret EEG (brain waves) signals sent from the brain and perform related operations by placing or wearing some detection equipment on the head. For example, Facebook uses electrocorticogram (ECoG) technology to record the signals of the cerebral cortex from the exposed surface of the brain. By tracking the signals of the brain area responsible for language and speech, and according to the movement rules of human vocal organs, the information is decoded into text sentences. , Its accuracy rate is about 61%.

The development of the above-mentioned two kinds of brain-computer technology is restricted by the ability to decode neural signals of the brain. This is because the human brain is an extremely complex neural network system (at least 87 billion neurons, 100 trillion neural connections). Most of the electrical signals sent by the brain cannot be accurately interpreted at present, especially if they are connected with complex thoughts and behaviors.

After research and analysis, the prior art brain-computer integration technology has obvious common shortcomings, that is, it cannot accurately interpret human brainwave signals with a complete level of meaning. In addition, the implantable brain-computer integration technology requires the implantation of foreign bodies through craniotomy, which has the risk of harming brain function, especially when humans do not have a deep understanding of their own brains. For example, the flexible electrode wires (threads) recently developed by Musk Neuralink are not suitable for direct experiments on the human brain. The second non-implantable brain-computer combination technology, by placing or wearing some detection equipment on the head, the brain nerve signals that can be interpreted are also very limited. Generally, they are limited to the cerebral cortex, and are easy to interact with eye movements and muscles. The electrical signals are confused. For example, Facebook's use of electrocorticogram (ECoG) technology is actually more based on the muscle movement rules of human vocal organs to decode text.

In the traditional minimally invasive brain stimulation technology, many technologies need to further improve the accuracy and enhance the effect. Due to the diffusion effect of electric field, transcranial direct electrical stimulation has limited temporal and spatial resolution in brain regions. Although deep brain electrical stimulation has certain curative effects, it also has shortcomings, including unpredictability and even side effects, because the stimulation lacks neuronal selectivity and specificity, and the treatment mechanism is often unclear. Although optogenetic stimulation has proved to have the advantages of neuron specificity and selectivity, it requires the expression of special light-sensing genes and proteins, and it is currently impossible to use it in the human brain. Repeated transcranial magnetic stimulation, because the magnetic field strength decays quickly with the increase of distance, the induced current can basically only act on the cerebral cortex. Ultrasound stimulation technology is almost non-invasive and seems to be able to reach some deep brain regions, but the accuracy and efficacy of this technology in stimulating brain nuclei need to be further explored.

In summary, the existing brain-computer integration technology mostly revolves around brainwave signals (EEG) in the interpretation of brain nerve signals. However, it should not be overlooked that the clearest interpretation of human brain signals so far is the output expression of our autonomous sensory terminals, such as the voice and language expression of our mouth and the body language expression of body behavior, rather than the letter. EEG signal recording with low noise ratio.

Summary of the invention

The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art, and provide a brain-computer combination system and method based on sensory transmission, which can perform non-invasive brain regulation technology based on the information transmitted by the sensory, and analyze and analyze the emotional cognitive state of the brain. Regulation can play a unique advantage.

According to the first aspect of the present invention, a brain-computer integration system based on sensory transmission is provided. The system includes:

The brain information terminal decoding unit is used to interpret the neural information transmitted by the target object through the sense organs in a non-invasive manner to obtain psychological and physiological data;

The psycho-emotional modeling unit is used to recognize the emotional cognitive state of the target object by using machine learning based on the collected psychological and physiological data;

The physical output signal stimulation unit is used to generate human-computer interaction information with sensory stimulation effects based on the emotional cognitive state, so as to prompt the target object to produce the desired emotional cognitive state.

In an embodiment, the brain information terminal decoding unit includes one or more of a video collection device, a sound collection device, a skin electro-skin collection device, and an electromyogram collection device.

In an embodiment, the information transmitted by the sensory organs includes one or more of facial expression information, sound information, eye movement information, heartbeat information, and electrical skin information.

In one embodiment, the emotional cognitive state includes one or more of emotional state, fatigue state, attention state, and stress state.

In one embodiment, the human-computer interaction information with sensory stimulation effects includes one or more of text communication, voice dialogue, game interaction, picture playback, and short video playback.

In one embodiment, the psychological and emotional modeling unit inputs the obtained psychological and physiological data to the trained deep learning model, and outputs the emotional cognitive state of the target object.

In one embodiment, the deep learning model includes a convolutional neural network and a cyclic neural network.

In an embodiment, the physical output signal stimulation unit generates interactive information with sensory stimulation effects according to the following steps:

Judging the type of human-computer interaction information with sensory stimulation effect that needs to be generated based on the emotional cognitive state;

By categorizing and sorting the existing human-computer interaction information types, and combining the adversarial generation network algorithm to synthesize new relevant human-computer interaction information, forming a human-computer interaction information database;

According to the characteristics of the type of human-computer interaction information that needs to be generated, the human-computer interaction information to be displayed to the target object is selected from the human-computer interaction information database.

According to the second aspect of the present invention, a brain-computer integration method based on sensory transmission is provided. The method includes: non-invasively interpreting the neural information transmitted by the target object through the senses to obtain psychological and physiological data; using machine learning to identify the emotional and cognitive state of the target object based on the collected psychological and physiological data; and based on the emotional cognition The state generates interactive information with sensory stimulus effect to promote the target object to produce the desired emotional cognitive state.

According to a third aspect of the present invention, an electronic device is provided. The electronic device includes the brain-computer combination system based on sensory transmission of the present invention, which is used to perform one or more of the following steps: based on one of the user's use frequency, use time, and focus orientation of the software program of the electronic device Or multiple items to provide the user with matching content; recognize the user’s emotional and cognitive information based on facial expression characteristics and voice communication status; use a sensor that recognizes finger bioelectric signals and photoplethysmography to calculate the user’s physiological data, Obtain autonomic nervous system information.

According to a fourth aspect of the present invention, an electronic device is provided. The electronic device includes the brain-computer integration system based on sensory transmission of the present invention, which is used to implement: use supervised machine learning to build an audio-visual emotion database based on existing data; filter its personality orientation and characteristics according to the audio-visual emotion database selected by the user; The user provides targeted audiovisual scenes.

Compared with the prior art, the present invention has the advantages of providing a new type of brain-computer integration technology system and application development, decoding and encoding the brain through the neural information flow of the sensory terminal, and combining deep learning and artificial intelligence on the computer side , To achieve non-invasive brain regulation technology based on sensory transmission. The sensory, non-invasive way of the present invention realizes brain-computer integration, and can play a unique advantage in the analysis and adjustment of human brain emotion and cognitive state, especially for patients with affective cognitive disorders when they implement cognitive behavioral therapy. A very good intelligent auxiliary tool, which has the effect of getting twice the result with half the effort.

Description of the drawings

The following drawings only schematically illustrate and explain the present invention, and are not used to limit the scope of the present invention, in which:

Fig. 1 is a schematic diagram of a basic theoretical model of brain-computer integration technology according to an embodiment of the present invention;

Figure 2 is a schematic diagram of a smart terminal device with a biosensing function according to an embodiment of the present invention;

Fig. 3 is a schematic diagram of an artificial intelligence-assisted cognitive behavioral therapy process according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of the basic working technical framework of the brain-computer integration technology according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions, design methods, and advantages of the present invention clearer, the following further describes the present invention in detail through specific embodiments with reference to the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, but not used to limit the present invention.

In all examples shown and discussed herein, any specific value should be construed as merely exemplary, rather than as a limitation. Therefore, other examples of the exemplary embodiment may have different values.

The technologies, methods, and equipment known to those of ordinary skill in the relevant fields may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be regarded as part of the specification.

For ease of understanding, the following will introduce the basic concept of the present invention, application transformation capabilities, comparison with the prior art, and actual application scenarios.

The new brain-computer combination technology proposed by the present invention controls the input and output of information through human's existing receptors (that is, various senses, such as vision, hearing, touch, etc.), which can cleverly bypass the current brain-computer combination that is difficult to accurately decode. Technical obstacles, the application of big data and machine algorithms to achieve non-invasive fine stimulation and regulation of the brain, this deep learning combined with artificial neural networks is expected to advance the existing brain-computer integration technology to a more accurate level. Using this sensory and non-invasive brain-computer combination new technology can achieve in-depth interpretation of the human brain's emotional and cognitive state and the regulation of related brain diseases, such as human emotional disorders such as anxiety, depression, and post-traumatic stress Obstacles can provide a brand-new intervention and treatment plan.

On the other hand, from an application point of view, brain-computer integration technology is not far from our real life. At present, almost one smart phone is one of the most vivid examples of brain-computer integration. Smart phones input a large amount of information to the cognitive center of the user's brain through the audiovisual system every day, which has a subtle and profound impact on the emotion, cognition and behavior controlled by the cerebral cortex. The stimulus produced by the smartphone to the brain is realized through the senses, and is non-invasive compared to other brain-computer interface technologies, such as intelligent calculation based on deep learning for such sensory information input and instant interpretation of individual brain feedback output , Is one of the typical applications of the new brain-computer integration technology provided by the present invention.

Compared with the two main brain-computer interface technologies in the prior art, the brain-computer integration technology provided by the present invention is closest to actual applications and has the strongest application transformation ability. This is because the present invention directly bypasses the poor signal-to-noise Interpretation of electroencephalography (EEG). It is reported that CTRL-Labs, which was recently acquired by Facebook in the United States, has also tried to use less invasive methods to achieve brain-computer integration. They are equipped with a wearable electromyogram (EMG) device to monitor various neuron signals on the user's wrist and convert these signals into digital signals. The device is like a smart wristband, with built-in lightweight skin sensors and nerve electrodes to collect signals, and combined with AI algorithms, it can distinguish each nerve impulse from the brain to the hand muscles. The device can convert people's thoughts into action signals, and wirelessly transmit the information to computers and smartphones via Bluetooth. In addition, the virtual reality exposure therapy (vertual reality exposure therapy, referred to as VERT) system used in psychotherapy, combined with computer audio-visual media technology, can provide visitors with a nearly real, immersive and interactive 3D virtual environment. The system has shown certain effects in the treatment of some special phobias, such as arachnophobia, acrophobia, flying phobia, etc. However, due to the relatively large production and process costs of virtual therapy, it has not been well promoted at present.

The two existing brain-computer interface technologies, including invasive and non-invasive, are still facing huge challenges in the practical application of brain science. For extremely complex brain neural network systems, both invasive and non-invasive brain-computer interface methods can only interpret part of the information in local areas of the brain, and it is difficult to read the true emotional state and cognitive thinking of human beings. Refer to FIG. 1, where A represents the invasive method in the existing brain-computer integration mode, B represents the non-invasive method in the existing brain-computer integration mode, and C represents the method provided by the present invention. It can be seen that in the system model of the entire brain neural network, both the invasive method represented by A and the non-invasive method represented by B can only obtain part of the information in some areas of the brain, even if the invasive method may one day be able to read deep Brain area information, non-invasive may one day be able to resolve many EEG signals in the cortex, but it is not as good as the method provided by the present invention, that is, direct reading of clearer output information at each terminal of the nervous system, especially for human cognition In terms of behavior interpretation, the advantages of the present invention are more obvious. Because the clearest interpretation of human brain information so far comes from our own sensory system, such as speech language expression and body language expression. Therefore, the brain-computer integration technology provided by the present invention pays more attention to the corresponding decoding of the input and output of our own sensory information, such as the decoding of information from various sensory limbs such as expressions, sounds, skin electrokines, and eye movements, and combines cognitive nerves Science, psychology, and the latest machine learning algorithms for human emotion calculation. The information directly transmitted by the various senses, including sight and hearing, etc., can directly understand the emotional state and cognitive response of the brain, especially in conjunction with the indirect signals expressed by the limbs such as bioelectricity. In addition, under the premise of effectively interpreting brain information through sensory terminals, effective sensory stimulus information can also be input to actively change the emotional and cognitive state of the brain, for example, combined with deep learning methods based on artificial neural networks. In short, the theoretical basis of the brain-computer integration technology provided by the present invention is that effective sensory information input can generate beneficial brain output, and machine learning such as deep learning is used to generate sufficient artificial intelligence to find effective information input.

In actual application scenarios, compared with the two existing brain-computer interface technologies, the brain-computer combination technology provided by the present invention will be the most feasible and safe. Refer to the application development examples shown in Figs. 2 and 3. Figure 2 is a smart device with biosensing function (referred to as BIAI-PC, BIAI-based Personal Computer in this article). When using the smart device, the machine can sense the user's state of use at the same time, and make beneficial cooperation as needed Or appropriate adjustments can be divided into working mode, leisure mode, and child mode. It can effectively solve the main problems of the modern information society, such as people's distraction when using smart devices, confusion between work and rest, and health hazards caused by long-term use of electronic devices.

Figure 3 is an artificial intelligence assistant tool that can effectively improve the effect of cognitive therapy (referred to as BIAI-CBT, BIAI-aided cognitive behavior therapy in this article). In the process of cognitive therapy, the most common problem is that the visitor's attention is not enough, it is not easy to enter the state of psychological communication, and it is difficult to accurately express their mental and cognitive status. The BIAI-CBT auxiliary tool developed by using the brain-computer interface technology provided by the present invention can recognize and understand the true thoughts and state of the visitor’s brain through the interpretation of the information output of the sensory terminal of the visitor, in addition to the effective stimulus information generated by machine learning The sensory input can assist the therapist to better correct the cognitive behavior of the visitor.

It should be understood that in some application scenarios, the technology provided by the present invention does not exclude the combination with the existing brain-computer interface technology, but the present invention emphasizes that it does not interpret the complex content of the entire brain by directly interpreting the neural signals of the local brain. It is to understand and change the brain through the decoding and coding of neural information at the various sensory terminals. As shown in Figure 4, the communication of neural information from sensory terminals such as vision, hearing, and somatic touch is one of the best means to interpret and influence brain information. Especially in combination with the accumulation of human big data and the establishment of model algorithms, the brain of the present invention Computer-to-computer integration technology is entirely possible to achieve excellent generalization capabilities, and is superior to the existing brain-computer interface technology in many application scenarios.

In summary, in view of the current interpretation of brain nerve signals, most of the research is carried out around the decoding of electroencephalogram (EEG), but the poor EEG signal-to-noise ratio has been difficult to solve the technical problem, the brain-computer integration technology provided by the present invention is more Pay attention to the decoding of sensory terminal nerve information, such as the nerve information from the sensory limbs such as facial expressions, voices, skin electricity and eye movements, and combine the latest artificial intelligence algorithms such as facial emotion recognition, natural language processing, and human bioelectric feature recognition. Calculation, processing and interaction.

The following will specifically introduce the brain-computer integration system based on sensory transmission of the present invention and its application examples.

According to an embodiment of the present invention, a brain-computer integration system based on sensory transmission is provided. The system includes: a brain information terminal decoding unit, which is used to non-invasively interpret the neural information transmitted by the target object via the senses to obtain psychological and Physiological data; psycho-emotional modeling unit, which is used to recognize the emotional cognitive state of the target object based on the collected psychological and physiological data using machine learning; physical output signal stimulation unit, which is used to generate the emotional cognitive state based on the emotional cognitive state The interactive information of sensory stimulation effects to promote the target object to produce the desired emotional cognitive state.

In the system of the present invention, the brain information terminal decoding unit mainly collects various physiological and psychological data through various fine and human-friendly receptors, including but not limited to invisible cameras, eye tracking devices, audio acquisition devices, and skin electricity. Collectors, etc., and related data processing software; the psycho-emotional modeling unit refers to the creation of a system involving emotion and cognitive state calculation through data collection, model building and algorithm optimization, combined with deep learning methods based on artificial neural networks Method: The physical output signal stimulation unit refers to some picture, sound, and video stimulus generation system and its supporting devices that are generated by combining relevant software and hardware, especially intelligent algorithms. In summary, the present invention provides a non-invasive brain-computer interface for decoding and recognition based on the nervous system terminal signal, that is, the sensory signal stream, without the need for craniotomy, and without being limited to EEG signals.

Specifically, the function execution of the brain-computer integration solution based on sensory transmission provided by the embodiment of the present invention is mainly divided into two parts, one is the input from the computer side, and the other is the output from the human brain, and the computer side cyclically reads the output feedback from the human brain. Later, more useful input can be made. Each part has machine intelligence algorithms mainly based on artificial neural networks.

Computer input mainly realizes integration and generation algorithms. For example, through adversarial generative network algorithm (GAN), various audiovisual input information (or human-computer interaction information) with sensory stimulus effects are synthesized according to big data, including various pictures, Music, short videos and even interactive games.

The output of the human brain mainly involves reading and processing algorithms, such as the collection and interpretation of people's expressions, sounds, eye movements, and skin electrical signals through machine algorithms such as CNN (Convolutional Neural Network) and RNN (Circular Neural Network).

For example, when the computer recognizes that the human brain needs to watch some happy short videos, the computer will determine the type of funny short videos that the human brain likes, and then use algorithms and the network to generate more and better funny short videos for input to the human brain. . In one embodiment, the synthesis process is to use the GAN algorithm to first arrange and score 1000-10000 funny short videos, and then synthesize more funny short videos with the same characteristics to form a video library. When the human brain shows that it needs a funny short video with a certain feature orientation, the computer side reads its feature and orientation, and then outputs more similar short videos. Regarding the process of recognizing the needs of the human brain on the computer side, various biological signals and behavior characteristics output from the human brain, including direct output of manual and voice, and indirect output of worry or fatigue signals through facial expressions, eye movements, and electrical skin , To accurately identify the current status and needs of users.

Based on the inventive concept of the brain-computer combination system provided by the present invention, various types of functional products can be realized, and the emotional and cognitive state of the target object can be flexibly recognized, and cooperation and assistance can be provided; or the mental and physical health of the target object can be tracked, actively Prevention and treatment of diseases, etc. The specific product form is still shown in Figure 2 and Figure 3.

For the intelligent terminal device with biosensing function in Fig. 2: In the perception mode, the user can work better when working, and can be better leisure when he is leisure. Effectively solve the problem that smart devices often confuse people's work and rest, interfere with each other, and influence each other. Combining the collected information characteristics of the device user's facial expression, body and voice, it can intelligently calculate the person's current emotional and cognitive state, and actively provide beneficial cooperation and assistance. Specific applications can be set in different modes or scenes, such as work scenes, learning modes, leisure states, and children's modes. Different modes can be automatically identified and switched. The specific technical solutions include: 1). In the intelligent perception mode, the user’s needs can be directly calculated and matched content can be provided through the user’s use frequency, use time and attention orientation on the software program; 2), the use of smart devices including mobile phones Existing hardware configurations such as, ipad and personal computers, combined with computer vision technology and natural language processing technology, can calculate most of the human emotion cognitive parameters through facial expression features and voice communication status; 3), in addition, combined with Install a sensor that can recognize finger bioelectric signals on the keyboard or touch screen, for example, combined with PPG (photoplethysmography) technology, which can further calculate the user's pulse, heart rate and other physiological parameters to obtain some important information of the autonomic nervous system . Through these technologies, the signal collection of various sense organs can be integrated to realize more efficient human-computer interaction and brain-computer integration.

For the artificial intelligence-assisted cognitive behavioral therapy process shown in Figure 3, it is necessary to establish a good consultation relationship as soon as possible and more efficient communication in the cognitive behavioral therapy process. Through the combination of artificial intelligence technology, in the cognitive behavioral therapy process, the cognitive image or potential consciousness in the brain of the visitor can be virtually presented on the computer, which will help the psychological communication and the treatment process to be smoother and more effective, which can be significantly reduced Difficulty in communication, shorten treatment time and improve correction effect. The theoretical basis of this kind of cognitive behavioral therapy, for example, is that a song, a picture, or even a short video can sometimes just hit the critical point of a person’s emotional cognitive state at a certain moment, causing the greatest resonance value, which can make The brain reacts strongly and brings the most possible neuroplasticity, or is the best time window for changing obsessions. At this time, the psychotherapist can give appropriate guidance and corrections, and you can get twice the result with half the effort. Therefore, the key point of the present invention is to find the audiovisual stimulus that best matches the visitor's mood (or image) through the machine algorithm within the effective time, which is conducive to the visitor's realization of the mind (or attention) concentration and open the heart to proceed with the problem in the shortest time. comminicate. The specific solutions include: 1) Based on existing data (such as big data of emotions and six desires), use supervised machine learning algorithms to build emotional libraries, including photo galleries, sound libraries, and short videos, which can be combined with Google, Bing, etc. to search for existing view data; 2) The visitor chooses an emotional library by himself, and scores and arranges multiple scenes (for example, 10) to filter their personality orientation and characteristics; 3) AI empowers them to artificially synthesize new audio-visual scenes to provide More, more effective, and more targeted audio-visual scenes; 4), around the closest content scene, the two sides communicate, discuss and correct several intuitive beliefs and core beliefs in cognitive behavior.

It should be noted that on the basis of the brain-computer combination system provided by the present invention, various new software and hardware functions are added, such as various health index monitors, combined with the latest artificial intelligence technology applications, etc., which can achieve more Various types of smart devices or electronic devices to identify the emotional cognitive state of the target object and provide targeted intervention or treatment. For example, emotional cognitive states include, but are not limited to, emotional states, fatigue states, attention states, and stress states. For example, the risk of Parkinson’s disease can be predicted early through the analysis of touch screen actions on smart devices. Motion scanning analysis can predict children's autism and Alzheimer's disease early, and the risk of depression can be predicted early through sound spectrum analysis. Smart devices include, but are not limited to, smart phones, computers, smart pets, smart medical instruments, and smart robots.

In summary, the new brain-computer combination technology proposed in the present invention is not limited to the decoding of brainwave signals, but pays more attention to the decoding of the terminal output signals of the entire nervous system, including human expressions, voices, eye movements, heartbeats and heartbeats. Decoding the information of the whole sensory limbs such as skin electricity, and using the latest artificial intelligence technologies such as facial expression recognition, natural language processing, and human bioelectric feature recognition. It is not only conducive to more accurate interpretation and adjustment of human brain emotion and cognitive status, but also conducive to the timely prediction, prevention and intervention of brain cognitive diseases.

It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even change the order, as long as it can be implemented. The required functions are sufficient.

The present invention may be a system, a method and/or a computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling a processor to implement various aspects of the present invention.

The computer-readable storage medium may be a tangible device that holds and stores instructions used by the instruction execution device. The computer-readable storage medium may include, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing, for example. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a printer with instructions stored thereon The protruding structure in the hole card or the groove, and any suitable combination of the above.

The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and spirit of the described embodiments, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles, practical applications, or technical improvements in the market of the various embodiments, or to enable other ordinary skilled in the art to understand the various embodiments disclosed herein.

Claims

A brain-computer integration system based on sensory transmission, including:

The brain information terminal decoding unit is used to interpret the neural information transmitted by the target object through the sense organs in a non-invasive manner to obtain psychological and physiological data;

The psycho-emotional modeling unit is used to recognize the emotional cognitive state of the target object by using machine learning based on the collected psychological and physiological data;

The physical output signal stimulation unit is used to generate human-computer interaction information with sensory stimulation effects based on the emotional cognitive state, so as to prompt the target object to produce the desired emotional cognitive state.
The brain-computer integration system based on sensory transmission according to claim 1, wherein the brain information terminal decoding unit includes one of a video acquisition device, a sound acquisition device, a skin electrokinesis acquisition device, and an electromyography acquisition device, or Many kinds.
The brain-computer integration system based on sensory transmission according to claim 1, wherein the information transmitted by the sensory organs includes one or more of expression information, sound information, eye movement information, heartbeat information, and skin electrical information .
The brain-computer integration system based on sensory transmission according to claim 1, wherein the emotional cognitive state includes one or more of emotional state, fatigue state, attention state, and stress state.
The brain-computer combination system based on sensory transmission according to claim 1, wherein the human-computer interaction information with sensory stimulation effects includes one of text communication, voice dialogue, game interaction, picture playback, and short video playback. Or multiple.
The brain-computer integration system based on sensory transmission according to claim 1, wherein the psycho-emotional modeling unit inputs the obtained psychological and physiological data to the trained machine learning or deep learning model, and outputs the target object's Affective cognitive state.
The brain-computer integration system based on sensory transmission according to claim 6, wherein the deep learning model includes a convolutional neural network and a cyclic neural network.
A brain-computer combination system based on sensory transmission is characterized in that the physical output signal stimulation unit generates interactive information with sensory stimulation effects according to the following steps:

Judging the type of human-computer interaction information with sensory stimulation effect that needs to be generated based on the emotional cognitive state;

By categorizing and sorting the existing human-computer interaction information types, and combining the adversarial generation network algorithm to synthesize new relevant human-computer interaction information, forming a human-computer interaction information database;

According to the characteristics of the type of human-computer interaction information that needs to be generated, the human-computer interaction information to be displayed to the target object is selected from the human-computer interaction information database.
A brain-computer combination method based on sensory transmission, including the following steps:

Interpret the neural information transmitted by the target object through the sense organs in a non-invasive way to obtain psychological and physiological data;

Use machine learning to identify the emotional and cognitive state of the target object based on the collected psychological and physiological data;

Based on the emotional cognitive state, interactive information with sensory stimulation effects is generated to prompt the target object to produce a desired emotional cognitive state.
The brain-computer combination method based on sensory transmission according to claim 9, wherein the psychological and physiological data includes facial expression information collected by a camera, eye movement information collected by an eye tracker, and collected by an audio acquisition device. One or more of the sound information, the electric skin information obtained by the electric skin collector, and the heartbeat information obtained by the bioelectric signal sensor.
An electronic device, comprising the brain-computer integration system based on sensory transmission according to any one of claims 1 to 8, the electronic device performing one or more of the following steps:

Provide users with matching content based on one or more of the user's use frequency, use time, and attention orientation of the software program of the electronic device;

Recognize the user’s emotional and cognitive information based on facial expression features and voice communication status;

Using the sensor that recognizes the bioelectric signal of the finger combined with photoplethysmography, the user's physiological data is calculated and the autonomic nervous system information is obtained.
An electronic device, comprising the brain-computer integration system based on sensory transmission according to any one of claims 1 to 8, the electronic device performing one or more of the following steps:

Use supervised machine learning based on existing data to build an audio-visual feeling database;

According to the audio-visual emotion library selected by the user, filter its personality orientation and characteristics;

Provide targeted audiovisual scenes to the user.
An electronic device, comprising a memory and a processor, and a computer program that can run on the processor is stored on the memory, wherein the processor implements the method of claim 9 when the program is executed. step.
The electronic device according to claim 13, the electronic device comprising a smart phone, a computer, a smart pet, a smart medical instrument, or a smart robot.
A computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps of the method according to claim 9.