CN114356074B - Animal brain-computer interface implementation method and system based on in-vivo fluorescence signals - Google Patents
Animal brain-computer interface implementation method and system based on in-vivo fluorescence signals Download PDFInfo
- Publication number
- CN114356074B CN114356074B CN202111447778.3A CN202111447778A CN114356074B CN 114356074 B CN114356074 B CN 114356074B CN 202111447778 A CN202111447778 A CN 202111447778A CN 114356074 B CN114356074 B CN 114356074B
- Authority
- CN
- China
- Prior art keywords
- brain
- fluorescence
- frame
- signal
- activity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000001727 in vivo Methods 0.000 title claims abstract description 39
- 241001465754 Metazoa Species 0.000 title claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 53
- 210000004556 brain Anatomy 0.000 claims abstract description 40
- 238000010171 animal model Methods 0.000 claims abstract description 24
- 102000004169 proteins and genes Human genes 0.000 claims abstract description 18
- 108090000623 proteins and genes Proteins 0.000 claims abstract description 18
- BHPQYMZQTOCNFJ-UHFFFAOYSA-N Calcium cation Chemical compound [Ca+2] BHPQYMZQTOCNFJ-UHFFFAOYSA-N 0.000 claims abstract description 12
- 229910001424 calcium ion Inorganic materials 0.000 claims abstract description 12
- 230000003993 interaction Effects 0.000 claims abstract description 12
- 241000700605 Viruses Species 0.000 claims abstract description 11
- 230000000694 effects Effects 0.000 claims description 52
- 210000002569 neuron Anatomy 0.000 claims description 52
- 210000004027 cell Anatomy 0.000 claims description 43
- 238000009792 diffusion process Methods 0.000 claims description 20
- 238000012544 monitoring process Methods 0.000 claims description 17
- 230000009467 reduction Effects 0.000 claims description 16
- 238000012937 correction Methods 0.000 claims description 14
- 230000033001 locomotion Effects 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 14
- 230000008859 change Effects 0.000 claims description 10
- 238000004458 analytical method Methods 0.000 claims description 9
- 230000006399 behavior Effects 0.000 claims description 9
- 230000007177 brain activity Effects 0.000 claims description 9
- 238000006073 displacement reaction Methods 0.000 claims description 9
- OYPRJOBELJOOCE-UHFFFAOYSA-N Calcium Chemical group [Ca] OYPRJOBELJOOCE-UHFFFAOYSA-N 0.000 claims description 8
- 229910052791 calcium Inorganic materials 0.000 claims description 8
- 239000011575 calcium Substances 0.000 claims description 8
- 239000011159 matrix material Substances 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 7
- 230000001419 dependent effect Effects 0.000 claims description 7
- 238000000799 fluorescence microscopy Methods 0.000 claims description 6
- 238000009499 grossing Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 238000011160 research Methods 0.000 claims description 5
- 238000009825 accumulation Methods 0.000 claims description 4
- 238000003706 image smoothing Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 4
- 238000002372 labelling Methods 0.000 claims description 2
- 230000001960 triggered effect Effects 0.000 claims description 2
- 230000006870 function Effects 0.000 description 6
- 239000000126 substance Substances 0.000 description 6
- 238000001917 fluorescence detection Methods 0.000 description 4
- 239000000523 sample Substances 0.000 description 4
- 230000005859 cell recognition Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 230000001537 neural effect Effects 0.000 description 3
- 238000011830 transgenic mouse model Methods 0.000 description 3
- 241000699660 Mus musculus Species 0.000 description 2
- 241000699670 Mus sp. Species 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 239000007850 fluorescent dye Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008035 nerve activity Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 239000013543 active substance Substances 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000037396 body weight Effects 0.000 description 1
- 239000013626 chemical specie Substances 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 150000002500 ions Chemical class 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 210000000337 motor cortex Anatomy 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 210000000653 nervous system Anatomy 0.000 description 1
- 230000001722 neurochemical effect Effects 0.000 description 1
- 239000011347 resin Substances 0.000 description 1
- 229920005989 resin Polymers 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000000392 somatic effect Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
Abstract
The invention provides an animal brain-computer interface realization method and system based on an in-vivo fluorescence signal, comprising the following steps: injecting a calcium ion indicator or indicator protein into the cortex or deep nucleus of the brain to express the indicator or indicator protein in cells by using in vivo viruses; after the indication medium enters the cells, implanting a brain interface in a target brain area of the experimental animal, recording, and transmitting a recording result to a signal processing host; after the data result is transmitted, the data result enters a decoding algorithm to identify a pattern rule, and an output interface outputs the pattern rule to a control end; after the control terminal receiving device receives the signals, recoding the signals finally realizes coding control on external equipment, and brain-computer interaction is formed. The invention can solve the problems that the signal source of the traditional brain-computer interface of the brain-computer signal is greatly influenced by external electrical noise, and the problems of single coded information and low coding efficiency.
Description
Technical Field
The invention relates to the technical field of electroencephalogram signal acquisition, in particular to an animal brain-computer interface realization method and system based on an in-vivo fluorescence signal.
Background
The brain-computer interface refers to a direct connection created between the human or animal brain and an external device, enabling information exchange between the brain and the device. At present, a common brain-computer interface adopts nerve activity electric signals to decode information, external electric interference encountered in living scenes is amplified, and the brain-computer interface is limited in use in free activity scenes. In recent years, in animal experiments, by using an in vivo fluorescence microscopic imaging technique, nerve activity changes can be recorded in detail by using a neurochemical probe. The method can accurately analyze the activity rules of different neurons, and simultaneously provides a good signal source for signal identification in a brain-computer interface for information decoding and encoding.
The invention patent with the publication number of CN106293088A discloses a brain-computer interface processing system and an implementation method thereof, wherein the brain-computer interface processing system comprises a brain-computer interface processing device and a software application platform, the brain-computer interface processing device comprises a dry electrode, an electroencephalogram signal acquisition and analysis module, a microprocessor, a wireless communication module and a processing and display terminal which are sequentially connected, the software application platform comprises an electroencephalogram signal storage module, an electroencephalogram signal on-line feedback module and an open API interface, and the microprocessor is respectively in information interaction with the wireless communication module, the electroencephalogram signal storage module, the electroencephalogram signal on-line feedback module and the open API interface; and the wireless communication module performs information interaction with the processing and display terminal.
The prior art has the following defects: the electroencephalogram signals in the common brain-computer interface are greatly influenced by external electrical noise, and the requirements on the realization environment are high; the brain-computer interface has low spatial resolution, and the coding capability of the brain-computer interface is limited; the recorded neuron signals cannot correspond for a long time, and interference is very easy to occur among the signals.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides an animal brain-computer interface realization method and system based on an in-vivo fluorescence signal.
According to the method and the system for realizing the animal brain-computer interface based on the in-vivo fluorescence signal, the scheme is as follows:
in a first aspect, an implementation method of an animal brain-computer interface based on an in-vivo fluorescence signal is provided, and the implementation method comprises the following steps:
Step S1: injecting a calcium ion indicator or indicator protein into the cortex or deep nucleus of the brain to express the indicator or indicator protein in cells by using in vivo viruses;
Step S2: after the indication medium enters cells, a brain interface is implanted in a target brain area of an experimental animal, the brain interface is recorded by an in-vivo micro fluorescent microscope, and a recording result is transmitted to a signal processing host through a signal acquisition host;
Step S3: after the data result is transmitted, the data result enters a decoding algorithm to identify a pattern rule and is output to a control end through an instruction output interface;
step S4: after the control terminal receiving device receives the signals, recoding the signals finally realizes coding control on external equipment, and brain-computer interaction is formed.
Preferably, after the fluorescent indication signal in the indicator is expressed, the image data is transmitted into the signal processing end for signal processing through the acquisition end, and the method specifically comprises the following steps: after being collected by the collecting equipment, the brain activity fluorescence original signals of the experimental animals are transmitted to the signal processing host for on-line processing; the original data is subjected to neuron activity extraction and identification through an algorithm, and mainly comprises noise reduction correction and cell marking.
Preferably, the noise reduction correction is to reduce signal noise interference caused by motion, and correct the positions of the same neuron at different time points in real time by applying an anisotropic diffusion denoising operation on the original image frame;
For each frame of image, I ε R HxW, the evolution follows the equation for a given diffusion time τ:
Where div is the divergence operator, And delta is the gradient and laplace operator, respectively, C is the diffusion coefficient matrix dependent on pixel and τ;
I represents the current frame; r HxW represents a set of frames; h represents a frame height; w represents the frame width;
Subsequently, a KTL tracker is used between different frames to estimate the displacement of potential angular features between two adjacent frames; after KTL tracker based registration, further unit registration is performed using diffeomorphic Log-Demons;
finally, the acquired image data is minimized in motion noise, and further subsequent analysis and processing are performed.
Preferably, the cell markers include: after the image smoothing process, the potential cell contour image needs to be subjected to cell marking:
First, a portion of the frames is randomly selected:
Where a is the frame number of the random profile set; s represents contour projection; t represents the maximum frame number;
The maximum projection map for the selected frame is then calculated:
Detecting that all local maximum points on the graph are S α, repeating the process for a plurality of times, wherein any two of the local maximum points do not contain the same frame;
and finally, collecting S α of all the maximum projection graphs and calculating the integration of the maximum projection graphs S with the complete contour data set as a final neuron contour seed set.
Preferably, after obtaining the neuron contours, the system will perform fluorescence activity monitoring and pattern learning and classification of the corresponding neurons, in particular:
First, the fluorescence brightness of the image data needs to be normalized, and the average brightness F t of the target cell contour is subtracted by the average background fluorescence brightness F 0 of the tie and divided by the background fluorescence brightness value F 0, and the calculation formula is as follows, wherein Δf t/F0 is abbreviated as Δf/F:
Secondly, denoising and smoothing the recorded calcium signals by using an exponential weighted moving average method;
thirdly, after the smoothed fluorescence intensity change value is obtained, detecting a fluorescence activity event on line;
Finally, synchronously collecting the behaviors of the experimental animal, pairing the behaviors of the animal with real-time fluorescence imaging results, performing pattern learning, extracting features and classifying patterns based on PCA and SVM algorithms, and performing classification learning on existing fluorescence activity patterns;
After pattern learning and classification, corresponding coding is carried out on the output instruction according to the classification pattern, the fluorescence activity pattern is judged online when the operation is executed, and the corresponding instruction is output to control the external equipment.
Preferably, after obtaining the neuron outline, a whole fluorescence observation method is selected to indicate the activity of the nucleus, and after obtaining the neuron outline, the average gray value of the whole frame image is calculated.
Preferably, the specific steps of calculating the average gray value are:
1) Counting the gray value of the pixel of each frame of data source image to carry out accumulation summation;
2) Calculating the total number of pixels of the image: n= VIEW WIDTH × VIEW HEIGHT;
3) Calculating the average gray value of the image: avg_value=sum/n;
After the average value is obtained, the change of the average fluorescence intensity is tracked in real time, a fluorescence intensity curve is drawn, and nuclear mass activity monitoring is carried out; and the activity threshold is set according to the research requirement, the threshold exceeding monitoring is carried out in real time in the actual working process, and once the threshold is exceeded, the command output is triggered, so that the external equipment is controlled or closed-loop regulation and control are carried out.
In a second aspect, there is provided an animal brain-computer interface system based on an in vivo fluorescence signal, the system comprising:
Module M1: injecting a calcium ion indicator or indicator protein into the cortex or deep nucleus of the brain to express the indicator or indicator protein in cells by using in vivo viruses;
Module M2: after the indication medium enters cells, a brain interface is implanted in a target brain area of an experimental animal, the brain interface is recorded by an in-vivo micro fluorescent microscope, and a recording result is transmitted to a signal processing host through a signal acquisition host;
Module M3: after the data result is transmitted, the data result enters a decoding algorithm to identify a pattern rule and is output to a control end through an instruction output interface;
Module M4: after the control terminal receiving device receives the signals, recoding the signals finally realizes coding control on external equipment, and brain-computer interaction is formed.
Preferably, after the fluorescent indication signal in the indicator is expressed, the image data is transmitted into the signal processing end through the acquisition end for signal processing, and the method specifically comprises the following steps: after being collected by the collecting equipment, the brain activity fluorescence original signals of the experimental animals are transmitted to the signal processing host for on-line processing; the original data is subjected to neuron activity extraction and identification through an algorithm, and mainly comprises noise reduction correction and cell marking.
Preferably, the noise reduction correction is to reduce signal noise interference caused by motion and correct the positions of the same neuron at different time points in real time by applying an anisotropic diffusion noise removal operation on the original image frame;
For each frame of image, I ε R HxW, the evolution follows the equation for a given diffusion time τ:
Where div is the divergence operator, And delta is the gradient and laplace operator, respectively, C is the diffusion coefficient matrix dependent on pixel and τ;
I represents the current frame; r HxW represents a set of frames; h represents a frame height; w represents the frame width;
Subsequently, a KTL tracker is used between different frames to estimate the displacement of potential angular features between two adjacent frames; after KTL tracker based registration, further unit registration is performed using diffeomorphic Log-Demons;
finally, the acquired image data is minimized in motion noise, and further subsequent analysis and processing are performed.
Compared with the prior art, the invention has the following beneficial effects:
1. the invention solves the problem that the signal source of the brain-computer interface of the traditional brain-computer signal is greatly influenced by external electrical noise by collecting the fluorescence of the neuron active probe as the source of the neural activity signal;
2. The invention decodes and codes large-scale data through large-scale neuron activity recognition, and solves the problems of single coding information and low coding efficiency.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
FIG. 1 is a diagram of the system of the present invention;
fig. 2 is a schematic diagram of the specific working principle of the present invention.
Reference numerals:
Acquisition front end 1 laboratory animal 2
Brain interface 3 signal acquisition host 4
First signal pattern 6 of signal processing host 5
Second signal pattern 7 pattern rule recognition 8
Control end receiving device 10 of instruction output interface 9
The external device 11 fluoresces the original signal 101
Cell recognition Module 102 cell recognition results 103
Pattern learning and classification 104 pattern library 105
Mode judgment instruction output module 106 average calculation module 107
Activity monitoring Module 108 threshold monitoring Module 109
Instruction output module 110
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present invention.
In order to expand brain signal sources of brain-computer interfaces, reduce the influence of external electrical noise on the functions of the brain-computer interfaces, enrich coding capacity of the brain-computer interfaces and avoid mutual interference between nerve electrical signals, the embodiment of the invention provides an animal brain-computer interface system based on in-vivo fluorescence signals, which can monitor activities of thousands of neurons at the same time and can perform complex coding without being interfered by external electrical noise; the system may decode complex local brain activity, interact with external devices 11, and control the devices.
An animal brain-computer interface system based on in-vivo fluorescence signals, active substances in neurons are marked through fluorescent probes, the activity modes and coding states of the neurons and a nervous system can be indirectly reflected, and the signals are used as signal sources to decode the neuron signals, so that the signals are richer than the electric signals, and the brain-computer interface signals are decoded and recoded. Referring to fig. 1 and 2, the system mainly includes: the device comprises a chemical substance indication medium, an in-vivo fluorescence detection acquisition device, a signal processing device and an output control terminal.
Chemical species-indicating agents are primarily realized by calcium ion indicators or other types of chemical species fluorescent probes, including all concentration-changing indicator probes for ions and compounds that are indicative of the functional state of nerve cells;
The in-vivo fluorescence detection device mainly comprises a miniature in-vivo fluorescence imaging microscope;
The signal processing device and the output control terminal are composed of a decoding and encoding algorithm based on a computer and software and hardware.
The system comprises the following modules: module M1: injecting a calcium ion indicator or indicator protein into the cortex or deep nucleus of the brain to express the indicator or indicator protein in cells by using in vivo viruses;
Module M2: after the indication medium enters cells, a brain interface is implanted in a target brain area of an experimental animal, the brain interface is recorded by an in-vivo micro fluorescent microscope, and a recording result is transmitted to a signal processing host through a signal acquisition host;
Module M3: after the data result is transmitted, the data result enters a decoding algorithm to identify a pattern rule and is output to a control end through an instruction output interface;
Module M4: after the control terminal receiving device receives the signals, recoding the signals finally realizes coding control on external equipment, and brain-computer interaction is formed.
After the fluorescent indication signal in the indicator is expressed, the image data is transmitted into the signal processing end through the acquisition end for signal processing, and the method specifically comprises the following steps: after being collected by the collecting equipment, the brain activity fluorescence original signals of the experimental animals are transmitted to the signal processing host for on-line processing; the original data is subjected to neuron activity extraction and identification through an algorithm, and mainly comprises noise reduction correction and cell marking.
The noise reduction correction is to reduce signal noise interference caused by motion, correct the positions of different time points of the same neuron in real time, and realize the noise reduction by applying the anisotropic diffusion denoising operation on the original image frame;
For each frame of image, I ε R HxW, the evolution follows the equation for a given diffusion time τ:
Where div is the divergence operator, And delta is the gradient and laplace operator, respectively, C is the diffusion coefficient matrix dependent on pixel and τ;
I represents the current frame; r HxW represents a set of frames; h represents a frame height; w represents the frame width;
A KTL tracker is then used between different frames to estimate the displacement of potential angular features between two adjacent frames. After KTL tracker based registration, further unit registration is performed using diffeomorphic Log-Demons;
finally, the acquired image data is minimized in motion noise, and further subsequent analysis and processing are performed.
The invention also provides an animal brain-computer interface implementation method based on the in-vivo fluorescence signal, which comprises the following steps:
Step S1: injecting a calcium ion indicator or indicator protein into the cortex or deep nucleus of the brain to express the indicator or indicator protein in cells by using in vivo viruses;
step S2: after the indication medium enters cells, a brain interface 3 is implanted in a target brain area of the experimental animal 2, the recording is carried out through an in-vivo micro fluorescence microscope, and the recording result is transmitted to a signal processing host 5 through a signal acquisition host 4;
Step S3: after the data result is transmitted, the data result enters a decoding algorithm to carry out pattern rule recognition 8 and is output to a control end through an instruction output interface 9;
step S4: after the control end receiving device 10 receives the signals, recoding the signals finally realizes coding control on the external equipment 11, and brain-computer interaction is formed.
Specifically, after the fluorescent indication signal in the indicator is expressed, the image data is transmitted into the signal processing end through the acquisition end for signal processing, and specifically comprises the following steps: the brain activity fluorescence original signal 101 of the experimental animal 2 is transmitted to the signal processing host 5 for on-line processing after being acquired by the acquisition equipment; the raw data is required to be extracted and identified by the algorithm of the cell identification module 102, and mainly comprises noise reduction correction and cell marking.
The noise reduction correction is to reduce signal noise interference caused by motion, correct the positions of the same neuron at different time points in real time, and realize the noise reduction by applying the anisotropic diffusion denoising operation on the original image frame;
For each frame of image, I ε R HxW, the evolution follows the equation for a given diffusion time τ:
Where div is the divergence operator, And delta is the gradient and laplace operator, respectively, C is the diffusion coefficient matrix dependent on pixel and τ;
I represents the current frame; r HxW represents a set of frames; h represents a frame height; w represents the frame width;
A KTL tracker is then used between different frames to estimate the displacement of potential angular features between two adjacent frames. After KTL tracker based registration, further unit registration is performed using diffeomorphic Log-Demons;
finally, the acquired image data is minimized in motion noise, and further subsequent analysis and processing are performed.
Cell markers include: after the image smoothing process, the potential cell contour image needs to be subjected to cell marking:
First, a portion of the frames is randomly selected:
Where a is the frame number of the random profile set; s represents contour projection; t represents the maximum frame number;
The maximum projection map for the selected frame is then calculated:
Detecting that all local maximum points on the graph are S α, repeating the process for a plurality of times, wherein any two of the local maximum points do not contain the same frame;
and finally, collecting S α of all the maximum projection graphs and calculating the integration of the maximum projection graphs S with the complete contour data set as a final neuron contour seed set.
After obtaining the neuron contour cell recognition result 103, the system will perform fluorescence activity monitoring and pattern learning and classification 104 on the corresponding neurons, specifically:
First, the fluorescence brightness of the image data needs to be normalized, and the average brightness F t of the target cell contour is subtracted by the average background fluorescence brightness F 0 of the tie and divided by the background fluorescence brightness value F 0, and the calculation formula is as follows, wherein Δf t/F0 is abbreviated as Δf/F:
Secondly, denoising and smoothing the recorded calcium signals by using an exponential weighted moving average method;
thirdly, after the smoothed fluorescence intensity change value is obtained, detecting a fluorescence activity event on line;
Finally, synchronously collecting the behaviors of the experimental animal 2, pairing the behaviors of the animal with real-time fluorescence imaging results, performing pattern learning, extracting features and classifying patterns based on PCA and SVM algorithms, and performing classification learning on existing fluorescence activity patterns;
after pattern learning and classification, a pattern library 105 is established, the output instructions are correspondingly encoded according to the classification patterns by a pattern judgment instruction output module 106, the fluorescence activity patterns are judged online when the operation is performed, and the corresponding instructions are output to control the external device 11.
Meanwhile, after the neuron contour cell identification result 103 is obtained, an integral fluorescence observation method can be selected in parallel to indicate the nuclear cluster activity, and after the neuron contour cell identification result 103 is obtained, an average gray value of the whole frame image is calculated in the average value calculation module 107.
The method comprises the following specific steps:
1) Counting the gray value of the pixel of each frame of data source image to carry out accumulation summation;
2) Calculating the total number of pixels of the image: n= VIEW WIDTH × VIEW HEIGHT;
3) Calculating the average gray value of the image: avg_value=sum/n;
After the average value is obtained, the change of the average fluorescence intensity is tracked in real time, a fluorescence intensity curve is drawn, and nuclear cluster activity monitoring is carried out in the activity monitoring module 108; and the activity threshold is set according to the research requirement, in the actual working process, the threshold monitoring module 109 carries out threshold exceeding monitoring in real time, and once the threshold is exceeded, the command output module 110 triggers the command output, so as to realize the control or closed-loop regulation and control of the external equipment 11.
Next, the present invention will be described in more detail.
The implementation process of the invention is as follows: the indicator or indicator protein is expressed intracellularly by injecting a calcium ion indicator or indicator protein into the cortex or deep nucleus of the brain. After the indication medium enters cells, a brain interface 3 is implanted in a target brain area of the experimental animal 2, the recording is carried out through an in-vivo miniature fluorescence microscope, the recorded result is transmitted to a signal processing host 5 through a signal acquisition host 4, the data comprise two forms, the fluorescence intensity change value can obtain a first signal mode 6 and the image data can obtain a second signal mode 7. After the data result is transmitted, the data result enters a decoding algorithm to carry out pattern rule recognition 8 and is output to a control end through an instruction output interface 9. After the control end receiving device 10 receives the signals, recoding the signals finally realizes coding control on the external equipment 11, and brain-computer interaction is formed.
After the fluorescence indication signal is expressed, the image data is transmitted into the signal processing end through the acquisition end to be subjected to signal processing, and the working principle is as follows:
the brain activity fluorescence original signal 101 of the experimental animal 2 is transmitted to the signal processing host 5 for on-line processing after being acquired by the acquisition equipment. The original data is required to be extracted and identified through an algorithm, and mainly comprises noise reduction correction and cell marking.
The main function of the noise reduction correction module is to reduce signal noise interference caused by movement and correct the positions of different time points of the same neuron in real time. This is achieved mainly by applying a spread-spectrum denoising operation on the original image frame. For each frame of image, I ε R HxW, the evolution follows the equation for a given diffusion time τ
Where div is the divergence operator,And delta is the gradient and laplace operator, respectively, C is the diffusion coefficient matrix dependent on pixel and τ;
I represents the current frame; r HxW represents a set of frames; h represents a frame height; w represents the frame width;
By choosing the specific form of C and τ, we can control the level of smoothing along or perpendicular to the boundary between the neuron and the background. C selecting classical Perona-Malik filter Preferably smoothing the low contrast region, wherein exp (-) represents an exponential function with e as the base; ali represents the value of the determinant of the matrix; k represents the boundary sensitivity.
Subsequently, a Kanade-Lucas-Tomasi (KTL) tracker was used between different frames to estimate the displacement of potential angular features between two adjacent frames. After KTL tracker based registration, further unit registration was performed using diffeomorphic Log-Demons. diffeomorphic Log-Demons is a non-parametric registration method that can find the displacement of all pixels by minimizing the global energy in the logarithmic domain:
It includes similarity, correspondence and regularization terms, where Sim () represents the similarity term; dist (-) represents a distance item; reg (-) represents the annotation item; s represents contour projection; f and M are fixed and moving images, respectively, u is the displacement/update field, σ i,σx and σ T represent noise levels, corresponding spatial uncertainty and the level of forward and backward. After the steps, the acquired image data is minimized in motion noise and can be subjected to subsequent analysis.
After the image smoothing process, cell labeling is required for the potential cell contour image. The contour set of all potential actual cell contour ROI centers generated initially, but contains a large number of false positive cell contours. The process first randomly selects a portion of the frames
Where a is the frame number of the random profile set; s represents contour projection; t represents the maximum frame number;
The maximum projection map for the selected frame is then calculated:
And detecting that all local maximum points on the graph are S α, repeating the above process for a plurality of times, wherein any two of the local maximum points do not contain the same frame. And finally, collecting S α of all the maximum projection graphs and calculating the integration of the maximum projection graphs S with the complete contour data set as a final neuron contour seed set.
After obtaining the neuron contour cell identification result 103, the system monitors fluorescence activity and learns and classifies patterns of corresponding neurons, firstly, the fluorescence brightness of image data needs to be normalized, average brightness F t of target cell contour is used for subtracting the average background fluorescence brightness F 0 of tie and dividing the average brightness F 0 by the average background fluorescence brightness value F 0, and a calculation formula is as follows, wherein Δf t/F0 is abbreviated as Δf/F:
secondly, denoising and smoothing the recorded calcium signals by using an Exponential Weighted Moving Average (EWMA) method;
And thirdly, after the smoothed fluorescence intensity change value is obtained, detecting the fluorescence activity event on line. The method mainly comprises the steps of defining a base line length, setting a detection window length, defining a rising rate threshold, defining a fluorescence activity event after exceeding the threshold, and collecting the fluorescence value of a peak point in real time.
Finally, synchronously collecting the behaviors of the experimental animal 2, pairing the behaviors of the animal with real-time fluorescence imaging results, performing pattern learning, extracting features and classifying patterns based on PCA and SVM algorithms, and performing classification learning on existing fluorescence activity patterns.
After pattern learning and classification, the output instructions are correspondingly encoded according to the classification patterns, and when the external device 11 is operated, the fluorescence activity patterns are judged online and the corresponding instructions are output to control the external device 11.
Meanwhile, after the identification result 103 of the neuron contour cells is obtained, the whole fluorescence observation method can be selected in parallel to indicate the activity of the nucleolus. Firstly, after obtaining the identification result 103 of the neuron contour cells, the average value calculation module 107 calculates the average gray value of the whole frame image, which includes the steps of counting all pixels of the whole frame image, accumulating and calculating the gray value sum of all pixels, wherein the average gray value is the quotient of the gray value sum and the number of pixels, and the steps are as follows:
1) Counting the gray value of the pixel of each frame of data source image to carry out accumulation summation;
2) Calculating the total number of pixels of the image: n= VIEW WIDTH × VIEW HEIGHT;
3) Calculating the average gray value of the image: avg_value=sum/n.
After the average value is obtained, the change of the average fluorescence intensity is tracked in real time, a fluorescence intensity curve is drawn, and the activity of the nucleus is monitored in the activity monitoring module 108. And the activity threshold is set according to the research requirement, in the actual working process, the threshold monitoring module 109 carries out threshold exceeding monitoring in real time, and once the threshold is exceeded, the command output module 110 triggers the command output, so as to realize the control or closed-loop regulation and control of the external equipment 11.
Workflow of software and hardware interworking: the system comprises a chemical substance indicating medium, an in-vivo fluorescence detection acquisition device, a signal processing device and an output control terminal. By monitoring the chemical substance activity of the brain, indicating the neuron activity, identifying the neuron activity patterns caused by different behavioral patterns, pattern identification and output are realized, and the external device 11 is controlled. By using the fluorescence indication signal as a signal source, the interference of external electric noise can be avoided, more visual and effective neuron activity decoding capability can be realized, and the coding effect of the brain-computer interface can be improved. The design is different from the existing brain-computer interface based on the brain-computer signal, realizes the identification of the neuron activity mode through the real-time activity of the fluorescent signal of the chemical substance, and forms interaction with external equipment, enriches the signal source of the brain-computer interface, and can realize the deep research on the neural activity code.
Brain-computer control experiments based on calcium ion signals:
a) Experimental animal 2: healthy adult C57bl/6 mice, clean grade, body weight 20-25g. The experimental animal 2 is fed in an independent environment of 12-12 h day-night alternation, the room temperature is maintained at 24+/-2 ℃, and the experimental animal is free to drink and ingest, and the experiment is carried out after the experimental animal is adapted to the environment for 1 week.
B) Experimental equipment: calcium signal indicates transgenic mice (GCaMP transgenic mice), specific calcium signal indicates protein carrier virus (GCaMP carrier virus driven by specific promoter), virus injection instrument, resin lens, fixed surgical equipment, brain-computer system.
C) The experimental steps are as follows: as shown in fig. 1 and fig. 2, taking a motor cortex as an example, a transgenic mouse or a specific neuron infected by a specific calcium signal indicator protein virus in a target brain region is selected to realize the indication marker of the calcium signal activity of a specific neuron group in the target brain region. The brain interface 3 is implanted into the target brain region by means of stereotactic positioning, and is fixed for a long period of time by using a fixing material. After the mice recover, the acquisition front end 1 and the signal acquisition host 4 are utilized to perform real-time acquisition analysis on the activity of the neuron calcium ions in the target brain region. The identification of the calcium ion activity of the first signal pattern 6 and the second signal pattern 7 is performed by the signal processing host 5. After the data result is transmitted, the data result enters a decoding algorithm to carry out pattern rule recognition 8, wherein the pattern rule comprises fluorescence intensity, neuron activation sequence and pattern rule, and the data result is output to a control end through an instruction output interface 9. After the control end receiving device 10 receives the signals, recoding the signals finally realizes coding control on the external equipment 11, so as to form brain-computer interaction, and control the external equipment 11 or regulate and control the corresponding functions of the body in a closed loop manner.
The embodiment of the invention provides an animal brain-computer interface realization method and system based on an in-vivo fluorescence signal, which uses a neuron active probe fluorescence signal as a brain-computer interface signal source, and solves the problem that the traditional brain-computer interface signal source is greatly influenced by external electrical noise; acquiring brain activity characteristics in different behaviors by a somatic fluorescence detection means; the dynamic change of chemical substances in local neurons of the brain of an animal is used for decoding and recoding the neural activity, so that the problems of single coding information and low coding efficiency are solved.
Those skilled in the art will appreciate that the invention provides a system and its individual devices, modules, units, etc. that can be implemented entirely by logic programming of method steps, in addition to being implemented as pure computer readable program code, in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Therefore, the system and various devices, modules and units thereof provided by the invention can be regarded as a hardware component, and the devices, modules and units for realizing various functions included in the system can also be regarded as structures in the hardware component; means, modules, and units for implementing the various functions may also be considered as either software modules for implementing the methods or structures within hardware components.
The foregoing describes specific embodiments of the present application. It is to be understood that the application is not limited to the particular embodiments described above, and that various changes or modifications may be made by those skilled in the art within the scope of the appended claims without affecting the spirit of the application. The embodiments of the application and the features of the embodiments may be combined with each other arbitrarily without conflict.
Claims (10)
1. An animal brain-computer interface implementation method based on an in-vivo fluorescence signal is characterized by comprising the following steps:
Step S1: injecting a calcium ion indicator or indicator protein into the cortex or deep nucleus of the brain to express the indicator or indicator protein in cells by using in vivo viruses;
Step S2: after the indication medium enters cells, a brain interface is implanted in a target brain area of an experimental animal, the brain interface is recorded by an in-vivo micro fluorescent microscope, and a recording result is transmitted to a signal processing host through a signal acquisition host;
Step S3: after the data result is transmitted, the data result enters a decoding algorithm to identify a pattern rule and is output to a control end through an instruction output interface;
step S4: after the control terminal receiving device receives the signals, recoding the signals finally realizes coding control on external equipment, and brain-computer interaction is formed.
2. The method for realizing an animal brain-computer interface based on an in-vivo fluorescence signal according to claim 1, wherein after the fluorescence indication signal in the indicator is expressed, the image data is transmitted into a signal processing end for signal processing through an acquisition end, and the method specifically comprises the following steps: after being collected by the collecting equipment, the brain activity fluorescence original signals of the experimental animals are transmitted to the signal processing host for on-line processing; the original data is subjected to neuron activity extraction and identification through an algorithm, and mainly comprises noise reduction correction and cell marking.
3. The method for realizing the animal brain-computer interface based on the in-vivo fluorescence signal according to claim 2, wherein the noise reduction correction is to reduce signal noise interference caused by motion and correct the positions of the same neuron at different time points in real time by applying an anisotropic diffusion noise removal operation on an original image frame;
For each frame of image, I ε R HxW, the evolution follows the equation for a given diffusion time τ:
Where div is the divergence operator, And delta is the gradient and laplace operator, respectively, C is the diffusion coefficient matrix dependent on pixel and τ;
I represents the current frame; r HxW represents a set of frames; h represents a frame height; w represents the frame width;
Subsequently, a KTL tracker is used between different frames to estimate the displacement of potential angular features between two adjacent frames; after KTL tracker based registration, further unit registration is performed using diffeomorphic Log-Demons;
finally, the acquired image data is minimized in motion noise, and further subsequent analysis and processing are performed.
4. The method of claim 3, wherein the cell labeling comprises: after the image smoothing process, the potential cell contour image needs to be subjected to cell marking:
First, a portion of the frames is randomly selected:
Where a is the frame number of the random profile set; s represents contour projection; t represents the maximum frame number;
The maximum projection map for the selected frame is then calculated:
Detecting that all local maximum points on the graph are S α, repeating the process for a plurality of times, wherein any two of the local maximum points do not contain the same frame;
and finally, collecting S α of all the maximum projection graphs and calculating the integration of the maximum projection graphs S with the complete contour data set as a final neuron contour seed set.
5. The method according to claim 4, wherein after obtaining the neuron profile, the system will perform fluorescence activity monitoring and pattern learning and classification of the corresponding neurons, in particular:
First, the fluorescence brightness of the image data needs to be normalized, and the average brightness F t of the target cell contour is subtracted by the average background fluorescence brightness F 0 of the tie and divided by the background fluorescence brightness value F 0, and the calculation formula is as follows, wherein Δf t/F0 is abbreviated as Δf/F:
Secondly, denoising and smoothing the recorded calcium signals by using an exponential weighted moving average method;
thirdly, after the smoothed fluorescence intensity change value is obtained, detecting a fluorescence activity event on line;
Finally, synchronously collecting the behaviors of the experimental animal, pairing the behaviors of the animal with real-time fluorescence imaging results, performing pattern learning, extracting features and classifying patterns based on PCA and SVM algorithms, and performing classification learning on existing fluorescence activity patterns;
After pattern learning and classification, corresponding coding is carried out on the output instruction according to the classification pattern, the fluorescence activity pattern is judged online when the operation is executed, and the corresponding instruction is output to control the external equipment.
6. The method for realizing the animal brain-computer interface based on the in-vivo fluorescence signal according to claim 4, wherein after obtaining the neuron outline, a whole fluorescence observation method is selected to indicate the activity of the nucleus, after obtaining the neuron outline, the average gray value of the whole frame image is calculated, wherein the method comprises the steps of counting all pixel points of the whole image, accumulating and calculating the gray value sum of all the pixel points, and the average gray value is the quotient of the gray value sum and the number of the pixel points.
7. The method for realizing the animal brain-computer interface based on the in-vivo fluorescence signal according to claim 6, wherein the specific steps of calculating the average gray value are as follows:
1) Counting the gray value of the pixel of each frame of data source image to carry out accumulation summation;
2) Calculating the total number of pixels of the image: n= VIEW WIDTH × VIEW HEIGHT;
3) Calculating the average gray value of the image: avg_value=sum/n;
After the average value is obtained, the change of the average fluorescence intensity is tracked in real time, a fluorescence intensity curve is drawn, and nuclear mass activity monitoring is carried out; and the activity threshold is set according to the research requirement, the threshold exceeding monitoring is carried out in real time in the actual working process, and once the threshold is exceeded, the command output is triggered, so that the external equipment is controlled or closed-loop regulation and control are carried out.
8. An animal brain-computer interface system based on in vivo fluorescence signals, comprising:
Module M1: injecting a calcium ion indicator or indicator protein into the cortex or deep nucleus of the brain to express the indicator or indicator protein in cells by using in vivo viruses;
Module M2: after the indication medium enters cells, a brain interface is implanted in a target brain area of an experimental animal, the brain interface is recorded by an in-vivo micro fluorescent microscope, and a recording result is transmitted to a signal processing host through a signal acquisition host;
Module M3: after the data result is transmitted, the data result enters a decoding algorithm to identify a pattern rule and is output to a control end through an instruction output interface;
Module M4: after the control terminal receiving device receives the signals, recoding the signals finally realizes coding control on external equipment, and brain-computer interaction is formed.
9. The animal brain-computer interface system based on in-vivo fluorescence signals according to claim 8, wherein after the fluorescence indication signals in the indicator are expressed, the image data are transmitted into the signal processing end through the acquisition end for signal processing, and specifically comprising: after being collected by the collecting equipment, the brain activity fluorescence original signals of the experimental animals are transmitted to the signal processing host for on-line processing; the original data is subjected to neuron activity extraction and identification through an algorithm, and mainly comprises noise reduction correction and cell marking.
10. The animal brain-computer interface system based on an in-vivo fluorescence signal according to claim 9, wherein said noise reduction correction is to reduce motion-induced signal noise interference and correct the positions of the same neuron at different time points in real time by applying a hetero-diffusion denoising operation on the original image frame;
For each frame of image, I ε R HxW, the evolution follows the equation for a given diffusion time τ:
Where div is the divergence operator, And delta is the gradient and laplace operator, respectively, C is the diffusion coefficient matrix dependent on pixel and τ;
I represents the current frame; r HxW represents a set of frames; h represents a frame height; w represents the frame width;
Subsequently, a KTL tracker is used between different frames to estimate the displacement of potential angular features between two adjacent frames; after KTL tracker based registration, further unit registration is performed using diffeomorphic Log-Demons;
finally, the acquired image data is minimized in motion noise, and further subsequent analysis and processing are performed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111447778.3A CN114356074B (en) | 2021-11-30 | 2021-11-30 | Animal brain-computer interface implementation method and system based on in-vivo fluorescence signals |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111447778.3A CN114356074B (en) | 2021-11-30 | 2021-11-30 | Animal brain-computer interface implementation method and system based on in-vivo fluorescence signals |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114356074A CN114356074A (en) | 2022-04-15 |
| CN114356074B true CN114356074B (en) | 2024-04-23 |
Family
ID=81098031
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111447778.3A Active CN114356074B (en) | 2021-11-30 | 2021-11-30 | Animal brain-computer interface implementation method and system based on in-vivo fluorescence signals |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114356074B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115357125A (en) * | 2022-10-19 | 2022-11-18 | 之江实验室 | Interactive brain-computer interface method and device carrying graphical interface |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110974219A (en) * | 2019-12-20 | 2020-04-10 | 北京脑陆科技有限公司 | Human brain idea recognition system based on invasive BCI |
| CN111406706A (en) * | 2019-01-04 | 2020-07-14 | 温州医科大学 | Method for constructing brain-computer interface behavioural model |
| US10795440B1 (en) * | 2017-04-17 | 2020-10-06 | Facebook, Inc. | Brain computer interface for text predictions |
| CN112107307A (en) * | 2020-08-24 | 2020-12-22 | 中国科学院上海微系统与信息技术研究所 | Preparation method and structure of high-flux implanted flexible nerve electrode |
| CN112462055A (en) * | 2015-09-14 | 2021-03-09 | Essenlix公司 | Device and system for analyzing a sample, in particular blood, and method for using same |
| US11013411B1 (en) * | 2018-06-13 | 2021-05-25 | The Research Foundation For The State University Of New York | Chromatic bioluminescence as a cellular level readout system of neural activity |
| CN113440143A (en) * | 2021-07-12 | 2021-09-28 | 北京华科恒生医疗科技有限公司 | Fluorescence sensing composite deep electrode |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015164300A1 (en) * | 2014-04-21 | 2015-10-29 | The General Hospital Corporation | Biomedical system variably configured based on estimation of information content of input signals |
-
2021
- 2021-11-30 CN CN202111447778.3A patent/CN114356074B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112462055A (en) * | 2015-09-14 | 2021-03-09 | Essenlix公司 | Device and system for analyzing a sample, in particular blood, and method for using same |
| US10795440B1 (en) * | 2017-04-17 | 2020-10-06 | Facebook, Inc. | Brain computer interface for text predictions |
| US11013411B1 (en) * | 2018-06-13 | 2021-05-25 | The Research Foundation For The State University Of New York | Chromatic bioluminescence as a cellular level readout system of neural activity |
| CN111406706A (en) * | 2019-01-04 | 2020-07-14 | 温州医科大学 | Method for constructing brain-computer interface behavioural model |
| CN110974219A (en) * | 2019-12-20 | 2020-04-10 | 北京脑陆科技有限公司 | Human brain idea recognition system based on invasive BCI |
| CN112107307A (en) * | 2020-08-24 | 2020-12-22 | 中国科学院上海微系统与信息技术研究所 | Preparation method and structure of high-flux implanted flexible nerve electrode |
| CN113440143A (en) * | 2021-07-12 | 2021-09-28 | 北京华科恒生医疗科技有限公司 | Fluorescence sensing composite deep electrode |
Non-Patent Citations (2)
| Title |
|---|
| 光纤式在体荧光显微成像系统在动态观测活体动物脑内神经元中的应用;石莹;陈露岚;姜民;;生理学报;20121225(06);全文 * |
| 脑―机接口技术研究;王萍;杨静丽;万柏坤;;医疗卫生装备;20071115(11);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114356074A (en) | 2022-04-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Kesav et al. | Efficient and low complex architecture for detection and classification of Brain Tumor using RCNN with Two Channel CNN | |
| Singh et al. | Transforming sensor data to the image domain for deep learning—An application to footstep detection | |
| MICrONS Consortium et al. | Functional connectomics spanning multiple areas of mouse visual cortex | |
| Dolz et al. | 3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study | |
| Xia et al. | Aquatic toxic analysis by monitoring fish behavior using computer vision: A recent progress | |
| JP7034069B2 (en) | Automatic classification of animal behavior | |
| Giovannucci et al. | Onacid: Online analysis of calcium imaging data in real time | |
| JP2019516157A (en) | Automatic classification method of animal behavior | |
| US9569845B2 (en) | Method and system for characterizing cell motion | |
| Froudarakis et al. | Object manifold geometry across the mouse cortical visual hierarchy | |
| Naga Srinivasu et al. | A comparative review of optimisation techniques in segmentation of brain MR images | |
| Wang et al. | An automated behavior analysis system for freely moving rodents using depth image | |
| Zhang et al. | A swarm intelligence based searching strategy for articulated 3D human body tracking | |
| CN114356074B (en) | Animal brain-computer interface implementation method and system based on in-vivo fluorescence signals | |
| Kriegel et al. | Cell shape characterization and classification with discrete Fourier transforms and self‐organizing maps | |
| Soleimani et al. | Improving ant colony optimization for brain MRI image segmentation and brain tumor diagnosis | |
| Gupta et al. | Analysis and identification of dermatological diseases using Gaussian mixture modeling | |
| Yang | An efficient automatic gait anomaly detection method based on semisupervised clustering | |
| Chang et al. | Behavioral clusters revealed by end-to-end decoding from microendoscopic imaging | |
| An et al. | Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL | |
| Xu et al. | A new cyborg rat auto navigation system based on finite state machine | |
| Sun et al. | Interactive medical image segmentation via point-based interaction and sequential patch learning | |
| Sandhya et al. | Segmentation of magnetic resonance brain images using the advanced ant colony optimization technique | |
| Johnston et al. | Robust population single neuronal calcium signal extraction using SCOUT allows for longitudinal analysis of behavior-associated neural ensemble dynamics | |
| Lee et al. | Comparison of Deep Learning and Image Processing for Tracking the Cognitive Motion of a Laboratory Mouse |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |