KR102096565B1 - Analysis method of convolutional neural network based on Wavelet transform for identifying motor imagery brain waves - Google Patents

Abstract

Disclosed is a convolution neural network analysis method based on wavelet transform for motion imagination brain signal recognition. The convolution neural network analysis technique based on wavelet transform for motion imagination brain signal recognition uses a brain wave analysis program of a computer connected to an electroencephalogram (EEG) measuring device connected to multiple EEG detection sensors to be worn on right and left heads. Brain wave measurement is performed during left-hand motion imagination and right-hand motion imagination by a brain wave measurement, amplification, A/D conversion, and wavelet transform computer interface (brain wave analysis program). Among EEG data signals measured by band pass filters (BPF2, BPF3) of the EEG measuring device, only mu (8 to 13 Hz alpha wave) and beta (13 to 30 Hz beta wave) band frequency domains are extracted from the output spectrum along the frequency axis. The motion image EEG signal is wavelet-transformed to extract and classify the characteristics of a motion imagination brain signal based on a one-dimensional convolution neural network (CNN) for a time-frequency image.

Description

Analysis method of convolutional neural network based on Wavelet transform for identifying motor imagery brain waves}

The present invention relates to a method for analyzing a brain image of a motor imagery (MI), and more specifically, a brain wave in a left hand motion and a right hand motion imagination using an EEG measuring device connected to a plurality of EEG detection sensors worn on the head. Measurement-> Amplification-> A / D conversion-> Wavelet conversion- EEG data is measured by computer interface (EEG analysis program), and EEG data measured through band pass filter (BPF2, BPF3) of EEG measurement device Convolution for time-frequency images by performing wavelet transform on the image of the motion image EEG by extracting only the band frequency regions of mu (8 ~ 13Hz alpha wave) and beta (13-30Hz beta wave) from the output spectrum along the frequency axis among signals. Convolutional neural network analysis method based on wavelet transform for motion imagination brain signal recognition, which analyzes motion imagination brain signals based on solution neural networks (CNN) It is done.

Brainwave (Electroencephalography, EEG) is a microscopic flow of bioelectricity that occurs when a signal is transmitted between the nervous system and the brain, and measures the electrical potential difference that occurs on the brain surface using an electrode. , Delta (δ) wave, theta (θ) wave, alpha (α) wave, beta (β) wave, gamma (γ) wave.

EEG can be classified according to its frequency and amplitude, and the EEG output from the human brain outputs a frequency of 0 to 30 Hz and shows an amplitude of approximately 20 to 200 μV.

EEG is artificially delta-δ wave (0.2 to 3.99 Hz), theta -θ wave (4 to 7.99 Hz), alpha -α wave (8 to 12.99 Hz), beta -β wave (13 ~ 29.99 Hz), gamma-g wave (30 ~ 50 Hz)

The delta wave has a frequency of 0.2 to 4 Hz and an amplitude of 20 to 200 V, and appears mainly in the deep sleep state of normal people or in newborns. In addition, there is a study that deltapa state produces a large amount of growth hormone at bedtime. The story of being tall when adolescents are sleeping is supported by these scientific facts.

Theta waves have a frequency of 4 to 8 Hz and an amplitude of 20 to 100 V, and appear emotionally stable or before falling into sleep. Theta waves occur in the perception of human perception and the boundary between dreams. This state creates an image of the mind like a dream and leads to a vivid memory.

Alpha waves have a frequency of 8 to 13 Hz and an amplitude of 20 to 60 V, appear in a relaxed state like meditation, and help relieve stress and improve concentration. When we close our eyes and relax, the EEG activity slows down. At this time, the brain produces alpha waves, and the brain becomes an alpha state. People who are healthy and stress-free have a lot of alpha wave activity.

The beta wave has a frequency of 13 ~ 30Hz and an amplitude of 2 ~ 20V, and it appears during activity such as tension and excitation. It helps to improve athletic ability and is an EEG when the consciousness is awake. Beta waves appear mainly when a person opens his eyes, walks, and focuses on the outside world. Beta waves appear mainly in the left and right frontal lobes of the head, and appear in all conscious activities, such as when awake and speaking, especially in anxiety, tension, and complicated computation.

Gamma waves have a frequency of 30 to 50 Hz and an amplitude of 2 to 20 V, and appear mainly when excited. It is said that gamma waves vibrate faster than beta waves, and are emotionally more nervous or related to advanced cognitive information processing such as reasoning and judgment.

Delta waves, theta waves, alpha waves, beta waves, and gamma waves are frequency domains of brain waves classified conveniently for convenience. Researchers analyzing the characteristics of the EEG observe the power spectrum distribution showing the power distribution for each frequency component of 0-50 Hz as a whole, and then analyze the frequency component to make the power spectrum distribution slightly different for each measurement area on the head surface. Aspect.

The cerebral cortex below the head surface is largely divided into the frontal lobe, parietal lobe, temporal lobe, and occipital lobe. For example, the occipital lobe, which corresponds to the back of the head, has primary visual cortex to handle primary visual information processing, and the parietal lobe, located near the parietal lobe, has somatosensory cortex to handle exercise / sensory related information processing.

1A and 1B are views showing a conventional EEG measuring device and method.

EEG measuring device is a head multi-point measuring type or hair band type EEG measuring device, which measures the EEG of the left and right frontal lobe, and EEG measurement->amplification-> A / D conversion-> Fast Fourier Transform (FFT) -computer By checking the output power spectrum along the frequency axis by the interface (EEG analysis program), features are extracted and EEG is measured and analyzed. When recording the EEG, the EEG signals include delta waves (δ waves: 0 to 4 Hz), theta waves (θ waves: 4 to 8 Hz), alpha waves (α waves: 8 to 13 Hz), and beta waves (β waves: 13 to 30 Hz) ), Gamma wave (γ wave: 30 Hz or more). ^* Delta waves dominate when sleeping, and theta waves dominate when you feel tired and depressed. Alpha waves dominate when your mind and body are resting, while beta waves dominate when you are busy or anxious. Gamma waves occur in the frontal and parietal lobes during extreme excitement or awakening.

For example, an EEG signal detected using a USB cable, or any one of Bluetooth or Wi-Fi communication from a hairband type EEG measuring device used as a medical device, or a computer or smartphone equipped with an EEG analysis program / It transmits to the tablet PC and analyzes the EEG by extracting the characteristics of the EEG through EEG recording and spectrum analysis by the EEG analysis program of the terminal (computer or smartphone / tablet PC).

Spectrum analysis is an analysis method for obtaining an energy distribution for each wavelength of each EEG, and a Fast Fourier Transform (FFT) method and a Discrete Wavelet Transform (DWT) method are used.

FIG. 1c shows i) normal, ii) left ear cell phone, iii) right ear cell phone, measured using wavelet transform of EEG signal to investigate the effect of cell phone use on human brain This diagram shows the measurement results showing the wavelet coefficient of a beta wave.

2 is a configuration diagram of an EEG measurement system using a conventional hairband type EEG detection sensor.

The EEG measurement system using the hairband type EEG detection sensor includes a hairband attached to the subject's head or forehead area; A tightening device for tightening the hair band to a circumferential length on the head where it is worn; At least one EEG detection sensor that detects an EEG signal from electrodes attached to the forehead area of the subject's head; After amplifying, A / D conversion, and fast Fourier conversion (FFT) by the control unit, the EEG signal detected by at least one EEG detection sensor is detected through an RS232 serial port or a USB cable, or through a wired or wireless communication such as Bluetooth or Wi-Fi. EEG (Electroencephalogram) is output to the EEG analysis application by sending it to a computer, smartphone, or tablet PC, and upon activation of neurons (nerve cells) in the hippocampus through the enthohinal cortex. After measuring the generated bioelectrical signal, EEG is encoded and analyzed by the computer's EEG analysis program.

At least one brain wave detection sensor does not attach a metal electrode coated with a conductive gel to the measurement area of the head, and uses a “fiber electrode for brain wave detection”.

Referring to FIG. 1, the hairband type EEG measuring device includes a plurality of EEG detection electrodes (fiber electrodes) 31, an EEG signal detection unit 32, an EEG amplifier 33, a filter 34, and A / D. It is composed of a converter 35, a control unit 36, a button unit 37, an LED unit 38, a power supply unit 40, a storage unit 43, and a wireless communication unit 44.

The power supply unit 40 is composed of a power supply unit 41 that receives AC 220V power, and an AC / DC converter 42 that converts AC 220V power into a predetermined DC driving voltage (DC 5V).

The headband type EEG measuring device includes at least one EEG detection electrode 31 that attaches a headband EEG measuring device to the forehead of the subject's head before bedtime, measures EEG signals from the head surface at bedtime, and uses fiber electrodes; EEG signal detection unit 32 for detecting the EEG signal measured from the external wave detection electrode attached to each measurement position; An amplifier for amplifying the fine EEG signal measured from the EEG signal detection unit 32 and removing Gaussian noise (EEG Amplifier) 33; Delta waves (LPF, 4 Hz or less), theta waves (BPF1, 4 to 8 Hz), alpha waves (BPF2, 8 to 13 Hz) using LPF, BPF1, BPF2, BPF3, and HPF by classification of brain waves having a frequency in the range of 0 to 50 Hz ), Beta wave (BPF3, 13 ~ 30Hz), gamma wave (HPF, 30Hz or more) Equipped with a number of filters (LPF, BPF1, BPF2, BPF3, HPF) to filter by the frequency range of EEG, sleep by button A filter 34 that filters sleep EEG using a low pass filter (LPF) of 4 Hz or less when it is set to detect EEG (Sleep EEG); An A / D converter 35 for converting the filtered analog EEG signal to a digital EEG signal; It controls the frequency of the EEG to be filtered according to the user's button setting, receives the digital EEG signal from the A / D converter 35, and stores it in real time. A control unit 36 that is controlled to be transmitted; Equipped with power ON / OFF button, delta wave (LPF, 4Hz or less), theta wave (BPF1, 4 ~ 8Hz), alpha wave (BPF2, 8 ~ 13Hz), beta wave (BPF3, 13 ~ 30Hz), gamma wave A button unit 37 for controlling filter settings for each frequency range of the EEG (HPF, 30 Hz or more); An LED unit 38 displaying a color when power ON / OFF of the EEG measuring device or sleep EEG is detected; A storage unit 43 for storing EEG signals measured from a plurality of EEG detection electrodes (fiber electrodes) of the hairband type EEG measurement device; And a wireless communication unit 44 that transmits the measured EEG signal to the EEG analysis application program 70 of the computer terminal using Bluetooth or Wi-Fi wireless communication.

The measured electroencephalogram (EEG) detects the brain wave 0.2 ~ 4Hz Delta wave closely related to the brain's learning memory consolidation and the EOG (electroocculogram, safety latitude) followed immediately by two or more times to improve learning. It is used for sleep EEG by detecting the EEG and detecting the 1st or 2nd EOG immediately after completion of the learning-related EEG check.

The terminal (PC, smart phone, Tablet PC) is a wireless communication unit 45 that receives the EEG signal measured from the headband type EEG measurement device through an RS232 serial port or a USB cable, or Bluetooth or Wi-Fi short-range wireless communication; A microcomputer 46 that controls each function to detect EEG by performing a fast Fourier (FFT conversion) of the A / D-converted EEG signal after amplification of multiple channels detected at predetermined positions in the brain; EEG analysis application (SLM App) (70) and a memory for storing the measured EEG signal by time (48); It has an LCD display unit 47 for outputting the analyzed EEG signal, and the EEG analysis application program 70 includes an LCD display unit 47 for outputting a user's EEG signal.

1. Non-invasive brain-computer interface (BCI)

Recently, a brain-computer interface (BCI) is a system that directly connects an EEG based on an electroencephalogram (EGG) and an external device. Electroencephalogram (EEG) is widely used in non-invasive brain-computer interface (BCI) research because it is easier to use and less expensive than other EEG activities. Brain-computer interface (BCI) research is primarily aimed at enabling people with severe physical disabilities to control the body using only brain waves [1]. Electroencephalogram (EEG), which is used in brain-computer interface research, is affected by a variety of specific stimuli and analyzes the signals using responses according to the stimuli. Event-related potential (ERP), steady-state visual evoked potential (SSVEP), and motor imagery (MI) are the most commonly used for specific stimuli.

To this end, this study used public open data for motion imagination.

Patent Registration No. 10-0282733 (Registration Date November 30, 2000), "Real-time EEG Measurement Device Using Headband", Founded Patent registration number 10-0450758 (Registration date September 20, 2004), "EEG measurement device and method", Korea Electronics and Telecommunications Research Institute

 Shangkai Gao, Yijun Wang, Xiaorong Gao and Bo Hong, “and Auditory Brain-Computer Interfaces.” In IEEE Transaction on Biomedical Engineering, vol 61, no 5, May 2014  Yousef Rezaei Tabar, Ugur Halici, “novel deep learning approach for classification of EEG motor imagery signals” in Journal of Neural Engineering, 2017  Pfurtscheller G and Da Silva F L 1999 Event-related EEG / MEG synchronization and desynch-ronization: basic principles Clin. Neurophysiol  Torrence, C. and G.P. Compo. “Practical Guide to Wavelet Analysis” Bull. Am. Meteorol. Soc., 79, 6178, 1998.

An object of the present invention for solving the above problems is to measure brain waves in left-hand movement and right-hand movement imagination using an EEG measuring device connected to a plurality of EEG detection sensors worn on the head-> amplification-> A / D conversion- > Wavelet transform-EEG is measured by computer interface (EEG analysis program), and EEG data signal measured by band pass filter (BPF2, BPF3) of EEG measuring device is selected from the output spectrum along the frequency axis. Based on the convolutional neural network (CNN) for time-frequency images by performing wavelet transformation of the motion image EEG signal by extracting only the 8-13 Hz alpha wave) and beta (13-30 Hz beta wave) band frequency domains. Provided is a wavelet transform based convolutional neural network analysis system and method for analyzing motion imagination brain signals.

In order to achieve the object of the present invention, a wavelet transform based convolutional neural network analysis method for motion imagination brain signal recognition, (a) EEG measuring device connected to a plurality of EEG detection sensors provided at multiple points of the right and left heads Transmitting a measurement brainwave from the multi-points of the right and left heads to the computer from the communication unit, and using the brainwave analysis program of the computer in communication with the brainwave measurement device; (b) in the left hand movement and right hand movement imagination, EEG measurement-> amplification-> A / D conversion-> wavelet conversion-transmitting the EEG data measured by the EEG analysis program of the computer; (c) A wavelet transform of a motion image EEG signal by extracting only the mu band and the beta band frequency domain from the output spectrum of the frequency axis among the EEG data signals measured through the band pass filters (BPF2, BPF3) of the EEG measuring device. Performing to generate a time-frequency image; (d) A reduced time-frequency two-dimensional image that reduces the motion image EEG signals of the mu band and the beta band in the frequency domain and the time domain to mxn, respectively, using a cubic interpolation method by a computer EEG analysis program. Generating a; And (e) analyzing a motion imaginary brain signal detected through feature extraction and classification using a 1D convolutional neural network (CNN) with respect to the reduced time-frequency 2D image. Includes,

Data on left and right hand movement imagination, respectively, responses to the imagination appear in different cortex of the brain, and motion imagination brain signals detected using the one-dimensional convolutional neural network (CNN) on the brain waves extracted using these features Extract and classify features.

The mu band has a frequency range of 8 to 13 Hz alpha wave, and the beta band has a frequency range of 13-30 Hz beta wave.

The wavelet transform uses a Morlet wavelet transform.

The wavelet transform extracts electroencephalogram (EEG) data having a sampling frequency of 250 Hz for 0.5 seconds, and a total of 125 samples have been used.After undergoing the pre-processing process of the Morlet wavelet transform, the results of 500 and 331 samples respectively in the time axis and frequency axis And based on this, an image of 331 × 500 was generated,

Mu bands and beta bands are extracted from the output spectrum along the frequency axis, and the image size extracted from the mu band is 40 × 500, and the image size extracted from the beta band is 55 × 500.

과

의 크기를 가지는 상기 축소된 시간-주파수 이미지를 만들며, In step (c), since the size of the frequency domain and the time domain of the two bands, the mu band and the beta band, is relatively large, the mu band 16 × 500 in the frequency domain using the cubic interpolation method, Reduced the beta band size to 15 × 500, not only in the frequency domain, but also in the time domain from 125 samples to 32 samples.

and

To create the reduced time-frequency image having the size of,

The above process is applied to C3, Cz, and C4, respectively, to make the frequency size N _F x 3, and through this process, the final image size is 93 × 32.

The filter used in the one-dimensional convolutional neural network (CNN) uses a one-dimensional convolution filter to analyze frequency characteristics over time, and the size of the filter is N _F = 93 x 3.

The 1D convolutional neural network (CNN) is composed of a convolutional layer, a pooling layer, and a fully connected layer (FC) to recognize the reduced time-frequency 2D image. And

The output function of the convolutional layer (Convolutional Layer) was applied to the ReLU (rectified linear unit) function,

The pooling layer used max-pooling, and max-pooling is performed by the FC layer (Fully Connected Layer) that classifies the output into two classes of left- and right-handed operations.

The parameters of the one-dimensional convolutional neural network (CNN) were trained by a back-propagation algorithm, and the error E was calculated after calculating the data of the data with the specified label and the output result. -descent method) to make the error E small.

In the step (e), in the convolutional neural network analysis step based on wavelet transform of the motion imagination brain signal, the influence of ERS and ERD of the C4 channel is strong in the left hand imagination, and the effect of ERS and ERD of the C3 channel in the right hand imagination. It appears strongly.

In step (e), when analyzing a convolutional neural network based on wavelet transform for recognizing the motion imaginary brain signal, the EEG analysis program of the computer uses a Matlab program.

The present invention uses an EEG measuring device connected to a plurality of EEG detection sensors worn on the head to measure EEG during left-hand movement and right-hand movement imagination-> amplification-> A / D conversion-> wavelet transform-computer interface (EEG analysis EEG is measured by the program) and among the EEG data signals measured through the band pass filters (BPF2, BPF3) of the EEG measuring device, mu (8 ~ 13Hz alpha wave) and beta (from the output spectrum along the frequency axis) 13-30Hz Beta Wave) Wavelet transform based on a convolutional neural network (CNN) for time-frequency images by performing wavelet transformation on the motion image EEG signal by extracting only the band frequency domain based on wavelet transform for motion imagination brain signal recognition Through analysis of the convolutional neural network, brain signals of motion imagination were analyzed.

Motion imagination study means moving the part of the body without moving the subject. The data used in this study are data for left- and right-hand movement imagination, and reactions according to the imagination appear in different cortex of the brain. Using this feature, the classification accuracy was improved by applying a convolutional neural network (CNN) to the extracted brain waves.

1A and 1B are views showing a conventional EEG measuring device and method.
FIG. 1c shows i) normal, ii) left ear cell phone, iii) right ear cell phone, measured using wavelet transform of EEG signal to investigate the effect of cell phone use on human brain This diagram shows the measurement results showing the wavelet coefficient of a beta wave.
2 is a configuration diagram of an EEG measurement system using a hairband type EEG detection sensor.
3 is a diagram showing an input image form for left-hand imagination.
4 is a diagram showing a convolutional neural network (CNN) model based on wavelet transform for recognizing motion imaginary brain signals according to the present invention.
5 is a flowchart showing a method of analyzing a convolutional neural network based on wavelet transform for recognizing a motion imaginary brain signal of the present invention.
6 is a view showing the results of the accuracy evaluation and the mean and standard deviation of the data analyzed by Matlab.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in detail the configuration and operation of the invention.

The present invention is a motion image analysis (motor imagery, MI) wavelet transform-based convolutional neural network analysis method for brain signal recognition using a brain wave measuring device connected to a plurality of EEG detection sensors worn on the head to imagine left and right hand movements EEG measurement-> amplification-> A / D conversion-> Wavelet Transform- EEG is measured by computer interface (computer's EEG analysis program) and band pass filter (BPF2, BPF3) of EEG measuring device Among the EEG data signals measured through, only the frequency bands of the mu (8 ~ 13Hz alpha wave) and beta (13 ~ 30Hz beta wave) bands are extracted from the output spectrum along the frequency axis to perform wavelet transformation of the motion image EEG signal. Motion Imagination Brain Signal, performing analysis to analyze motion imagination brain signals based on Convolutional Neural Network (CNN) for time-frequency images Provided is a convolutional neural network analysis system and method based on wavelet transform for recognition.

The left hand motion is related to the right brain imagination brain signal, and the right hand motion is related to the left brain imagination brain signal.

As the EEG measuring device for detecting the EEG signal, an EEG measuring device equipped with a plurality of electrodes of a headset type used during cycling, or an EEG measuring device equipped with a plurality of electrodes of a non-invasive hairband type may be used.

EEG measuring device for detecting the EEG signal is provided at each point of the left and right head, a plurality of EEG detection sensors having an electrode of 32 to 128 channels for detecting the EEG signal; An EEG amplifier unit for amplifying each of the fine EEG signals measured from a plurality of EEG detection sensors having electrodes of the 32 to 128 channels; Delta waves (LPF, 4 Hz or less), theta waves (BPF1, 4 to 8 Hz), alpha waves (BPF2, 8 to 13 Hz) by operating LPF, BPF1, BPF2, BPF3, and HPF by classification of EEG with frequencies in the range of 0 to 50 Hz ), A plurality of filter units (LPF, BPF1, BPF2, BPF3, HPF) to filter by frequency range of the EEG of beta waves (BPF3, 13 ~ 30Hz), gamma waves (HPF, 30Hz or more); An A / D converter that converts the analog EEG signals filtered for each EEG channel into digital EEG signals; By selecting the button of the EEG measuring device, mu (8 ~ 13Hz) is performed by band pass filters (BPF2, BPF3) in the output spectrum along the frequency axis among the EEG data signals detected by the multiple EEG detection sensors of each measurement point of the head. Alpha wave) and beta (13-30Hz beta wave) includes a communication unit that extracts only the frequency domain of the band and transmits the frequency output values of the mu band and the beta band to a computer.

The communication unit of the EEG measuring device and the communication unit of the computer may be connected by a USB cable, or Bluetooth or Wi-Fi may be used.

The computer has an EEG analysis program, receives EEG data for each measurement electrode through wired / wireless communication, and is located on a frequency axis among EEG data signals measured through the band pass filters (BPF2, BPF3) of the EEG measurement device. A communication unit configured to extract wavelet transformed frequency output values by extracting only the mu band (8 to 13 Hz alpha wave) and beta band (13 to 30 Hz beta wave) frequency domains from the output spectrum; A control unit including an EEG signal processing unit for wavelet transforming the EEG data digitally to provide output values for each frequency; A storage unit that stores measurement EEG data and stores an EEG analysis program for extracting and classifying features of a motion imaginary brain signal based on a Convolutional Neural Network (CNN); And an LCD display unit that outputs EEG data having output values for each frequency over time.

Motion imagination study means moving the part of the body without moving the subject. The data used in this study are data for left- and right-hand movement imagination, and each imaginative response appears in different cortex of the brain. The goal was to improve classification accuracy by applying a one-dimensional convolutional neural network (CNN) to the extracted brain waves (alpha and beta waves) using these features [2].

Then, a 2D image in the form of a time axis and a frequency axis was created for the EEG signal over time, and a 1D convolutional neural network (CNN) was applied to the time-frequency 2D image.

Convolutional neural network (CNN) is a convolutional layer (Convolutional Layer), three layers including a pooling layer (Pooling Layer) and an FC layer (Fully Connected Layer) are used.

At this time, in the time-frequency image form, wavelet transform is applied, and in the output spectrum along the frequency axis among the EEG data signals measured using the bandpass filters (BPF2, BPF3) of the EEG measuring device rather than applying all frequency domains Only the frequency band of the mu band (8 ~ 13Hz alpha wave) and beta band (13 ~ 30Hz beta wave) were extracted to create a time-frequency two-dimensional image.

The data used in this study was analyzed using the BCI Competition Ⅳ dataset 2b, which is widely used in brain-computer interface (BCI) studies worldwide.

Referring to FIG. 5, a method of analyzing a convolutional neural network based on wavelet transform for recognizing a motion imaginary brain signal according to the present invention includes (a) an EEG measuring device connected to a plurality of EEG detection sensors provided at multiple points of the right and left heads. Transmitting a measurement brainwave at a multi-point of the right and left heads to a computer through a communication unit, and using the brainwave analysis program of the computer in communication with the brainwave measurement device; (b) in the left hand movement and right hand movement imagination, EEG measurement-> amplification-> A / D conversion-> wavelet conversion-transmitting the EEG data measured by the EEG analysis program of the computer; (c) Mu bands (8 to 13 Hz alpha waves) and beta bands (13 to 30 Hz beta waves) from the output spectrum of the frequency axis among the EEG data signals measured through the band pass filters (BPF2 and BPF3) of the EEG measuring device. ) Extracting only the frequency domain to perform wavelet transform of the motion image EEG signal to generate a time-frequency image; (d) Reduced time-frequency two-dimensional reduction of the motion image EEG signal of the mu band and the beta band in the frequency domain and the time domain to mxn size, respectively, using the cubic interpolation method by the EEG analysis program of the computer. Generating an image; And (e) analyzing a motion imaginary brain signal detected through feature extraction and classification using a 1D convolutional neural network (CNN) with respect to the reduced time-frequency 2D image. Includes,

The left and right hand movement imagination data shows the response of each imagination in different cortex of the brain, and using these features, the convolutional neural network (CNN) is used to extract the characteristics of the movement imagination brain signal. Classify.

The mu band has a frequency range of 8 to 13 Hz alpha wave, and the beta band has a frequency range of 13-30 Hz beta wave.

The wavelet transform uses a Morlet wavelet transform.

For reference, Wavelet Transform is a technique designed to overcome the limitation of the discontinuity of the signal of the Fourier Transform, which was used to analyze the frequency characteristics in the conventional one-dimensional signal on the time axis, scale and translation of the mother function. It is an improved technique to grasp the time-frequency spatial characteristics in the Fourier transform, where only the characteristics of the frequency domain are known by adjusting (position). Wavelet Transform is based on the shape of the parent function Haar, Morlet. It can be divided into bumps, etc.

The Morlet Wavelet Transform expression is defined as follows. Ψ means the parent function. t means translation of the parent function, time axis shift, and s means scale.

The wavelet transform extracts electroencephalogram (EEG) data having a sampling frequency of 250 Hz for 0.5 seconds, and a total of 125 samples have been used.After undergoing the pre-processing process of the Morlet wavelet transform, the results of 500 and 331 samples respectively in the time axis and frequency axis And based on this, an image of 331 × 500 was generated,

In the output spectrum along the frequency axis, the mu band (8 to 13 Hz alpha wave) and beta band (13 to 30 Hz beta wave) were extracted. The image size extracted from the mu band was 40 × 500, and the image size extracted from the beta band. It is characterized in that 55 × 500.

과

의 크기를 갖는 상기 축소된 시간-주파수 이미지를 만들며, In step (d), since the size of the frequency domain and the time domain of the two bands, the mu band and the beta band, is too large, the mu band 16 × 500, beta in the frequency domain using the cubic interpolation method The band was reduced to 15 × 500, and not only in the frequency domain, but also in the time domain from 125 samples to 32 samples.

and

To create the reduced time-frequency image having a size of

The above process is applied to C3, Cz, and C4, respectively, to make the frequency size N _F x 3, and through this process, the final image size is 93 × 32.

The filter used in the one-dimensional convolutional neural network (CNN) uses a one-dimensional convolution filter that analyzes frequency characteristics over time, and the size of the filter is N _F = 93 x 3.

The 1D convolutional neural network (CNN) is composed of a convolutional layer, a pooling layer, and a fully connected layer to recognize the reduced time-frequency 2D image,

The output function of the convolutional layer (Convolutional Layer) was applied to the ReLU (rectified linear unit) function,

The pooling layer used max-pooling, and max-pooling is performed by the FC layer (Fully Connected Layer) that classifies the output into two classes: left hand and right hand.

The parameters of the one-dimensional convolutional neural network (CNN) were trained by a back-propagation algorithm, and the error E was calculated after calculating the data of the data with the specified label and the output result. -descent method) to make the error E small.

In the step (e), in the convolutional neural network analysis step based on wavelet transform of the motion imagination brain signal, the influence of ERS and ERD of the C4 channel is strong in the left-hand imagination, and the effect of ERS and ERD of the C3 channel in the right-hand imagination. It appeared strong.

In step (e), when analyzing a convolutional neural network based on wavelet transform for recognizing the motion imaginary brain signal, the EEG analysis program of the computer used a Matlab program.

2. Wavelet Transform Based Convolutional Neural Network Analysis Method for Motion Imagination Brain Signal Recognition

2.1 Image Format

In a Brain-Computer Interface (BCI) study, in an embodiment, EEG detection-amplification-A / D conversion by an EEG measurement device from an EEG measurement detection sensor attached to a multi-point location of the head in the EEG measurement device- A total of six EEG channels were used for detecting EEG through wavelet transformation, three EEG (Electroencephalography) data and three EOG (electrooculogram) data, where we have EEG data C3, Cz. , Only C4 was used. The previous three channels are the most prominent features of motion imagination in EEG data [3].

In a specific frequency mu band (8 to 13 Hz, alpha wave) from the extracted EEG, when the motion is imagined, a form in which the EEG energy decreases appears, which is called event related desynchronization (ERD).

In the beta band (13 ~ 30Hz, beta wave), the form of increasing energy appears, and this is called event related synchronization (ERS). In addition, ERS and ERD have different channels that respond to each in the left-hand imagination and the right-hand imagination in motion imagination.

In the left-hand imagination, the effects of ERS and ERD on the C4 channel are strongly influenced.

Wavelet transform was used in this study to make EEG data signals into two-dimensional images in the time axis and frequency axis. For reference, wavelet transform is a transform method used to express time and frequency components for signals whose frequency components change in time, such as chirped signals, ECGs (Electrocardiographs), and video signals. In this embodiment, the measured EEG signal is referred to as a wavelet, and any one small wavelet that is locally present in the head is patterned to shift or randomize it through a scale of enlargement and reduction. It is expressed as a waveform.

3 is a diagram showing an input image form for left-hand imagination.

EEG data having a sampling frequency of 250 Hz were extracted for 0.5 second, and a total of 125 samples were used. The wavelet transform was a Morlet wavelet transform [4]. After the previous pre-processing, 500 samples and 331 samples were obtained for the time axis and the frequency axis, respectively, and based on this, an image shape of 331 × 500 was produced. Then, mu bands and beta bands were extracted from the output spectrum along the frequency axis of the EEG signal. The image size extracted from the mu band is 40 × 500, and the image size extracted from the beta band is 55 × 500.

과

의 크기를 가지는 이미지를 만들 수 있었다.Since the size of the mu band and the beta band is relatively large, the size of the mu band is reduced to 16 x 500 and beta band 15 x 500 using the cubic interpolation method. Since the size of the time domain as well as the frequency domain is too large, the same method was used to reduce the size from 125 samples to 32 samples. And,

and

I was able to create an image with the size of.

The above procedure is applied to C3, Cz, and C4, respectively, to make the frequency size N _F x 3. After all of these steps, the final image will be 93 × 32 in size.

For reference, interpolation is a method for estimating a value between two end points. Here, the cubic interpolation method means a method of estimating a value between the two end points through a cubic polynomial that is not a straight line but a curved line.

In FIG. 3, when analyzing the image pattern according to the right hand movement imagination, the influence of ERD and ERS in the right hand movement imagination must be strongly shown in the C3 channel. However, in the C3 channel, it was confirmed that the characteristics of the ERD were stronger than other channels, but the effect of ERS could not be confirmed. This image was analyzed by a convolutional neural network (CNN) in the next step.

2.2 Convolutional Neural Network (CNN)

For reference, the existing multi-layer neural network (MLNN) is composed of an input layer, a hidden layer, and an output layer, and recognizes 16 x 16 characters (handwriting recognition) It is mainly used for.

Unlike this, the convolutional neural network (CNN) is mainly used to recognize a video image, and is composed of a convolutional layer, a pooling layer, and a fully connected layer. The convolutional neural network (CNN) repeats convolution (convolution) and sub-sampling (pooling, and pooling means sampling and resizing) to reduce the amount of data and feature extraction by convolution. Classification is performed by a neural network (NN), and finally, it is used for feature extraction, object classification, and object detection technology in the image in the Fully Connected Layer. For example, the FC layer (Fully Connected Layer) classifies whether a feature of an object extracted from an input image including a vehicle corresponds to a vehicle, a truck, or a bus. CNN has a structure suitable for 2D data learning.

In image processing, the convolution layer calculates the input image by moving the weighted filter by one pixel of the matrix after covering the input image with a weighted 2D filter [window, (eg, 3x3 window)]. After multiplying the weight of the value and the filter (window), the sum is set as the pixel value of the output image, and the function to reduce the image is provided.

In the pooling layer, subsampling (also called pooling, pooling means sampling and resizing) is a process of reducing the size of the screen, and max pooling, a method of selecting the maximum value of the relevant area (eg, 3x3 area), was used. For reference, in the convolutional layer, the size of the output data is maintained as it is in the convolutional layer, and the size is adjusted only in the pooling layer.

There are Max-Pooling and Average pooling for pooling, Max-Pooling is a method for finding the maximum value in the area, and Average-Pooling is a method for calculating the average value for the area. In image recognition, Max-Pooling is mainly used. In this way, a feature map including robust features (distortion, deformation) is generated by extracting multiple features of the image. Finally, the FC layer (Fully Connected Layer) classifies objects in the image.

When recognizing a video image, the convolution layer uses a 2D convolution filter, and the pooling layer uses subsampling (pooling) to reduce the size of the image.

However, in this study, a 1-dimensional convolution filter was applied because the frequency characteristics of the EEG were analyzed over time.

4 is a diagram showing a convolutional neural network (CNN) model based on wavelet transform for recognizing motion imaginary brain signals according to the present invention.

The 1D convolutional neural network (CNN) of the present invention is composed of a convolutional layer, a pooling layer, and a fully connected layer to recognize a time-frequency 2D image. ,

The output function of the convolutional layer (Convolutional Layer) was applied to the ReLU (rectified linear unit) function,

The pooling layer used max-pooling, and the max-pooling is performed by the FC layer (Fully Connected Layer) that classifies the output into two classes of left- and right-handed operations.

After all, Convolutional Neural Network (CNN) applies Convolution, ReLU, POOL, and Dropu-out alternately, and the object is finally classified by the FC network (Fully connected network).

The filter uses a one-dimensional convolution filter that analyzes frequency characteristics over time. Filter size of N _F = 93 x 3 was applied.

The overall model of the one-dimensional convolutional neural network (CNN) can be seen in FIG. 4. For the output function of the convolutional layer, the ReLU (rectified linear unit) function was applied. ReLU is defined as follows.

ReLU is an abbreviation of Rectified Linear Unit, and it is a function that is frequently used as an activation function. The ReLU function is used a lot because it has the advantage that the previously used sigmoid function can solve the gradient vanishing problem where the gradient converges to 0 when performing the back propagation algorithm. In the form of the ReLU function, if the input is less than 0, the output is 0. If the input is greater than 0, the output is output as it is.

식(α는 학습계수)을 사용하여 함수의 기울기로 최소값을 찾아내는 1차 근삿값 발견용 최적화 알고리즘이다. 기본 아이디어는 함수의 접선의 기울기(경사)를 구하여 프로그램에 의해 기울기가 낮은 쪽으로 계속 이동시켜 극값에 이를 때까지 반복시키는 것이다 The output of the convolutional layer reduces the size of the image by performing subsampling by the Max-Pooling Layer. Max-Pooling is performed in the FC layer (Fully Connected Layer), which classifies the output into two classes: left hand and right hand. The parameters of the convolutional neural network (CNN) were trained by a back-propagation algorithm. The error E should make the error E small using the gradient-descent method after calculating the data with the specified label and the output result. For reference, the gradient descent method is

This is an optimization algorithm for first-order approximation to find the minimum value by using the equation (α is the learning coefficient) as the slope of the function. The basic idea is to find the slope (slope) of the tangent of the function and continue to move it to the lower slope by the program and repeat it until it reaches the extreme value.

은 학습률(learning rate)을 의미한다.

는 필터 k에 대한 가중치의 행렬이며, b_K는 필터에 대한 bias 값을 의미한다. here,

Means learning rate.

Is a matrix of weights for filter k, and b _K means a bias value for filter.

Back-propagation algorithm is an algorithm that is widely used in artificial neural networks. When the result (y) and the desired result (o) are different through learning data, the error (e) is output. The error (e) is reduced by updating the weight of the difference between the result (y) and the desired result (o). At this time, it is called a back propagation algorithm because the direction of training and the direction of updating the weight of the error are opposite directions. Back propagation algorithms are used in multi-layer perceptrons by compensating for the shortcomings of perceptrons in the past.

The gradient-descent method is a method of finding the minimum value of a cost function from arbitrary data, and is currently used in studies such as neural networks and machine learning (ML). The gradient descent method is a method of measuring a gradient value in arbitrary data to find a portion having a lower gradient, that is, a portion having zero.

3. Results

In this study, a convolutional neural network analysis method based on wavelet transform for motion imagination brain signal recognition was implemented in the Matlab program environment. 90% of the data used was used as training data, and the remaining 10% was used as test data. Here, accuracy was evaluated by applying 10 × 10 fold cross validation. The batch size was set to 50 and the epoch was set to 300 and analyzed.

6 is a view showing the results of the accuracy evaluation and the mean and standard deviation of the data analyzed by Matlab.

Motion imagination study means moving the part of the body without moving the subject. For example, the motion of the left hand is related to the brain signal of the right brain, and the motion of the right hand is associated with the brain signal of the left brain.

The left and right hand movement imagination data shows the response of each imagination in different cortex of the brain, and using these features, the convolutional neural network (CNN) is used to extract the characteristics of the movement imagination brain signal. Classified.

The data used in this study are data for left- and right-hand movement imagination, and each imaginative response appears in different cortex of the brain. Using this feature, classification accuracy was improved by applying a one-dimensional convolutional neural network (CNN) to the extracted EEG (alpha wave, beta wave).

In the convolutional neural network analysis step based on wavelet transform of the motion imagination brain signal, the influence of ERS and ERD of the C4 channel is strongly shown in the left-hand imagination, and the influence of ERS and ERD of the C3 channel is strongly shown in the right-hand imagination.

As described above, the present invention has been described with reference to preferred embodiments of the present invention, but the present invention is within the scope of those skilled in the art without departing from the technical spirit and scope of the present invention as set forth in the claims below. It will be understood that various modifications or variations can be carried out.

: 학습률(learning rate)

: 필터 k에 대한 가중치의 행렬
b_K : 필터에 대한 bias 값 E: 오차CNN: Convolutional Neural Network

: Learning rate

: Matrix of weights for filter k
b _K : bias value for filter E: error

Claims (10)

(a) transmits a measurement brainwave at multiple points of the right and left heads to a computer through a communication unit from a brainwave measurement device connected to a plurality of brainwave detection sensors provided at multiple points of the right and left heads, and communicates with the brainwave measurement device Using the computer's EEG analysis program;
(b) in the left hand movement and right hand movement imagination, EEG measurement->amplification-> A / D conversion-> wavelet conversion-transmitting the EEG data measured by the EEG analysis program of the computer;
(c) Among the EEG data signals measured through the band pass filters (BPF2, BPF3) of the EEG measuring device, only the mu-band and beta-band frequency domains are extracted from the output spectrum of the frequency axis to convert wavelet transforms of motion image EEG signals. Performing to generate a time-frequency image;
(d) Reduced time-frequency two-dimensional reduction of the motion image EEG signal of the mu band and the beta band in the frequency domain and the time domain to mxn size, respectively, using the cubic interpolation method by the EEG analysis program of the computer. Generating an image; And
(e) analyzing a motion imaginary brain signal detected through feature extraction and classification using a 1D convolutional neural network (CNN) on the reduced time-frequency 2D image based on wavelet transform; And
The left and right hand movement imagination data shows the response according to the imagination in different cortex of the brain, and the motion imagination brain signal detected using the one-dimensional convolutional neural network (CNN) on the brain waves extracted using these features A convolutional neural network analysis method based on wavelet transform for recognizing motion imaginary brain signals, extracting and classifying features. According to claim 1,
The mu band has a frequency range of 8 to 13 Hz alpha wave, and the beta band has a frequency range of 13 to 30 Hz beta wave, a wavelet transform based convolutional neural network analysis method for motion imagination brain signal recognition. According to claim 1,
The wavelet transform is a convolutional neural network analysis method based on wavelet transform for recognizing motion imaginary brain signals using Morlet wavelet transform. According to claim 3,
The wavelet transform extracts electroencephalogram (EEG) data having a sampling frequency of 250 Hz for 0.5 seconds, and a total of 125 samples are used, and after undergoing the pre-processing process of the Morlet wavelet transform, the results of 500 and 331 samples of the time axis and the frequency axis, respectively. And based on this, an image of 331 × 500 was generated,
Mu band and beta band are extracted from the output spectrum along the frequency axis, and the image size extracted from the mu band is 40 × 500, and the image size extracted from the beta band is 55 × 500. Convolutional Neural Network Analysis Method based on Wavelet Transform. According to claim 4,
In step (d) above
Since the size of the frequency domain and the time domain of the two bands, the mu band and the beta band, is relatively large, the cu band interpolation method is used to generate a mu band of 16 × 500 and a beta band of 15 × 500 in the frequency domain. And reduced from 125 samples to 32 samples in the time domain, not just in the frequency domain.

and

To create the reduced time-frequency image having the size of,
The above process is applied to C3, Cz, and C4 to make the frequency size N _F x 3, and through this process, the final image size is 93 × 32, based on wavelet transform for motion imagination brain signal recognition. Neural network analysis method. According to claim 1,
The filter used in the one-dimensional convolutional neural network (CNN) uses a one-dimensional convolution filter to analyze frequency characteristics over time, and the size of the filter uses N _F = 93 x 3, a motion imaginary brain signal Convolutional neural network analysis method based on wavelet transform for recognition. According to claim 1,
The 1D convolutional neural network (CNN) is composed of a convolutional layer, a pooling layer, and a fully connected layer to recognize the reduced time-frequency 2D image,
The output function of the convolutional layer (Convolutional Layer) was used the ReLU (rectified linear unit) function,
The pooling layer uses max-pooling, and max-pooling is performed by the FC layer (Fully Connected Layer) that classifies the output into two classes of left- and right-handed motions, and based on wavelet transform for motion imagination brain signal recognition. Convolutional neural network analysis method. The method of claim 7,
The parameters of the one-dimensional convolutional neural network (CNN) were trained by a back-propagation algorithm, and the error E was calculated after calculating the data of the data with the specified label and the output result. -descent method) to make the error E small, wavelet transform based convolutional neural network analysis method for motion imagination brain signal recognition. According to claim 1,
In step (e),
In the convolutional neural network analysis step based on wavelet transform of the motion imagination brain signal, the left hand imagination has a strong influence of ERS and ERD of the C4 channel, and the right hand imagination shows a strong influence of the ERS and ERD of the C3 channel. Convolutional neural network analysis method based on wavelet transform for signal recognition. According to claim 1,
In step (e),
When analyzing a convolutional neural network based on wavelet transform for recognizing the motion imaginary brain signal, the computer's EEG analysis program uses a Matlab program, a wavelet transform based convolutional neural network analysis method for recognizing the motion imaginary brain signal.

Images