KR102292678B1 - System for classificating mental workload using eeg and method thereof - Google Patents

Abstract

The present technology discloses a brain cognitive load classification system and method. According to a specific example of the present technology, each multi-level feature of each layer is derived through a predetermined number of convolutions on input data of a three-dimensional image including spectral and spatial information of an input EEG signal, and the derived multi-level Each weight of each multi-level feature is derived based on the parameters optimized through features and learning, and a log of each multi-level feature is derived by multiplying the derived weight and multi-level feature, and the log of each derived multi-level feature is derived. The angular loss of each multi-level feature is calculated by applying the classification loss function to Accordingly, the classification accuracy of the cognitive load of the brain can be improved, and the learning speed can be increased by determining the optimized weight by learning the weight based on the EEG signal, thereby improving the performance of the system.

Description

SYSTEM FOR CLASSIFICATING MENTAL WORKLOAD USING EEG AND METHOD THEREOF

The present invention relates to a brain cognitive load classification system and method, and more particularly, to classify brain cognitive load through a three-dimensional convolutional neural network (CNN) algorithm based on spectral and spatial information of EEG signals measured for a given task. It's about the technology that made it possible.

Most CNN algorithms perform classification using only deep features extracted from the last convolutional layer. Deep features are essential information for accurate classification, and high-level structural information is included, and low-level features extracted from intermediate convolutional layers also contain rich local structural information.

Because local structure information improves deep learning performance, the multi-level feature fusion method is applied to various applications.

In addition, we propose a multi-step functional fusion method using this local structure information for various applications. For example, multi-level features are collected for the music auto-tagging problem, and the collected multi-level features are utilized for different time scale features.

The human brain functionally has two properties: separation and integration, where separation is a local cognitive load that processes specialized information locally, and integration is a global cognitive load that performs global integration of information across the entire brain.

In order to accurately represent the cognitive load of the brain, it is necessary to classify the low-level characteristic of the EEG signal representing the local cognitive load and the high-level characteristic of the EEG signal representing the global cognitive load.

However, there has been no CNN-based algorithm that accurately classifies cognitive load from multi-level features derived from EEG signals in the past.

delete

Korean Patent Publication No. 2020-0129505 (Neuroscience-based learning and memory method and system and recording medium)

Therefore, an object of the present invention is to provide a brain cognitive load classification system and method that can increase the learning speed based on the EEG signal based on the CNN-based multi-level feature fusion technique and improve the classification accuracy of the brain cognitive load. do it with

Brain cognitive load classification system according to an aspect of an embodiment for achieving the above object

a data collection unit for acquiring at least one EEG signal for at least one given task;

a pre-processing unit for outputting the EEG signal of the data collection unit as input data of a three-dimensional image including spectral and spatial information;

a learning unit that learns the input data of the preprocessor and outputs an optimized rescaling parameter, a weight, a bias parameter, a hidden layer parameter, and a classifier; and

The optimized rescaling parameter, the weight, and the CNN performing unit for classifying the brain cognitive load into one of a plurality of classifiers by performing a three-dimensional convolutional neural network (CNN) algorithm based on the input data of the classifier and the preprocessor. .

Preferably, the classification of the brain cognitive load may be output as one level.

Preferably, the pre-processing unit,

a filter module that removes noise components from the acquired EEG signal in the form of vibration in a predetermined frequency range through filtering and restores it to a single source signal through source localization;

an estimation module for outputting a density of spectral power in a predetermined frequency range using a fast Fourier transform and wavelet on the EEG signal of the single source of the time series;

A mapping module for obtaining spatial information of an EEG signal by converting a three-dimensional electrode into a two-dimensional image by applying an azimuthal equidistant projection by definition that an electrode is located at a proportionally corrected distance from the central point; and

Outputs input data of a three-dimensional image including spectrum and spatial information by interpolating the spectral power of the empty space between the electrode and the electrode using cubic spline interpolation for the spectral power of the corresponding electrode in each space It may include an interpolation module that

Preferably, the CNN performing unit,

A predetermined number of convolutional layers, polling layers, and fully connected layers,

A certain number of convolutional layers is

It may be provided to derive a layer including a predetermined number of multi-level features by activation based on an exponential linear unit for input data of a three-dimensional image,

The polling layer is

It may be provided to apply average polling and maximum polling to some layers representing medium and low-level features of a predetermined number of layers to output the average and maximum polling features for each channel,

The fully connected layer is

Each weight representing the importance of each multi-level feature is derived using the activation function for the log of the multi-level feature derived from the product of each multi-level feature and each classifier, the hidden layer parameter, and the bias parameter.

Estimate the brain cognitive load by the sum of the product of each of the derived weights and the predetermined activation function value and the rescaling parameter for the logit of the multi-level feature, and classify the estimated brain cognitive load into one of a plurality of predetermined classifiers can be

Preferably, the learning unit

Derive the angular loss of the multi-level feature from the classification loss function for the log of each multi-level feature,

It may be provided to derive each weight of each multi-level feature optimized as a minimized solution of the derived angle loss and deliver it to the 3D CNN execution unit.

Preferably, the solution to minimize the angular loss is

It may be provided to derive the gradient technique, which is a partial differential equation of the feature vector for the angular loss of each multi-level feature.

Brain cognitive load classification method according to another aspect of an embodiment,

a data collection step of acquiring at least one EEG signal for at least one given task;

a pre-processing step of outputting the EEG signal of the data collection unit as input data of a two-dimensional image including spectral and spatial information;

a learning step of learning the input data of the preprocessor and outputting an optimized rescaling parameter, a weight, a bias parameter, a hidden layer parameter, and a classifier; and

A CNN performing step of classifying the brain cognitive load into one of a plurality of classifiers by performing a three-dimensional convolutional neural network (CNN) algorithm based on the optimized rescaling parameters, weights, and spectral and spatial information of the classifier and the preprocessor can do.

Preferably, the classification of the brain cognitive load may be output as one level.

Preferably, the pretreatment step is

removing a noise component through filtering from the obtained EEG signal in the form of vibration in a predetermined frequency range and restoring a signal of a single source through source localization;

outputting a density of spectral power in a predetermined frequency range using a fast Fourier transform and wavelet of the EEG signal of the single source of the time series;

obtaining spatial information of an EEG signal by converting a three-dimensional electrode into a two-dimensional image by applying an azimuthal equidistant projection according to the definition that an electrode is located at a proportionally corrected distance from the central point; and

Outputs input data of a three-dimensional image including spectrum and spatial information by interpolating the spectral power of the empty space between the electrode and the electrode using cubic spline interpolation for the spectral power of the corresponding electrode in each space may include the step of

Preferably, the CNN performing step is

A predetermined number of convolutional layers, polling layers, and fully connected layers,

A certain number of convolutional layers is

It may be provided to derive a layer including a predetermined number of multi-level features by activating it based on an exponential linear unit with respect to the input data of the three-dimensional image,

The polling layer is

It may be provided to apply average polling and maximum polling to some layers exhibiting low-level features among a predetermined number of layers and output the average and maximum polling features for each channel,

The fully connected layer is

Each weight representing the importance of each multi-level feature is derived using the activation function for the log of the multi-level feature derived from the product of each multi-level feature and each classifier, the hidden layer parameter, and the bias parameter.

Estimate the brain cognitive load by the sum of the product of each of the derived weights and the predetermined activation function value and the rescaling parameter for the logit of the multi-level feature, and classify the estimated brain cognitive load into one of a plurality of predetermined classifiers can be

Preferably, the learning step

Derive the angular loss of the multi-level feature from the classification loss function for the log of each multi-level feature,

It may be provided to derive each weight of each multi-level feature optimized as a solution for minimizing the derived angle loss and deliver it to the 3D CNN execution step.

Preferably, the solution to minimize the angular loss is

It may be provided to derive the gradient technique, which is a partial differential equation of the feature vector for the angular loss of each multi-level feature.

According to an embodiment, each multi-level feature of each layer is derived through a predetermined number of convolutions on input data of a three-dimensional image including spectral and spatial information of the input EEG signal, and the derived multi-level feature and Based on the parameters optimized through learning, each weight of each multi-level feature is derived, and the log of each multi-level feature is derived by multiplying the derived weight and multi-level feature, and classified into the derived log of each multi-level feature The angular loss of each multi-level feature is calculated by applying the loss function, and each multi-level feature with the calculated angular loss is fully combined to classify the brain cognitive load for the input EEG signal into one of a number of classifiers. The classification accuracy of the cognitive load of the brain can be improved, and the learning speed can be increased by determining the optimized weight by learning the weight based on the EEG signal, thereby improving the performance of the system.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and serve to further understand the technical spirit of the present invention together with the detailed description of the present invention to be described later, so the present invention is a matter described in such drawings should not be construed as being limited only to
1 is an overall configuration diagram of a brain cognitive load classification system according to an embodiment.
2 is an exemplary diagram showing a given operation of one embodiment.
3 is an exemplary diagram illustrating an electrode for acquiring an EEG signal according to an embodiment.
4 is a detailed configuration diagram of a preprocessor according to an embodiment.
5 is an exemplary diagram illustrating a spectral density of an EEG signal according to an embodiment.
6 is an exemplary diagram showing the structure of a CNN according to an embodiment.
7 is an exemplary diagram illustrating a weight derivation process according to an embodiment.
8 is a graph showing a CDF of a weight for each image according to an embodiment.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the drawings.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

When a part "includes" a certain element throughout the specification, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. Also, as used herein, the term “unit” refers to a hardware component such as software, FPGA, or ASIC, and “unit” performs certain roles. However, "part" is not meant to be limited to software or hardware. A “unit” may be configured to reside on an addressable storage medium and may be configured to refresh one or more processors.

Thus, by way of example, “part” includes components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functionality provided within components and “parts” may be combined into a smaller number of components and “parts” or further divided into additional components and “parts”.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art can easily implement them. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description will be omitted.

The brain cognitive load classification system to which an embodiment is applied may include arbitrary dogs in any suitable configuration for each component. In general, computing and communication systems are presented in a wide variety of configurations, and the drawings do not limit the scope of the present disclosure to any particular configuration. While the drawings illustrate one operating environment in which the various features disclosed in this patent document may be used, such features may be used in any other suitable system.

Hereinafter, an embodiment will be described in detail with reference to the accompanying drawings.

In an embodiment, the cognitive load of the brain for a task includes a separate cognitive load that performs a local operation of the brain and a global cognitive load that performs an integrated operation of the brain, and each separate cognitive load is represented by a low-level characteristic of an EEG signal, Global cognitive load is expressed from high-level features of EEG signals.

Accordingly, in one embodiment, each multi-level feature of each layer is derived through a predetermined number of convolutions on input data of a two-dimensional image including spectral and spatial information of the input EEG signal, and the derived multi-level features and learning are performed. Each weight of each multi-level feature is derived based on the parameters optimized through The angular loss of each multi-level feature is calculated by applying a function, and each multi-level feature with the calculated angular loss is completely combined to classify the brain cognitive load for the input EEG signal as one of a number of classifiers. Accordingly, the brain cognitive load is subdivided and accurately classified.

1 is a diagram for estimating a brain cognitive load using an EEG signal according to an embodiment. Referring to FIG. 1 , the cognitive load classification system of the brain according to an embodiment is a convolutional neural network (CNN), which is a type of deep learning. It may be provided to classify the brain cognitive load into one of a plurality of classifiers based on a multi-level feature fusion technique using It may include at least one of the CNN performing unit 140 .

The data collection unit 110 may include collecting EEG signals measured according to the performance of a task determined by Sternberg.

2 is an exemplary view showing a task determined by Sternberg, and FIG. 3 is an exemplary view showing an electrode installed on the scalp to detect an EEG signal for each stimulus of FIG. 2, and FIGS. 2 and 3 Referring to, the three stimuli T3, A1, and G7 are randomly sequentially performed at a predetermined (for example, 0.5 second) cycle, and at this time, an electroencephalogram (EEG) for each stimulus is installed on the participant's scalp by a predetermined number (eg, an electroencephalogram). 30) electrodes. Until such an EEG signal is obtained, the above-described predetermined number of stimuli are repeatedly performed sequentially.

After that, the last given stimulus G7 completely disappears, and then, in order to maintain the EEG signal for a given stimulus for a certain period of time (2 seconds), a predetermined number of each stimulus is maintained for a certain period of time.

EEG signals for these stimuli are acquired at a sampling rate of a predetermined frequency (500 Hz) period. When the magnitude of the extracted EEG signal exceeds a predetermined range, the EEG signal exceeding the predetermined range is removed, and the EEG signal within the predetermined range is transmitted to the preprocessor 120 .

The preprocessor 120 converts the EEG signal measured from each input electrode into a 3D image including spectral and spatial information.

4 is a view showing a detailed configuration of the pre-processing unit 120, the pre-processing unit 120 may be performed based on MATLAB. Accordingly, the preprocessor 120 may include at least one of a filter module 121 , an estimation module 123 , a mapping module 125 , and an interpolation module 127 with reference to FIG. 4 .

The filter module 121 band-pass filters the obtained EEG signal in a predetermined frequency range (4Hz to 30Hz) for a predetermined frequency band (0.5Hz to 40Hz), then removes a DC component and a noise component, and sets the filtered EEG signal to a predetermined value. Downsampling is performed to a frequency (100 Hz).

The EEG signal obtained here is a signal in which brain activity of the cerebral cortex is reflected, and the cerebral cortex includes a frontal lobe, a temporal lobe, a parietal lobe, and an occipital lobe. Accordingly, the EEG signal can be expressed as spectral power and spatial characteristics for each stimulus.

In order to extract the spectral power and spatial information for each of these stimuli, an EEG signal, which is the sum of signals generated from multiple sources, is acquired based on a single electrode. Laplacian filtering may be performed to recover the superimposed EEG signal as a signal from a single source.

의 인접된 전극

에 대해 가중치의 합을 감산함으로써, 단일 소스의 뇌파 신호를 도출할 수 있으며, 단일 소스의 뇌파 신호

는 다음 식1을 만족한다.Accordingly, the filter module 121 performs Laplacian filtering on the EEG signal from which the DC component and the noise component are removed, so that each electrode

adjacent electrodes of

By subtracting the sum of weights for , a single source EEG signal can be derived, and a single source EEG signal

satisfies Equation 1 below.

[Equation 1]

는 i 번째 전극의 뇌파 신호이고,

는 시점 t에서 i번째 전극과 가장 인접된 전극이며,

는 i 번째 전극과 j 번째 전극의 역에 의해 결정된 가중치로서, 주어진 임의의 i 에 대해

를 만족하도록 정규화된다.here,

is the EEG signal of the i-th electrode,

is the electrode closest to the i- th electrode at time t,

is the weight determined by the inverse of the i- th electrode and the j- th electrode, for any given i

is normalized to satisfy

는 각 전극에 대해 가장 인접된 전극 세트로 정의된다. here,

is defined as the closest set of electrodes for each electrode.

In addition, the Laplacian-filtered single source EEG signal is transmitted to the estimation module 123 .

The estimation module 123 converts the EEG signal of a single source of the time series into an EEG signal in the frequency domain using a fast Fourier transform, and performs Welch technique on the EEG signal in the frequency domain to a predetermined frequency range (2Hz to 40Hz) We can output the density of the spectral power of . The density of the spectral power may be calculated in various ways, such as a wavelet method and a multitaper method, but is not limited thereto. 5 is an exemplary diagram showing the density of spectral power of EEG signals for each stimulus.

Meanwhile, in order to obtain spatial information about the EEG signal, the mapping module 125 may be provided to convert the 3D electrode of the scalp into a 2D image. That is, the mapping module 125 may convert the 3D electrode into a 2D image by applying an azimuthal equidistant projection by definition that the electrode is at a proportionally corrected distance from the center point.

Thereafter, the mapping module 125 maps the projected point to a region divided into predetermined sizes, and then allocates spectral power to the EEG signal of the corresponding electrode of each frequency for each space.

And, the spectral power of the corresponding electrode for each space is transmitted to the interpolation module 127 .

The interpolation module 127 interpolates the spectral power of the empty space between the electrode and the electrode by using cubic spline interpolation for the spectral power of the corresponding electrode for each space, so that each EEG obtained from 30 electrodes The signal can be converted into a three-dimensional brainwave image containing spectral and spatial information.

A predetermined number of input data among a plurality of input data of a plurality of three-dimensional brainwave images including spectral and spatial information for each of these EEG signals is transmitted to the learning unit 130 as learning data.

, 가중치

, 바이어스 파라미터

, 히든 레이어 파라미터

, 및 분류기

중 적어도 하나이다.The learner 130 may derive parameters optimized for a plurality of parameters with respect to the training data. Here, a plurality of parameters are rescaling parameters

, weight

, bias parameter

, hidden layer parameters

, and classifier

at least one of

, 가중치

, 바이어스 파라미터

, 히든 레이어 파라미터

, 및 분류기

와 전처리부(120)의 다수의 입력 데이터 중 학습 데이터를 제외한 나머지 입력 데이터는 테스트 데이터로 3차원 콘볼루션 신경망(3차원 CNN) 수행부(140)로 전달된다.In addition, the optimized parameter rescaling parameter of the learning unit 130

, weight

, bias parameter

, hidden layer parameters

, and classifier

and the remaining input data excluding the training data among the plurality of input data of the preprocessor 120 are transmitted to the 3D convolutional neural network (3D CNN) performing unit 140 as test data.

6 is a diagram showing a 3D CNN structure. Referring to FIG. 6 , the 3D CNN performing unit 140 performs a predetermined number based on a 3D kernel in order to learn spectral and spatial information for an EEG signal. (preferably four) may be provided in a CNN structure including a convolutional layer 131 , a polling layer 135 , and a fully connected layer 137 .

Here, each linear layer of the convolutional layer 131 is activated and output based on an exponential linear unit (ELU), and the exponential linear unit (ELU) can be expressed by Equation 2 below.

[Equation 2]

In an embodiment, each layer may be activated and output based on an exponential linear unit, but may be activated and output based on a rectified linear unit (ReLYU) if necessary, but is not limited thereto.

에 대해 포화값을 제어하기 위한 양의 하이퍼 파라미터이고, 고정된 하이퍼 파라미터의 경우 β=1 이다.where β is negative input data

It is a positive hyperparameter for controlling the saturation value for , and β=1 for a fixed hyperparameter.

As shown in FIG. 6 , the low-level characteristics of Layers 1 to 3 among the convolutional layers 131 provided with a plurality of Layers 1 to 4 are parameters to be estimated in order to connect between Layers 1 to 3, respectively. the number is greatly increased.

Accordingly, in order to reduce the order of the low-level features, the 3D CNN performing unit 140 applies average polling and maximum polling to each of the low-level features for each channel to generate a polling layer 135 for each channel, and each generated The average polling and maximum polling characteristics per channel are fed to the fully connected layer 137 .

Here, the low-level feature represents the separate cognitive load of the brain, the high-level feature represents the integrated cognitive load of the brain, and the 3D CNN performing unit 140 of an embodiment performs the high-level feature and the low-level feature by average polling and maximum polling. By integrating features, it is possible to classify the cognitive load of the brain.

로 변환한 다음 각 멀티 레벨 특징의 로지트(Logit)을 연산한다. 여기서, 로지트(logit)는 후술될 활성화 함수인 소프트맥스(softmax) 함수 이전에 비정규화된 최종 점수를 말한다. 7 is a view showing the calculation process of a multi-layered perceptron that derives a weight for classifying the final brain cognitive load into one by fusing low-level features and low-level features. Referring to FIG. 7, first, a three-dimensional CNN is performed. The unit 140 converts each multi-level feature that is the output data of each layer into a one-dimensional multi-level feature vector.

, and then calculate the logit of each multi-level feature. Here, the logit refers to the final score denormalized before the softmax function, which is an activation function to be described later.

및 각 분류기

의 열에 대해 L₂ 정규화를 수행한 다음 각 정규화된 항을 곱하여 각 멀티 레벨 특징의 로지트

을 도출한다. 이때 각 멀티 레벨 특징의 로지트

는 L₂ 정규화를 통해 임의의 i 및 j에 대해 -1과 +1 사이의 값을 가진다. 즉, 각 멀티 레벨 특징의 로지트

는

,

를 만족한다.That is, the three-dimensional CNN performing unit 140 performs each multi-level feature.

and each classifier

_{Perform L 2} regularization on the columns of , then multiply each normalized term to obtain the logit of each multilevel feature.

to derive In this case, the logit of each multi-level feature

has a value between -1 and +1 for any i and j through _{L 2 normalization.} That is, the logit of each multi-level feature

Is

,

is satisfied with

에 하나의 히든 레이어

와, 바이어스

및 활성화 함수

를 이용하여 각 멀티 레벨 특징에 대한 가중치

를 도출할 수 있다. 여기서 일 실시 예에서 활성화 함수

는 Softmax 함수이다. 즉, 각 멀티 레벨 특징에 대한 가중치

는 다음 식 2로 나타낼 수 있다.And, the three-dimensional CNN performing unit 140 is a log of multi-level features

one hidden layer on

wow, bias

and activation function

weights for each multi-level feature using

can be derived. Here, in one embodiment, the activation function

is the Softmax function. That is, the weights for each multi-level feature

can be expressed by Equation 2 below.

[Equation 3]

는 입력값인 멀티 레벨 특징

이고,

는 L₂ 정규화된 멀티 레벨 특징이며,

는 하나의 히든 레이어이며,

는 바이어스 파라미터이고,

는 가중치이며,

는 softmax 함수이다. 여기서, 가중치

는 각 멀티 레벨 특징의 중요도로 가중치

의 합은 1이다.here,

is the input multi-level feature

ego,

is the L ₂ normalized multi-level feature,

is one hidden layer,

is the bias parameter,

is the weight,

is a softmax function. Here, the weight

is weighted by the importance of each multi-level feature

The sum of is 1.

는 바이어스 파라미터

와 멀티 레벨 특징의 곱의 합과 히든 레이어

에 의해 결정되며, 여기서, 바이어스 파라미터

및 히든 레이어

은 학습 가능한 파라미터로 학습부(130)로부터 최적화된 바이어스 파라미터 및 히든 레이어이다.and weight

is the bias parameter

and the sum of products of multi-level features and hidden layers

is determined by , where the bias parameter

and hidden layers

is a learnable parameter, and is a bias parameter and a hidden layer optimized by the learning unit 130 .

는 3차원 뇌파 신호로부터 추출된 정규화된

의 종속되어 적응적으로 결정된다.and weight

is the normalized extracted from the 3D EEG signal.

is dependent and adaptively determined.

는 식 3으로 정의된다.The cognitive load of the j-th class is derived as the sum of each weighted output, and the cognitive load estimate of the j-th class

is defined by Equation 3.

[Equation 4]

는 분류기

의 i 번째 열로 정의되고,

는 가중치

의 i 번째 성분으로 정의되며,

는 양의 실수를 가지는 리스케일링 파라미터이다. 리스케일링 파라미터

는 네트워크 커버리지에 의해 유도된다.here,

is a classifier

is defined as the i-th column of

is the weight

is defined as the i-th component of

is a rescaling parameter with a positive real number. rescaling parameters

is derived by the network coverage.

은 학습부(130)로 전달되며, 이에 일 실시 예는 인지부하 추정값에 대한 학습을 통해 인지부하 추정값에 대한 분류 정확도를 향상시킬 수 있다.Cognitive load estimate, which is a single feature value fused with these high-level and low-level features

is transmitted to the learning unit 130 , and according to an embodiment of the present disclosure, classification accuracy of the estimated cognitive load may be improved through learning of the estimated cognitive load.

를 도출함에 있어, 일 실시 예는 입력된 멀티 레벨 특징의 로지트는 분류기와 멀티 레벨 특징의 방향 및 크기로 분해된다. 즉,

That is, the cognitive load estimate for each multi-level feature is weighted to classify it into the corresponding class.

In deriving , according to an embodiment, the log of the input multi-level feature is decomposed into a classifier and the direction and magnitude of the multi-level feature. in other words,

는 L₂ 정규화된 각 멀티레벨 특징 벡터

의 놈(norm)이고, L₂ 정규화된 벡터의 놈은 하나이며, 분류를 위해 각도 정보는 사용된다. 그리고 각 멀티 레벨 특징의 분류 결과는 가중치의 곱으로 가중된다. 이에 멀티 레벨 특징의 각도 손실(angular loss)이 가중되어 발생되고 이러한 가중된 분류 각도 손실은 가중 계수의 문제를 해인 그래디언트 보상 방법에 의해 최소화된다. here,

is L ₂ each normalized multilevel feature vector

is the norm of , and the norm of the L ₂ normalized vector is one, and the angle information is used for classification. And the classification result of each multi-level feature is weighted by the product of weights. Accordingly, the angular loss of the multi-level feature is weighted and generated, and this weighted classification angular loss is minimized by the gradient compensation method which solves the problem of the weighting coefficient.

는 softmax 함수인 교차 엔트로피 손실 함수로 사용되고, softmax 함수는 다음 식 5로 나타낼 수 있다.i.e. the classification loss function

is used as a cross-entropy loss function, which is a softmax function, and the softmax function can be expressed by Equation 5 below.

[Equation 5]

는 입력값인 심층 특징으로 정의되고,

는 심층 특징

의 인코딩된 클래스 레벨이며,

는 j 번째 열의 바이어스 파라미터이며,

은 분류의 클래스 수이다. 그리고 단순화하기 위해, 배치 사이즈(batch size)는 1로 설정되고,

은 마지막 콘볼루션으로부터 추출되는 특징으로서, 활성화 함수 없이 네트워크의 최종 출력 특징의 로지트는

로 표현될 수 있다.here,

is defined as a deep feature as an input,

is an in-depth feature

is the encoded class level of

is the bias parameter of the j-th column,

is the number of classes in the classification. And for simplicity, the batch size is set to 1,

is the feature extracted from the last convolution, and the logit of the final output feature of the network without an activation function is

can be expressed as

로 나타낼 수 있다. 여기서, 분류 벡터

와 특징 벡터

는 L₂ 정규화된다. However, the logit of all multi-level features is

can be expressed as where the classification vector

with feature vector

is L ₂ normalized.

로 설정되면, 멀티 레벨 특징의 로지트는 다음 식 6으로 나타낼 수 있다.And, for any i, the bias parameter

If set to , the logit of the multi-level feature can be expressed by Equation 6 below.

[Equation 6]

는 분류 벡터

와 특징 벡터

사이의 각도이다. 이러한 멀티 레벨 특징의 로지트는 각도

로 나타낼 수 있다. 여기서 L₂ 정규화 추정값은 분류 벡터

와 특징 벡터

사이의 각

에 종속된다. here,

is a classification vector

with feature vector

is the angle between The logit of these multi-level features is the angle

can be expressed as where the L ₂ normalized estimate is the classification vector

with feature vector

angle between

is dependent on

은 식 7으로 나타낼 수 있다.Therefore, rearranging Equation 6 gives the angular loss of multi-level features.

can be expressed by Equation 7.

[Equation 7]

와 각 멀티 레벨 특징의 로지트

의 곱으로 멀티 레벨 특징의 각도 손실

이 도출되며, 이에 도출된 멀티 레벨 특징의 각도 손실

은 다음 식 8로 도출될 수 있다.Next, weights are used to assign importance to multi-level features.

and logit of each multi-level feature

angular loss of multi-level features as a product of

is derived, and the angular loss of the derived multi-level feature

can be derived from the following Equation 8.

[Equation 8]

은 멀티 레벨 특징 벡터

의 로지트이고 각도 손실

은 멀티 레벨 특징의 벡터

의 중요도를 나타내는 가중치

와 멀티 레벨 특징 벡터의 로지트

의 곱에 의해 결정된다.here,

is a multi-level feature vector

is the logit of and the angle loss

is a vector of multi-level features

weight indicating the importance of

and logit of multi-level feature vectors

is determined by the product of

을 곱하고 역전파 알고리즘에 의해 결정된 리스케일링 파라미터의 학습으로 로지트의 스케일을 재조정한다. 이에 최종 분류 손실 함수는 로지트의 재스케일링에 의거 얻을 수 있으며 최종 분류 손실 함수는 다음 식 9를 만족한다.Also, when each feature vector and each classifier column are L ₂ normalized, the cross entropy loss function has a low bound (low bound). And, the loss function converges to a very large value through tens of millions of iterations, and the network does not converge. Accordingly, in one embodiment, in order to solve this problem, the rescaling parameter in the log of each multi-level feature is

Multiply by and rescale the logit by learning the rescaling parameters determined by the backpropagation algorithm. Accordingly, the final classification loss function can be obtained by logit rescaling, and the final classification loss function satisfies Equation 9 below.

[Equation 9]

는 학습 가능한 파라미터이다. 이에 각 멀티 레벨 특징의 분류는 각 정보에 의해 결정되고, 각 멀티 레벨 특징의 중요도는 가중치에 의해 결정된다. Here, the rescaling parameter

is a learnable parameter. Accordingly, the classification of each multi-level feature is determined by each piece of information, and the importance of each multi-level feature is determined by a weight.

의 각 로지트에 가중치

를 곱하고, 이에 멀티 레벨 특징의 학습 속도는 가중치

에 의존된다. 즉, 가중치

가 높을수록 훈련 속도는 빨라지고, 가중치

가 작으면 멀티레벨 특징의 학습은 수렴되지 아니한다.That is, each multi-level feature vector

weight for each log of

multiplied by , and the learning rate of multi-level features is weighted

depend on i.e. weight

The higher the value, the faster the training speed, and the

If is small, the learning of multi-level features does not converge.

는 다음 식 10으로 정리된다.The final classification loss function of Equation 9 to solve this problem

is summarized in Equation 10 below.

[Equation 10]

에 대한 분류기

의 그래디언트(gradient)는 다음 식 11로 도출될 수 있다.Here, the multi-level feature

classifier for

The gradient of can be derived from Equation 11 below.

[Equation 11]

here,

는 멀티 레벨 특징

,

의 벡터이고,

이다. here,

features multi-level

,

is a vector of

am.

의 그래디언트는 가중치

에 대한 그래디언트와 특징 벡터

에 대한 그래디언트로 나누어진다. 가중치

에 특징 벡터

를 곱하기 때문에 가중치

의 학습 속도와 유사한 멀티레벨 특징

의 학습 속도의 조절이 가능하다.here,

The gradient of is weighted

Gradients and feature vectors for

is divided by the gradient for . weight

in the feature vector

weight because it is multiplied by

Multilevel features similar to the learning rate of

It is possible to control the learning rate of

가 다른 가중치에 비해 극도로 작은 경우 멀티 레벨 특징의 벡터

을 추출함에 있어 역전파 알고리즘 수행 과정에서 멀티레벨 특징 벡터의 학습 속도를 보상하기 위해 가중치

의 역수의 곱하여 멀티 레벨 특징

에 대한 그래디언트를 도출하며, 도출된 멀티 레벨 특징에 대한 그래디언트는 다음 식 13으로 나타낼 수 있다.i.e. weight

A vector of multilevel features if is extremely small compared to the other weights

To compensate for the learning speed of multilevel feature vectors in the process of performing the backpropagation algorithm in extracting

Multi-level features by multiplying the reciprocal of

A gradient is derived for , and the gradient for the derived multi-level feature can be expressed by the following Equation 13.

[Equation 13]

를 추출하는데 사용되는 콘볼루션 계층과 완전 연결 계층의 가중치

는 그래디언트 보정에 영향을 받으며, 가중치

및 분류기

를 추출되는데 사용되는 히든 레이어

및 바이어스 파라미터

는 경사 보정에 영향을 받지 아니한다.Here, a vector of multi-level features

The weights of the convolutional layer and the fully connected layer used to extract

is affected by gradient correction,

and classifier

hidden layer used to extract

and bias parameters

is not affected by slope correction.

The three-dimensional CNN performing unit 140 combines the sum of products of each multi-level feature and weights with minimized angular loss based on the full coupling layer 147 to convert the brain cognitive load into one of a plurality of classifiers for the input EEG signal. classify

Accordingly, in one embodiment, each multi-level feature of each layer is derived through a predetermined number of convolutions on input data of a two-dimensional image including spectral and spatial information of an input EEG signal, and the derived multi-level feature and Based on the parameters optimized through learning, each weight of each multi-level feature is derived, and the log of each multi-level feature is derived by multiplying the derived weight and multi-level feature, and classified into the derived log of each multi-level feature The angular loss of each multi-level feature is calculated by applying the loss function, and each multi-level feature with the calculated angular loss is fully combined to classify the brain cognitive load for the input EEG signal into one of a number of classifiers. It is possible to increase the learning speed based on the EEG signal and improve the classification accuracy of the cognitive load of the brain.

The performance (Accuracy) of the system according to this embodiment and the performance of the system of the existing SVM, KNN, and multi-level convergence technique are shown in Table 1 below.

[Table 1]

According to Table 1, it can be seen that the performance of the system according to an embodiment is improved than that of the conventional system.

In addition, the classification accuracy of the existing multilevel fusion method having a fixed weight and the classification accuracy of the multilevel fusion method according to the weight derived according to an embodiment are shown in Table 2 below.

[Table 2]

별 CDF(Cumulative Distribution Function)를 보인 그래프로서, 도 8을 참조하면, 일 실시 예는 뇌파 신호에 의거 결정된 히든 레이어 파라미터

과 멀티 레벨 특징

의 곱으로 가중치

가 도출되므로 뇌파 신호에 따라 상이한 가중치

가 도출된다. 이에 도 8을 참조하면, 중간 멀티 레벨 특징

,

에 대한 가중치

,

는 주로 0과 0.05 사이에 분포되며 매우 작다. 그러나, 첫 번째 및 마지막 멀티 레벨 특징

,

의 가중치

,

는 각각 0.2와 0.8 사이에 분포되어 있으므로 상대적으로 높다는 것을 알 수 있다.8 shows each weight

As a graph showing a star CDF (Cumulative Distribution Function), referring to FIG. 8 , an embodiment shows a hidden layer parameter determined based on an EEG signal.

and multi-level features

weighted by the product of

is derived, so different weights depending on the EEG signal

is derived Referring to FIG. 8, the intermediate multi-level feature

,

weight for

,

is mainly distributed between 0 and 0.05 and is very small. However, the first and last multi-level features

,

weight of

,

is distributed between 0.2 and 0.8, respectively, so it can be seen that they are relatively high.

를 결정함에 따라 시스템 성능이 향상된다. Accordingly, it can be estimated that layer 1 of the first convolution has a low-level feature, and layer 4 of the fourth convolution has a high-level feature. Therefore, optimized weights for EEG signals

The system performance is improved by determining

That is, one embodiment converts an EEG signal including both a local structure as a separate cognitive load and a global structure as a global cognitive load into a 3D image including spectral and spatial information, and then converts each multi By extracting the level features and multiplying the extracted multi-level features by weights, the importance of the multi-level features is determined, and the brain cognitive load is classified according to the determined importance of the multi-level features. At this time, the weights are optimized through learning about EEG signals. As a result, the performance of the system is improved, and the classification accuracy of brain cognitive load is further improved compared to other existing algorithms.

Although the present invention has been described in detail through representative embodiments above, those of ordinary skill in the art to which the present invention pertains will understand that various modifications are possible within the limits without departing from the scope of the present invention with respect to the above-described embodiments. will be. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by all changes or modifications derived from the claims and equivalent concepts as well as the claims to be described later.

Each multi-level feature of each layer is derived through a predetermined number of convolutions on input data of a three-dimensional image including spectral and spatial information of the input EEG signal, and parameters optimized through the derived multi-level features and learning Each weight of each multi-level feature is derived based on The angular loss of multi-level features is calculated, and each multi-level feature with the calculated angular loss is completely combined to classify the brain cognitive load for the input EEG signal into one of a number of classifiers. Accuracy of operation of brain cognitive load classification system and method that can improve accuracy, learn weights based on EEG signals and determine optimized weights, increase learning speed, and improve system performance It can bring very great progress in terms of reliability and performance efficiency, and it is an invention that has industrial applicability because it has a sufficient possibility of commercialization or sales of an image classification tool, as well as a degree that can be clearly implemented in reality.

Claims (12)

a data collection unit for acquiring at least one EEG signal for at least one given task;
a preprocessing unit for outputting the EEG signal of the data collection unit as input data of a three-dimensional image including spectral and spatial information;
a learning unit that learns the input data of the preprocessor and outputs a rescaling parameter, a weight, a bias parameter, a hidden layer parameter, and a classifier; and
A CNN performing unit for classifying a brain cognitive load into one of a plurality of classifiers by performing a three-dimensional convolutional neural network (CNN) algorithm based on the rescaling parameters, weights, and input data of the classifier and the preprocessor,
The learning unit,
A predetermined number of multi-level features are derived by activating the input data of the three-dimensional image of the preprocessor based on an exponential linear unit,
Deriving the angular loss of the multi-level feature from the classification loss function for the log of the multi-level feature derived by the product of each of the derived multi-level features and each classifier,
Brain cognitive load classification system, characterized in that by deriving each weight of each multi-level feature as a minimized solution of the derived angle loss, and delivering the derived weight of each multi-level feature to the CNN execution unit. The method of claim 1, wherein the classification of the brain cognitive load is
Brain cognitive load classification system, characterized in that output as one level. According to claim 1, wherein the pre-processing unit,
a filter module that removes noise components through filtering from the acquired EEG signals in the form of vibrations in a predetermined frequency range and restores them to a single source signal through source localization;
an estimation module for outputting a density of spectral power in a predetermined frequency range using a fast Fourier transform and wavelet for a single source EEG signal of a time series;
A mapping module for obtaining spatial information of an EEG signal by converting a three-dimensional electrode into a two-dimensional image by applying an azimuthal equidistant projection by definition that an electrode is located at a proportionally corrected distance from the central point; and
Outputs input data of a three-dimensional image including spectrum and spatial information by interpolating the spectral power of the empty space between the electrode and the electrode using cubic spline interpolation for the spectral power of the corresponding electrode in each space Brain cognitive load classification system, characterized in that it comprises an interpolation module. According to claim 3, wherein the CNN performing unit,
A predetermined number of convolutional layers, polling layers, and fully connected layers,
A certain number of convolutional layers is
It may be provided to derive a layer including a predetermined number of multi-level features by activation based on an exponential linear unit for input data of a three-dimensional image,
The polling layer is
It may be provided to apply average polling and maximum polling to some layers representing medium and low-level features of a predetermined number of layers to output the average and maximum polling features for each channel,
The fully connected layer is
Each weight representing the importance of each multi-level feature is derived using the activation function for the log, hidden layer parameter, and bias parameter of the multi-level feature derived from the product of each multi-level feature and each classifier, and then
Estimate the brain cognitive load by the sum of the product of each of the derived weights and the predetermined activation function value and the rescaling parameter for the logit of the multi-level feature, and classify the estimated brain cognitive load into one of a plurality of predetermined classifiers Brain cognitive load classification system, characterized in that. delete The method according to claim 1, wherein the solution to minimize the angular loss is
Brain cognitive load classification system, characterized in that it is provided to derive the gradient technique, which is a partial differential equation of the feature vector for the angular loss of each multi-level feature. In the brain cognitive load classification method executed based on the brain cognitive load classification system of claim 1,
a data collection step of acquiring, in the data collection unit, at least one EEG signal for at least one given task;
a pre-processing step of outputting, in the pre-processing unit, input data of a two-dimensional image including spectral and spatial information with respect to the EEG signal of the data collection unit;
a learning step of, in a learning unit, learning the input data of the preprocessor and outputting a scaling parameter, a weight, a bias parameter, a hidden layer parameter, and a classifier; and
In the CNN performing unit, a 3D convolutional neural network (CNN) algorithm is performed based on the rescaling parameters, weights, and spectral and spatial information of the classifier and the preprocessor to classify the brain cognitive load into one of a plurality of classifiers. comprising steps,
The learning step is
A predetermined number of multi-level features are derived by activating the input data of the three-dimensional image of the preprocessor based on an exponential linear unit,
Deriving the angular loss of the multi-level feature from the classification loss function for the log of the multi-level feature derived by the product of each of the derived multi-level features and each classifier,
Brain cognitive load classification method, characterized in that by deriving each weight of each multi-level feature as a minimized solution of the derived angle loss, and delivering the derived weight of each multi-level feature to the CNN execution unit. The method of claim 7, wherein the classification of the brain cognitive load is
Brain cognitive load classification method, characterized in that output as one level. The method of claim 7, wherein the pretreatment step
removing a noise component through filtering from the obtained EEG signal in the form of vibration in a predetermined frequency range and restoring a signal of a single source through source localization;
outputting a density of spectral power in a predetermined frequency range using a fast Fourier transform and wavelet of a single source EEG signal of a time series;
obtaining spatial information of an EEG signal by converting a three-dimensional electrode into a two-dimensional image by applying an azimuthal equidistant projection according to the definition that an electrode is located at a proportionally corrected distance from the central point; and
Outputs input data of a three-dimensional image including spectrum and spatial information by interpolating the spectral power of the empty space between the electrode and the electrode using cubic spline interpolation for the spectral power of the corresponding electrode in each space Brain cognitive load classification method comprising the step of. 10. The method of claim 9, wherein the CNN performing step
A predetermined number of convolutional layers, polling layers, and fully connected layers,
A certain number of convolutional layers is
It may be provided to derive a layer including a predetermined number of multi-level features by activating it based on an exponential linear unit with respect to the input data of the three-dimensional image,
The polling layer is
It may be provided to apply average polling and maximum polling to some layers representing medium and low-level features of a predetermined number of layers to output the average and maximum polling features for each channel,
The fully connected layer is
Each weight representing the importance of each multi-level feature is derived using the activation function for the log, hidden layer parameter, and bias parameter of the multi-level feature derived from the product of each multi-level feature and each classifier, and then
Estimate the brain cognitive load by the sum of the product of each of the derived weights and the predetermined activation function value and the rescaling parameter for the logit of the multi-level feature, and classify the estimated brain cognitive load into one of a plurality of predetermined classifiers Brain cognitive load classification method, characterized in that. delete 8. The method of claim 7, wherein the minimized solution of the angular loss is
Brain cognitive load classification method, characterized in that it is provided to derive the gradient technique, which is a partial differential equation of the feature vector for the angular loss of each multi-level feature.

Images