KR20210035530A - Device and method for learning connectivity - Google Patents

Abstract

An embodiment of the present invention relates to a device for learning connectivity, which can determine connectivity among a plurality of objects by virtue of deep neural network learning and classify input signals based on the determined connectivity, wherein the device comprises: a membership extraction unit for extracting membership which indicates presence of connectivity among the objects from raw signals acquired from the objects; a graph sampling unit for sampling connectivity among the objects from the extracted membership to create a graph structure representing inter-object connectivity; a feature extraction unit for extracting signal features from the raw signals; and a classification unit for classifying input signals into any one of a plurality of classes based on the graph structure created and the signal features extracted. According to the present invention, signals can be classified with enhanced accuracy through direct learning based on raw signals without prior knowledge.

Description

Connectivity learning device and connectivity learning method {DEVICE AND METHOD FOR LEARNING CONNECTIVITY}

The present invention relates to connectivity learning, and more particularly, to a connectivity learning device and connectivity learning method for learning connectivity between a plurality of objects.

Analysis of brain activity through brain images such as electroencephalography (EEG) is important for understanding human mental states and thoughts. This analysis of brain activity is essential in a variety of applications, including the brain-computer interface, emotion recognition, and diagnosis of mental illness. The brain is made up of several functional areas, and activation patterns across several functional areas provide valuable information about the state of mind. Therefore, it is effective for brain signal analysis to study the relationship between regions represented by a pattern called functional connectivity (also referred to as'connectivity' for simplicity in this specification). Since brain regions are not in Euclidean space, graphs are the most natural and suitable data structure for connectivity.

However, how to measure the degree of connectivity, how to define an appropriate graph structure, and how to define an appropriate function for signals from other brain regions are still unresolved problems. These issues are usually determined manually based on prior knowledge. For example, a correlation or causal matrix between signals in different regions can be used as a measure of connectivity. And, a graph can be constructed by connecting a pair of brain regions showing a high connectivity value, and finally, the power or entropy of a signal can be used as a feature of each region (ie, a peak of the graph). However, solving these problems manually is not the best option. In fact, these issues apply equally to brain signaling data as well as other data related to graph structures such as social networks and chemicals.

The problem to be solved by the present invention is to provide a connectivity learning apparatus capable of performing signal classification with improved accuracy through direct learning based on a source signal even without any prior knowledge.

Another problem to be solved by the present invention is to provide a connectivity learning device capable of expressing the overall state of a network having interconnectivity between a plurality of objects rather than connectivity between one object pair.

Another problem to be solved by the present invention is to determine the connectivity between a plurality of objects even when information on how much interconnectivity exists between a plurality of objects, that is, an explicit loss function for a graph structure is not given. It is to provide a connected learning device that can do.

Another problem to be solved by the present invention is to provide a connectivity learning method performed by the connectivity learning device.

A connectivity learning device according to an embodiment of the present invention for solving the above problem is a connectivity learning device for determining connectivity between a plurality of objects by learning a deep neural network and classifying an input signal based on the determined connectivity. And a membership extracting unit for extracting a membership indicating the existence of connectivity between the objects from the original signals obtained from the plurality of objects; A graph sampling unit that samples the connectivity between the objects from the extracted membership and generates a graph structure expressing the connectivity between the objects in a graph; A feature extraction unit for extracting a signal feature from the original signal; And a classification processor for classifying the input signal into any one of a plurality of classes based on the generated graph structure and the extracted signal characteristic.

In one embodiment, the connectivity between the objects may have two or more types. In addition, the membership extracting unit may extract the membership for two or more connectivity layers representing different types of connectivity.

In an embodiment, the graph sampling unit may generate one graph layer for each connectivity layer, and the graph structure may include two or more graph layers. In addition, the graph sampling unit may generate the graph structure by deterministic binarization that samples connectivity between the objects based on whether a value indicated by the extracted membership is equal to or greater than a threshold value.

In one embodiment, the raw signal or the input signal is Functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and Functional Near Infrared Spectroscopy (fNIRS) acquired from a plurality of sensors. It may include one or more of.

In an embodiment, the feature extractor may extract signal features from the original signal through a convolution operation and a max pooling operation. In addition, the convolution operation may be performed using an extended convolutional layer having a plurality of time intervals.

A connectivity learning method according to another embodiment of the present invention for solving the above problem is a connectivity learning method for determining connectivity between a plurality of objects by learning a deep neural network and classifying an input signal based on the determined connectivity. And a membership extraction step of extracting membership indicating the existence of connectivity between the objects from the original signals obtained from the plurality of objects; A graph sampling step of sampling the connectivity between the objects from the extracted membership and generating a graph structure representing the connectivity between the objects in a graph; A feature extraction step of extracting a signal feature from the original signal; And a classification processing step of classifying the input signal into any one of a plurality of classes based on the generated graph structure and the extracted signal characteristic.

In one embodiment, the connectivity between the objects may have two or more types. In addition, in the membership extraction step, the membership may be extracted for two or more connectivity layers representing different types of connectivity.

In an embodiment, in the graph sampling step, one graph layer is generated for each connectivity layer, and the graph structure may include two or more graph layers. In addition, in the graph sampling step, the graph structure may be generated by deterministic binarization of sampling connectivity between the objects based on whether a value indicated by the extracted membership is equal to or greater than a threshold value.

In one embodiment, the raw signal or the input signal is Functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and Functional Near Infrared Spectroscopy (fNIRS) acquired from a plurality of sensors. It may include one or more of.

In an embodiment, in the feature extraction step, a signal feature may be extracted from the original signal by a convolution operation and a max pooling operation. In addition, the convolution operation may be performed using an extended convolutional layer having a plurality of time intervals.

According to an embodiment of the present invention, a new deep learning model that automatically extracts graph structures and signal features representing a plurality of connectivity between a plurality of objects and performs signal classification using the extracted graph structures and signal features is used. By doing so, it is possible to perform signal classification with improved accuracy through direct learning based on the raw signal even without any prior knowledge, and to express the overall state of the network having interconnectivity between a plurality of objects, and the loss function is A connectivity learning apparatus and connectivity learning method capable of determining connectivity between a plurality of objects even when not given may be provided.

1 is a block diagram of an apparatus for learning connectivity according to an embodiment of the present invention.
2 is a conceptual diagram showing the configuration of the membership extraction unit 10 according to an embodiment of the present invention.
3 is a conceptual diagram showing the configuration of the graph sampling unit 20 according to an embodiment of the present invention.
4 is a conceptual diagram showing the configuration of a feature extraction unit 30 according to an embodiment of the present invention.
5 is a conceptual diagram showing the configuration of the classification processing unit 40 according to an embodiment of the present invention.
6A to 6D illustrate a raw signal and an extracted graph according to an embodiment of the present invention.
7 is a graph showing the names and locations of 32 EEG electrodes for graphical representation used in the present invention.
8A to 8C are graphs showing examples of graph structures obtained according to an embodiment of the present invention.
9 is a flowchart illustrating a connection learning method according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The embodiments of the present invention are provided to more completely describe the present invention to those of ordinary skill in the art, and the following examples may be modified in various other forms, and the scope of the present invention is as follows. It is not limited to the examples. Rather, these embodiments are provided to make the present disclosure more faithful and complete, and to fully convey the spirit of the present invention to those skilled in the art.

Hereinafter, embodiments of the present invention will be described with reference to the drawings schematically showing ideal embodiments of the present invention. In the drawings, for example, the size and shape of members may be exaggerated for convenience and clarity of description, and in actual implementation, variations of the illustrated shape may be expected. Accordingly, embodiments of the present invention should not be construed as being limited to the specific shape of the member or region shown herein.

이고, X 는 N 개의 센서(일 실시예에서 EEG 전극)로부터 수집된 시계열 신호 세트이고, y는 대응하는 클래스 레이블이다. The present invention can build a neural network model that classifies given signal data using a connectivity structure generated by a deep learning model. The given data is expressed as (X, y ), where

And X is the set of time series signals collected from N sensors (EEG electrodes in one embodiment), and y is the corresponding class label.

The estimated graph structure is a multi-layer graph having weights and directions without self-loop, and the graph structure G may be expressed by Equation 1.

(수학식 1)

(Equation 1)

V representing a vertex can be expressed by Equation 2, E representing the existence of an edge can be expressed by Equation 3, and W representing the weight can be expressed by Equation 4.

(수학식 2)

(Equation 2)

(수학식 3)

(Equation 3)

(수학식 4)

(Equation 4)

로 가정되었고, K는 그래프 레이어의 개수를 제어하는 하이퍼 파라미터이다.E ^k represents the existence of an edge between the pair of vertices in the k-th graph layer, and W ^k represents the weight of the edge between the pair of vertices.

Is assumed, and K is a hyper parameter that controls the number of graph layers.

1 is a block diagram of an apparatus 1 for learning connectivity according to an embodiment of the present invention.

Referring to FIG. 1, the connectivity learning apparatus 1 may include a membership extracting unit 10, a graph sampling unit 20, a feature extracting unit 30, and a classification processing unit 40. The connectivity learning apparatus 1 may determine connectivity between a plurality of objects by learning a deep neural network, and classify an input signal based on the determined connectivity.

Connectivity between objects may include a causal relationship or a correlation, and may include two or more types. When the connectivity between objects includes two or more types, it may be represented by a plurality of connectivity layers, as described later.

The membership extracting unit 10 may extract a membership indicating the existence of connectivity between objects from a raw signal obtained from a plurality of objects, and output the extracted membership to the graph sampling unit 20. In an embodiment, the object may include a sensor or sensing element capable of sensing or detecting various signals.

The raw signal or input signal may include one or more of Functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and Functional Near Infrared Spectroscopy (fNIRS). The raw signal or the input signal is not limited to a signal related to the brain, and in other embodiments may include an image or an audio signal related to the heart. In another embodiment, the original signal or the input signal may include a signal related to weather data such as an air volume or wind speed, or an image signal indicating vehicle traffic on a road.

When the connectivity between objects is of two or more types, the membership extracting unit 10 may extract membership for each of the two or more connectivity layers representing different types of connectivity. In addition, in this case, the graph sampling unit 20 may generate one graph layer for each connectivity layer.

The graph sampling unit 20 samples the connectivity between objects from the membership extracted by the membership extraction unit 10 to generate a graph structure expressing the connectivity between the objects as a graph, and classifies the generated graph structure into the classification processing unit 40 ) Can be printed.

In addition, the graph sampling unit 20 may generate a graph structure by deterministic binarization that samples the connectivity between objects based on whether the value represented by the membership extracted by the membership extraction unit 10 is greater than or equal to a threshold value. have.

The feature extraction unit 30 may extract a signal feature from an original signal obtained from a plurality of objects and output the extracted signal feature to the classification processor 40. The feature extraction unit 30 may extract a signal feature from the original signal through a convolution operation and a max pooling operation. The convolution operation performed by the feature extraction unit 30 may be performed through extended convolutional layers having a plurality of time intervals.

2 is a conceptual diagram showing the configuration of the membership extraction unit 10 according to an embodiment of the present invention.

The membership extraction unit 10 calculates potential membership from the input time series data. The potential membership (h _ij ) represents the probability of the existence of an edge from the vertex vi to the vertex vj for each graph layer, and can be calculated by Equations 5 and 6. The membership extraction unit 10 uses a vertex-to-edge operation, an edge-to-vertex operation, and a fully-connected network.

(수학식 5)

(Equation 5)

(수학식 6)

(Equation 6)

는 연결 연산(concatenation operation)을 나타낸다. f1 부터 f4까지는 지수적 선형 유닛(exponential linear unit: Djork-Arnι Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In Proceedings of the 3rd International Conference on Learning Representations, pages 1-14, 2015 참조)을 가지며, 배치 정규화(batch normalization: Sergey Ioffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, pages 448-456, 2015 참조)를 갖는 완전 연결 네트워크이다.In Equation 5 and Equation 6

Denotes a concatenation operation. From f1 to f4 exponential linear units (Djork-Arnι Clevert, Thomas Unterthiner, and Sepp Hochreiter.Fast and accurate deep network learning by exponential linear units (ELUs).In Proceedings of the 3rd International Conference on Learning Representations, pages 1-14, 2015 ), and batch normalization: accelerating deep network training by reducing internal covariate shift.In Proceedings of the 32nd International Conference on Machine Learning, pages 448- 456, 2015 ).

3 is a conceptual diagram showing the configuration of the graph sampling unit 20 according to an embodiment of the present invention.

The graph sampling unit 20 may generate a graph structure from the layer membership information through probabilistic or deterministic sampling. In the present invention, three methods are considered for graph sampling: stochastic sampling (STO), deterministic thresholding (DET), and continuous sampling (CON).

The probabilistic sampling method probabilistically assigns a potential edge from vertex vi to vertex vj to one of the K graph layers. Since the sampled graph weights are discontinuous, a Gumbel-softmax reparameterization technique (refer to documents 28 and 29) is used to provide continuous relaxation and enable gradient calculations.

(수학식 7)

(Equation 7)

는 랜덤 벡터이고, 이 랜덤 벡터의 각 요소들은 IID(Independent and Identically Distributed) 속성을 갖고, 표준 Gumbel 분포를 따른다.

는 샘플링 평활도(smoothness)를 제어하는 소프트맥스 온도이다. 본 실시예에서는 0.5로 설정하였다. vi로부터 vj로의 비가중 간선(

)은 수학식 8에 의해 얻을 수 있다.here,

Is a random vector, and each element of this random vector has an IID (Independent and Identically Distributed) attribute, and follows a standard Gumbel distribution.

Is the softmax temperature that controls the sampling smoothness. In this example, it was set to 0.5. unweighted edges from vi to vj (

) Can be obtained by Equation 8.

(수학식 8)

(Equation 8)

Since the estimated graph includes a plurality of graph layers, different types of connectivity information may be modeled for each graph layer. The probabilistic sampling method limits each edge to belong to only one graph layer, but the deterministic binarization method alleviates this limitation so that a pair of vertices has edges in multiple layers through binarization.

(수학식 9)

(Equation 9)

In Equation 9, r is a threshold value, and in the example, it is set to 0.5. Discrete variables can be differentiated during learning using the same continuous relaxation technique used in the probabilistic sampling method.

는 k 번째 그래프 계층에서 vi에서 vj까지의 간선 가중치로 직접 사용된다. 따라서 이 방법은 세 가지 방법 중 가장 일반적인 형태의 그래프 구조, 즉 가중치 및 방향성을 갖는 멀티 레이어 그래프를 생성한다.The previous two methods constitute an unweighted graph, but if the continuous sampling method is used, different degrees of connectivity can be maintained in different graph layers by using the edge weight as a continuous value. To do this, we have a continuous value between 0 and 1 obtained from the Gumbel-softmax operation.

Is used directly as the edge weight from vi to vj in the k-th graph layer. Therefore, this method creates the most common type of graph structure among the three methods, that is, a multi-layer graph with weights and directions.

There may be no direct relationship between a particular pair of vertices. To enable this, one of the graph layers may be assigned as a skip layer. Since the skip layer is discarded when the graph is transmitted to the classification processing unit 40, the trunk line belonging to this layer is omitted from the graph used for classification.

4 is a conceptual diagram showing the configuration of a feature extraction unit 30 according to an embodiment of the present invention.

The feature extraction unit 30 may extract signal features from the original signal by 1-D convolution and max pooling operations. In order to capture the dynamic information of the signal, the present invention adopts the 1-D version of the extended inception module including convolutional layers with various dilation rates. The extended convolution layer with a low expansion rate captures features that appear between surrounding samples corresponding to fast-changing high-frequency information, and the extended convolution layer with a high expansion rate captures features that change slowly for a large temporal window. .

(수학식 10)

(Equation 10)

The signal feature Ui can be calculated by Equation 10. Here, T'is the reduced signal length, and F is the feature dimension.

5 is a conceptual diagram showing the configuration of the classification processing unit 40 according to an embodiment of the present invention.

를 출력할 수 있다. 먼저, 신호 특징들에 대해 정점-간선 연산이 수행되고, 그 결과는 메시지 전달(message passing) 연산을 통해 그래프 구조와 결합되고 집계 연산(aggregation operation) 및 간선-정점 연산이 수행된다.The classification processing unit 40 includes a graph neural network (GNN), and class labels predicted by performing classification using signal features and a built graph

Can be printed. First, a vertex-edge operation is performed on signal features, and the result is combined with a graph structure through a message passing operation, and an aggregation operation and an edge-vertex operation are performed.

(수학식 11)

(Equation 11)

는 입력 및 출력이 각각 F-차원 및 F'-차원, 즉

인 ReLU(Rectified Linear Unit)를 갖는 완전 연결된 네트워크에 의해 모델링된다. 마지막으로 이 결과는 스킵 커넥션을 통해 신호 특징들과 연결되어 벡터화 후에 완전 연결된 네트워크에 제공된다.In Equation 11, t is a time index, s=2 when a skip layer is used, and s=1 when a skip layer is not used.

Is the input and output are F-dimensional and F'-dimensional, respectively

It is modeled by a fully connected network with ReLU (Rectified Linear Unit). Finally, this result is linked to the signal features via a skip connection, vectorized and provided to a fully connected network.

(수학식 12)

(Equation 12)

이다.)(here

to be.)

Hereinafter, specific embodiments of the present invention will be described.

In this embodiment, one of the largest databases on the emotional and mental state of humans, the Database for Emotion Analysis using Physiological Signals (DEAP) data set was used. The data set of the example includes 32 channel EEG records collected from 32 subjects while watching 40 video stimuli and the corresponding emotion rating by the subject. In the embodiment, a video identification task was performed to classify a set of EEG signals given by a deep learning model into one of 40 video stimuli.

Specifically, each 1 minute long EEG signal of the DEAP data set is divided into 3 second long segments overlapping 2 seconds long (T=384). As a result, a total of 74,240 32-channel EEG signal sets (32 subjects x 40 videos x 58 segments) are obtained. The obtained signal sets are randomly separated into training, validation and test data sets holding 80%, 10% and 10% of the total data set, respectively.

The deep learning model of this example is PyTorch ( Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In Proceedings of the NIPS. 2017 Autodiff Workshop: The Future of Gradient-based Machine Learning Software and Techniques, pages 1-4, 2017 ), and Adam optimizer ( Diederik P. Kingma and Jimmy Ba. Adam: a method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, pages 1-15, 2015 ).

Details of the model structure and learning parameters will be described below.

Graph membership extraction: f1, f2 and f3 are fully connected two-layer networks with 256 hidden neurons and 256 output neurons. Each fully connected layer has an exponential linear unit, and batch normalization is used for the output layer. f4 contains 3 fully connected layers. Among the three layers, there are 256 hidden neurons with exponential linear units in the first two layers, and K output neurons in the last layer.

Feature Extraction: Each extended inception module consists of four 1-D convolutional layers with a kernel size of 3, each with a scaling factor of 1, 2, 4 and 8. Each layer has 8 output channels, so F = 8 × 4 = 32. The maximum pooling size is set to α = 4. Three extended start modules are connected in a line so that the length of the signal extension is T'=6.

(k = 1,?, K)는 256 개의 은닉 뉴런과 ReLU를 갖는 256 개의 출력 뉴런을 갖춘 2 레이어의 완전 연결 네트워크이다. g₂는 ReLU(Rectified Linear Unit)를 갖는 256 개의 은닉 뉴런과 40 개의 소프트 맥스 출력 뉴런을 갖는 2 레이어의 완전 연결 네트워크이다.Neural network graph:

( k = 1,?, K) is a two-layer fully connected network with 256 hidden neurons and 256 output neurons with ReLU. g ₂ is a two-layer fully connected network with 256 hidden neurons with ReLU (Rectified Linear Unit) and 40 soft max output neurons.

Learning: The deep learning model according to the embodiment of the present invention has a learning rate of 0.0001 for 30 epochs for the purpose of minimizing cross entropy loss by Adam optimizer (Diederik P. Kingma and Jimmy Ba. Adam: a method for stochastic optimization.In Proceedings of the 3rd International Conference on Learning Representations, pages 1-15, 2015). The batch size is 32. The training procedure took about 10-12 hours using one NVIDIA K80 GPU. Test accuracy was measured using the network that showed the best validation accuracy during the training process. The experiment was repeated 5 times with different random seeds and the average performance was determined.

[Table 1]

Table 1 shows a deep learning model (a deterministic binarization method for graph sampling and three graph layers without a skip layer) according to an embodiment of the present invention and classification accuracy of the conventional method. Traditional classifier including k-nearest neighbor (k-NN) and random forest, ChebNet-based method in which the graph structure is determined by the physical distance between electrodes and signal entropy is used as a feature ( Soobeom Jang, Seong-Eun Moon, and Jong-Seok Lee.EEG-based video identification using graph signal modeling and graph convolutional neural network.In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3066-3070, 2018 ) were tested. . In addition, in the deep learning model according to an embodiment, a model excluding graph structure extraction (indicated as "GNN only" in Table 1) was also tested. From Table 1, it is clearly understood that the deep learning model according to an embodiment of the present invention performs much better than other methods. The deep learning model of the present invention has better performance than the ChebNet-based method, which indicates that data-driven graph and function extraction are effective. In addition, it can be understood that the graph structure is important for EEG data modeling because performance is greatly degraded if graph extraction is omitted (ie, in the case of “GNN only”) in the deep learning model of the present invention.

[Table 2]

Table 2 shows the accuracy of the model for various combinations of the graph sampling method, the number of graph layers (K), and the presence or absence of skip layers. The results show that the number of graph layers is the most important parameter. The single layer graphs considered in most existing studies are not sufficient, and it is very advantageous to model the interactions between separate graph layers and different types of regions. Of the three graph sampling methods, the deterministic binarization method showed the best performance except for the case of K=1, and the continuous sampling method showed excellent performance when the number of graph layers was large. In the probabilistic sampling method, it was found that the performance is limited by selecting only one edge of the graph layer. The presence of the skip layer only has a minor effect on performance, especially when the number of graph layers is large.

6A to 6D illustrate a raw signal and an extracted graph according to an embodiment of the present invention.

6A is a visualization of the raw signal for subject #1, and FIG. 6B is an extracted graph for subject #1. 6C is a visualization of the raw signal for subject #2, and FIG. 6D is an extracted graph for subject #2. Different colors represent different classes. The plots of FIGS. 2B and 2D are enlarged versions of the red boxed areas for better visualization.

The extracted graph structure was analyzed using the t-SNE technique (Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605, 2008). 6A to 6D compare the t-SNE visualization of the original EEG signal and adjacency matrices of all graph layers in graphs extracted for two subjects. Different colors here represent different classes. In Figs. 6A and 6C, different classes of raw signals are mixed, so it is not easy to distinguish them. On the other hand, in Figs. 6B and 6D, graphs of the same class are closely grouped, which greatly contributes to classification.

Since there is no basis for the graph structure, it is difficult to evaluate the accuracy of the extracted graph. Accordingly, as one method for evaluating the quality of the extracted graph, the consistency of repeated experiments was tested. That is, dissimilarity is measured for all pair-wise combinations between graph structures obtained from different repeated experiments. A low level of dissimilarity, i.e., a high level of consistency, between the obtained graph structures indicates that it is reliable and meaningful. The whole procedure for calculating the dissimilarity of two graph structures is summarized in [Algorithm 1]. In [Algorithm 1], the distance function (Dist) is a function that calculates the distance by taking the adjacency matrix of two graph layers as input. , In the examples, the sum of the absolute differences was adopted. There is no problem of isomorphism because the vertices are clearly identified by distinct EEG electrodes and differences can be calculated. The average value of the distances calculated in all repeat pairs was divided by the total number of possible edges and subtracted from 1 to obtain the final consistency score.

[Algorithm 1]

[Table 3]

[Table 3] shows the results of the consistency analysis as a percentage. Except for the case of K=1, a high level of consistency (approximately 75-90%) is observed under other conditions. This indicates that the graph extraction process works stably, and it can be expected that the extracted graph contains a meaningful data representation.

7 is a graph showing the names and locations of 32 EEG electrodes for graphical representation used in the present invention. An example of a graph structure to be described later is based on 32 EEG electrodes shown in FIG. 7.

8A to 8C are graphs showing examples of graph structures obtained according to an embodiment of the present invention.

In the case of the example in which the best result was obtained among the examples of the present invention (i.e., K=3, no skip layer, deterministic binarization), the graph structure obtained in order to understand the learned expression in terms of the emotional cognitive response is shown. Investigated. Since the obtained graph structure is different for each iteration, a representative graph structure including the edge that appears most frequently (top 10%) among several graphs obtained in the repeated experiment was obtained. 8A is a representative graph including the trunk line that is most frequently activated across all video stimuli. 8B and 8C correspond to a representative graph structure for a video stimulus having the highest valence and a video stimulus having the lowest incentive, respectively. The size of the vertex represents the input degree (in-degree). The out-degree is similar at all vertices, so the marking is omitted.

In the first layer (red) of Fig. 8A, strong activation towards the left temporal lobe is observed. The temporal lobe is involved in the processing of complex visual stimuli such as scenes. The amygdala, which plays an important role in emotional processing, is also located in the medial temporal lobe. Thus, the first layer represents a mental state exposed to emotional visual stimuli. Functional connectivity related to visual content and emotion processing in the first layer is also observed in FIGS. 8B and 8C. The first layer of FIG. 8C includes a large number of trunk lines entering the frontal and occipital lobes, respectively, associated with emotional processing and sensory processing of visual stimuli. In the case of the first layer of FIG. 8B, the connectivity is largely related to the content of the video stimulus. This video includes rhythmic dance, which is probably understood to contribute to the incoming connection to the central frontal region known to be involved in motion-related perception.

The second layer (green) of FIGS. 8B and 8C shows clearly distinguished patterns from each other. In Fig. 8b the right part of the front area receives more incoming connections than the left part, which is the opposite of Fig. 8c. In previous related research papers, it has been continuously reported that the asymmetry of the right and left frontal lobes is strongly related to induction. That is, one side of the forebrain is more activated than the other side in the emotional processing process, and the more activated side changes according to the polarity of the emotion, which is consistent with the observed pattern of the second layer of FIGS. 8B and 8C. In the second layer of FIG. 8A, incoming trunk lines are spread over the entire brain, which is thought to be a result of aggregation of patterns appearing in positive and negative induced video stimuli. Therefore, it can be inferred that the second layer has learned the emotional characteristics of the brain signals.

The third layer (blue) of FIGS. 8A to 8C shows relatively few connections, and seems to mainly complement other layers. In all cases, the frontal region of the brain receives a large number of connections. There are several incoming trunk lines in FIG. 8B in the frontal lobe, which is thought to be due to the same reason as in the case of the first layer (ie, motion related).

9 is a flowchart illustrating a connection learning method according to an embodiment of the present invention.

Referring to FIG. 9, the connectivity learning method according to an embodiment of the present invention may include a membership extraction step (S10), a graph sampling step (S20), a feature extraction step (S30), and a classification processing step (S40).

Membership extraction step (S10), graph sampling step (S20), feature extraction step (S30), and classification processing step (S40) are the membership extraction unit 10 of the connectivity learning device 1 described with reference to FIG. 1, graph sampling It may be performed by the unit 20, the feature extraction unit 30, and the classification processing unit 40, respectively, and a membership extraction step (S10), a graph sampling step (S20), a feature extraction step (S30), and a classification processing step ( For a description of S40), a description of the membership extracting unit 10, the graph sampling unit 20, the feature extracting unit 30, and the classification processing unit 40 may be referred to.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and that various substitutions, modifications, and changes are possible within the scope of the technical spirit of the present invention. It will be obvious to those who have knowledge.

Claims (16)

As a connectivity learning device for determining connectivity between a plurality of objects by learning a deep neural network and classifying an input signal based on the determined connectivity,
A membership extracting unit for extracting a membership indicating the existence of connectivity between the objects from the original signals obtained from the plurality of objects;
A graph sampling unit that samples the connectivity between the objects from the extracted membership and generates a graph structure expressing the connectivity between the objects in a graph;
A feature extraction unit for extracting a signal feature from the original signal; And
Classification processing unit for classifying an input signal into any one of a plurality of classes based on the generated graph structure and the extracted signal characteristics
Connectivity learning device comprising a. The method of claim 1,
Connectivity learning device having two or more types of connectivity between the objects. The method of claim 2,
The connectivity learning device for extracting the membership with respect to two or more connectivity layers representing different types of connectivity. The method of claim 3,
The graph sampling unit generates one graph layer for each connectivity layer, and the graph structure includes two or more graph layers. The method of claim 4,
The graph sampling unit generates the graph structure by deterministic binarization that samples connectivity between the objects based on whether a value indicated by the extracted membership is equal to or greater than a threshold value. The method of claim 1,
The raw signal or the input signal includes at least one of Functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and Functional Near Infrared Spectroscopy (fNIRS) acquired from a plurality of sensors. Connected learning device. The method of claim 1,
The feature extraction unit extracts a signal feature from the original signal by a convolution operation and a max pooling operation. The method of claim 7,
The device for learning connectivity, wherein the convolution operation is performed using an extended convolutional layer having a plurality of time intervals. As a connectivity learning method for determining connectivity between a plurality of objects by learning a deep neural network and classifying an input signal based on the determined connectivity,
A membership extraction step of extracting a membership indicating the existence of connectivity between the objects from the original signals obtained from the plurality of objects;
A graph sampling step of sampling the connectivity between the objects from the extracted membership and generating a graph structure expressing the connectivity between the objects in a graph;
A feature extraction step of extracting a signal feature from the original signal; And
Classification processing step of classifying the input signal into any one of a plurality of classes based on the generated graph structure and the extracted signal characteristic
Connectivity learning method comprising a. The method of claim 9,
The connectivity learning method having two or more types of connectivity between the objects. The method of claim 10,
In the membership extraction step, the membership is extracted for two or more connectivity layers representing different types of connectivity. The method of claim 11,
In the graph sampling step, one graph layer is generated for each connectivity layer, and the graph structure includes two or more graph layers. The method of claim 12,
In the graph sampling step, the graph structure is generated by deterministic binarization of sampling connectivity between the objects based on whether a value indicated by the extracted membership is equal to or greater than a threshold value. The method of claim 9,
The raw signal or the input signal includes at least one of Functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and Functional Near Infrared Spectroscopy (fNIRS) acquired from a plurality of sensors. How to learn connectedness. The method of claim 9,
In the feature extraction step, a signal feature is extracted from the original signal by a convolution operation and a max pooling operation. The method of claim 15,
The convolutional learning method is performed using an extended convolutional layer having a plurality of time intervals.

Images