WO2021060748A1

WO2021060748A1 - Connectivity learning device and connectivity learning method

Info

Publication number: WO2021060748A1
Application number: PCT/KR2020/012235
Authority: WO
Inventors: 이종석; 장수범
Original assignee: 연세대학교 산학협력단
Priority date: 2019-09-24
Filing date: 2020-09-10
Publication date: 2021-04-01
Also published as: KR102298709B1; KR20210035530A

Abstract

A connectivity learning device according to an embodiment of the present invention is a connectivity learning device for determining connectivity between multiple objects by deep neural network learning, and classifying an input signal on the basis of the determined connectivity. The device may comprise: a membership extraction unit for extracting membership representing the existence of connectivity between the multiple objects from a raw signal obtained from the objects; a graph sampling unit for sampling the connectivity between the objects from the extracted membership, and generating a graph structure expressing the connectivity between the objects as a graph; a feature extraction unit for extracting a signal feature from the raw signal; and a classification processing unit for classifying an input signal as one class among multiple classes on the basis of the generated graph structure and the extracted signal feature.

Description

Connectivity learning device and connectivity learning method

The present invention relates to connectivity learning, and more particularly, to a connectivity learning device and connectivity learning method for learning connectivity between a plurality of objects.

Analysis of brain activity through brain images such as electroencephalography (EEG) is important for understanding human mental states and thoughts. This analysis of brain activity is essential in a variety of applications, including the brain-computer interface, emotion recognition, and diagnosis of mental illness. The brain is made up of several functional areas, and activation patterns across several functional areas provide valuable information about the state of mind. Therefore, it is effective for brain signal analysis to study the relationship between regions represented by a pattern called functional connectivity (also referred to as'connectivity' for simplicity in this specification). Since brain regions are not in Euclidean space, graphs are the most natural and suitable data structure for connectivity.

However, how to measure the degree of connectivity, how to define an appropriate graph structure, and how to define an appropriate function for signals from other brain regions are still unresolved problems. These issues are usually determined manually based on prior knowledge. For example, a correlation or causal matrix between signals in different regions can be used as a measure of connectivity. And, a graph can be constructed by connecting a pair of brain regions showing a high connectivity value, and finally, the power or entropy of a signal can be used as a feature of each region (ie, a peak of the graph). However, solving these problems manually is not the best option. In fact, these issues apply equally to brain signaling data as well as other data related to graph structures such as social networks and chemicals.

The problem to be solved by the present invention is to provide a connectivity learning apparatus capable of performing signal classification with improved accuracy through direct learning based on a source signal even without any prior knowledge.

Another problem to be solved by the present invention is to provide a connectivity learning device capable of expressing the overall state of a network having interconnectivity between a plurality of objects rather than connectivity between one object pair.

Another problem to be solved by the present invention is to determine the connectivity between a plurality of objects even when information on how much interconnectivity exists between a plurality of objects, that is, an explicit loss function for a graph structure is not given. It is to provide a connected learning device that can do.

Another problem to be solved by the present invention is to provide a connectivity learning method performed by the connectivity learning device.

A connectivity learning device according to an embodiment of the present invention for solving the above problem is a connectivity learning device for determining connectivity between a plurality of objects by learning a deep neural network and classifying an input signal based on the determined connectivity. And a membership extracting unit for extracting a membership indicating the existence of connectivity between the objects from the original signals obtained from the plurality of objects; A graph sampling unit that samples the connectivity between the objects from the extracted membership and generates a graph structure expressing the connectivity between the objects in a graph; A feature extraction unit for extracting a signal feature from the original signal; And a classification processor for classifying the input signal into any one of a plurality of classes based on the generated graph structure and the extracted signal characteristic.

In one embodiment, the connectivity between the objects may have two or more types. In addition, the membership extracting unit may extract the membership for two or more connectivity layers representing different types of connectivity.

In an embodiment, the graph sampling unit may generate one graph layer for each connectivity layer, and the graph structure may include two or more graph layers. In addition, the graph sampling unit may generate the graph structure by deterministic binarization that samples connectivity between the objects based on whether a value indicated by the extracted membership is equal to or greater than a threshold value.

In one embodiment, the raw signal or the input signal is Functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and Functional Near Infrared Spectroscopy (fNIRS) acquired from a plurality of sensors. It may include one or more of.

In an embodiment, the feature extractor may extract signal features from the original signal through a convolution operation and a max pooling operation. In addition, the convolution operation may be performed using an extended convolutional layer having a plurality of time intervals.

A connectivity learning method according to another embodiment of the present invention for solving the above problem is a connectivity learning method for determining connectivity between a plurality of objects by learning a deep neural network and classifying an input signal based on the determined connectivity. And a membership extraction step of extracting membership indicating the existence of connectivity between the objects from the original signals obtained from the plurality of objects; A graph sampling step of sampling the connectivity between the objects from the extracted membership and generating a graph structure representing the connectivity between the objects in a graph; A feature extraction step of extracting a signal feature from the original signal; And a classification processing step of classifying the input signal into any one of a plurality of classes based on the generated graph structure and the extracted signal characteristic.

In one embodiment, the connectivity between the objects may have two or more types. In addition, in the membership extraction step, the membership may be extracted for two or more connectivity layers representing different types of connectivity.

In an embodiment, in the graph sampling step, one graph layer is generated for each connectivity layer, and the graph structure may include two or more graph layers. In addition, in the graph sampling step, the graph structure may be generated by deterministic binarization of sampling connectivity between the objects based on whether a value indicated by the extracted membership is equal to or greater than a threshold value.

In an embodiment, in the feature extraction step, a signal feature may be extracted from the original signal by a convolution operation and a max pooling operation. In addition, the convolution operation may be performed using an extended convolutional layer having a plurality of time intervals.

According to an embodiment of the present invention, a new deep learning model that automatically extracts graph structures and signal features representing a plurality of connectivity between a plurality of objects and performs signal classification using the extracted graph structures and signal features is used. By doing so, it is possible to perform signal classification with improved accuracy through direct learning based on the raw signal even without any prior knowledge, and to express the overall state of the network having interconnectivity between a plurality of objects, and the loss function is A connectivity learning apparatus and connectivity learning method capable of determining connectivity between a plurality of objects even when not given may be provided.

1 is a block diagram of an apparatus for learning connectivity according to an embodiment of the present invention.

2 is a conceptual diagram showing the configuration of the membership extraction unit 10 according to an embodiment of the present invention.

3 is a conceptual diagram showing the configuration of the graph sampling unit 20 according to an embodiment of the present invention.

4 is a conceptual diagram showing the configuration of a feature extraction unit 30 according to an embodiment of the present invention.

5 is a conceptual diagram showing the configuration of the classification processing unit 40 according to an embodiment of the present invention.

6A to 6D illustrate a raw signal and an extracted graph according to an embodiment of the present invention.

7 is a graph showing the names and locations of 32 EEG electrodes for graphical representation used in the present invention.

8A to 8C are graphs showing examples of graph structures obtained according to an embodiment of the present invention.

9 is a flowchart illustrating a connection learning method according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The embodiments of the present invention are provided to more completely describe the present invention to those of ordinary skill in the art, and the following examples may be modified in various other forms, and the scope of the present invention is as follows. It is not limited to the examples. Rather, these embodiments are provided to make the present disclosure more faithful and complete, and to fully convey the spirit of the present invention to those skilled in the art.

Hereinafter, embodiments of the present invention will be described with reference to the drawings schematically showing ideal embodiments of the present invention. In the drawings, for example, the size and shape of members may be exaggerated for convenience and clarity of description, and in actual implementation, variations of the illustrated shape may be expected. Accordingly, embodiments of the present invention should not be construed as being limited to the specific shape of the member or region shown herein.

The present invention can build a neural network model that classifies given signal data using a connectivity structure generated by a deep learning model. The given data is expressed as (X, y ), where

And X is the set of time series signals collected from N sensors (EEG electrodes in one embodiment), and y is the corresponding class label.

The estimated graph structure is a multi-layer graph having weights and directions without self-loop, and the graph structure G may be expressed by Equation 1.

[Equation 1]

V representing a vertex can be expressed by Equation 2, E representing the existence of an edge can be expressed by Equation 3, and W representing the weight can be expressed by Equation 4.

[Equation 2]

[Equation 3]

[Equation 4]

E ^k represents the existence of an edge between the pair of vertices in the k-th graph layer, and W ^k represents the weight of the edge between the pair of vertices.

Is assumed, and K is a hyper parameter that controls the number of graph layers.

1 is a block diagram of an apparatus 1 for learning connectivity according to an embodiment of the present invention.

Referring to FIG. 1, the connectivity learning apparatus 1 may include a membership extracting unit 10, a graph sampling unit 20, a feature extracting unit 30, and a classification processing unit 40. The connectivity learning apparatus 1 may determine connectivity between a plurality of objects by learning a deep neural network, and classify an input signal based on the determined connectivity.

The connectivity between the objects includes a causal relationship or a correlation, and may include two or more types. When the connectivity between objects includes two or more types, it may be represented by a plurality of connectivity layers, as described later.

The membership extracting unit 10 may extract a membership indicating the existence of connectivity between objects from a raw signal obtained from a plurality of objects, and output the extracted membership to the graph sampling unit 20. In an embodiment, the object may include a sensor or sensing element capable of sensing or detecting various signals.

The raw signal or input signal may include one or more of Functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and Functional Near Infrared Spectroscopy (fNIRS). The original signal or the input signal is not limited to a signal related to the brain, and may include an image or an audio signal related to the heart in another embodiment. In another embodiment, the original signal or the input signal may include a signal related to weather data such as an air volume or wind speed, or an image signal indicating vehicle traffic on a road.

When the connectivity between the objects is of two or more types, the membership extraction unit 10 may extract membership for each of two or more connectivity layers representing different types of connectivity. In addition, in this case, the graph sampling unit 20 may generate one graph layer for each connectivity layer.

The graph sampling unit 20 samples the connectivity between objects from the membership extracted by the membership extraction unit 10 to generate a graph structure expressing the connectivity between the objects in a graph, and classifies the generated graph structure into a classification processing unit ( 40).

In addition, the graph sampling unit 20 generates a graph structure by deterministic binarization that samples the connectivity between objects based on whether the value indicated by the membership extracted by the membership extraction unit 10 is equal to or greater than a threshold value. I can.

The feature extraction unit 30 may extract a signal feature from an original signal obtained from a plurality of objects and output the extracted signal feature to the classification processor 40. The feature extraction unit 30 may extract a signal feature from the original signal through a convolution operation and a max pooling operation. The convolution operation performed by the feature extraction unit 30 may be performed through extended convolutional layers having a plurality of time intervals.

The membership extraction unit 10 calculates potential membership from the input time series data. The potential membership (h _ij ) represents the probability of the existence of an edge from the vertex vi to the vertex vj for each graph layer, and can be calculated by Equations 5 and 6. The membership extraction unit 10 includes vertex-edge operation and edge-vertex operation (for'vertex-edge operation and edge-vertex operation, 2018 editions Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zeme. Co-authored'Neural Relational Inference for Interacting Systems' In Proceedings of the 35th International Conference on Machine Learning, pages 2678 to 2687, the disclosures of which are incorporated herein by reference in their entirety), and a full connection ( Use a fully-connected) network.

[Equation 5]

[Equation 6]

In Equation 5 and Equation 6

(For'exponential linear units', see'exponential linear unit: Fast and accurate deep network learning by exponential linear units (ELUs)' in the 2015 edition of Djork-Arnι Clevert, Thomas Unterthiner, and Sepp Hochreiter' In Proceedings of The 3rd International Conference on Learning Representations, pages 1 to 14, the disclosures of which are incorporated herein in their entirety by reference), and batch normalization (for'batch normalization', 2015 edition of Sergey Ioffe and Christian Szegedy co-author'Batch normalization: accelerating deep network training by reducing internal covariate shift' In Proceedings of the 32nd International Conference on Machine Learning, pages 448 to 456, the disclosures of which are disclosed in the present specification by reference. Included) is a fully connected network.

The graph sampling unit 20 may generate a graph structure from the layer membership information through probabilistic or deterministic sampling. In the present invention, three methods are considered for graph sampling: stochastic sampling (STO), deterministic thresholding (DET), and continuous sampling (CON).

The probabilistic sampling method probabilistically assigns a potential edge from vertex vi to vertex vj to one of the K graph layers. Since the sampled graph weights are discontinuous, the Gumbel-softmax reparameterization technique ('Categorical reparameterization with Gumbel-softmax' In Proceedings, 2017 edition of Eric Jang, Shixiang Gu, and Ben Poole) of the 5th International Conference on Learning Representations, pp. 1-12 and'The concrete distribution: a continuous relaxation of discrete random variables' In Proceedings of the 5th International Conference by Chris J. Maddison, Andriy Mnih, and Yee Whye Teh, 2017 edition. on Learning Representations, pages 1 to 20, the disclosure of which is incorporated herein in its entirety by reference) to provide continuous relaxation and to enable gradient calculations.

[Equation 7]

here,

Is a random vector, and each element of this random vector has an IID (Independent and Identically Distributed) attribute, and follows a standard Gumbel distribution.

Is the softmax temperature that controls the sampling smoothness. In this example, it was set to 0.5. unweighted edges from vi to vj (

) Can be obtained by Equation 8.

[Equation 8]

Since the estimated graph includes a plurality of graph layers, different types of connectivity information may be modeled for each graph layer. The probabilistic sampling method limits each edge to belong to only one graph layer, but the deterministic binarization method alleviates this limitation so that a pair of vertices has edges in multiple layers through binarization.

[Equation 9]

In Equation 9, r is a threshold value, and in the example, it is set to 0.5. Discrete variables can be differentiated during learning using the same continuous relaxation technique used in the probabilistic sampling method.

The previous two methods constitute an unweighted graph, but if the continuous sampling method is used, different degrees of connectivity can be maintained in different graph layers by using the edge weight as a continuous value. To do this, we have a continuous value between 0 and 1 obtained from the Gumbel-softmax operation.

Is used directly as the edge weight from vi to vj in the k-th graph layer. Therefore, this method creates the most common type of graph structure among the three methods, that is, a multi-layer graph with weights and directions.

There may be no direct relationship between a particular pair of vertices. To enable this, one of the graph layers may be assigned as a skip layer. Since the skip layer is discarded when the graph is transmitted to the classification processing unit 40, the trunk line belonging to this layer is omitted from the graph used for classification.

The feature extraction unit 30 may extract signal features from the original signal by 1-D convolution and max pooling operations. In order to capture the dynamic information of the signal, the present invention adopts the 1-D version of the extended inception module including convolutional layers with various dilation rates. The extended convolution layer with a low expansion rate captures features that appear between surrounding samples corresponding to fast-changing high-frequency information, and the extended convolution layer with a high expansion rate captures features that change slowly for a large temporal window. .

[Equation 10]

The signal feature U _i can be calculated by Equation 10. Here, T'is the reduced signal length, and F is the feature dimension.

The classification processing unit 40 includes a graph neural network (GNN), and class labels predicted by performing classification using signal features and a built graph

Can be printed. First, a vertex-edge operation is performed on signal features, and the result is combined with a graph structure through a message passing operation, and an aggregation operation and an edge-vertex operation are performed.

[Equation 11]

In Equation 11, t is a time index, s=2 when a skip layer is used, and s=1 when a skip layer is not used.

Is the input and output are F-dimensional and F'-dimensional, respectively

It is modeled by a fully connected network with ReLU (Rectified Linear Unit). Finally, this result is linked to the signal features via a skip connection, vectorized and provided to a fully connected network.

[Equation 12]

(here

to be.)

Hereinafter, specific embodiments of the present invention will be described.

In this embodiment, one of the largest databases on the emotional and mental state of humans, the Database for Emotion Analysis using Physiological Signals (DEAP) data set was used. The data set of the example includes 32 channel EEG records collected from 32 subjects while watching 40 video stimuli and the corresponding emotion rating by the subject. In the embodiment, a video identification task was performed to classify a set of EEG signals given by a deep learning model into one of 40 video stimuli.

Specifically, each 1 minute long EEG signal of the DEAP data set is divided into 3 second long segments overlapping 2 seconds long (T=384). As a result, a total of 74,240 32-channel EEG signal sets (32 subjects x 40 videos x 58 segments) are obtained. The obtained signal sets are randomly separated into training, validation and test data sets holding 80%, 10% and 10% of the total data set, respectively.

The deep learning model of this example is PyTorch (Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer co-authored'Automatic differentiation in PyTorch In Proceedings of the NIPS 2017 Autodiff Workshop: The Future of Gradient-based Machine Learning Software and Techniques', pages 1 to 4, the disclosures of which are incorporated herein by reference in their entirety) Adam optimizer ('Adam optimizer', 2015 edition of Diederik P. Kingma and Jimmy Ba co-author'Adam: a method for stochastic optimization' In Proceedings of the 3rd International Conference on Learning Representations, published on pages 1 to 15 And the corresponding disclosure has been learned using (the whole is incorporated herein by reference).

Details of the model structure and learning parameters will be described below.

Graph membership extraction: f1, f2 and f3 are fully connected two-layer networks with 256 hidden neurons and 256 output neurons. Each fully connected layer has an exponential linear unit, and batch normalization is used for the output layer. f4 contains 3 fully connected layers. Among the three layers, there are 256 hidden neurons with exponential linear units in the first two layers, and K output neurons in the last layer.

Feature Extraction: Each extended inception module consists of four 1-D convolutional layers with a kernel size of 3, each with a scaling factor of 1, 2, 4 and 8. Each layer has 8 output channels, so F = 8 × 4 = 32. The maximum pooling size is set to α = 4. Three extended start modules are connected in a line so that the length of the signal extension is T'=6.

Neural network graph:

( k = 1,..., K) is a two-layer fully connected network with 256 hidden neurons and 256 output neurons with ReLU. g ₂ is a two-layer fully connected network with 256 hidden neurons with ReLU (Rectified Linear Unit) and 40 soft max output neurons.

Learning: The deep learning model according to the embodiment of the present invention has a learning rate of 0.0001 for 30 epochs for the purpose of minimizing cross entropy loss. Diederik P. Kingma and Jimmy Ba. Adam: a method for stochastic optimization', In Proceedings of the 3rd International Conference on Learning Representations, pages 1 to 15, the disclosures of which are incorporated herein by reference in their entirety It was learned using). The batch size is 32. The training procedure took about 10-12 hours using one NVIDIA K80 GPU. Test accuracy was measured using the network that showed the best validation accuracy during the training process. The experiment was repeated 5 times with different random seeds and the average performance was determined.

	k-NNk-NN	Random forestRandom forest	ChebNetChebNet	GNN onlyGNN only	본 발명The present invention
AccuracyAccuracy	48.50%48.50%	51.34%51.34%	65.27%65.27%	44.70%44.70%	91.23%91.23%

Table 1 shows a deep learning model (a deterministic binarization method for graph sampling and three graph layers without a skip layer) according to an embodiment of the present invention and classification accuracy of the conventional method. Traditional classifier including k-nearest neighbor (k-NN) and random forest, ChebNet-based method in which the graph structure is determined by the physical distance between electrodes and signal entropy is used as a feature ( Soobeom Jang, Seong-Eun Moon, and Jong-Seok Lee.EEG-based video identification using graph signal modeling and graph convolutional neural network.In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3066-3070, 2018 ) were tested. . In addition, in the deep learning model according to an embodiment, a model excluding graph structure extraction (indicated as "GNN only" in Table 1) was also tested. From Table 1, it is clearly understood that the deep learning model according to an embodiment of the present invention performs much better than other methods. The deep learning model of the present invention has better performance than the ChebNet-based method, which indicates that data-driven graph and function extraction are effective. In addition, it can be understood that the graph structure is important for EEG data modeling because performance is greatly degraded if graph extraction is omitted (ie, in the case of “GNN only”) in the deep learning model of the present invention.

#Layers(K)#Layers(K)	1+skip1+skip	22	2+skip2+skip	33	3+skip3+skip
STOSTO	69.28%69.28%	73.98%73.98%	76.03%76.03%	86.86%86.86%	86.65%86.65%
DETDET	55.91%55.91%	86.61%86.61%	83.14%83.14%	91.23%91.23%	91.04%91.04%
CONCON	58.31%58.31%	76.29%76.29%	77.84%77.84%	90.08%90.08%	89.43%89.43%

Table 2 shows the accuracy of the model for various combinations of the graph sampling method, the number of graph layers (K), and the presence or absence of skip layers. The results show that the number of graph layers is the most important parameter. The single layer graphs considered in most existing studies are not sufficient, and it is very advantageous to model the interactions between separate graph layers and different types of regions. Of the three graph sampling methods, the deterministic binarization method showed the best performance except for the case of K=1, and the continuous sampling method showed excellent performance when the number of graph layers was large. In the probabilistic sampling method, it was found that the performance is limited by selecting only one edge of the graph layer. The presence of the skip layer only has a minor effect on performance, especially when the number of graph layers is large.

6A to 6D illustrate a raw signal and an extracted graph according to an embodiment of the present invention. In FIGS. 6A to 6D, the color color may refer to the color drawing of the original application.

6A is a visualization of the raw signal for subject #1, and FIG. 6B is an extracted graph for subject #1. 6C is a visualization of the raw signal for subject #2, and FIG. 6D is an extracted graph for subject #2. Different colors represent different classes. The plots of FIGS. 2B and 2D are enlarged versions of the areas marked with red boxes B0 and B1 for better visualization.

The extracted graph structure was analyzed using the t-SNE technique (Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605, 2008). 6A to 6D compare the t-SNE visualization of the original EEG signal and adjacency matrices of all graph layers in graphs extracted for two subjects. Different colors here represent different classes. In Figs. 6A and 6C, different classes of raw signals are mixed, so it is not easy to distinguish them. On the other hand, in Figs. 6B and 6D, graphs of the same class are closely grouped, which greatly contributes to classification.

Since there is no basis for the graph structure, it is difficult to evaluate the accuracy of the extracted graph. Accordingly, as one method for evaluating the quality of the extracted graph, the consistency of repeated experiments was tested. That is, dissimilarity is measured for all pair-wise combinations between graph structures obtained from different repeated experiments. A low level of dissimilarity, i.e., a high level of consistency, between the obtained graph structures indicates that it is reliable and meaningful. The whole procedure for calculating the dissimilarity of two graph structures is summarized in [Algorithm 1]. In [Algorithm 1], the distance function (Dist) is a function that calculates the distance by taking the adjacency matrix of two graph layers as input. , In the examples, the sum of the absolute differences was adopted. There is no problem of isomorphism because the vertices are clearly identified by distinct EEG electrodes and differences can be calculated. The average value of the distances calculated in all repeat pairs was divided by the total number of possible edges and subtracted from 1 to obtain the final consistency score.

[Algorithm 1]

Algorithm 1 Computing graph dissimilarity

Input: Dist(·,·),

,

Output: D*, P*

1: M = GetPerm(1,...,K)

▷ Make a set of permutations in the lexicographic order

2: P* ← (1,...,K), D* ←

▷ Initial permutation and distance

3: for P in M do ▷ Pick a permutation of (1,...,K)

4:

▷ Permute the graph layers in W _(m) with P

5:

6: if D <D* then

7: D* ← D

8: P* ← P

9: end if

10: end for

#Layers(K)#Layers(K)	1+skip1+skip	22	2+skip2+skip	33	3+skip3+skip
STOSTO	43.91%43.91%	89.32%89.32%	89.67%89.67%	83.41%83.41%	79.10%79.10%
DETDET	61.23%61.23%	88.01%88.01%	85.31%85.31%	77.79%77.79%	77.66%77.66%
CONCON	55.87%55.87%	84.01%84.01%	76.58%76.58%	81.64%81.64%	82.11%82.11%

[Table 3] shows the results of the consistency analysis as a percentage. Except for the case of K=1, a high level of consistency (approximately 75-90%) is observed under other conditions. This indicates that the graph extraction process works stably, and it can be expected that the extracted graph contains a meaningful data representation.

7 is a graph showing the names and locations of 32 EEG electrodes for graphical representation used in the present invention. An example of a graph structure to be described later is based on 32 EEG electrodes shown in FIG. 7.

In the case of the example in which the best result was obtained among the examples of the present invention (i.e., K=3, no skip layer, deterministic binarization), the graph structure obtained in order to understand the learned expression in terms of the emotional cognitive response is shown. Investigated. Since the obtained graph structure is different for each iteration, a representative graph structure including the edge that appears most frequently (top 10%) among several graphs obtained in the repeated experiment was obtained. 8A is a representative graph including the trunk line that is most frequently activated across all video stimuli. 8B and 8C correspond to a representative graph structure for a video stimulus having the highest valence and a video stimulus having the lowest incentive, respectively. The size of the vertex represents the input degree (in-degree). The out-degree is similar at all vertices, so the marking is omitted.

In the first layer (red, L1) of Fig. 8A, strong activation towards the left temporal lobe is observed. The temporal lobe is involved in the processing of complex visual stimuli such as scenes. The amygdala, which plays an important role in emotional processing, is also located in the medial temporal lobe. Accordingly, the first layer L1 represents a mental state exposed to an emotional visual stimulus. Functional connectivity related to visual content and emotion processing in the first layer is also observed in FIGS. 8B and 8C. The first layer L1 of FIG. 8C includes a large number of trunk lines entering the frontal and occipital lobes, respectively, associated with emotional processing and sensory processing of visual stimuli. In the case of the first layer L1 of FIG. 8B, the connectivity is largely related to the content of the video stimulus. This video includes rhythmic dance, which is probably understood to contribute to the incoming connection to the central frontal region known to be involved in motion-related perception.

The second layers (green, L2) of FIGS. 8B and 8C show patterns that are clearly distinguished from each other. In Fig. 8b the right part of the front area receives more incoming connections than the left part, which is the opposite of Fig. 8c. In previous related research papers, it has been continuously reported that the asymmetry of the right and left frontal lobes is strongly related to induction. That is, one side of the forebrain is more activated than the other side in the emotional processing process, and the more activated side changes according to the polarity of the emotion, which is consistent with the observed pattern of the second layer (L2) of FIGS. 8B and 8C. . In the second layer (L2) of FIG. 8A, incoming trunk lines are spread over the entire brain, which is thought to be a result of aggregation of patterns appearing in positive and negative induced video stimuli. Accordingly, it can be inferred that the second layer L2 has learned the emotional characteristics of the brain signals.

The third layer (blue, L3) of FIGS. 8A to 8C shows relatively few connections, and seems to mainly complement other layers. In all cases, the frontal region of the brain receives a large number of connections. There are several incoming trunk lines in FIG. 8B in the frontal lobe, which is thought to be due to the same reason as in the case of the first layer (ie, motion related).

Referring to FIG. 9, the connectivity learning method according to an embodiment of the present invention may include a membership extraction step (S10), a graph sampling step (S20), a feature extraction step (S30), and a classification processing step (S40).

Membership extraction step (S10), graph sampling step (S20), feature extraction step (S30), and classification processing step (S40) are the membership extraction unit 10 of the connectivity learning device 1 described with reference to FIG. 1, graph sampling It may be performed by the unit 20, the feature extraction unit 30, and the classification processing unit 40, respectively, and a membership extraction step (S10), a graph sampling step (S20), a feature extraction step (S30), and a classification processing step ( For a description of S40), a description of the membership extracting unit 10, the graph sampling unit 20, the feature extracting unit 30, and the classification processing unit 40 may be referred to.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and that various substitutions, modifications, and changes are possible within the scope of the technical spirit of the present invention. It will be obvious to those who have knowledge.

Claims

As a connectivity learning device for determining connectivity between a plurality of objects by learning a deep neural network and classifying an input signal based on the determined connectivity,

A membership extracting unit for extracting a membership indicating the existence of connectivity between the objects from the original signals obtained from the plurality of objects;

A graph sampling unit that samples the connectivity between the objects from the extracted membership and generates a graph structure expressing the connectivity between the objects in a graph;

A feature extraction unit for extracting a signal feature from the original signal; And

Classification processing unit for classifying an input signal into any one of a plurality of classes based on the generated graph structure and the extracted signal characteristics

Connectivity learning device comprising a.
The method of claim 1,

Connectivity learning device having two or more types of connectivity between the objects.
The method of claim 2,

The connectivity learning device for extracting the membership with respect to two or more connectivity layers representing different types of connectivity.
The method of claim 3,

The graph sampling unit generates one graph layer for each connectivity layer, and the graph structure includes two or more graph layers.
The method of claim 4,

The graph sampling unit generates the graph structure by deterministic binarization that samples connectivity between the objects based on whether a value indicated by the extracted membership is equal to or greater than a threshold value.
The method of claim 1,

The raw signal or the input signal includes at least one of Functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and Functional Near Infrared Spectroscopy (fNIRS) acquired from a plurality of sensors. Connected learning device.
The method of claim 1,

The feature extraction unit extracts a signal feature from the original signal by a convolution operation and a max pooling operation.
The method of claim 7,

The device for learning connectivity, wherein the convolution operation is performed using an extended convolutional layer having a plurality of time intervals.
As a connectivity learning method for determining connectivity between a plurality of objects by learning a deep neural network and classifying an input signal based on the determined connectivity,

A membership extraction step of extracting membership indicating the existence of connectivity between the objects from the original signals obtained from the plurality of objects;

A graph sampling step of sampling the connectivity between the objects from the extracted membership and generating a graph structure representing the connectivity between the objects in a graph;

A feature extraction step of extracting a signal feature from the original signal; And

Classification processing step of classifying the input signal into any one of a plurality of classes based on the generated graph structure and the extracted signal characteristic

Connectivity learning method comprising a.
The method of claim 9,

The connectivity learning method having two or more types of connectivity between the objects.
The method of claim 10,

In the membership extraction step, the membership is extracted for two or more connectivity layers representing different types of connectivity.
The method of claim 11,

In the graph sampling step, one graph layer is generated for each connectivity layer, and the graph structure includes two or more graph layers.
The method of claim 12,

In the graph sampling step, the graph structure is generated by deterministic binarization of sampling connectivity between the objects based on whether a value indicated by the extracted membership is equal to or greater than a threshold value.
The method of claim 9,

The raw signal or the input signal includes at least one of Functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and Functional Near Infrared Spectroscopy (fNIRS) acquired from a plurality of sensors. How to learn connectedness.
The method of claim 9,

In the feature extraction step, a signal feature is extracted from the original signal by a convolution operation and a max pooling operation.
The method of claim 15,

The convolutional learning method is performed using an extended convolutional layer having a plurality of time intervals.