JP2020027399A - Computer system - Google Patents

Abstract

To perform pooling processing in a convolution operation on a graph.SOLUTION: A computer system executes a convolution neural network on a graph. The convolution neural network on the graph includes one or more convolution layers and one or more pooling layers. One or more processors update a value of each node by a convolution operation based on a kernel having the size of a prescribed hop number in each convolution layer, and update a value of each node by pooling processing based on the value of each node and a node within the pooling range of the prescribed hop number from each node in each pooling layer. The size of a kernel of a convolution layer of a subsequent stage of the pooling layer is larger than the size of a kernel of a convolution layer of a front stage.SELECTED DRAWING: Figure 9

Description

  The present disclosure relates to a computer system.

  To efficiently design and operate social infrastructure and cities, etc., process data in the real world and cyber space, analyze or predict the state of social infrastructure, etc., or control or guide social infrastructure, etc. Technology is attracting attention. The data to be processed includes sensing data of the environment such as temperature and humidity, log data of a machine such as an automobile, and log data of a human or an organization such as mail and SNS.

  In recent years, machine learning using a neural network has attracted attention as a data analysis technique. The neural network can easily realize high-speed and accurate data analysis. Various types of data, including system data as described above, are graph data that can be represented by a graph structure.

  For example, WO 2016/174725 (Patent Document 1) discloses a technique capable of constructing a neural network of graph data. Specifically, Patent Literature 1 is a computer that executes an arithmetic process using a neural network including a plurality of layers including one or more neurons, and connects a plurality of nodes and a plurality of nodes. A storage unit that stores graph data composed of one or more edges, and sample data that stores one or more values input to the neural network, and a construction unit that constructs a neural network using the graph data. The construction unit generates one or more neurons of each of a plurality of layers based on the plurality of nodes included in the graph data, and based on one or more edges included in the graph data, Disclose constructing a neural network by creating connections between one or more neurons in each of a plurality of layers For example, summary). Non-Patent Document 1 discloses an example of a relational graph convolution network.

International Publication No. 2016/174725

M. Schlichtkrull et al., “Modeling Relational Data with Graph Convolutional Networks”, arXiv preprint arXiv: 1703.06103, 2017.

  Among various types of neural networks, a convolutional neural network (CNN) is known as a neural network particularly useful for image processing. The graph data has a relationship, and it is useful if the graph data can be analyzed by the CNN.

  In order to realize so-called deep learning in which the CNN is configured by a deep network, it is necessary to perform pooling or, more generally, an operation of reducing the degree of freedom of the parameter toward an upper layer. If a large number of convolutional layers are superimposed without a pooling layer, only the number of parameters increases without reducing the degree of freedom that can be expressed (no regularization is applied), which causes over-learning when learning the parameters.

  In the conventional CNN of grid data (CNN on a grid), it is common to reduce the image size by an operation called pooling after a convolution operation (and application of a nonlinear activation function). However, it is difficult to realize the pooling operation in the CNN of the graph data.

  If the pooling operation in the grid data is simply made to correspond to the graph data, it is considered that nodes of the graph data should be appropriately deleted to reduce the size of the graph data. However, it is not clear how to select a node to be deleted and a node to be left.

  Therefore, a technique for implementing pooling that reduces the degree of freedom of parameters in a CNN on a graph is desired.

  One aspect of the present disclosure is a computer system that executes a convolutional neural network on a graph, the computer system including one or more processors and one or more storage devices, wherein the convolutional neural network on the graph includes one or more And one or more pooling layers, the one or more storage devices store kernel weight data for the one or more convolution layers, and the one or more processors, at each convolution layer, The value of each node is updated by a convolution operation based on a kernel having a size of a predetermined number of hops, and in each pooling layer, the value of each node is set within a pooling range of a predetermined number of hops from the value of each node and each node. Updated by the pooling process based on the node value, the kernel size of the convolutional layer after the pooling layer is Larger than the size of the kernel of the convolution layer.

  According to an embodiment of the present disclosure, pooling processing can be appropriately performed in a convolution operation on a graph.

1 shows an example of the configuration of a computer system. 1 shows an example of a neural network. 4 shows an example of a convolutional neural network. 4 shows a configuration example of a CNN on a graph. An example of graph data is shown. One node and a node connected to the node are shown. It is a part of the weight data and indicates a value related to the convolution of the node. 5 schematically illustrates updating of a state value of a node. 3 shows details of a first convolutional layer, a first pooling layer, a second convolutional layer and a second pooling layer of CNN on a graph. 4 shows an example of graph data input to a CNN on a graph. 4 shows an example of a convolution operation of a first convolution layer. The example of the attribute of graph data is shown. 3 shows a part of the weight data of the present example. The update of the state value of the edge by the first convolution layer is shown. 4 schematically shows a pooling process by a first pooling layer. 16 illustrates an example of an applicable Laplacian row example in graph data having the graph structure illustrated in FIG. 15. 4 schematically illustrates a convolution operation for a node by a second convolution layer. It is a flowchart explaining the learning process which a computer performs. 1 shows an example of a computer system that executes a distributed CNN. 3 shows examples of layers and columns of dispersed CNN. 4 shows a conceptual configuration example of a distributed CNN. 5 shows a propagation rule of an application node of the function f. Shows the propagation rule for countable nodes. 3 shows a propagation rule of a multiplication node. The propagation rule of a variable replication node (branch node) is shown. A configuration example of a distributed CNN and its processing will be described.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that this embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention. In each of the drawings, common components are denoted by the same reference numerals.

[System configuration]
FIG. 1 shows an example of the configuration of a computer system. As shown in FIG. 1, the computer system 100 includes a plurality of computers 101 and a storage system 102, and the plurality of computers 101 and the storage system 102 are connected via a network 103 so that they can communicate with each other. . The computer system 100 of the present embodiment includes three computers 101. In FIG. 1, only one computer is indicated by reference numeral 100 as an example. The number of computers and storage systems is arbitrary. The computer system 100 may be composed of one computer 101.

  The network 103 may be a WAN (Wide Area Network), a LAN (Local Area Network), a SAN (Storage Area Network), or the like. Note that the type of the network 103 is not limited. The network connecting each computer 101 to the storage system 102 and the network connecting each computer 101 may be different networks.

  The computer 101 includes a processor 110, a memory 111, a communication interface 112, and an input / output interface 118, and each component is connected to each other via a bus 114. The input / output device 119 is connected to another device via the input / output interface 118. The processor 110 includes one or more CPUs 115 that execute arithmetic processing. In FIG. 1, only one CPU is indicated by reference numeral 115. The CPU 115 realizes the functions of the computer 101 by executing a program stored in the memory 111. Further, processing executed on the computer 101 is executed by one or more CPUs 115. Note that one CPU 115 may execute a plurality of processes. Note that the CPU 115 may be an arithmetic unit such as an FPGA or a GPU.

  The memory 111 stores a program executed by the processor 110 and information used by the program. Further, the memory 111 includes a memory space allocated to one process executed by the processor 110. The memory space may be secured on a memory area of a plurality of memories 111, or may be secured on a memory area of one memory 111. Further, the memory 111 may include a memory space for a plurality of processes. The programs and information stored in the memory 111 will be described later.

  The communication interface 112 communicates with an external device via the network 103. The processor 110 accesses another device via the communication interface 112. The input / output interface 118 mediates communication between another device and the input / output device 119 via the network 103.

  The storage system 102 includes a disk interface 133 and a plurality of storage devices 117, and each device is connected to each other via a bus 135. The disk interface 133 is an interface for connecting to a plurality of storage drives 134. The storage drive 134 is a storage device that stores various data, and is, for example, an HDD or an SSD. In FIG. 1, only one storage drive is designated by reference numeral 134.

  The memory 111 stores a program that implements the data processing unit 120. The processor 110 that executes the data processing unit 120 performs a learning process by backward error propagation and an analysis process by forward propagation. In the learning process, the processor 110 uses the learning data 122 to determine the weight of an edge in the constructed neural network. In the analysis process, the processor 110 performs predetermined analysis such as classification and regression by inputting data to be analyzed into the constructed neural network.

  Note that the data processing unit 120 may be composed of a plurality of program modules. For example, the data processing unit 120 may include a construction module that constructs a neural network, a learning processing module that performs a learning process, and an analysis processing module that performs an analysis process. Further, each computer 101 may execute each program module.

[neural network]
FIG. 2 shows an example of a neural network. The neural network 200 shown in FIG. 2 includes three layers: an input layer 201, an intermediate layer 202, and an output layer 203. Each layer is composed of one or more neurons 211. Further, the neurons 211 in each layer are connected to at least one neuron 211 in another layer. Specifically, the neurons 211 of the input layer 201 are connected to at least one neuron 211 of the intermediate layer 202, and the neurons 211 of the intermediate layer 202 are connected to at least one neuron 211 of the output layer 203. The side of the input layer 201 is the former stage, and the side of the output layer 203 is the latter stage.

  Edge 212 represents the output of data between neurons 211. In the analysis processing, the neural network 200 sequentially propagates input data from the input layer 201 to the output layer 203 via the intermediate layer 202. The value input to each neuron 211 is multiplied by the weight assigned to the edge, and output to the neuron 211 of another layer connected by the edge 212. The input to input 211 may include a bias. The weight indicates the strength of the connection between the neurons 211 and is determined by a learning process described later. In the learning process, the weight of each layer is updated by back propagation of the loss signal.

  Returning to FIG. 1, the memory 111 stores the graph data 121, the learning data 122, and the weight data 123. The graph data 121 is data having a graph structure including nodes corresponding to arbitrary elements and edges connecting the nodes. The learning data 122 is data used in a learning process using a neural network. The weight data 123 is information for managing a weight that is a processing result of the learning processing of the neural network.

  The graph data 121 and the learning data 122 are stored in, for example, the storage system 102. The processor 110 acquires the graph data 121 and the learning data 122 from the storage system 102, and converts the acquired graph data 121 and learning data 122. The data is loaded into the memory 111.

[CNN on grid]
FIG. 3 shows an example of a convolutional neural network (CNN). The computer system 100 of the present disclosure executes a CNN on a graph. Referring to FIG. 3, a typical CNN 300 on a grid will be described. The CNN on the grid processes input data in a matrix (grid) such as image data.

  In the example of FIG. 3, an image 301 of a 28 × 28 gray scale handwritten numeral is input. The CNN 300 applies a 5x5 convolution filter and a ReLU function (302) to generate six 24x24 feature maps 303. The CNN 300 applies 2 × 2 maximum pooling to these feature maps (304) to generate a feature map 305 with the image size reduced by half.

  The CNN 300 further executes a 5x5 convolution filter and a ReLU function (306) to generate a 10 8x8 feature map 307. The CNN 300 applies 2 × 2 maximum pooling to these feature maps (308) to generate a feature map 309 with the image size reduced by half. After that, the perceptron 310 including the two fully connected layers finally outputs a 10-dimensional vector. The output 10-dimensional vector is converted into a one-hot vector expression by a SoftMax function, and represents the likelihood of the input image being a number from 0 to 9.

[CNN on graph (forward propagation)]
Hereinafter, the CNN on the graph of the present disclosure will be described. The computer system 100 executes the CNN on the graph. FIG. 4 shows a configuration example 400 of the CNN on the graph. FIG. 4 shows forward propagation (analysis processing) in the CNN 400, which is executed by the data processing unit 120. The CNN 400 on the graph processes data on a common graph structure in each layer.

  The graph structure includes a plurality of nodes and a plurality of edges each connecting two nodes. The grid structure is one of graph structures, and each node is connected to an adjacent node by an edge. The graph structure according to the present disclosure includes an arbitrary plurality of nodes and an arbitrary number of edges. One node is connected to one or more arbitrary number of edges.

  The graph data is data having a graph structure, and a state value is assigned to each node. State values can be represented by one or more dimensional vectors. As shown in FIG. 4, the graph data 121 is input to the CNN 400 on the graph. As described above, each node of the graph data 121 has a state value, and an edge between nodes is defined.

  The CNN 400 of FIG. 4 includes two convolutional layers and two pooling layers, like the CNN 300 on the grid described with reference to FIG. Specifically, the CNN 400 includes a first convolution layer 411, a first pooling layer 412, a second convolution layer 413, a second pooling layer 414, and a Global Average Pooling (GAP) layer 415. The first convolution layer 411 performs a convolution operation on the input graph data 121 on the graph structure of the graph data 121. As a result, two feature maps 402 are generated.

  On the graph structure of the first pooling layer 412 and the graph data 121, pooling processing is performed on each of the feature maps 402, and the feature maps 402 are converted into feature maps 403 with reduced degrees of freedom in expression. The second convolution layer 413 performs a convolution operation on the feature map 403 on the graph structure of the graph data 121. Thereby, three feature maps 404 are generated.

  On the graph structure of the second pooling layer 414 and the graph data 121, pooling processing is performed on each of the feature maps 404, and the feature maps 404 are converted into the feature maps 405 with reduced degrees of freedom in expression. The feature map 405 is input to the global average pooling layer 415. The global average pooling layer 415 averages the values of all neurons for each channel.

  Although details of the processing of the convolutional layer and the pooling layer will be described later, the convolutional layer and the pooling layer respectively update the state value of each node of the graph data. Therefore, the graph structure of the data input to each layer is maintained in the output data. Specifically, the graph structures of the input graph data 121 and the feature maps 402 to 405 are common. In each layer, there is a neuron corresponding to each node, and the neuron outputs a new value of the corresponding node. In another example, the graph structure of the input data in one or more layers may be different from the graph structure of the output data.

  FIG. 5 shows an example of the graph data 121. The graph data 121 shown in FIG. 5 includes edge structure information 500 and node structure information 510. The edge structure information 500 includes an edge ID 501, a node ID 502, and an edge attribute 503. The edge ID 501 indicates the ID of each edge of the graph data. The node ID 502 indicates the ID of each node of the graph data. The edge attribute 503 indicates attribute information assigned to each edge. The edge attribute 503 includes information on the direction and type of the edge.

  The node structure information 510 includes a node ID 511, an edge ID (OUT) 512, an edge ID (IN) 513, and a node attribute 514. The node ID 511 is the same as the node ID 502. The edge ID (OUT) 512 indicates an ID of an edge flowing out of the node. The node is the start node of the edge indicated by the edge ID (OUT) 512. The edge ID (OUT) 513 indicates an ID of an edge flowing into the node indicated by the node ID 511. The node is the end point of the edge indicated by the edge ID (IN) 513. The node attribute 514 indicates attribute information of the node. The node attribute 514 includes information on the type of the node.

  Various types of information can be represented by graph data, for example, map information, molecular structure information, social network information, transportation network information, and the like can be represented by graph data. For example, the position of the map is represented by a node, and the road is represented by an edge. The attribute information of the node representing the land includes a real estate value, a use form of the land such as a station or a park, and the like. The attribute information of an edge representing a road includes a line name, a road width, an attribute of a land (node) to be connected, a direction, and the like. An attribute type is defined for each of the land and the road according to the attribute information of the land and the road. For example, it is possible to estimate the real estate value of a given land by learning the graph CNN using attribute information representing the existing real estate value as teacher data.

  The convolution operation on a graph can be roughly classified into a method using a graph Fourier transform and a more direct method called Relational Graph Convolution Network (R-GCN). Among them, the method using the graph Fourier transform has an advantage that the theoretical background is clear, but has a disadvantage that it can be applied only to a simple undirected graph without overlapping edges.

  On the other hand, the R-GCN is an intuitive extension of the convolution operation on the grid, and can be naturally defined even when a directed graph edge or, more generally, an attribute is given to a node or an edge. is there. Although the features of the present disclosure can be applied to any of the methods, an example of the R-GCN will be described below.

[Example of R-GCN]
A known example of the R-GCN will be described with reference to FIGS. The convolution operation updates the state value of the node based on the state values of the node and other surrounding nodes. Hereinafter, an example of a process of updating the state value of one node will be described. FIG. 6 shows one node 600 and nodes 611 to 614 and 621 to 623 connected to the node 600. The edge connecting each of the node 600 and the nodes 611 to 614 has an attribute R1. An edge connecting each of the node 600 and the nodes 621 to 623 has an attribute R2.

  FIG. 7 shows a part of the weight data 123, which is a value related to the convolution of the node 600. The weight data 123 includes an edge attribute type 601 and a weight 602. The edge attribute type 601 indicates an identifier of the type of the edge attribute. In this example, the edge attribute type 601 is different from the type indicated by the edge attribute 503 of the graph data 121. These may be the same. The weight 602 indicates the weight of each edge attribute type. The value indicated by the weight 602 is updated in learning (training) of the CNN 400. The bias of the convolution operation of the state value of the node 600 in this example is set to 0.

  As shown in FIGS. 6 and 7, the attribute type of the edge is defined by the attribute of the edge and the direction of the edge with respect to the node 600. Specifically, the edge attribute type of the edge connecting the nodes 612 to 614 and the node 600 is R1 (IN). The edge attribute type of the edge connecting the node 611 and the node 600 is R1 (OUT). The edge attribute type of the edge connecting the nodes 622 and 623 and the node 600 is R2 (IN). The edge attribute type of the edge connecting the node 621 and the node 600 is R2 (OUT). The start point and end point of the edge 631 are a self-loop of the node 600, and the edge attribute type is SELF.

  FIG. 8 schematically shows the update of the state value of the node 600. In the convolution operation, first, the sum of the state values of the nodes having the same edge attribute type is calculated. Specifically, it is as follows. The edge attribute type of the nodes 612 to 614 is R1 (IN). The convolution operation calculates the total sum 811 of the state values 801 of the nodes 612 to 614. The edge attribute type of the node 611 is R1 (OUT). The convolution operation calculates the sum 812 of the state values 802 of the node 611.

  The edge attribute type of the nodes 622 and 623 is R2 (IN). The convolution operation calculates the sum 813 of the state values 803 of the nodes 622 and 623. The edge attribute type of the node 621 is R2 (OUT). The convolution operation calculates the sum 814 of the state values 804 of the node 621. The convolution operation calculates the sum 815 of the state values 805 of the node 600 with respect to the self-loop edge 631.

  Next, the convolution operation calculates the product sum of the sum of the edge attribute types and the corresponding weight. The weight of the edge attribute type is shown in the weight data 123 of FIG. That is, in the convolution operation, the sum of the edge attribute types is multiplied by the weight indicated by the weight data 123, and the sum is calculated (821). The convolution operation outputs a state value 831 obtained by applying the ReLU function 822 to the sum of products.

  In the example of the CNN on the graph described with reference to FIGS. 6 to 8, the updated value of the state value of the target node is determined from the state value of the target node and the state values of the nodes adjacent to the target node. The adjacent node is a node one hop from the target node. The hop number indicates the distance between nodes and indicates the number of edges between nodes. When there are a plurality of routes from one node to another node, the hop number of each route is defined.

[Treatment of each layer of CNN on graph]
FIG. 9 shows details of the first convolution layer 411, the first pooling layer 412, the second convolution layer 413, and the second pooling layer 414 of the CNN 400 on the graph. The first convolutional layer 411 calculates the update value of the state value of each node from the state value of the node and the node within one hop using the ReLU function. The first pooling layer 412 executes pooling processing within one hop. The details of the pooling process will be described later.

  The second convolutional layer 413 calculates the updated value of the state value of each node from the state values of the node and the nodes within two hops using the ReLU function. The second pooling layer 414 performs pooling processing within two hops. Instead of performing pooling processing within two hops, one hop pooling may be performed twice.

  As shown in FIG. 9, the range of the convolution operation of the second convolution layer 413 is wider than the range of the convolution operation of the first convolution layer 411. Further, while the first pooling layer 412 executes pooling processing within one hop, the second pooling layer 414 executes pooling processing within two hops. This means that the range of the pooling process of the second pooling layer 414 is wider than the range of the pooling process of the first pooling layer 412.

  As described with reference to FIG. 3, the CNN 300 on the conventional grid performs a pooling process (usually a maximum value pooling process) after performing a convolution operation (convolution operation + application of a non-linear activation function). Reduce image size. In contrast, the pooling layers 412 and 414 of the CNN 400 on the graph of the present example execute pooling processing without reducing the graph structure. As will be described later, the pooling process in the CNN 400 on the graph substantially performs pooling by taking into account regularization of making values close to neighboring nodes while keeping the size of the graph constant.

  The second convolution layer 413 immediately after the first pooling layer 412 looks at a wide range, that is, enlarges the kernel size of the convolution filter because the values of neighboring nodes approach each other in the pooling layer. Thereby, the convolution operation after the pooling processing becomes more effective. The pooling layer 414 after the second convolution layer 413 realizes more effective pooling processing by expanding the pooling range because the kernel size of the convolution filter of the second convolution layer 413 is expanded.

  In the example shown in FIG. 9, the range of the first convolution layer 411 is within one hop, but the range of the first convolution layer 411 may be wider. As long as the range of the second convolution layer 413 is wider than the range of the first convolution layer 411, their relation is not limited. For example, the range of the second convolutional layer 413 may be 3 hops or more.

  Examples of CNNs 400 on the graph can include more than three convolutional and pooling layers, respectively. For example, the range (the number of hops) of the convolution operation of the convolution layer is, for example, a constant multiple of the immediately preceding convolution layer. In the example where the number of hops is twice, the range of the convolution operation of the convolution layer increases to 1 hop, 2 hops, 4 hops, and 8 hops. In the example where the number of hops is 1.5 times, the range of the convolution operation of the convolution layer increases to 1 hop, 2 hops, 3 hops, 4 hops, and 6 hops. The range (the number of hops) of the pooling processing of the pooling layer is made to match, for example, the number of hops of the immediately preceding convolutional layer.

  Hereinafter, an example of the process of the CNN 400 on the graph will be described. FIG. 10 shows an example of graph data 121 input to the CNN 400 on the graph. The graph data 121 includes nodes N1 to N13 and a plurality of edges connecting the nodes. Although not shown in FIG. 10, a state value is defined for each of the nodes N1 to N13, and an edge attribute type is defined for each of the edges.

[First convolution layer]
FIG. 11 shows an example of the convolution operation of the first convolution layer 411. FIG. 11 shows an example of updating the state value of the node N1. The first convolution layer 411 performs a convolution operation within one hop. Nodes within one hop from node N1, that is, nodes connected to node N1 by one edge are nodes N2, N6, N9, N10, and N12. The first convolutional layer 411 collects the state values of the nodes N2, N6, N9, N10, and N12, and calculates the updated state value of the node N1 from the state values and the state value of the node N1.

  FIG. 12 shows an example of the attribute of the graph data 121. An edge attribute type is defined for each edge of the graph data, and is indicated by a code consisting of P and a number. For example, the edge attribute type of the edge between the node N1 and the node N12 is R1, the edge attribute type of the edge between the node N1 and the node N2 is R2, and the edge attribute of the edge between the node N1 and the node N6. The type is R3. The edge of the edge attribute type R1 is represented by a solid line, the edge of the edge attribute type R2 is represented by a broken line, and the edge of the edge attribute type R3 is represented by a two-dot chain line.

  FIG. 13 shows a part of the weight data 123 of the present example. The weight data 123 defines the weight W0 of the current state value of the target node, the weight of the 1-hop edge attribute type, and the weight of the 2-hop edge attribute type. The 1-hop edge attribute types match the edge attribute types R1, R2, and R3 of the respective edges. The 2-hop edge attribute type is represented by two Ps and two numbers, and indicates a set of edge attribute types of two edges.

  More specifically, the 2-hop edge attribute type indicates an edge attribute type of an edge close to the target node and an edge attribute type of an edge far from the target node. For example, in the edge attribute type R1R2, the edge attribute type of the edge between the target node and the adjacent node is R1, and the edge attribute type of the edge between the adjacent node of the target node and the node two hops away is R2. It represents that.

  FIG. 14 shows updating of the state value of the node N1 by the first convolution layer 411. The first convolutional layer 411 includes the state value V1 of the target node N1 and the state values V2, V6, V9, V10, and V12 of the nodes within one hop of the target node N1, that is, the adjacent nodes N2, N6, N9, N10, and N12. To get. Edge attribute types of edges between the target node N1 and the adjacent nodes N2, N6, N9, N10, and N12 are R2, R3, R1, R2, and R1.

As shown by the weight data 123 in FIG. 13, the weight of the state value of the own node is W0, and the weights of the edge attribute types R1, R2, and R3 are W1, W2, and W3, respectively. In the example of the convolution operation, the state value of the node is multiplied by a different weight for each edge attribute type of the edge and added to calculate the sum of all edge attribute types. That is, the convolution operation for the node N1 is represented by the following equation. The bias was set to 0.
V1 '= W0 * V1 + W1 * (V9 + V12) + W2 * (V2 + V10) + W3 * V6

  The first convolution layer 411 inputs the state value V1 ′ resulting from the convolution operation to ReLU, which is a nonlinear function, and holds the output as a new state value V1 of the node N1. Note that a non-linear function (for example, a sigmoid function) different from ReLU may be used. The first convolutional layer 411 performs the above-described processing for all nodes.

[First pooling layer]
Next, the processing of the first pooling layer 412 will be described. In a conventional CNN on a grid, it is common to reduce the image size by a pooling process after a convolution operation (+ application of a non-linear activation function). The pooling layer of the CNN 400 on the graph according to the present disclosure executes a process corresponding to the conventional pooling process without changing the graph structure of the input graph data 121, that is, the pooling process on the graph.

  The pooling process is a process of making values close to each other in a neighboring node, thereby reducing the degree of freedom of expression. Hereinafter, examples of pooling processing on some graphs will be described. Several techniques are known for the pooling process, but the commonly used techniques are average pooling and maximum pooling. Therefore, hereinafter, the average value pooling and the maximum value pooling on the graph will be described.

  In the pooling process, similar to the convolution operation, the state value of each node is updated while maintaining the graph structure. The pooling process determines a new state value of the target node from the state values of the target node and nodes near the target node. FIG. 15 schematically shows the pooling process by the first pooling layer 412. In FIG. 15, attention is focused on two nodes N1 and N2. The first pooling layer 412 updates the state value of the node N1 so as to approach the state values of the adjacent nodes N2, N6, N9, N10, and N12. Similarly, the first pooling layer 412 updates the state value of the node N2 so as to approach the state values of the adjacent nodes N1, N3, N4, N5.

  The example of the average value pooling on the graph is based on the state values of the target node and the adjacent node, so that the new state value of the target node becomes the average value of the new state values of the target node and the adjacent node. The new state value of the target node is determined. For example, the first pooling layer 412 acquires the current (pre-update) state values of the node N1 and its neighboring nodes N2, N6, N9, N10, N12, and calculates an average value thereof. The first pooling layer 412 determines the average value as a new state value of the node N1. The first pooling layer 412 may calculate an average value of weighted node state values according to the edge attribute type.

Another example of the average value pooling on the graph determines a new state value of the target node according to the following formula. N indicates a set of nodes of the graph data 121, and E indicates a set of edges. p _i is the output of the pooling process node i, v _i is the input of pooling processing node i. v _i is the output of the previous convolution (state value). β is a hyperparameter, I is a unit matrix, and L is a Laplacian matrix. D is the total number of edges in the graph data 121. The reason why the second term on the right-hand side is divided by D is to make the strength of the constraint per edge uniform, but a configuration that does not divide by D (D = 1) is also conceivable.

The output p _i of the first pooling layer 412 of node i is a value close to the input v _i from the first term, and a value close to the output value p _j neighboring nodes than the second term. As the value of β increases, the contribution of the second term increases, and the value approaches a wider range on the graph. That is, β is a parameter corresponding to the kernel size of the pooling process. The above equation can perform an average pooling that is more general and appropriate.

  FIG. 16 shows an example of an applicable Laplacian row example L in the graph data 121 having the graph structure shown in FIG. The pooling formula and the Laplacian row example L shown in FIG. 16 are used when all edges are uniformly pooled, ignoring the edge attribute type. The pooling process may be performed ignoring the edge attribute type.

  Alternatively, the pooling process may be performed based on the edge attribute type. Thereby, more appropriate pooling processing can be executed. The first pooling layer 412 executes pooling processing for each edge attribute type. Specifically, an example of the pooling process based on the edge attribute type follows the following formula.

E ₁ , E ₂ , and E ₃ indicate sets of edges of edge attribute types R1, R2, and R3, respectively. β ₁ , β ₂ , and β ₃ are hyperparameters (pooling kernel size) of edge attribute types R1, R2, and R3, respectively. L ₁ , L ₂ , and L ₃ are pooling Laplacian matrices of edge attribute types R1, R2, and R3, respectively.

  Next, an example of maximum value pooling on a graph will be described. The example of the maximum value pooling on the graph selects the maximum value of the state values of the nodes within the pooling range. For example, the first pooling layer 412 obtains the current (pre-update) state values of the node N2 and its neighboring nodes N1, N3, N4, N5, and selects the maximum value among them. The first pooling layer 412 determines the maximum value as a new state value of the node N2. The first pooling layer 412 may select a node to which an update value is to be given based on the weight according to the type of the edge attribute.

  Another example of maximum value pooling on a graph performs average value pooling after applying the Smooth Maximum to input values (current state values of nodes). Several functions are known as Smooth Maximum, for example, Log MeanExp can be used. Another example of maximum value pooling on a graph determines a new state value of the target node according to the following formula.

  ρ is a parameter indicating the strength of the Smooth Maximum, and at the limit of ∞, LogMeanExp matches the maximum value function. Other variables are as described for mean pooling. The maximum value pooling represented by [Equation 3] is used when all edges are uniformly pooled, ignoring the edge attribute type. The pooling process may be performed ignoring the edge attribute type.

  As in the description of the average value pooling, the first pooling layer 412 may execute the pooling process based on the edge attribute type. The first pooling layer 412 uses [Equation 2] instead of [Equation 1]. Thereby, more appropriate pooling processing can be executed. The first pooling layer 412 executes pooling processing for each edge attribute type.

[Second convolution layer]
Next, the processing of the second convolution layer 413 will be described. As described above, the first pooling layer 412 reduces the degree of freedom in expressing data by adopting regularization in which values are made closer between neighboring nodes while keeping the size of the graph structure constant. The second convolution layer 413 refers to a wide range because the value of a neighboring node approaches in the first pooling layer 412, that is, expands the kernel size of the convolution filter (the range of nodes referred to in the convolution operation).

  FIG. 17 schematically illustrates the convolution operation for the node N1 by the second convolution layer 413. A similar convolution operation is performed for all nodes. The second convolutional layer 413 acquires the current (pre-update) state value of a node within a 2-hop range from the node N1. The nodes within the two-hop range are composed of one-hop nodes and two-hop nodes from the target node. Note that a configuration using only a 2-hop node instead of using a node within a 2-hop range, that is, a 1-hop node and a 2-hop node, is also conceivable.

  In the example illustrated in FIG. 17, the nodes N2, N6, N9, N10, and N12 are one-hop nodes from the node N1. The nodes N3 to N5 and N7 to N13 are two-hop nodes from the node N1. The nodes N9, N10, and N12 are one-hop nodes from the node N1 and are also two-hop nodes. The nodes N9, N10, and N12 may be defined as nodes of one hop from the node N1, and may be excluded from nodes of two hops from the node N1.

  The second convolutional layer 413 is defined for each edge attribute type by the state value of the target node N1, the state value of the node one hop from the target node N1, the state value of the node two hops from the target node N1, and the weight data 123. Calculate the product sum of the weights. As described with reference to FIG. 13, the weight data 123 defines the weight of the own node, the weight of the edge attribute type of one edge, and the weight of the edge attribute type of a pair of two edges. Although not shown, the weight data 123 further defines the weight of the edge attribute type of the set of three or more types of edges. Similarly to the first convolution layer 411, the state value of the target node obtained by the convolution operation is input to ReLU, and the output is held as a new state value of the target node. Note that the convolution operation of the second convolution layer 413 can include a bias.

[Second pooling layer]
The second pooling layer 414 performs pooling processing within a two-hop range, as shown in FIG. For example, the second pooling layer 414 executes the pooling process performed by the first pooling layer 412 twice. Accordingly, pooling processing according to the processing range of the second convolution layer 413 can be performed.

[Whole area average pooling layer]
Next, the processing of the entire area average pooling layer 415 will be described. The CNN 400 on the graph handles graph data having various graph structures with different numbers of nodes and edges. The fully connected layer in the conventional CNN is based on the assumption that the number of neurons in the input layer is fixed, and cannot be directly applied to the CNN 400 on the graph. CNN 400 on the graph of this disclosure uses global average pooling. If the graph structure of the input graph data is constant, instead of the global average pooling layer, an all-connected layer that collects the values of all neurons in one place and performs an affine operation to calculate the final output value May be used.

  The global average pooling layer 415 averages the values of all neurons for each input channel. Therefore, the output of the global average pooling layer 415 is a vector of the number of input channels. For example, an affine transformation layer (full coupling layer) is arranged at a stage subsequent to the global average pooling layer 415, and an output of a finally required dimension is obtained. Although the global average pooling layer 415 has a significantly smaller number of parameters than the fully coupled layer, it can exhibit performance equal to or higher than that of the fully coupled layer.

[Learning (training) by backpropagation]
In the following, learning of parameters by backpropagation of errors by the CNN 400 on the graph will be described. The learning process is executed by the data processing unit 120. Backpropagation reverses the nodes of the graph in forward propagation. As described above, the second convolution layer 413 increases the kernel size of the convolution filter. However, since the values of the neighboring nodes are approaching in the first pooling layer 412, simply increasing the kernel size increases the degree of freedom in parameter setting, causing over-learning, or causing a bad setting problem and causing a parameter setting problem. There is a possibility that you cannot learn at all.

  Therefore, for the second convolution layer 413 after pooling, regularization is performed such that the kernel size of the convolution filter is increased and, similarly to the immediately preceding first pooling layer 412, the weights of the neighbors on the convolution filter are made closer. By doing so, the degrees of freedom of the first pooling layer 412 and the immediately following second convolution layer 413 become uniform, and the CNN on the so-called deep learning graph is obtained by stacking a number of layers of the convolution operation and the pooling processing set. Can be properly configured.

  Here, learning (parameter backpropagation) of the parameters of the CNN 400 on the graph optimizes a plurality of weights in the convolutional layer by using an equation including the same regularization term as the pooling layer one stage before. More specifically, the error function E (w) of the entire CNN 400 includes the regularization term of the convolutional layer.

The above equation shows an error function E (w) of a configuration including three or more convolutional layers. E ₀ (w) is a normal error function that does not include the regularization term of the convolutional layer. The second term on the right side is a regularization term of the second convolutional layer, and the third term is a regularization term of the third convolutional layer. w2 _{, i} , w2 _{, j} indicate adjacent nodes (joined by one edge) at a node two hops away from a certain node. w3 _{, i} , w3 _{, j} indicate adjacent nodes at a node three hops away from a certain node. β is the same as β used in the pooling layer. The error function E (w) includes the regularization terms of the convolutional layers after the second convolutional layer.

  As described above, the kernel size of the convolution filter increases in the second convolution layer, the third convolution layer, and the subsequent stages. By increasing the order of β in the second and third terms on the right side of the above equation to increase the effect of the regularization term, the degree of freedom of the actual parameter setting in each convolutional layer is kept constant.

  As described with reference to FIG. 13, the weight data 123 includes the weight of the own node, the weight of the edge attribute type of one edge, the weight of the edge attribute type of a pair of two edges, and the weight of three or more types of edges. The weight of a set of edge attribute types is defined. At this time, for example, the second term on the right side, that is, the regularization term of the second convolutional layer, calculates the sum of the squares of the difference in weight between adjacent nodes in a 2-hop node from each of all the nodes.

The second term on the right side will be described using the node N1 as an example. The nodes of two hops from the node N1 are the nodes N3 to N5 and N7 to N13. For w2 _{, i} , all the weights of these nodes are substituted. For w2 _{, j} , all the weights of all the nodes adjacent to the node of w2 _{, i} are substituted. The calculation of the second term of the node N1 (omitting β / 2) is expressed as follows.

An example of the node N3 will be described. When looking at the node N3 from the node N1, the weights of the edge attribute types of the set of two edges are W12, W23 (via the node N2), and W23 (via the node N10). Therefore, w2, _i of the node N3 is W12, W23, W23. Among the nodes that are two hops from the node N1, the nodes adjacent to the node N3 are the nodes N4, N10, and N12. Therefore, weights obtained by viewing these nodes from the node N1 are substituted for w2 _{, j} .

  For example, when looking at the node N1 to the node N4, the weight of the edge attribute type of the pair of two edges is W22. When looking at the node N10 from the node N1, the weights of the edge attribute types of the pair of two edges are W11 and W12. Looking at the node N1 to the node N12, the weight of the edge attribute type of the pair of two edges is W22.

  The second term calculates the same processing for the node N1 as described above for the node 3 also for the nodes N4, N5, N7 to N13. Further, the same calculation is performed for nodes other than the node N1. In the third term, similarly to the second term, calculation is performed for a node of three hops from each of all nodes.

  In another example, the regularization term may be calculated based on the relationship between edge attribute types instead of the graph structure. For example, for the second term, a calculation may be performed such that an edge close to the target node has the same edge attribute type and the same weight. For example, in each of the weight groups (W11, W12, W13), (W21, W22, W23), and (W31, W32, W33), the second term is defined so that the weights become closer. The third and subsequent terms are similarly defined according to the types of edge attributes of three or more hops.

  The gradient value of the second convolutional layer is obtained by differentiating the error function E (w) with w as in the following equation.

The first term on the right side in [Equation 6] indicates an error signal (normal error back propagation) from the upper layer. The second term on the right side shows the differentiation of the normalization term. The gradient values of other convolutional layers can be expressed by a similar expression. The learning process updates the weight w of the convolutional layer according to the gradient value. For example, a known method such as SGD (stomatic gradient descent) and ADAM can be used. As described above, in learning by error back propagation, the weight w is updated with a value obtained by adding a differential term (βLw _{, n} ) of a regularization term to a normal error signal back-propagated from the upper layer as a gradient. This makes it possible to add a constraint to the convolutional layer (filter) according to the immediately preceding pooling.

  Next, the entire learning process will be described. FIG. 18 is a flowchart illustrating an example of a learning process performed by the computer 101. The processor 110 executes a neural network construction process (S201). The construction process of the neural network will be described later.

  Next, the processor 110 acquires the learning data 122 from the storage system 102 and stores it in the memory 111 (S202). The learning data 122 includes a plurality of samples. Next, the processor 110 starts a loop process of the error back propagation process (S203). In this loop processing, the processor 110 repeatedly executes the processing from step S203 to step S207. For example, when the error calculated by the error back-propagation processing executed in the loop processing becomes equal to or less than a preset threshold value, or when the error back-propagation processing is executed a predetermined number of times, The loop processing ends.

  The processor 110 starts loop processing of the sample (step S705). In the sample loop process, the process inside the loop is executed for each of the plurality of samples acquired in step S202. Further, the processor 110 performs an error back propagation process on one sample (S205).

  In the error back-propagation processing, the processor 110 takes a sample as an input, compares an output result of the CNN 400 on the graph with respect to the input with teacher data corresponding to the sample, and reduces an error between the two data. The weight is updated sequentially from the upper layer of the CNN 400.

  In the sample loop processing, the processor 110 performs the above-described processing on each of a plurality of samples. In the error back-propagation processing of step S205, the weight is updated only once for any sample data. On the other hand, in the loop processing of the error backpropagation processing, the error backpropagation processing is executed a plurality of times in order to minimize the error. Specifically, the processor 110 repeatedly executes the backpropagation processing on each of the plurality of samples until a predetermined condition is satisfied. In the sample loop process, the weight is updated for a plurality of samples.

  Next, the processor 110 determines whether or not the backpropagation processing has been performed on all the sample data (S206). When it is determined that the backpropagation processing has not been performed on all the sample data, the processor 110 returns to step S204 and performs the same processing. If it is determined that the backpropagation processing has been performed on all the sample data, the processor 110 determines whether a predetermined condition has been satisfied (S207).

  When it is determined that the predetermined condition is not satisfied, the processor 110 returns to step S203 and performs the same processing. When it is determined that the predetermined condition is satisfied, the processor 110 stores the learning result in the memory 111 or the storage system 102 (Step S208). The learning result includes information indicating the structure of the constructed neural network, weight information obtained by learning, and the like. The weight is obtained for each edge attribute type, and is used as the weight data 123 in the analysis processing.

[Construction of CNN on graph]
The CNN 400 on the graph handles graph data having various graph structures with different numbers of nodes and edges. The data processing unit 120 constructs the CNN 400 according to the graph data 121 to be analyzed. The data processing unit 120 constructs each of the convolutional layer and the pooling layer according to the graph structure of the graph data 121. The data processing unit 120 acquires weights for each of the edge attribute types from the learning result, and forms the weight data 123.

  As described above, according to the present embodiment, by performing pooling in the CNN on the graph, the representable degree of freedom (the degree of freedom of the parameter) can be reduced toward the upper layer, and the excess in learning the parameter can be obtained. It is possible to reduce the probability of learning and the problem of setting a defect. In addition, it is possible to configure a CNN on a graph for performing so-called deep learning by increasing the number of CNN layers.

  Hereinafter, the distributed CNN will be described. In the distributed CNN, a plurality of subsystems (for example, computers) connected via a network execute the CNN. The subsystem can communicate (exchange information) only with the subsystem adjacent to the subsystem on the network. The network structure connecting the subsystems does not need to be fixed, and may change over time. In the distributed CNN, each subsystem estimates the value of the system state function independently, for example. With distributed CNN, the estimates of all subsystems match the values of the true system state function.

  In another example, each subsystem may take action in response to a respective environmental input and internal state. Each subsystem can find an action that optimizes the value of the state function of the overall system. For example, a factory (computer) connected via a network communicates only with each adjacent factory, and determines the power consumption of the factory so as to minimize the total profit of all factories. The factory or its computer is a subsystem.

  In addition, the distributed CNN can be applied to vehicle route optimization by inter-vehicle communication. A car or its calculator (eg, a car navigation system) is a subsystem. Each car independently determines a route from the current position and the destination. Each car communicates with neighboring cars so that many cars constitute one distributed CNN as a whole. Each vehicle regresses (estimates) the average traveling time by the variance CNN, using the route of the vehicle as an input.

  Each vehicle independently updates the route (= input) in a direction to reduce the output value (= average travel time) of the distributed CNN by backpropagation. At the same time, each vehicle independently updates a parameter (weight) by error backpropagation so as to reduce the error between the true average travel time obtained from the road traffic information and the output value of the distributed CNN. Each car leaves the decentralized CNN once it reaches its destination.

[System configuration]
FIG. 19 shows an example of a computer system that executes a distributed CNN. The computer system includes a plurality of computers 101A to 101M, which are connected via a network. Neighboring computers that can communicate are connected by an edge. The configuration of the computer is as described in the first embodiment. The network configuration in FIG. 19 is an example, and in another example, computers arranged in a grid may be able to communicate only with adjacent computers.

  The distributed CNN does not handle neurons constituting the CNN collectively for each layer (convolution layer, pooling layer, etc.), but collects neurons at the same position in a plurality of layers (hereinafter referred to as columns) to form one sub-cell. Calculate with the system. As a result, the entire system of many subsystems operates as a single CNN.

  FIG. 20 shows an example of dispersed CNN layers and columns. The distributed CNN includes layers 701 to 704 such as a convolution layer and a pooling layer. FIG. 20 shows one column 711 as an example. One column is executed by one subsystem. In the computer system shown in FIG. 19, one of the computers performs the calculation of the column neurons.

  As understood from the above description, the CNN on the graph described in the first embodiment can be applied to the distributed CNN. The network structure of the computer system shown in FIG. 19 matches the graph structure of the graph data 121 described in the first embodiment. Each of the computers 101A to 101M performs an arithmetic process on a corresponding node in the graph data 121 in each layer (convolution layer or pooling layer) configuring the CNN.

  FIG. 21 shows a conceptual configuration example of a distributed CNN. FIG. 21 shows three computers (subsystems) 101B, 101C, and 101D as an example. The computer 101D is adjacent to the computers 101B and 101C and can communicate with each other. The computers 101B, 101C, and 101D execute columns 711B, 711C, and 711D, respectively. The squares that make up the columns indicate neurons.

  Each subsystem holds, as an internal state, a vector in which the output values of the neurons on the column in charge of the subsystem are collected. Each subsystem inputs its own internal state vector and the internal state vector of the adjacent subsystem obtained by communication with the adjacent subsystem (the value of the neuron on the column at the adjacent position), and performs the operation of each layer, and Update the internal state vector of.

  In a computer system that executes a distributed CNN, there is no central control system that controls the entire system, and there is no global synchronization mechanism for the entire system. Each subsystem communicates with an adjacent subsystem and updates an internal state vector of the subsystem, and the other subsystems execute asynchronously. The subsystem updates the internal state vector of its own subsystem based on the internal state vector of another subsystem and the internal state vector of its own subsystem.

  The communication with the adjacent subsystem and the update of the internal state vector of the subsystem are not always performed at the same frequency.In one subsystem, the communication frequency with the adjacent subsystem is high, while in the other subsystem, the communication frequency is high. The internal state vector may be updated frequently. When a certain subsystem has a plurality of adjacent subsystems, the frequency of communication with each adjacent subsystem may not be the same.

  The computer system that executes the distributed CNN asynchronously performs communication between adjacent subsystems and updates the internal state vector. Therefore, when executing the update of the internal state vector, the subsystem cannot obtain the value of the neuron on the adjacent column (the column executed by the adjacent subsystem) required for the convolution operation.

  Therefore, the subsystem stores the value of the neuron on the adjacent column (internal state vector) obtained in the previous communication with the adjacent subsystem in the memory, and stores the value stored when the internal state vector is updated. Is used to perform a convolution operation. This configuration makes it possible to perform error back propagation asynchronously, as described later.

[Convolution]
Hereinafter, processing of each layer of the distributed CNN will be described. The computer system can execute the processing of each layer in the CNN 400 on the graph described in the first embodiment. First, the processing of the convolution layer will be described. Each subsystem executing the distributed CNN performs a convolution operation of a column in charge of the own system. Each subsystem executes the processing of the convolutional layer described in the first embodiment in the column in charge.

  As described in the first embodiment, the convolution operation uses the value of the target node (neuron) and the value of the node having a predetermined number of hops from the target node. In the distributed CNN, each subsystem collects values of nodes other than the target node (neuron) from other subsystems.

  As described above, each subsystem stores the value of the neuron on the adjacent column obtained at the time of communication with the adjacent subsystem, and performs the convolution operation using the stored value when updating the internal state vector. I do. The weight (coefficient of the kernel) of the convolution operation is common to all subsystems. For example, the subsystems can share the same weight data by communicating with each other.

  The subsystem needs the values of all columns within the kernel range (hop count) of the convolution operation. As described in the first embodiment, the kernel size in the convolution chapter after the pooling layer is increased. For a convolution operation with a kernel size of 2 hops or more, such as the processing of the second convolution layer 413, the subsystem uses the values of the columns (internal state vectors) of the non-adjacent subsystems via the adjacent subsystems. Obtained by multi-hop relay. Information transmission of n hops requires n times of communication.

[Pooling]
Next, the pooling process will be described. In the distributed CNN, since each subsystem treats neurons at the same position from the lower layer to the upper layer of the CNN as columns, the number of neurons is constant in all layers. This is similar to the CNN on the graph described in the first embodiment. In the distributed CNN, similarly to the pooling process described in the first embodiment, each subsystem executes a pooling process that reduces the degree of freedom of expression without reducing the number of neurons (the number of nodes).

  Both the average value pooling and the maximum value pooling described in the first embodiment can be applied to the pooling process in the distributed CNN. As described in the first embodiment, the pooling process requires the state value of the adjacent node. In the distributed CNN, a subsystem calculates a new internal state vector of its own system from an internal state vector of its own system and an internal state vector acquired from an adjacent subsystem.

[Full join]
Next, all coupling will be described. The centralized CNN calculates the final output value by repeating the convolution operation and pooling several times, then collecting all neuron values at one place by an all-connected layer and performing an affine operation. On the other hand, in the distributed CNN, each subsystem operates autonomously, and the values of all subsystems cannot be collected in one place.

  Therefore, the computer system according to the present disclosure uses a known algorithm for calculating an average value by variance. The algorithm for obtaining the average value by the variance is, for example, a distributed ADMM (Alternating Direction Method of Multipliers) or an average agreement algorithm.

  Specifically, each subsystem first multiplies the output value of the neuron on its own column by the weight coefficient of the corresponding fully connected layer. Next, the output value of the fully connected layer can be obtained by calculating the average of the values of the multiplication results of all the subsystems in a variance using an algorithm described later. In this method, an average value is output instead of the sum of the values obtained by multiplying the value of the neuron by the weight coefficient. Since the weight coefficient is learned from the learning data (teacher data), this difference does not matter. When the weighting factor learned in the conventional centralized CNN is used in the distributed CNN, the value of the weighting factor held in each subsystem may be multiplied by the number of all subsystems in advance.

In the following, two examples of an algorithm for calculating an average value by variance will be described. The first example is an average calculation algorithm using a distributed ADMM. Using a dispersion ADMM by solving the following optimization problem, it is possible to calculate the mean value ν of N values x _i in a distributed.

  In the method using the distributed ADMM, the optimization is always performed so as to satisfy the above equation. Therefore, even when the value xi or the network topology changes dynamically during the operation, the average at that time follows the change. The value ν can be output. Therefore, it is possible to calculate an average value for a value at any specified time.

A second example is an average agreement algorithm. Mean agreement algorithm, while keeping the sum of the values x _i of holding in each subsystem constant distributes the values x _i between subsystems, finally, the same value that the value x _i of all the subsystems To converge. Since the sum of values x _i is always constant, the convergence value is the average of the values x _i the initial value. Among several types of means algorithm, the simplest mean agreement algorithm, for example, when communicating subsystem i and subsystem j, updates a value x _i and the value x _j according to the following equation.

Thereby, every time the communication between the neighbors is performed, the variance of the value x _i is reduced while keeping the sum of the values x _i of the entire system constant, so that all the values x _{i eventually} become the same value, that is, will converge to an average value of the initial value of each value x _i.

The average consensus algorithm calculates the average value of the initial values of each value x _i because the sum of the values x _{i in} the entire system changes when the value of the value x _i or the network topology changes dynamically during the operation. Cannot calculate correctly. On the other hand, since the convergence value of the average agreement algorithm is always the average value of the values x _i of the entire system at that time, it is useful for learning the weight coefficient of the convolution filter by the back propagation of the error.

[Learning (training) by backpropagation]
As described above, in the distributed CNN, the subsystem stores the value of the neuron on the adjacent column obtained during the previous communication with the adjacent subsystem, and performs convolution operation or the like using the stored value. To update the internal state vector by performing forward propagation calculation. By using the value used during forward propagation at the time of error backpropagation calculation, the subsystem can update parameters by error backpropagation without synchronization with other subsystems.

  The calculation of the backpropagation is performed by following the nodes (neurons) of the forward propagation graph in reverse. 22 to 25 show examples of propagation rules for each type of node on the graph. Specifically, FIGS. 22 to 25 show an application node 771, a countable node 772, a multiplication node 773, and a variable replication node (branch node) 774 of the function f, respectively. The propagation rule of forward propagation is shown by a solid line, and the propagation rule of back propagation is shown by a broken line. The inputs to the forward propagation are x and y. In the distributed CNN, a device is required to realize the variable replication node 774 among them.

  The variable replication node 774 is equivalent to using the same variable a plurality of times when calculating forward propagation. In the distributed CNN, a plurality of subsystems perform calculations in a distributed manner. Therefore, in the backpropagation rule of the variable replication node 774 in FIG. 25, the error signals dx1 and dx2 from the upper layer are columns that are handled by subsystems other than the own subsystem. Can not get the signal from.

  Therefore, in the error backpropagation of the variable replication node 774, each subsystem independently calculates the derivative of the parameter and updates the parameter value using only the error signal obtained inside the column assigned to the own subsystem. I do. Thereafter, each subsystem calculates the average of the parameter values held by all the subsystems using the average agreement algorithm. In the average consensus algorithm, the convergence value is always the average value of the values of all the subsystems at that time, so that learning, that is, updating of the parameter value can be continuously performed.

  Note that while the mathematically correct differential coefficient is the sum of all error signals from the upper layer, this method updates the parameters by considering the average value of all error signals from the upper layer as the differential coefficient. Therefore, the learning rate (learning rate), which is a hyperparameter of the learning algorithm used for updating the parameter, is multiplied in advance by the number of all subsystems.

  Similarly, when there is another node after the branch node in the backward error propagation (the previous stage in the forward propagation), similarly, in the branch node, each subsystem generates an error signal by using only the information of the column in charge of its own subsystem. The parameter is back-propagated and the subsequent parameters are updated independently, and then the average of the parameter values is calculated by the average agreement algorithm.

[Configuration Example of Distributed CNN]
Hereinafter, a configuration example of the distributed CNN of the present disclosure will be described. FIG. 26 shows a configuration example 900 of the distributed CNN and its processing. Hereinafter, processing executed by one subsystem will be described. First, forward propagation will be described. In the forward propagation, the first convolution layer 901 performs the convolution operation and the ReLU, similarly to the first convolution layer 411 of the first embodiment. The first convolutional layer 901 obtains the value of the target neuron from its own internal state vector, and obtains the value of the adjacent neuron from the internal state vector of the adjacent subsystem stored in advance. The first convolution layer 901 executes a convolution operation of these values and ReLU to update its own internal state vector.

  The first pooling layer 902 performs a pooling process, like the first convolution layer 411 of the first embodiment. The first pooling layer 902 acquires the value of the target neuron updated by the first convolution layer 901 from its own internal state vector, and obtains the first neuron value of the adjacent neuron from the internal state vector of the adjacent subsystem stored in advance. The output value of the convolution layer 901 is obtained. The first pooling layer 902 performs pooling of these values, in this example, maximum value pooling, and updates its own internal state vector.

  The second convolution layer 904 performs the convolution operation and the ReLU similarly to the second convolution layer 413 of the first embodiment. The second convolutional layer 904 acquires the value of the target neuron updated in the first pooling layer 902 from its own internal state vector, and obtains the value of the neighboring neuron from the internal state vector of the subsystem in two hops stored in advance. To obtain the output value of the first pooling layer. Data from subsystems two hops away is forwarded through neighboring subsystems. The second convolution layer 904 performs a convolution operation of these values and ReLU to update its own internal state vector.

  The second pooling layer 906 performs a pooling process, similarly to the second pooling layer 414 of the first embodiment. The second pooling layer 906 acquires the value of the target neuron updated in the second convolution layer 904 from its own internal state vector, and obtains the value of the adjacent neuron from the internal state vector of the adjacent subsystem stored in advance. get. The second pooling layer 906 performs pooling of these values, in this example, maximum value pooling, and updates its own internal state vector.

  The second pooling layer 906 further acquires the value of the neuron updated in the immediately preceding pooling process from the own internal state vector, and acquires the value of the adjacent neuron from the internal state vector of the adjacent subsystem stored in advance. . The second pooling layer 906 performs pooling of these values, in this example, maximum value pooling, and updates its own internal state vector.

  The total coupling layer 907 calculates an average value by the distributed ADMM. The fully connected layer 907 calculates an average value by the distributed ADMM from the value of the target neuron of the own internal state vector and the value of the target neuron of the internal state vector of another subsystem stored in advance.

  Next, error back propagation will be described. In backpropagation, each subsystem is provided with teacher data. Each subsystem updates the weight data (including the bias) independently of the other subsystems, using the data held internally. Each subsystem calculates an average value of the updated weight data by an average agreement algorithm. As a result, the weight data of all subsystems is shared.

  Note that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described above. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of one embodiment can be added to the configuration of another embodiment. Also, for a part of the configuration of each embodiment, it is possible to add, delete, or replace another configuration.

  In addition, each of the above configurations, functions, processing units, and the like may be partially or entirely realized by hardware, for example, by designing an integrated circuit. In addition, the above-described configurations, functions, and the like may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as a program, a table, and a file for realizing each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card or an SD card. Further, the control lines and the information lines are shown to be necessary for the explanation, and not all the control lines and the information lines are necessarily shown on the product. In practice, almost all components may be considered to be interconnected.

101 computer, 110 processor, 111 memory, 123 weight data, 400 CNN, 411 first convolution layer, 412 first pooling layer, 413 second convolution layer, 414 second pooling layer, 900 distributed CNN

Claims (11)

A computer system for executing a convolutional neural network on a graph, comprising:
One or more processors,
One or more storage devices;
The convolutional neural network on the graph is
One or more convolutional layers,
One or more pooling layers;
The one or more storage devices store kernel weight data of the one or more convolutional layers,
The one or more processors include:
In each convolution layer, the value of each node is updated by a convolution operation based on a kernel having a size of a predetermined number of hops,
In each pooling layer, the value of each node is updated by pooling processing based on the value of each node and the value of a node within a pooling range of a predetermined number of hops from each node,
The computer system, wherein the size of the kernel of the convolutional layer at the subsequent stage of the pooling layer is larger than the size of the kernel of the convolutional layer of the preceding stage. The computer system according to claim 1, wherein
The computer system, wherein the pooling range of the subsequent pooling layer of the convolutional layer is wider than the pooling range of the preceding pooling layer. The computer system according to claim 1, wherein
The computer system, wherein the one or more processors perform the pooling process of the pooling range of the predetermined hop number by repeating the pooling process of the one hop range a number of times equal to the predetermined hop number. The computer system according to claim 1, wherein
The computer system, wherein the one or more processors include a regularization term of the one or more convolution layers in an error function in error propagation learning of the convolutional neural network on the graph. The computer system according to claim 1, wherein
The computer system, wherein the one or more processors perform average pooling after applying smooth maximum to an input value in the pooling process. The computer system according to claim 1, wherein
The computer system, wherein the one or more processors execute a global average pooling at a stage subsequent to the one or more convolutional layers and the one or more pooling layers. The computer system according to claim 1, wherein
The computer system includes a plurality of subsystems connected by a network,
The plurality of subsystems communicate only with adjacent subsystems,
A computer system, wherein each of the plurality of subsystems calculates a column composed of neurons at the same position in the convolutional neural network on the graph. The computer system according to claim 7, wherein
Each of the plurality of subsystems comprises:
Holding an internal state vector indicating the value of the column,
Obtaining the internal state vector from other subsystems, including neighboring subsystems,
A computer system for calculating the column using its own internal state vector and internal state vectors of other subsystems. The computer system according to claim 7, wherein
The computer system, wherein the plurality of subsystems calculate an average value of the values of the columns by variance in a stage subsequent to the one or more convolutional layers and the one or more pooling layers. The computer system according to claim 7, wherein
Each of the plurality of subsystems comprises:
Independent of the other subsystems, updating the weights of the one or more convolutional layers by learning by backpropagation,
A computer system for calculating the updated average value of the weights by communication with another subsystem. A method, wherein a computer system executes a convolutional neural network on a graph,
The computer system includes:
One or more processors,
One or more storage devices;
The convolutional neural network on the graph is
One or more convolutional layers,
One or more pooling layers;
The one or more storage devices store kernel weight data of the one or more convolutional layers,
The method comprises:
In each convolution layer, the value of each node is updated by a convolution operation based on a kernel having a size of a predetermined number of hops,
In each pooling layer, the value of each node is updated by pooling processing based on the value of each node and the value of a node within a pooling range of a predetermined number of hops from each node,
The method, wherein the size of the kernel of the subsequent convolutional layer is larger than the size of the kernel of the previous convolutional layer.

Images