KR20200062743A - Method and reconfigurable interconnect topology for multi-dimensional parallel training of convolutional neural network - Google Patents

Abstract

Disclosed is a reconfigurable interconnect topology between a method for multi-dimensional parallel training of convolutional neural networks and an apparatus for performing the same. According to an embodiment of the present invention, the method for multi-dimensional parallel training for a Winograd conversion convolution comprises the following steps of: converting input data including an input characteristics map into a plurality of tiles; distributing and delivering the converted tiles to a plurality of workers arranged in two dimensions by a plurality of clusters and a plurality of groups, wherein the converted tiles are distributed and delivered for each of the clusters; performing data parallelism on the input data distributed and delivered, through workers of each of the clusters; and performing intra-tile parallelism on an element unit of the tiles applied to the groups, respectively, among the input data distributed and delivered through the workers of each of the clusters.

Description

METHOD AND RECONFIGURABLE INTERCONNECT TOPOLOGY FOR MULTI-DIMENSIONAL PARALLEL TRAINING OF CONVOLUTIONAL NEURAL NETWORK}

The following embodiments relate to a multidimensional parallelization method of convolutional neural network training and a reconfigurable connection structure between devices performing the same.

Accelerated neural network learning is very important for exploring the design space of neural networks. Data parallel training is commonly used to accelerate the learning of Convolutional Neural Networks (CNNs) where input batches are distributed across multiple workers. For example, Korean Patent Publication No. 10-2016-0144467 discloses a technique for parallelizing the training of convolutional neural networks. However, the communication time of the weight gradient may limit the scalability of parallel training when the batch size is limited.

Providing a new multi-dimensional parallel training (MPT) of Winnograd domain convolution that utilizes both the existing data parallel training and intra-tile (or element-wise) parallel processing available in the Winograd domain. It provides a multi-dimensional parallel learning method and a computer device for performing the multi-dimensional parallel learning method.

Multi-dimensional parallel learning method to provide 2D organization of workers with hybrid topology using ring topology for collective communication and high connection topology for tile transmission in order to efficiently support communication between workers using MPT and A computer device for performing the multidimensional parallel learning method is provided.

Provided is a multi-dimensional parallel learning method that provides dynamic clustering that enables an operator to optimize the balance of two types of communication (weighted gradient and tile transmission) in an MPT structure, and a computer device that performs the multi-dimensional parallel learning method.

A multi-dimensional parallel learning method providing a prediction of activation of spatial domain neurons from quantized values of the Winograd domain data without loss of accuracy in order to reduce the bandwidth required for tile collection communication and a computer device performing the multi-dimensional parallel learning method Provides

A multi-dimensional parallel training method for Winograd transform convolution, the method comprising: converting input data including an input characteristic map into a plurality of tiles; Distributing and transmitting the converted plurality of tiles to a plurality of workers arranged in two dimensions by a plurality of clusters and a plurality of groups, and distributing the converted plurality of tiles for each of the plurality of clusters; Performing data parallelism on the distributedly transmitted input data through each worker of the plurality of clusters; And performing intra-tile parallelism on an element unit of the plurality of tiles applied to each of the plurality of groups among the plurality of groups of distributed input data through each of the plurality of groups of workers. It provides a multi-dimensional parallel learning method comprising a.

According to one side, the step of performing the intra-tile parallel processing may include performing the intra-tile parallel processing through a dot product of element units disposed at each of the same locations in the plurality of tiles in the Winograd domain dot product. It can be characterized as.

According to another aspect, workers included in each of the plurality of groups are interconnected through a ring topology supporting collective operation of weights, and workers included in each of the plurality of clusters are all-to-one for tile collection/distribution. Features that are interconnected via a high-connectivity topology (e.g., 2D flattened butterfly (FBFLY) topology or Dragonfly topology) that efficiently supports all-to-all traffic Can be done with

According to another aspect, the multi-dimensional parallel learning method may include the number of the plurality of clusters and the number of the plurality of groups through a host connecting the plurality of groups to each other according to at least one of the size and weight size of the input characteristic map. It may further include the step of dynamically changing.

According to another aspect, a source operator among the plurality of workers transmits a quantized value of a tile element for prediction before transmitting an actual value to a destination worker, and the target worker quantizes the quantized value. It can be characterized by predicting activation of a spatial domain neuron from a value and skipping unnecessary tile collection.

According to another aspect, each of the plurality of workers may be implemented by an expandable NDP (Near-Data Processing) architecture including a control unit, a calculation unit, and a communication unit.

In combination with a computer device, a computer program stored in a computer readable recording medium is provided to execute the multidimensional parallel learning method in a computer device.

A computer-readable recording medium in which a computer program for executing the multidimensional parallel learning method is executed is provided.

A computer device performing a multi-dimensional parallel training method for Winograd transform convolution, comprising: at least one processor implemented to execute readable instructions in the computer device; and At least one processor converts input data including an input characteristic map into a plurality of tiles, and distributes the converted plurality of tiles to a plurality of workers arranged in two dimensions by a plurality of clusters and a plurality of groups. Delivered, the converted plurality of tiles are distributed and distributed for each of the plurality of clusters, and data parallelism is performed on the distributed and transmitted input data through each of the plurality of clusters. It is characterized in that intra-tile parallelism is performed on an element unit of the plurality of tiles applied to each of the plurality of groups, among the plurality of groups, through the workers of each of the groups. To provide a computer device.

We can provide a new multi-dimensional parallel training (MPT) of Winnograd domain convolution that utilizes both intra-tile (or element-by-element) parallel processing and data available in the Winnograd domain.

To effectively support communication between workers using MPT, it is possible to provide a 2D organization of workers with hybrid topologies that use ring topologies for collective communication in data parallel processing and high connection topologies for tile transmission.

In the MPT architecture, you can provide dynamic clustering (rewrite of the connection structure between workers) that allows the operator to optimize the balance of two types of communication (weighted gradients and tile transfer).

To reduce the bandwidth required for tile collection, it is possible to provide prediction of activation of spatial domain neurons from quantized values of the Winograd domain data without loss of accuracy.

1 shows operations (FLOPs) (a) and memory access (L1) for Winograd transform convolution with direct convolution of 4 x 4 and tile size (Wino T4) and tile size (Wino T6) of 6 x 6 These are graphs for comparison of (b) data cache connections.
FIG. 2 is a diagram showing three steps of a convolutional layer having a Winograd transformation.
FIG. 3 is a diagram illustrating a winograd layer directly updating a weight in a winograd domain.
FIG. 4 is a diagram illustrating an example in which a 4D tensor in a spatial domain is internally transformed of tiles in a Winograd domain.
5 is a diagram illustrating an example of a dot product represented by element unit matrix multiplication.
6 is a diagram showing an example of direct convolution.
7 is a diagram showing an example of transform convolution (inner product).
8 is a diagram illustrating an example of data parallel learning.
9 is a diagram illustrating an example of multidimensional parallel learning according to an embodiment of the present invention.
10 is a graph illustrating comparison of traffic transmitted for each operator for repetition of two different layers in one embodiment of the present invention.
FIG. 11 is a graph illustrating the traffic transmitted for each worker each time the learning of FractalNet having different parallel strategies is repeated, according to an embodiment of the present invention.
12 is a diagram illustrating an example of various parallelization configurations using dynamic clustering in an embodiment of the present invention.
13 is a high-level block diagram (a) of a scalable NDP architecture with p =256, N _g =16, N _c =16, and three different from dynamic clustering in one embodiment of the present invention. It is the figure which showed the structure (b), (C), (D).
14 is a diagram illustrating an example of non-uniform quantization of a Winograd domain element with respect to quantization of a Winograd domain tile value predicting activation in an embodiment of the present invention.
15 is a diagram illustrating an example of quantization logic for a single-precision float in relation to quantization of a Winnograd domain tile value predicting activation in an embodiment of the present invention.
16 is a diagram illustrating an example of an activation prediction flow using quantization in an embodiment of the present invention.
17 is graphs showing examples of actual and predicted ratios of inactive tiles/lines for two different data sets in one embodiment of the present invention.
18 is a block diagram illustrating an example of an NDP added to a logic layer of each memory module, which is composed of control, operation, and communication units, in one embodiment of the present invention.
19 is a block diagram illustrating an example of P2P communication logic in an embodiment of the present invention.
20 is a block diagram illustrating an example of collective communication logic in an embodiment of the present invention.
21 is a block diagram showing an example of a computer device according to an embodiment of the present invention.
22 is a flowchart illustrating an example of a data processing method according to an embodiment of the present invention.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various other forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more fully describe the present invention to those skilled in the art. The shape and size of elements in the drawings may be exaggerated for a more clear description.

Accelerated neural network learning is very important for exploring the design space of neural networks. Data parallel processing is commonly used to accelerate the learning of Convolutional Neural Networks (CNNs) where input batches are distributed across multiple workers. However, the communication time of the weight gradient can limit the scalability of a suitable batch size. In embodiments of the present invention, multi-dimensional parallel training of a convolutional layer utilizing both data parallel processing and intra-tile parallel processing that can be used in Winograd transform convolution , MPT). Workers consist of two dimensions. One uses intra-tile parallel processing, while the other uses data parallel processing. MPT reduces the amount of communication required for the weight gradient because the weight gradient is communicated only in the data parallel processing dimension. However, the Winograd transformation essentially requires more data access, and the proposed MPT structure also introduces a new type of communication called tile transfer/gather/scatter of the Winograd dimensional functional maps. do. In one embodiment, scalable near-data processing (LIM) that utilizes a memory-centric network organization to provide high connectivity between operators to accelerate tile transfer, while minimizing the cost of data access through 3D stack memory. NDP) structure. In order to minimize the tile collecting communication overhead, in one embodiment, the prediction of the activation of spatial domain neurons is used to eliminate the communication of tiles converted to non-activated neurons. In addition, in one embodiment, dynamic clustering of a memory-centric network structure that reconstructs a connection topology between workers for each convolution layer is proposed to balance the communication required for weight gradient and tile transmission. According to the evaluation in one embodiment, the proposed MPT using the NDP structure accelerates learning by up to 2.7×, 21× compared to data parallel learning of the NDP structure and a multi-GPU (Graphics Processing Unit) system.

1. Introduction

Rapid learning of neural networks accelerates the exploration of neural networks' design space and enables large-scale neural network research. In one embodiment, the focus is on the Convolutional Neural Network (CNN), which is widely used for image recognition and classification. Data parallel learning is widely used in the convolutional layer of CNN, where input batches are distributed among multiple workers. Model parallel processing may be used in some neural networks, but in one embodiment, data parallel processing may be generally used and focuses on data parallel processing. In parallel training of data, every worker can generate a partial sum (equal to weight and size) of weighted gradients for its own subset of input batches. Weight gradients are reduced or accumulated for all workers, and the updated weights can be broadcast to all workers using stochastic gradient descent (SGD). Using a fixed amount of data generated per worker, the optimal collective communication algorithm for the ring topology (pipeline reduction/broadcast) results in a constant communication time regardless of the number of workers. However, an optimal (pipe lined) implementation for communication may limit scalability to a fixed total batch size. As the number of workers increases, the size of the worker batch or the amount of computation allocated to each worker decreases, and the communication represents a larger portion of the overall execution time. One common approach to data parallel processing is to increase the input batch size. However, previous studies have shown that the large size of the batch can degrade the quality of the trained model, and show that the convergence rate is reduced. In one embodiment, a synchronization SGD having an appropriately sized batch (ie, 128-256) used in CNN is assumed, but an alternative method is proposed to enable scalable learning.

In particular, one embodiment utilizes a new type of parallelism (intra-tile parallelism that can be used in transformed convolutions such as Winograd transform and Fast Fourier Transform (FFT)) to provide better scalability as the number of workers increases. Can provide. Transform convolution has been proposed to reduce computational complexity by changing the convolution operation in the spatial domain to a simpler dot product in the transform domain. When transforming the Winograd, the feature map and weights are converted into Winograd domain tiles with each tile size T × T. The element-wise dot product of the Winograd domain may be composed of independent matrix multiplication of T ² compared to direct convolution consisting of one large matrix multiplication.

By utilizing elemental unit (or intra-tile) independence, an embodiment proposes an MPT that combines intra-parallel data processing with data parallel processing for convolution of Winogram. In addition, in one embodiment, a two-dimensional organization that properly utilizes MPT is proposed. Here, workers in the same column may form a group for parallel processing of data, and workers in the same row may configure a cluster for intra-tile parallel processing. The input arrangement is distributed throughout the cluster (or in groups), while the elements of the Winograd domain tile are distributed throughout the group. Since there is no need to communicate weights between different groups (ie, different parts of the tile), communication for weight reduction/broadcast is limited (isolated) within each group. Therefore, the size of the weight gradient generated per worker is reduced by the number of groups and scalability can be improved.

MPT reduces weight communication (transmission) between workers, but can create additional challenges. MPT includes a new type of communication within a cluster to collect/scatter the Winograd domain characteristic map, which is referred to as tile transmission in embodiments of the present invention. In order to efficiently support tile transmission, embodiments of the present invention propose a hybrid topology. The hybrid topology utilizes a low-diameter (high-radix) topology for tile communication across the group. Alternatively, a ring topology is used for weight communication through the cluster. However, the optimal configuration (i.e., the number of workers for each dimension) can vary depending on the communication requirements of the convolutional layer. To enable structural flexibility, embodiments of the present invention propose dynamic clustering (or reconfigurable topology) of workers to reconstruct the interconnection network based on two-dimensional pre-computed traffic. In order to reduce the traffic for tile collection, embodiments of the present invention propose a method of predicting the activation of neurons without loss of accuracy from quantized values of the Winnograd domain data elements.

While the amount of computation decreases in the Winograd domain, the tiles and weights of the Winograd domain are larger than the feature maps and weights of the spatial domain, resulting in more data access. The comparison of operations and memory accesses for five different layers of direct convolution and two Winograd transform convolutions is illustrated in FIG. 1. According to these results, Winograd transform convolution can reduce the operation to 2.8×, while on average it increases the amount of data access to 4.4×. As a result, embodiments of the present invention propose a scalable NDP structure that utilizes the high-bandwidth of the 3D stack memory for increased data access and creates a memory-centric structure. In particular, embodiments of the present invention include the following.

● We propose a new MPT for Winnograd Domain Convolution that utilizes both intra-tile (or element-wise) parallel processing and data that can be used in the Winnograd domain.

● To efficiently support communication between workers using MPT, we propose a 2D organization of workers with hybrid topologies that use ring topologies for collective communication in data parallel processing and high connection topologies for tile transmission.

● In the MPT architecture, dynamic clustering is proposed, which enables the operator to optimize the balance of two types of communication (weighted gradient and tile transmission).

● To reduce the bandwidth required for tile collection, we propose a prediction of activation of spatial domain neurons from quantized values of the Winograd domain data without loss of accuracy.

2. Background

This study describes the convolution algorithm using the following symbols.

x , y , w : spatial domain input, output property map, weight

X , Y , W : Winograd domain input, output characteristic map, weight

I , J , B : Number of inputs, number of output channels, batch size

A. Convolution Layer

Convolutional layer learning consists of three steps: forward propagation, reverse propagation and weight update. Forward propagation (forward propagation, hereinafter 'fprop') for the input characteristic map x _i are weights w _{i, j} and the convolution ReLU (Rectified Linear Unit) and passes through the non-linear activation function f output characteristic map y _j is calculated as do.

에 대한 손실 함수 L의 그라디언트가 출력 그라디언트

및 플립 가중치 (w ^f )의 컨볼루션에 의해 연산된 후, 활성 함수 (f´)의 미분으로 전달된다.While the backward propagating (backward propagation, hereinafter 'bprop'), input

The gradient of the loss function L for the output gradient

And after being calculated by the convolution of the flip weight ( w ^f ), it is transferred as a derivative of the active function ( f ´).

가 연산된다.During the weight update phase (hereinafter referred to as'updateGrad '), the weight gradient is generated by convolution between the output gradient and the input characteristic map.

Is calculated.

The weight is updated by adding a weight gradient multiplied by the learning rate (or by calculating the average value).

B. Winograd layer

In order to reduce the number of products by replacing the convolution operation with the element-wise dot product operation (⊙), a Winograd algorithm for 2D convolution has been proposed. The 2D Winograd transformation can be expressed as F ( m × m , r × r ), and can be expressed as Equation 4 below.

Here, w is an r × r size weight, y is an m × m size output, and G , B , and A are coefficient matrices. Theoretically, compared to direct convolution, F (4 × 4, 3 × 3) reduces the product by a factor of four by transforming the convolution operation internally.

In order to apply the Winograd algorithm to the convolutional layer, the prior art proposed a tile-based approach. In the input characteristic map, each tile is converted into a plurality of tiles having a size of T × T (where T = m + r -1), and weights are also converted to tiles having the same size of T × T. The dot product between the plurality of input tiles and the weight tile generates output tiles, and the output tiles are inversely transformed into an output characteristic map. At this time, the overall operation for fprop can be expressed by a matrix equation as shown in Equation 5 below.

Here, ( u , v ) denotes elements located in the u- th row and v- th column of the tile.

와

로 교체되는 것을 제외하고 fprop와 유사한 연산 및 데이터 세트를 가진다. 위노그라드 레이어는 최근에 제안되었으며, 이 제안에서 가중치는 도 3에서와 같이 위노그라드 도메인에서 직접 업데이트될 수 있다. 이러한 점은 위노그라드 레이어가 본 발명의 실시예들에서 사용되는 작은 가중치/타일 크기에 대해 학습된 네트워크의 품질을 감소시키지 않는 반면, 위노그라드 도메인 가중치가 공간 도메인 가중치에 비해 큰 사이즈를 가져 정확성을 증가시킴을 보여준다. Nvidia의 현재 cuDNN은 위노그라드 변환 컨볼루션을 지원하지만, 위노그라드 변환 컨볼루션이 정확도에 영향을 미치지 않는 사이즈인 3 × 3 및 5 × 5 로 그 사이즈가 제한된다. 그러나, 가중치/타일 사이즈가 증가하면, 수치적 불안정성이 증가하여 정확성에 영향을 줄 수 있다. 최근 연구는 향상된 위노그라드 변환 매트릭스를 통하여 향상된 안정성과 정확성을 달성하는 방법을 활용하였다. 이러한 방법을 통해 정확성이 향상되면, 본 발명의 실시예들에서 제안하는 다중 차원 병렬 구조가 더욱 큰 가중치 사이즈로 확장될 수 있다.Figure 2 shows the fprop , updateGrad of the convolution layer using the Winograd transformation. bprop has input and output respectively

Wow

It has a similar operation and data set as fprop , except that it is replaced by. The Winograd layer was recently proposed, and in this proposal, the weight can be directly updated in the Winograd domain, as shown in FIG. 3. This does not reduce the quality of the learned network for the small weight/tile size used in the embodiments of the present invention, while the Winograd domain weight has a larger size compared to the spatial domain weight, thus providing accuracy. Increase. Nvidia's current cuDNN supports Winograd transform convolution, but its size is limited to 3 × 3 and 5 × 5, which are the sizes that Winograd transform convolution does not affect accuracy. However, as the weight/tile size increases, numerical instability increases, which can affect accuracy. A recent study utilized a method to achieve improved stability and accuracy through an improved Winograd transformation matrix. If the accuracy is improved through this method, the multi-dimensional parallel structure proposed in the embodiments of the present invention can be extended to a larger weight size.

C. Related Research

Accelerator: Another CNN accelerator architecture has been proposed. This prior art maximizes data reuse and power efficiency for inference on a single chip. While most accelerators have been proposed with an emphasis on reasoning, a heterogeneous server architecture has recently been proposed for neural network learning for scalability and efficient computation. In order to efficiently utilize available communication bandwidth, scalable parallel learning using multiple GPUs has been widely studied, including ring communication for data parallel processing or acceleration of collective communication using a tree topology. In contrast, embodiments of the present invention propose to reduce the amount of communication as the number of workers increases by using intra tile parallelism in transformed convolution and creating independent group operations groups.

Scalable parallel learning: Another approach for parallel learning is to increase the batch size, but neural network models trained with large batch sizes tend to be less generalized (called the "generalization gap"). Recently, deep learning has been actively conducted to increase the batch size while maintaining the quality of the learning model by linearly adjusting the learning rate according to the batch size. However, a large batch size can reduce the range of learning speeds that provide stable convergence with acceptable test accuracy. This is because a smaller batch size provides more weight values. Large batch-sized learning can take longer to provide the same quality as a trained network. Due to the limited adjustment of the batch size, the ratio of operation to communication increases as the number of workers increases (or the computing power of each worker increases as the architecture evolves). In order to provide high scalability with a reasonable batch size, embodiments of the present invention reduce communication time through the proposed multidimensional parallel processing and hybrid interconnect structure.

3D Integration: Recent advances in 3D integration technology enable 3D stacked dies with through-silicon vias (TSV). A hybrid memory cube (HMC) is an example of a 3D stack memory in which a logic die is present at the bottom, and includes processing elements for near data processing (NDP). Each HMC module that becomes a router can be interconnected by a high-speed link to expand the system and create a memory-centric network. Proximity data processing for CNN has recently been proposed as a way to utilize the high bandwidth and energy efficiency of 3D stack memory. However, the previous study focuses on the reasoning of a single memory module, while embodiments of the present invention focus on learning about multiple memory modules. Several recent studies have evaluated multiple memory modules, but the number of memory modules is limited to a few (up to 4), and the communication overhead between memory modules is not properly evaluated. In contrast, embodiments of the present invention focus on scalable learning using a large number of workers (memory modules) and focus on the impact of communication on scalability. The embodiments of the present invention assume a high-bandwidth serial link connection between operators. NVlink is an example of such an interconnect, and recently announced interconnect standards in the industry, including CCIX, OpenCAPI, and GenZ, aim to interconnect interconnect accelerators and/or memory modules to provide high-bandwidth. This is an example of interconnection.

3. Scalable parallel learning

Hereinafter, MPT for increasing scalability by combining data parallel processing and intra-tile parallel processing will be described, and the scalability problem of MPT will be described.

A. Parallel Processing in Transform Convolution

The dot product of the transformation convolution can be represented as shown in FIG. 4. Each input characteristic map is converted into t tiles along with the size of each tile of T × T , thereby generating a total of B × t tiles per input channel. By decomposing each tile having a size of T × T into T ² elements, the dot product of the tiles can be represented by T ² independent element unit matrix multiplication as shown in Equation 5 and FIG. 5.

In the Winograd domain dot product, no operation is performed between elements disposed at different locations in the tile (ie, different ( u , v )). This independence or parallelism that can be used in transformation convolution can be utilized for parallelization with minimal communication. In this embodiment, this is called intra-tile parallelism. 6 and 7 provide different views of intra-tile parallel processing compared to direct convolution. With direct convolution, most input data elements (or neurons) are shared to compute different output elements, and partitioned execution by element requires large input data redundancy. In contrast, transformations can change convolution internally on an element-by-element basis and separate input data into element-by-element, enabling efficient element-by-element (or intra-tile) parallel execution. 6 and 7 show an example of the updateGrad step, but different characteristics of direct convolution and transform convolution are applied to the fprop and bprop steps.

B. Multi-dimensional Parallel Training (MPT)

또는

)를 감소시키고 업데이트된 가중치를 브로드캐스팅해야 한다. 따라서, updateGrad 단계에서 각 작업자가 전달할 통신의 양은 거의 고정되어 있고 작업자의 수(p)에 관계없이 가중치 사이즈 |w|에 비례한다.For parallel learning of CNN, data parallel learning is widely used in the convolutional layer. Parallel training of data does not require inter-worker communication in the fprop and bprop phases, but in the updateGrad phase the weight gradients generated by all workers (

or

) And broadcast the updated weights. Therefore, in the updateGrad phase, the amount of communication to be delivered by each worker is almost fixed, and regardless of the number of workers (p), the weight size | proportional to w |

In this embodiment, an MPT combining data parallel processing and intra-tile parallel processing is proposed. When using MPT, as shown in FIGS. 8 and 9, workers existing in the same row form a cluster, and workers are arranged in two dimensions, and workers in the same column form a group. Data parallelism is applied to the entire cluster, as input batches are distributed throughout the N _c cluster, and intra-tile parallelism is applied to the entire group to distribute the elements of the Winograd domain tile to the N _g group. Therefore, in the case of N _c × N _g = p and N _g = 1, only data parallel processing is utilized for the entire operator. As can be seen in FIGS. 5 and 7, no operation is performed between different tile elements or between different groups for dot products. Therefore _, each part of the Winograd domain weights ( W _{u , v} ) is used only within the associated group. Tile transmission occurs in the entire group (or in each cluster) only for transformation between spatial domain neurons and Winograd domain tiles in the fprop and bprop steps described later. In the updateGrad phase, each worker generates a weight gradient for a portion of the weight assigned to the worker group, and communication of the weight gradient occurs within each group. As a result, the size of the weight gradient, which should be reduced and broadcast per worker, is reduced by N _g .

C. Challenges of MPT

MPT reduces communication for weight updates, but introduces a new type of communication. In the fprop and bprop phases, the input tiles are divided into elements and distributed for intra-tile parallel processing to workers in the cluster. This is called tile scattering . Each worker loads the weight elements and computes the output elements for a specific portion of the tile assigned to the worker group. The different parts of each output tile are combined to form a complete tile for inverse transformation to a spatial domain property map (or neuron) to communicate the tiles within the cluster. This is called tile gathering . Spatial domain neurons pass through the nonlinear activation and pooling (if any) layer and become the input of the next layer. FIG. 10 compares the amount of traffic transmitted by each worker for each repetition of two different layers in Table 1 below.

Abbr. CNN layer I , J x ( y ) dim. w dim. Early ResNet-34 conv2_x 128, 128 56 × 56 3 × 3 Mid-1 ResNet-34 conv4_x 256, 256 14 × 14 3 × 3 Late-1 ResNet-34 conv5_x 512, 512 7 × 7 3 × 3 Mid-2 WRN-40-10 conv3 320, 320 16 × 16 3 × 3 Late-2 WRN-40-10 conv4 640, 640 8 × 8 3 × 3

를 가정하였다. 도 10(b)에 표현된 바와 같이, 작은 특성 맵과 더 큰 가중치 크기가 존재하는 후반(late) 레이어의 경우, MPT는 가중치 통신을 크게 감소시키고, 타일 전송은 많은 수의 작업자(p)를 가지는 전체 통신량의 작은 부분으로 구성된다. 비교적 큰 특성 맵 크기를 갖는 레이어에서(예를 들어, Mid-1), MPT는 여전히 가중치 통신을 현저히 감소시키지만, 타일 전송은 많은 양의 통신을 초래하여 MPT의 이점을 감소시킨다(도 10(a)).Table 1 shows examples of five convolutional layers used in evaluation according to an embodiment of the present invention.

Was assumed. As shown in FIG. 10(b), in the case of a late layer having a small characteristic map and a larger weight size, MPT greatly reduces weight communication, and tile transmission significantly reduces the number of workers p. The branches consist of a small portion of the total traffic. In a layer with a relatively large characteristic map size (e.g. Mid-1), MPT still significantly reduces weight communication, but tile transmission results in large amounts of communication, reducing the benefits of MPT (Figure 10(a). )).

이며, p가 크면 거의 일정하다. MPT의 경우, 가중치는 N _g 그룹으로 나누어지고, 각 작업자는 그룹 당 N _c - 1 홉 순회(traversals)를 이용하여

가중치 그라디언트만 전송한다. 따라서, 가중치 그라디언트에 대한 작업자 1 인당 통신량은

가 된다. 타일 전송의 경우, 입력 배치(B의 크기)가 N _c 클러스터로 분할되고, 타일이 각 클러스터 내부의 N _g 그룹으로 분할된다. 따라서, 각 작업자는

타일 데이터를 보유하고, 타일 데이터의

부분을 동일한 클러스터의 다른 작업자에게 전송한다. 타일 전송을 위한 작업자 1인당 통신량은

가 된다.In order to understand the scalability of MPT in the added communication, data parallel processing and traffic from MPT can be analyzed first. For parallel training of data using a ring topology, each worker | W | The weight gradient of size is transmitted to the next worker, and the hop count is p -1. Therefore, the traffic per worker is

And p is large, almost constant. In the case of MPT, weights are divided into N _g groups, and each worker uses N _c -1 hop traversals per group.

Only the weight gradient is transmitted. Therefore, the traffic per worker for the weight gradient is

Becomes. For tile transmission, the input arrangement (the size of B ) is divided into N _c clusters, and the tiles are divided into N _g groups within each cluster. Therefore, each worker

Holds the tile data, and the tile data

The part is sent to other workers in the same cluster. The traffic per worker for tile transmission is

Becomes.

를 가정한다. 작업자 수가 적으면, 타일 전송 오버 헤드로 인해 데이터 병렬 학습과 비교할 때 실제로 통신 오버 헤드가 MPT보다 높다. 데이터 병렬 학습의 경우, 고정된 총 배치 크기로 작업자 수가 증가함에 따라 각 작업자에게 할당된 연산량이 감소하기 때문에, 작업자 당 통신량은 더 큰 p로 일정하게 유지되고 확장성이 떨어지게 된다. MPT의 경우, 작업자 당 통신량은 가중치 그라디언트에 대한

와 타일 전송에 대한

에 비례하므로 작업자의 수(p)가 증가함에 따라 작업자 1 인당 통신량은 감소한다.FIG. 11 shows an example in which the amount of data (communication amount) delivered for each worker is displayed whenever FractalNet learning is repeated in all layers. At this time, the fixed batch size is 256

Suppose When the number of workers is small, the communication overhead is actually higher than that of MPT compared to data parallel learning due to tile transmission overhead. In the case of data parallel learning, since the amount of computation allocated to each worker decreases as the number of workers increases with a fixed total batch size, the communication amount per worker remains constant at a larger p and the scalability decreases. In the case of MPT, the traffic per worker is the weight gradient.

For tile transfer

Since it is proportional to the number of workers ( p ) increases, the traffic per worker decreases.

The traffic required for the weight and tile transmission is significantly different depending on the structure of the convolutional layer, and the size of the two parallel processing domains (ie, N _g , N _c ) determines the traffic for the weight and tile transmission. In addition, the weight and the communication amount of tiles are in a trade-off relationship. As there are more groups, weight communication decreases, but tile transmission increases. If the number of groups decreases, weight communication increases, but tile transmission decreases. For example, in a single group, only data parallel processing remains and tile transmission is completely eliminated. Therefore, the optimal configuration of the MPT is to minimize the total traffic change for each convolution layer. In this embodiment, dynamic clustering that can reconstruct MPT per layer to minimize overall communication is proposed. In addition, Fig. 11 shows the result of MPT using dynamic clustering, and the required communication volume is reduced. For a small number of workers, the benefits of dynamic clustering are important because fixed interconnects result in inefficient use of bandwidth across different layers. However, the advantage of dynamic clustering still leads to reduced communication by 1.4× for p =256. Hereinafter, details of dynamic clustering using a hybrid topology are presented, and in section V, an activation prediction method for predicting activation of a neuron and bypassing a collection of tiles converted to non-active neurons to reduce the traffic of tile collection is presented. . 12 shows the application of dynamic clustering and activation prediction to MPT.

4. Dynamic clustering

크기 데이터에 대해 집단 연산을 수행한다. 클러스터 내부의 16명의 작업자가 타일 수집/분산을 위해 올-투-올 트래픽(all-to-all traffic)을 효율적으로 지원하는 고밀도 네트워크인 2D FBFLY(flattened butterfly)가 사용된다.In this embodiment, the proposed dynamic clustering that can reconstruct the number of groups N _g and clusters N _c to match the characteristics of the neural network layer is described. The high-level architecture is shown in Figure 13(a) and consists of workers or "memory modules" interconnected through a memory-centric network. 13 shows an architecture with 256 workers organized through two domains (groups and clusters) to match the MPT proposed in this embodiment. Workers are interconnected with ring interconnects in each group through a memory-centric network to support the collective computation of weights. Compared to data parallel learning, MPT is multiple and produces short rings. That is, in the case of the p worker, the length of the ring is reduced from p (data parallel) to N _c (MPT), resulting in an N _g ring. Here, each ring of the N _g ring is only

Perform a group operation on the size data. A 2D flattened butterfly (FBFLY), a high-density network that efficiently supports all - to - all traffic, is used for tile collection/distribution by 16 workers inside the cluster.

As described above, the group number N _g and the cluster number N _c create a trade-off of traffic for weight gradients and tile collection/scattering. A smaller N _g (and a larger N _c ) is preferred for the initial layer with a larger feature map size to minimize tile transfer, and a later layer with a larger weight size to minimize the amount of weight gradient. Larger N _g (and smaller N _c ) are preferred. In order to enable flexible configuration of workers for each layer, this embodiment proposes dynamic clustering to reconfigure interconnects to maximize scalability while modifying the number of groups and clusters and minimizing communication time.

In order to dynamically change the topology (or modify N _g and N _c ), this embodiment utilizes a connection through a host connecting several groups to each other. When N _c increases (or N _g decreases), connections through the host are utilized to increase the size of the group. Since the neural network has a fixed layer structure and the required traffic and link bandwidth can be calculated in advance, the optimal configuration for each layer to minimize communication time is predetermined and does not change. Reconstruction between the two layers modifies the path (or target) of tile transmission and weight communication, and no additional data transmission or overhead occurs.

The evaluation supports three types of workers:

● 16 N _g , 16 N _c : No routing through the host.

● 4 N _g , 64 N _c : gr0⇔gr3, gr4⇔gr7, gr8⇔gr11, gr12⇔gr15

● 1 N _g , 256 N _c : gr0⇔gr15, gr3⇔gr4, gr7⇔gr8, gr11⇔gr12

In the above,'⇔' represents an additional connection formed between two groups through a host ('Host CPU' in FIG. 13(a)). The first configuration (16 N _g , FIG. 13(b)) can be used to minimize weight communication. In the case of an F (2×2, 3×3) transform with a tile size of 4×4, a single element of the tile is assigned to each group. In order to interconnect 16 workers in each cluster, a 2D FBFLY topology is used, and tile data can be transmitted at a maximum of 2 hop counts. The second configuration (4 N _g , FIG. 13(c)) has more weight communication than the configuration using 16 N _g , but has less tile transmission. For an F (2 × 2, 3 × 3) transformation with a tile size of 4 × 4, 4 elements (a single line of 2D tiles) are assigned to each group, and a 1D Winograd transformation is executed before transmitting the tile data. The tile transmission can be further reduced. Details are described in the next section V. Four fully connected workers (eg, workers 3, 7, 11, 15 in FIG. 13(a)) form a cluster, and tile data may be transmitted in a single hop. The third configuration (1 N _g , FIG. 13(d)) returns to data parallel processing without communication for tile transmission, but has the largest weighted traffic. Depending on the characteristics of each layer, a more optimal configuration is selected to minimize communication time and maximize performance.

5. Activation prediction

Using the proposed MPT, tile transmission-collection/scattering of the Winograd domain characteristic map (tile) is essential. Hereinafter, a method for minimizing communication required for tile transmission will be described. To reduce the traffic for tile scattering, zero value skipping is used, and an activation prediction method is proposed for tile collection.

A. Activation prediction without loss of accuracy

In order to reduce the traffic of tile collection, this embodiment proposes a method for predicting activation without loss of accuracy. In this embodiment, it is assumed that a rectified linear unit (ReLU) is a nonlinear activation function that transmits only neurons having a positive value to the next layer. If all the neurons converted in a tile are not activated, the inverse transformation of the tile may be omitted. In order to skip unnecessary tile collection, the present embodiment proposes that the source worker transmits the quantized value of the tile element for prediction before transmitting the actual value to the destination worker. Because transformation is involved, the main challenge is to predict activation of spatial domain neurons from quantized values of the Winograd domain data (tiles). In the proposed method, the target operator performs conservative prediction by calculating the estimated value of the neuron and the maximum possible quantization error together. Since multiple quantization values are multiplied and added to the coefficients, quantization errors are accumulated in the transformation process.

In experiments using the ImageNet data set and the trained CNN, we observed that the values of the Winnograd domain tiles follow a normal distribution. In order to properly follow the normal distribution of actual values, in this embodiment, non-uniform quantization is used as in FIG. The range of real values is divided into several regions, and each region has the same number of steps. Real values out of range are indicated by overflow. When the real value area is changed, the step size Δ is doubled, and the step size is determined by the standard deviation σ of the real value. To implement non-uniform quantization for a single precision float number, a simple hardware implementation that precomputes and stores logarithmic values of simple integer arithmetic logic, bit shifters, and 1/Δ is shown in FIG. 15. Is shown in

16 shows the entire flow of activation prediction, and may be classified as 1D prediction or 2D prediction according to the amount of data allocated to each group. If the number of groups is small, each worker can include the entire line (row or column) of tiles, and the first 1D transform can be computed on the original worker using real values. After the first 1D transformation of the original worker, the quantization value of the 1D transformation output is transmitted to the target worker for prediction. This prediction is called 1D prediction. In contrast, in a large number of groups, each worker does not include the entire line of tiles, and the original worker sends the quantization value of the output element (part of the tile) to the target worker for prediction of activation. This prediction is called 2D prediction.

In the subject worker, a 2D transform is applied to both the quantization value and the quantization resolution to calculate the estimated value of the neuron and the maximum (positive) possible error of quantization. Resolution represents the quantization step size (1/2/4/8Δ in FIG. 14) or the maximum difference between the real value and the quantization value. During the first 1D transformation, positive (+) and negative (-) maximum possible errors are calculated. The Winograd transformation is basically a matrix multiplication using a co-efficient matrix as in Equation (4). The positive (negative) maximum possible error of the first 1D transformation is calculated by adding only positive (negative) terms during matrix multiplication. During the second 1D transformation, the final maximum positive error is calculated by multiplying the positive (negative) maximum error of the first 1D transformation by the positive (negative) coefficient. To perform conservative prediction without loss of accuracy, neurons are expected to be inactive only if “estimated value + max. error” is less than zero. Thus, the proposed method does not allow false-negatives to be predicted to be non-activated for activated neurons.

B. Prediction accuracy and tile transmission

17 shows the actual and predicted ratios of inactive tiles and lines using the F (2×2, 3×3) transform. In this embodiment, this ratio is measured using two data sets, CIFAR and ImageNet, with pre-trained weights. The inactive tile (line) indicates that the neurons transformed in the tile (line) are predicted to be all inactive in the 2D prediction case (1D prediction case). The dotted line in Fig. 17 shows the ratio measured by the actual value, and represents the upper limit of prediction. The non-uniform quantization of the four regions agrees well with the distribution of the Winograd domain values and provides the best prediction results for all test cases. Using 64 quantization levels for 2D prediction (6 bits) and 32 quantization levels for 1D prediction (5 bits), activation prediction can reduce tile collection communication by 34.0% and 78.1%, respectively. Overall, since the first 1D transformation is performed on the source worker and the accumulation of quantization errors is reduced by bypassing the target worker, 1D prediction exhibits higher prediction accuracy compared to 2D prediction.

For scattering of the input tile, the zero value of the input tile can be omitted. Zero skip reduces input tile scattering communication by 39.3% and 64.7% for 2D prediction and 1D prediction, respectively. The skipped data is filled with zero values in the operator calculation dot product. Information of the data skipped for zero skip and tile data predicted to be deactivated for activation prediction are shared between the target worker and the original worker through an activation map of input and output tiles.

6. Extensible Near-Data Processing (NDP) architecture

In this embodiment, NDP can be used as an operator. As shown in FIG. 1, the Winograd transform convolution can reduce computation, but can increase the amount of data accessed. In order to enable a high bandwidth of memory, in this embodiment, a 3D stack memory having a through-silicon visa (TSVs) and a logic layer enabling NDP may be used. In this embodiment, a scalable NDP architecture that implements the proposed MPT, dynamic clustering and activation prediction is discussed. 18 is a block diagram illustrating an example of an NDP added to a logic layer of each memory module composed of control, operation, and communication units in an embodiment of the present invention.

A. Control Units

When starting CNN learning, the host builds a task graph of the given CNN structure and parameters (such as the batch size and transformation algorithm). Nodes represent a single task and edges between nodes can represent dependencies on data. In this embodiment, a task can be created as an operation block suitable for a calculation unit (systolic array) and operated simultaneously. Therefore, one convolutional layer may be composed of several task nodes. To reduce synchronization overhead, an update counter based on dependency checking may be used in this embodiment. The property map can have a dependency on the previous layer, and the weight can have a dependency on the previous iteration. After each task is completed, the update counter can be incremented, and the next task can be started when all the updater counters connected to the previous tasks are updated. Task graphs can be stored in each NDP, and under NDP control, the task scheduler can load tasks in a predefined order and start computation and program DMA when dependency checking passes.

B. Computation Units

Since most of the neural network layers can be mapped to matrix multiplication, computational arrays are commonly used in CNN hardware accelerators. This embodiment also assumes a computational array in the proposed architecture. In this embodiment, assuming that the logic layer is manufactured at 28 nm, it is assumed that the 64×64 FP32 MAC unit occupies 31% of the logic layer area. The number of MAC units can be determined by considering not only the area overhead but also the available DRAM bandwidth and computational balance. The computational array receives input data streams from both sides of the array boundary, and one of the two input streams needs to be modified to compute a new output. Thus, half of the input data is unchanged and reused in the on-chip buffer, while the other half of the input data can be fetched from DRAM in the worst case scenario. Since one side of the input data stream is 64-line and 4-byte wide per lane, 256 GB of data bandwidth per second is required at a logic frequency of 1 GHz. This matches well with the bandwidth of 320G per second of 3D stack memory. For overlapping operation and communication, the input buffer has double buffering, and the output buffer has a capacity of 128 KB, while 512 KB for each instance of the input buffer (2 MB total) to completely store the weight of the hierarchical structure of the convolutional layer. Can have a size The area for the input and output buffers was estimated at CACTI6.5 and found to be 3.19 mm ² at 28 nm (4.7% of the overhead area). Vector processors can add programming capabilities to the NDP operation unit for pre- and post-processing of matrix multiplication, such as activation (ReLU), pooling, and simple additions between feature maps (such as residual connections or join operations). And the command sequence can be loaded from the task descriptor. In this embodiment, a vector processor based on a scratch-pad memory can be used. Here, the vector processor may load input data directly from the scratch-pad memory or directly store output data in the scratch-pad memory. Layers of neural networks are usually large in input/output data and can perform static and regular operations. Pre- and post-processing of convolution operations are mostly operations on streaming data (one operation for each data), such as applying ReLU/pooling to the output feature map. Scratch-pad memory with this characteristic (wide and aligned input data) can efficiently support a wide range of vector processing devices. The vector processor's scratch-pad memory can also provide a dual buffering architecture (e.g., 512KB size for each buffer) where the DMA stores the output data in DRAM and preloads the input data.

C. Communication Unit

19 and 20 show the architecture of the two communication processing elements. As described above, unicast communication logic used for tile transmission such as peer-to-peer (P2P) may include a transformation unit and quantization/prediction logic. After activation prediction, data packing (or compression) of unnecessary tile data may be performed before transmitting tile data. Also, instead of moving the data itself to reduce the movement of data in the register, a pointer-based shift register can be used. The output tile can be copied from the output buffer of the computational array to the free data buffer. At this time, each buffer may be designated for each target operator. The output activation map stores the result of the activation prediction and partially moves (shifts) the pointer register so that unnecessary data (inactive tiles) can be compressed. For tile collection, the data indicated by the point register can be packetized and transmitted to the target worker. For scattering using zero skip, the skipped data may be filled with zeros at the receiving side of the worker by referring to the input activation map shared between the source worker and the target worker.

For collective communication of weights, ring-based collective operation can be used in the proposed hardware architecture (FIG. 20). The message may be divided into several chunks and transmitted in parallel, as referred to as pipeline transmission. Each chunk can be mapped to a single packet inside a memory-centric network. In order to avoid blocking the entire ring by slow workers and to maximize bandwidth, the collective action of multiple messages can be processed simultaneously. Chunks of the same message arrive in a predefined order, but chunks of different messages can arrive in any order. In order to properly process chunks arriving in a predefined order, several reduce blocks and communication buffers can be implemented. At this time, a specific message may be assigned to each reduction block. When a chunk arrives from a link or local operation unit, the decrement block checks whether the chunk is in the connected communication buffer, and updates the chunk in the communication buffer (if the chunk exists) or stores it (if the chunk does not exist). . When the next worker is ready to receive the chunk, the updated chunk can be sent to the next worker. The next worker or target worker can be determined by the ring direction and dynamic clustering configuration. After the weight gradient reduction is complete, all weights can be updated and broadcast before being stored in DRAM.

21 is a block diagram showing an example of a computer device according to an embodiment of the present invention. The multi-dimensional parallel learning method according to the present embodiment may be performed by at least one computer device, and each of the at least one computer device may include an internal configuration such as the computer device 2100 illustrated in FIG. 21.

21, the computer device 2100 may include a memory 2110, a processor 2120, a communication interface 2130, and an input/output interface 2140. The memory 2110 is a computer-readable recording medium, and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, a non-destructive large-capacity recording device such as a ROM and a disk drive may be included in the computer device 2100 as a separate permanent storage device separate from the memory 2110. Also, an operating system and at least one program code may be stored in the memory 2110. These software components may be loaded into the memory 2110 from a computer-readable recording medium separate from the memory 2110. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD/CD-ROM drive, and memory card. In other embodiments, software components may be loaded into memory 2110 through communication interface 2130 rather than a computer-readable recording medium. For example, software components may be loaded into memory 2110 of computer device 2100 based on a computer program installed by files received over network 2160.

The processor 2120 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 2120 by memory 2110 or communication interface 2130. For example, the processor 2120 may be configured to execute an instruction received according to program code stored in a recording device such as the memory 2110.

The communication interface 2130 may provide a function for the computer device 2100 to communicate with each other through the network 2160. For example, requests, commands, data, files, etc. generated by the processor 2120 of the computer device 2100 according to program codes stored in a recording device such as the memory 2110 are controlled by the communication interface 2130. 2160) to other devices. Conversely, signals, commands, data, files, and the like from other devices may be received by the computer device 2100 through the communication interface 2130 of the computer device 2100 via the network 2160. Signals, instructions, data, etc. received through the communication interface 2130 may be transferred to the processor 2120 or the memory 2110, and files and the like may be further stored by the computer device 2100 (described above) Permanent storage device).

The input/output interface 2140 may be a means for interfacing with the input/output device 2150. For example, the input device may include a device such as a microphone, keyboard or mouse, and the output device may include a device such as a display or speaker. As another example, the input/output interface 2140 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 2150 may be configured as a computer device 2100 and a single device.

Also, in other embodiments, the computer device 2100 may include fewer or more components than those in FIG. 21. However, there is no need to clearly show most prior art components. For example, the computer device 2100 may be implemented to include at least some of the input/output devices 2150 described above, or may further include other components such as a transceiver, a database, and the like.

The communication method is not limited, and a communication method that utilizes a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcast network) that the network 2160 may include, as well as Bluetooth or NFC (Near Field Communication). Short-range wireless communications such as can also be included. For example, the network 2160 includes a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , Any one or more of the networks such as the Internet. Further, the network 2160 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. It is not limited.

22 is a flowchart illustrating an example of a data processing method according to an embodiment of the present invention. The data processing method according to the present embodiment may be performed by the computer device 2100 described above as an example. For example, the processor 2120 of the computer device 2100 may be implemented to execute control instructions according to code of an operating system included in the memory 2110 or code of at least one program. Here, the processor 2120 is a computer device 2100 so that the computer device 2100 performs steps 2210 to 2240 included in the method of FIG. 22 according to a control command provided by a code stored in the computer device 2100. Can be controlled.

In operation 2210, the computer device 2100 may convert input data including an input characteristic map into a plurality of tiles. For example, the computer device 2100 may convert the input characteristic map to t tiles having a tile size of T × T , thereby generating a total of B × t tiles per input channel.

In step 2220, the computer device 2100 distributes the converted plurality of tiles to a plurality of workers arranged in two dimensions by a plurality of clusters and a plurality of groups, and converts the converted plurality of tiles into a plurality of clusters. It can be distributed and delivered. At this time, workers included in each of the plurality of groups are interconnected through a ring topology that supports collective calculation of weights, and workers included in each of the plurality of clusters are all-to-all traffic (all-) for tile collection/distribution. It can be interconnected through a high-connectivity topology that efficiently supports to-all traffic. As an example, the high connectivity topology may include a 2D flattened butterfly (FBFLY) topology or a dragonfly topology (FBFLY).

According to an embodiment, the computer device 2100 may dynamically change the number of the plurality of clusters and the number of the plurality of groups through a host connecting the plurality of groups to each other according to at least one of the size and weight size of the input characteristic map. have. In one example, the host may correspond to the'Host CPU' shown in Figure 13 (a).

In operation 2230, the computer device 2100 may perform data parallelism on distributedly transmitted input data through workers of each of the plurality of clusters.

In operation 2240, the computer device 2100 performs intra-tile parallel processing of element units of a plurality of tiles applied to each of a plurality of groups among input data distributed through a plurality of groups of respective workers. parallelism). For example, the computer device 2100 may perform the intra-tile parallel processing through the dot product of the element unit disposed at the same location of each of the plurality of tiles in the Winograd domain dot product.

At this time, when a source worker among a plurality of workers transmits a tile to a destination worker, the quantized value of the tile element for prediction is transmitted before the actual value is transmitted, and the target worker quantizes the above. Predicting activation of spatial domain neurons from the value can skip unnecessary tile collection.

Further, each of the plurality of workers may be implemented by an expandable NDP (Near-Data Processing) architecture including a control unit, a calculation unit, and a communication unit. For example, each of the plurality of workers stores a task graph created on the host, loads the tasks in a predefined order based on the task graph, and controls a control counter based on dependency checking, for matrix multiplication It may include a computational unit including a Systolic Array and a vector processor for adding a programming function for pre- and post-processing of the matrix multiplication and a communication unit including unicast communication logic and ring-based collective communication logic. .

As described above, according to embodiments of the present invention, a new multi-dimensional parallel learning of Winnograd domain convolution that utilizes both data and intra-tile (or element-by-element) parallel processing that can be used in the Winograd domain Training, MPT). In addition, to efficiently support communication between workers using MPT, it is possible to provide a 2D organization of workers with a hybrid topology using a ring topology for collective communication and a high connection topology for tile transmission in data parallel processing. In addition, the MPT architecture can provide dynamic clustering (rewrite of the connection structure between workers) that allows the operator to optimize the balance of two types of communication (weighted gradients and tile transfer). In addition, it is possible to provide prediction of activation of spatial domain neurons from quantized values of the Winograd domain data without loss of accuracy in order to reduce the bandwidth required for tile collection.

The system or device described above may be implemented as a hardware component, or a combination of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable gate arrays (FPGAs). , A programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers. The processing device may perform an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and/or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The medium may be a computer that continuously stores executable programs or may be temporarily stored for execution or download. In addition, the medium may be various recording means or storage means in the form of a combination of single or several hardware, and is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks, And program instructions including ROM, RAM, flash memory, and the like. In addition, examples of other media may include an application store for distributing applications or a recording medium or storage medium managed by a site, server, or the like that supplies or distributes various software. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

In the multi-dimensional parallel training method for Winograd transform convolution,
Distributing and transmitting input data to a plurality of workers, which are arranged in two dimensions by a plurality of clusters and a plurality of groups, by distributing the plurality of tiles for each of the plurality of clusters;
Performing data parallelism on the distributedly transmitted input data through each worker of the plurality of clusters; And
Performing intra-tile parallelism on the element unit of the plurality of tiles applied to each of the plurality of groups among the plurality of groups of distributed input data through the workers of each of the plurality of groups.
Multidimensional parallel learning method comprising a. According to claim 1,
The step of performing the intra-tile parallel processing,
A multi-dimensional parallel learning method characterized in that the intra-tile parallel processing is performed through a dot product of element units arranged at the same location in each of the plurality of tiles in the Winnograd domain dot product. According to claim 1,
Workers included in each of the plurality of groups are interconnected through a ring topology supporting collective calculation of weights,
Workers included in each of the plurality of clusters are interconnected through a high-connectivity topology that efficiently supports all-to-all traffic for tile collection/distribution.
Multi-dimensional parallel learning method characterized in that. According to claim 1,
Dynamically changing the number of the plurality of clusters and the number of the plurality of groups through a host connecting the plurality of groups to each other according to at least one of the size and weight size of the input characteristic map.
Multi-dimensional parallel learning method further comprising a. According to claim 1,
Among the plurality of workers, a source worker transmits a quantized value of a tile element for prediction before transmitting a real value to a destination worker, and the target worker uses a spatial domain neuron from the quantized value. A multi-dimensional parallel learning method characterized by skipping unnecessary tile collection by predicting activation. According to claim 1,
Each of the plurality of workers is implemented in a scalable NDP (Near-Data Processing) architecture including a control unit, a calculation unit, and a communication unit. A computer program stored in a computer-readable recording medium in combination with a computer device for executing the method of any one of claims 1 to 6 in a computer device. A computer-readable recording medium on which a computer program for executing the method of any one of claims 1 to 6 is recorded on a computer device. In the computer device for performing a multi-dimensional parallel training (Width-of-the-Line) method for Winograd transform convolution,
At least one processor implemented to execute readable instructions on the computer device
Including,
By the at least one processor,
Distributing and transmitting input data to a plurality of tiles in a plurality of workers arranged in two dimensions by a plurality of clusters and a plurality of groups, and distributing and transmitting the plurality of tiles for each of the plurality of clusters,
Through the workers of each of the plurality of clusters, data parallelism is performed on the input data that is distributed and transmitted,
Performing intra-tile parallelism for element units of the plurality of tiles applied to each of the plurality of groups among the plurality of groups of distributed input data through the workers of each of the plurality of groups.
Computer device characterized in that. The method of claim 9,
Each of the plurality of workers,
A control unit that stores a task graph created in the host, loads tasks in a predefined order based on the task graph, and controls an update counter based on dependency checking;
An operation unit including a systolic array for matrix multiplication and a vector processor for adding a programming function for pre- and post-processing of the matrix multiplication; And
Communication unit including unicast communication logic and ring-based collective communication logic
Computer device comprising a.

Images