CN118095351A - Collaborative processing device and method for layer normalization calculation - Google Patents

Abstract

The disclosure relates to the technical field of neural networks, and in particular relates to a cooperative processing device and method for layer normalization calculation. The device comprises K layers of normalization processing units, wherein K is a positive integer greater than 1; the K layer normalization processing units are used for sharing respective local accumulation results and carrying out cooperative processing of layer normalization calculation, the local accumulation results comprise accumulation results of parameters related to input data, and the input data is local data input to the layer normalization processing units in original data. According to the embodiment of the disclosure, cooperative processing of layer normalization calculation can be realized by sharing respective local accumulation results through multiple nodes (namely K layer normalization processing units), so that distributed calculation acceleration is realized, and the layer normalization calculation efficiency of the neural network is improved.

Description

Collaborative processing device and method for layer normalization calculation

Technical Field

The present disclosure relates to the field of neural network technology, and in particular to a collaborative processing device and method for layer normalization calculation.

Background technique

The layer normalization operator is a commonly used nonlinear operator in neural network algorithms. The layer normalization operator reduces the data, making the neural network more stable during training and helping to improve the performance of the neural network. During the reduction calculation process, it is necessary to count all the intermediate results of the calculation to obtain the calculation parameters, so as to complete the reduction calculation. Therefore, the reduction calculation is one of the keys affecting the algorithm's calculation performance.

In the related art, for large language models based on transformer networks, reduction calculations have become a bottleneck in computing performance due to their large scale and complex computing requirements. In large model algorithms, there are a large number of matrix-vector multiplication operations, and the results of these calculations need to be reduced. However, due to the long vectors, it takes a lot of time to complete them within one computing unit, which greatly reduces the computing performance of the computing system. Currently, no reasonable and effective processing method has been provided.

Summary of the invention

In view of this, the present disclosure proposes a collaborative processing device and method for layer normalization calculation.

According to one aspect of the present disclosure, a collaborative processing device for layer normalization calculation is provided, the device comprising K layer normalization processing units, where K is a positive integer greater than 1;

The K layer normalization processing units are used to share their respective local accumulation results and perform collaborative processing of layer normalization calculations. The local accumulation results include accumulation results of parameters related to input data, and the input data is local data in the original data input to the layer normalization processing unit.

In a possible implementation, each of the layer normalization processing units further includes a layer normalization calculation processing unit and a layer normalization parameter router;

The layer normalization calculation processing unit is used to preprocess the input data to obtain the local accumulation result;

The layer normalization parameter router is used to send the local accumulation result to other layer normalization processing units for sharing, and receive K-1 local accumulation results sent by other layer normalization processing units;

The layer normalization calculation processing unit is also used to use a layer normalization operator to perform a reduction calculation on the input data to obtain a calculation result based on the shared K local accumulation results.

In another possible implementation, the K layer normalization parameter routers are connected using a ring routing structure.

In another possible implementation, each of the layer normalization processing units further includes a memory unit and a direct memory access unit;

The memory unit is used to store the input data;

The direct memory access unit is used to obtain the stored input data and dispatch the input data to the layer normalization calculation processing unit;

The layer normalization calculation processing unit is also used to obtain the scheduled input data.

In another possible implementation, each of the layer normalization calculation processing units includes a calculation unit, a preprocessing accumulation unit and a preprocessing calculation unit;

The preprocessing accumulation unit is used to preprocess the input data to obtain the local accumulation result;

The preprocessing accumulation unit is further used to accumulate the shared K local accumulation results to obtain a global accumulation result;

The preprocessing calculation unit is used to perform parameter calculation according to the global accumulation result to obtain the intermediate parameters of the layer normalization calculation;

The calculation unit is used to perform a reduction calculation using the layer normalization operator according to the intermediate parameters and the input data to obtain the calculation result.

In another possible implementation, the preprocessing accumulation unit includes a selector, an accumulation unit and a local counter, and the local accumulation result includes a parameter accumulation sum and a local accumulation number.

The selector is used to send the input data to the accumulating unit;

The accumulating unit is used to accumulate the parameters related to the input data to obtain the parameter cumulative sum;

The local counter is used to count the accumulation times of the parameters related to the input data to obtain the local accumulation times.

In another possible implementation, the preprocessing accumulation unit further includes an accumulation completion unit.

The accumulation completion unit is used to generate a first valid signal when the local accumulation result is accumulated, and the first valid signal is used to instruct the accumulation unit to output the local accumulation result to the corresponding layer normalization parameter router.

In another possible implementation, the pre-processing accumulation unit further includes waiting for a remote unit,

The local counter is also used to control the waiting remote unit to generate a second valid signal, and the second valid signal is used to instruct the selector to send the received local accumulation results shared by other layer normalization calculation processing units to the accumulation unit;

The accumulation unit is further used to accumulate the received K-1 local accumulation results with the local local accumulation result to obtain the global accumulation result.

In another possible implementation, the pre-processing accumulation unit further includes a remote completion unit.

The remote completion unit is used to control the accumulation completion unit to generate a third valid signal when the global accumulation result is accumulated, and the third valid signal is used to instruct the accumulation unit to transmit the global accumulation result to the preprocessing calculation unit for calculation;

The accumulation completion unit is also used to reset the waiting remote unit, the local counter and the accumulation unit.

In another possible implementation, the layer normalization parameter router includes a remote parameter queue unit and a remote parameter register;

The remote parameter queue unit is used to receive the local accumulation results sent by other layer normalization processing units;

The remote parameter register is used to store the local accumulation results sent by other layer normalization processing units, and input the local accumulation results to the layer normalization calculation processing unit through the target data channel. The target data channel is the data channel between the layer normalization parameter router and the layer normalization calculation processing unit.

In another possible implementation, the layer normalization parameter router further includes a local parameter register and a data selector;

The local parameter register is used to store the local accumulation result of the current layer normalization processing unit;

The data selector is used to send the local accumulation result to the layer normalization parameter routers of other layer normalization processing units for sharing.

In another possible implementation, the layer normalization parameter router further includes a data selector, a local register, and a remote register;

The local register is used to set the value of the local register to a first value when the local accumulation result is input into the layer normalization calculation processing unit;

The remote register is used to set the value of the remote register to a first value when the local accumulation result is sent to other layer normalization parameter routers through the data selector;

The remote parameter register is further configured to clear the remote parameter register to zero when the values of the local register and the remote register are both the first value.

In another possible implementation, the local accumulation result is transmitted between the layer normalization parameter routers in the form of a data packet, and the remote parameter queue unit is further used to:

receiving the data packet sent by the other layer normalization processing units, wherein the data packet includes a plurality of microchips;

The multiple micro-slices in the data packet are spliced to obtain the local accumulation result.

In another possible implementation, the input data of each layer normalization processing unit includes multiple data, the parameters related to the input data include multiple target parameters, the local accumulation result includes the cumulative sum of the multiple target parameters and the number of local accumulations, the target parameter is the square of the data, and the number of local accumulations is the total number of accumulations of the multiple target parameters; or,

The parameters related to the input data include a plurality of the data and a plurality of the target parameters, and the local accumulation result includes a cumulative sum of the plurality of the data, a cumulative sum of the plurality of the target parameters, and the number of local accumulations.

According to another aspect of the present disclosure, a collaborative processing method for layer normalization calculation is provided, which is used in the device provided by the first aspect or any possible implementation of the first aspect, and the method includes:

Each of the layer normalization processing units preprocesses the input data to obtain the local accumulation result;

Each of the layer normalization processing units shares the local accumulation result with other layer normalization processing units;

Each of the layer normalization processing units performs collaborative processing of layer normalization calculations based on the shared K local accumulation results.

In a possible implementation, each of the layer normalization processing units includes a layer normalization parameter router, and each of the layer normalization processing units shares the local accumulation result with other layer normalization processing units, including:

For each of the layer normalization processing units, the local accumulation result is sent to other layer normalization processing units for sharing through the layer normalization parameter router;

The method further comprises:

For each of the layer normalization processing units, K-1 local accumulation results sent by other layer normalization processing units are received through the layer normalization parameter router.

In another possible implementation, the K layer normalization parameter routers are connected using a ring routing structure.

In another possible implementation, each of the layer normalization processing units includes a memory unit, a direct memory access unit, and a layer normalization calculation processing unit; and the method further includes:

For each of the layer normalization processing units, obtaining the input data stored in the memory unit through the direct memory access unit;

dispatching the input data to the layer normalization calculation processing unit through the direct memory access unit;

For each of the layer normalization processing units, the scheduled input data is obtained through the layer normalization calculation processing unit.

In another possible implementation, each of the layer normalization calculation processing units includes a calculation unit, a preprocessing accumulation unit and a preprocessing calculation unit;

Each of the layer normalization processing units preprocesses the input data to obtain a local accumulation result, including:

For each of the layer normalization processing units, preprocessing the input data by the preprocessing accumulation unit to obtain the local accumulation result;

Each of the layer normalization processing units performs collaborative processing of layer normalization calculation according to the shared K local accumulation results, including:

For each of the layer normalization processing units, the shared K local accumulation results are accumulated through the preprocessing accumulation unit to obtain a global accumulation result;

After the accumulation is completed, the preprocessing calculation unit performs parameter calculation according to the global accumulation result to obtain the intermediate parameters of the layer normalization calculation;

The calculation result is obtained by performing a reduction calculation using the layer normalization operator according to the intermediate parameters and the input data by the calculation unit.

In another possible implementation, the preprocessing accumulation unit includes a selector, an accumulation unit, and a local counter, the local accumulation result includes a parameter accumulation sum and a local accumulation number, and the preprocessing of the input data by the preprocessing accumulation unit to obtain the local accumulation result includes:

Sending the input data to the accumulation unit through the selector;

The accumulation unit accumulates the parameters related to the input data to obtain the parameter accumulation sum, and the local counter counts the accumulation times of the parameters related to the input data to obtain the local accumulation times.

In another possible implementation, the preprocessing accumulation unit further includes an accumulation completion unit, and the method further includes:

When the local accumulation result is obtained by accumulation, a first valid signal is generated by the accumulation completion unit, and the first valid signal is used to instruct the accumulation unit to output the local accumulation result to the corresponding layer normalization parameter router.

In another possible implementation, the preprocessing accumulation unit further includes waiting for a remote unit, and the method further includes:

Controlling the waiting remote unit to generate a second valid signal through the local counter, wherein the second valid signal is used to instruct the selector to send the received local accumulation result shared by other layer normalization calculation processing units to the accumulation unit;

The step of accumulating the shared K local accumulation results by the preprocessing accumulation unit to obtain a global accumulation result includes:

The accumulation unit accumulates the received K-1 local accumulation results and the local local accumulation result to obtain the global accumulation result.

In another possible implementation, the preprocessing accumulation unit further includes a remote completion unit, and the method further includes:

When the global accumulation result is accumulated, the accumulation completion unit is controlled by the remote completion unit to generate a third valid signal, wherein the third valid signal is used to instruct the accumulation unit to transmit the global accumulation result to the preprocessing calculation unit for calculation;

The waiting remote unit, the local counter and the accumulation unit are reset and cleared by the accumulation completion unit.

In another possible implementation, the layer normalization parameter router includes a remote parameter queue unit and a remote parameter register; the receiving, through the layer normalization parameter router, K-1 local accumulation results sent by other layer normalization processing units includes:

For each of the K-1 local accumulation results, receiving the local accumulation result sent by other layer normalization processing units through the remote parameter queue unit;

The method further comprises:

storing the local accumulation result in the remote parameter register;

The local accumulation result is input into the layer normalization calculation processing unit through the target data channel via the remote parameter register, and the target data channel is the data channel between the layer normalization parameter router and the layer normalization calculation processing unit.

In another possible implementation, the layer normalization parameter router further includes a local parameter register and a data selector, and the sending of the local accumulation result to other layer normalization processing units for sharing through the layer normalization parameter router includes:

Storing the local accumulation result of the current layer normalization processing unit through the local parameter register;

The local accumulation result is sent to the layer normalization parameter routers of other layer normalization processing units through the data selector for sharing.

In another possible implementation, the layer normalization parameter router further includes a data selector, a local register, and a remote register; and the method further includes:

When the local accumulation result is input into the layer normalization calculation processing unit, the value of the local register is set to a first value;

When the local accumulation result is sent to other layer normalization parameter routers through the data selector, the value of the remote register is set to a first value;

When the values of the local register and the remote register are both the first value, the remote parameter register is cleared.

In another possible implementation, the local accumulation result is transmitted between the layer normalization parameter routers in the form of a data packet, and the receiving, through the remote parameter queue unit, the local accumulation result sent by the other layer normalization processing units includes:

Receiving the data packet sent by the other layer normalization processing units through the remote parameter queue unit, wherein the data packet includes a plurality of micro slices;

The multiple micro-slices in the data packet are spliced together through the remote parameter queue unit to obtain the local accumulation result.

In another possible implementation, the input data of each layer normalization processing unit includes multiple data.

The parameters related to the input data include multiple target parameters, the local accumulation result includes the cumulative sum of the multiple target parameters and the number of local accumulations, the target parameter is the square of the data, and the number of local accumulations is the total number of accumulations of the multiple target parameters; or,

The parameters related to the input data include a plurality of the data and a plurality of the target parameters, and the local accumulation result includes a cumulative sum of the plurality of the data, a cumulative sum of the plurality of the target parameters, and the number of local accumulations.

The disclosed embodiment designs a layer normalization computing architecture for collaborative processing of multiple nodes (i.e., K layer normalization processing units), and designs a preprocessing mechanism according to the layer normalization computing requirements. The preprocessing mechanism is utilized to perform local data processing in each node (i.e., each layer normalization processing unit). When the local data processing is completed and the local accumulation result is obtained, multiple nodes share their respective local accumulation results to realize collaborative processing of layer normalization computing without sharing intermediate data, thereby realizing distributed computing acceleration and improving the efficiency of layer normalization computing in neural networks.

Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

FIG1 shows a schematic diagram of the architecture of a collaborative processing device for layer normalization calculation provided by an exemplary embodiment of the present disclosure.

FIG2 shows a schematic diagram of the architecture of a collaborative processing device for layer normalization calculation provided by another exemplary embodiment of the present disclosure.

FIG3 shows a schematic diagram of the architecture of a layer normalization processing unit provided by an exemplary embodiment of the present disclosure.

FIG. 4 shows a schematic diagram of the architecture of a preprocessing accumulation unit provided by an exemplary embodiment of the present disclosure.

FIG5 shows a schematic diagram of the architecture of a preprocessing accumulation unit provided by another exemplary embodiment of the present disclosure.

FIG6 shows a schematic diagram of the architecture of a layer normalization parameter router provided by an exemplary embodiment of the present disclosure.

FIG. 7 shows a flowchart of a collaborative processing method for layer normalization calculation provided by an exemplary embodiment of the present disclosure.

Fig. 8 is a block diagram of a collaborative processing device for layer normalization calculation according to an exemplary embodiment.

Detailed ways

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

First, the application scenarios involved in the embodiments of the present disclosure are introduced. Please refer to Figure 1, which shows an architectural diagram of a collaborative processing device for layer normalization calculation provided by an exemplary embodiment of the present disclosure. The device 10 can be implemented as a computing device, and the computing device can be a terminal or a server. The device 10 includes K acceleration units, K is a positive integer greater than 1, and each acceleration unit may include a layer normalization processing unit, that is, the device 10 may include K layer normalization processing units (i.e., layer normalization processing unit 1, layer normalization processing unit 2, ... layer normalization processing unit K).

An accelerator is a hardware device used to accelerate a specific type of computing task. An accelerator can be a Field Programmable Gate Array (FPGA) device, a Graphic Processing Unit (GPU), a Central Processing Unit (CPU), a Neural Processing Unit (NPU), etc.

Each layer normalization processing unit is used to process the computational acceleration of the layer normalization operator in the neural network calculation and the reduction calculation of the layer normalization operator. Reduction calculation refers to the aggregation operation of a set of data in the neural network, such as sum, average, maximum, etc. In the layer normalization operator, the reduction calculation can be used to calculate the mean and variance of the data in order to normalize the input data.

The disclosed embodiment designs a dedicated acceleration unit according to the computing requirements of the layer normalization operator, and realizes the collaborative processing of layer normalization calculations by multiple nodes (i.e., K layer normalization processing units) through a preprocessing mechanism. When performing multi-node collaborative computing, distributed computing can be achieved by only sharing their respective local accumulation results.

Optionally, each of the K layer normalization processing units is used to obtain input data, where the input data is local data in the original data input to the layer normalization processing unit; preprocess the input data to obtain a local accumulation result, where the local accumulation result includes the accumulation result of parameters related to the input data; share the local accumulation result with other layer normalization processing units; and perform collaborative processing of layer normalization calculations based on the shared K local accumulation results.

The original data may be data whose length is greater than a preset length threshold. The original data may include N data, and the original data is divided into K groups of input data, each group of input data includes multiple data, and N and K are both positive integers greater than 1. For example, N is 1000, K is 10, the original data includes 1000 data, and the original data is divided into 10 groups of input data, each group of input data includes 100 data. One data may be an activation value in a large language model. The embodiments of the present disclosure are not limited to this.

Optionally, K groups of input data are respectively input into K layer normalization processing units, so that each of the K layer normalization processing units obtains a group of input data. In other words, the original data includes the input data of each of the K layer normalization processing units, and there is no intersection between the input data of each of the K layer normalization processing units.

Each layer normalization processing unit is used to preprocess the input data. The preprocessing includes accumulation and calculation of the input data, such as squaring each data in the input data, accumulating the square of the input data, and counting the number of accumulations of the square of the input data, or directly accumulating the input data to obtain a local accumulation result.

Among them, the local accumulation result includes the accumulation result of the parameters related to the input data. Since the input data of each layer normalization processing unit includes multiple data, optionally, the parameters related to the input data may include multiple target parameters, and correspondingly, the local accumulation result includes the cumulative sum of the multiple target parameters and the number of local accumulations, wherein one target parameter is the square of one data, and the number of local accumulations is the total number of accumulations of the multiple target parameters. Optionally, the parameters related to the input data may also include multiple data and multiple target parameters, and the local accumulation result includes the cumulative sum of the multiple data, the cumulative sum of the multiple target parameters, and the number of local accumulations.

Each layer normalization processing unit sends the local local accumulation results to other layer normalization processing units for sharing, and receives K-1 remote local accumulation results sent by other layer normalization processing units, so that each layer normalization processing unit obtains K shared local accumulation results, and the K local accumulation results include the local local accumulation results and the received remote K-1 local accumulation results, that is, the K local accumulation results are shared among the K layer normalization processing units.

The data sharing method among K layer normalization processing units includes but is not limited to several possible implementation methods: in one possible implementation method, each layer normalization processing unit sends the local local accumulation result to other K-1 layer normalization processing units respectively; in another possible implementation method, the K layer normalization processing units share data through their respective layer normalization parameter routers, and the K layer normalization parameter routers can be connected by a ring routing structure. Based on the layer normalization parameter routers connected by the ring routing structure, each layer normalization processing unit sends the local local accumulation result to the layer normalization processing unit of the next level, so that the layer normalization processing unit of the next level transmits the received local accumulation result to the layer normalization processing unit of the next level, and so on, thereby sharing the local accumulation result with other K-1 layer normalization processing units.

In a possible implementation, as shown in FIG2 , each layer normalization processing unit includes a layer normalization calculation processing unit and a layer normalization parameter router connected to the layer normalization calculation processing unit. That is, the device 10 includes K layer normalization calculation processing units (i.e., layer normalization calculation processing unit 1, layer normalization calculation processing unit 2, ... layer normalization calculation processing unit K) and K layer normalization parameter routers (i.e., layer normalization parameter router 1, layer normalization parameter router 2, ... layer normalization parameter router K). The K layer normalization parameter routers may be connected by a ring routing structure or other routing structures. This is not limited in the embodiments of the present disclosure.

Each layer normalization calculation processing unit is used to obtain input data, pre-process the input data to obtain a local accumulation result.

Each layer normalization parameter router is used to obtain the parameters (ie, local accumulation results) generated by the connected layer normalization calculation processing unit in the protocol calculation, and send the obtained parameters to other layer normalization processing units through other connected layer normalization parameter routers.

Each layer normalization parameter router is also used to receive parameters (i.e., local accumulation results) generated by other layer normalization processing units in the reduction calculation through other connected layer normalization parameter routers. In other words, the parameters (i.e., local accumulation results) generated in the reduction calculation are shared among the K layer normalization processing units of the device 10, without the intervention of the system-level scheduling controller, and can be completed automatically, thereby improving the data synchronization efficiency of the device 10 and reducing the calculation time.

Each of the K layer normalization processing units is also used to perform collaborative processing of layer normalization calculations based on the shared K local accumulation results. Optionally, each layer normalization processing unit is also used to accumulate the shared K local accumulation results to obtain a global accumulation result, and use a layer normalization operator to perform a reduction calculation on the input data based on the global accumulation result to obtain a calculation result.

The global accumulation result accumulated by the normalization processing unit of each layer can be the same. The global accumulation result is the sum of the local accumulation results corresponding to each of the K groups of input data. The K groups of input data constitute the original data, that is, the K groups of input data include a total of N data. In one possible implementation, the global accumulation result may include the accumulation sum of the N target parameters and the total number of accumulations of the N target parameters N. In another possible implementation, the global accumulation result includes the accumulation sum of the N data, the accumulation sum of the N target parameters, and the total number of accumulations of the N target parameters N.

It should be noted that the calculation result obtained by reducing the input data using the layer normalization operator can be referred to the relevant details in the following embodiments, which will not be introduced here.

The embodiment of the present disclosure designs a solution that supports multi-node collaborative work and parameter sharing in response to the need for the device 10 to calculate the layer normalization operator in the neural network scenario. Providing the ability of multi-node collaborative computing for the layer normalization operator can improve the efficiency of the reduction computing in the layer normalization operator and improve the computing performance of the computing device.

The neural network can be a deep learning network, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), etc. For example, the neural network is a neural network of a large language model.

The layer normalization operator is used to perform layer normalization calculation on the data, and the layer normalization operator includes a layer normalization (LayerNormalization, LayerNorm) operator or a root mean square layer normalization (Root Mean Square LayerNormalization, RMS-LayerNorm) operator. The data can be data to be layer normalized and outputted by any layer in the neural network, for example, the data can be the activation value outputted by the fully connected layer in the neural network. It should be understood that the data can be any data in the neural network that needs to be layer normalized, and the embodiments of the present disclosure are not limited to this.

In summary, the disclosed embodiment designs a layer normalization computing architecture for collaborative processing of multiple nodes (i.e., K layer normalization processing units), and designs a preprocessing mechanism according to the layer normalization computing requirements. The preprocessing mechanism is used to perform local data processing in each node (i.e., each layer normalization processing unit). When the local data processing is completed and the local accumulation result is obtained, multiple nodes share their respective local accumulation results to achieve collaborative processing of layer normalization calculations without sharing intermediate data, thereby achieving distributed computing acceleration and improving the efficiency of layer normalization calculations in neural networks.

In a possible implementation, the architecture diagram of each layer normalization processing unit is shown in Figure 3. The internal architecture of the layer normalization processing unit may include five parts, namely, a memory unit, a direct memory access unit, a control unit, a layer normalization calculation processing unit, and a layer normalization parameter router. The layer normalization calculation processing unit may include three parts, namely, a calculation unit, a preprocessing accumulation unit, and a preprocessing calculation unit. According to actual needs, the internal units of the layer normalization processing unit may be increased or decreased, and the embodiments of the present disclosure are not limited to this.

When the layer normalization processing unit performs calculation processing, data access is performed between the memory unit and the layer normalization calculation processing unit through the direct memory access unit. The input data stored in the memory unit is obtained through the direct memory access unit; the input data is dispatched to the layer normalization calculation processing unit through the direct memory access unit; and the dispatched input data is obtained through the layer normalization calculation processing unit.

After the input data enters the layer normalization calculation processing unit, the input parameters are preprocessed, that is, the preprocessing accumulation unit is used to preprocess the input data to obtain a local accumulation result, and to accumulate the shared K local accumulation results to obtain a global accumulation result. After the accumulation is completed, the preprocessing calculation unit is used to perform parameter calculation according to the global accumulation result to obtain the intermediate parameters of the layer normalization calculation. Optionally, the intermediate parameters may include the average value of the N data included in the original data.

After the preprocessing process of layer normalization is completed and the intermediate parameters of the final layer normalization calculation are obtained, the calculation unit uses the layer normalization operator to reduce the input data according to the intermediate parameters and input data to obtain the calculation result. During the preprocessing process, if the calculation needs to be performed simultaneously in multiple layer normalization processing units, the local accumulation results are shared through the layer normalization parameter router. The control unit controls the data calculation and data sharing operations in the entire layer normalization processing unit.

The layer normalization operator may be a LayerNorm operator or a RMS-LayerNorm operator, that is, the method provided in the embodiment of the present disclosure may support the calculation of these two layer normalization operators. Schematically, the calculation formula of the LayerNorm operator is as follows:

Wherein, _xi is the i-th data in the original data, _xi2 is the square of the i-th data ^, i.e., the i-th target parameter, i is a positive integer ranging from 1 to N, and N is a positive integer greater than 1. are all preset parameter values, and μ is the average value of N data.

In the LayerNorm operator, the global accumulation result includes the sum of N data. , the cumulative sum of N target parameters/> and the total number of N target parameters accumulated, the intermediate parameters may include μ and /> .

Schematically, the calculation formula of the RMS-LayerNorm operator is as follows:

In the LayerNorm operator, the global accumulation result includes the sum of N target parameters. , and the total number of N target parameters accumulated, the intermediate parameters may include/> .

When each layer normalization processing unit performs layer normalization calculation, the input data in the memory unit is first dispatched to the layer normalization calculation processing unit through the direct memory access unit, and the input data is accumulated and calculated through the layer normalization calculation processing unit. For the operation of multi-node collaborative calculation, it is necessary to transfer the accumulated local accumulation results to other layer normalization processing units through the parameter sharing network (i.e., the parameter sharing network composed of K layer normalization parameter routers), and receive the local accumulation results of other layer normalization processing units for further accumulation to obtain the final global accumulation result. After the accumulation is completed, the global accumulation result will be sent to the preprocessing calculation unit to perform parameter calculation to obtain the intermediate parameters of the layer normalization calculation. After the parameter calculation is completed, the intermediate parameters will be written into the parameter register of the preprocessing calculation unit to complete the preprocessing operation. Then the input data is read from the memory unit, and the layer normalization operator is used to perform the reduction calculation according to the intermediate parameters obtained by the preprocessing to obtain the calculation result, and the calculation result is written back to the memory unit through the direct memory access unit.

In a possible implementation, the preprocessing accumulation unit can be divided into two parts. The first part is responsible for the accumulation and sharing of multiple data included in the input data of the LayerNorm operator, and the second part is responsible for the accumulation of the target parameter (i.e., the square of the data) in the LayerNorm operator or the RMS-LayerNorm operator, the calculation of the total number of accumulations N, and the sharing. The two parts are introduced below.

The first part is responsible for the accumulation and shared processing of data in the LayerNorm operator in layer normalization, the preprocessing accumulation unit, please refer to Figure 4, which shows an architectural schematic diagram of a preprocessing accumulation unit provided by an exemplary embodiment of the present disclosure, and the architecture can be implemented as a part of the preprocessing accumulation unit. The internal architecture of the preprocessing accumulation unit may include the following parts, namely, a selector, an accumulation unit, a local counter, an accumulation completion unit, a waiting remote unit, and a remote completion unit. According to actual needs, the internal units of the preprocessing accumulation unit can be increased or decreased, and the embodiments of the present disclosure are not limited to this.

When the preprocessing accumulation unit performs accumulation processing, there are two different processing methods: local accumulation processing and global accumulation processing. Local accumulation processing is used to process local parameter accumulation, and global accumulation processing is used to calculate parameter accumulation between different layer normalization processing units. When performing local accumulation processing, the input data is sent to the accumulation unit through the selector; the accumulation unit accumulates multiple data included in the input data to obtain a local accumulation result, and the local accumulation number is counted by the local counter, and the local accumulation number is equal to the total number of data included in the local input data. In other words, each time an accumulation operation is performed, the local counter will perform an accumulation count. When the last accumulation operation is completed, that is, the local accumulation result is accumulated, the accumulation completion unit generates a first valid signal, and the first valid signal is used to instruct the accumulation unit to output the local accumulation result to the corresponding layer normalization parameter router. At this time, the local accumulation result in the accumulation unit will be output to the corresponding layer normalization parameter router through the data channel, and shared to other layer normalization processing units through the layer normalization parameter router.

When performing global accumulation processing, after the above-mentioned local accumulation and sharing processing, the local counter is controlled to wait for the remote unit to generate a second valid signal, and the second valid signal will act on the selector. The second valid signal is used to instruct the selector to send the received local accumulation results shared by the normalization processing units of other layers to the accumulation unit. Optionally, the state of the selector includes a first state and a second state, and the first state is different from the second state. When the selector is in the default first state, the selector is used to receive input data through the first data channel; when the selector receives the second valid signal, the state of the selector is switched from the first state to the second state. When the selector is in the second state, the selector is used to receive the local accumulation results shared by the normalization processing units of other layers through the second data channel. The first data channel is different from the second data channel. The embodiment of the present disclosure does not limit the setting method of the selector.

When the accumulation unit receives the local accumulation result, the received local accumulation result and the local local accumulation result are accumulated again. When the last accumulation is completed, that is, the accumulation of K local accumulation results is completed to obtain the global accumulation result, the accumulation completion unit is controlled by the remote completion unit to generate a third valid signal, and the third valid signal is used to instruct the accumulation unit to transmit the global accumulation result to the preprocessing calculation unit for calculation. At this time, the global accumulation result in the accumulation unit is transmitted to the preprocessing calculation unit of the next level through the data channel for the next step of calculation. Then, the waiting remote unit, the local counter and the accumulation unit are reset and cleared by the accumulation completion unit to facilitate the next calculation. At this point, the accumulation and sharing of multiple data included in the input data of the LayerNorm operator are completed. The entire process does not require the participation of the system-level scheduling controller, and multiple layer normalization processing units can work in a distributed and collaborative manner.

The second part is responsible for the accumulation of target parameters in the LayerNorm operator or the RMS-LayerNorm operator, the calculation of the total number of accumulations N, and the shared processing. The first part and the second part can be two parts that are not connected to each other or connected. The embodiments of the present disclosure are not limited to this. Please refer to Figure 5, which shows an architectural schematic diagram of a preprocessing accumulation unit provided by another exemplary embodiment of the present disclosure, and the architecture can be implemented as a part of the preprocessing accumulation unit. The internal architecture of the preprocessing accumulation unit may include the following parts, namely, a selector, an accumulation unit, a local counter, an element counter, an accumulation completion unit, a waiting remote unit, and a remote completion unit. According to actual needs, the internal units of the preprocessing accumulation unit can be increased or decreased, and the embodiments of the present disclosure are not limited to this.

In the case where the preprocessing accumulation unit performs accumulation processing of the target parameters, the input data (including multiple data) is first sent to the square processing unit to perform square calculation of multiple data to obtain multiple target parameters, and then the multiple target parameters are sent to the accumulation unit through the selector, and the accumulation unit accumulates the multiple target parameters to obtain a local accumulation result. The accumulation processing of the target parameters is also divided into local accumulation processing and global accumulation processing. The accumulation calculation process of the target parameters can be analogously referred to the accumulation calculation process of the above data. In the layer normalization calculation, it is necessary to know how many data are involved in the calculation, so it is necessary to count the number of accumulation operations. Since the number of accumulations of data and target parameters is the same, and the accumulation calculation of target parameters is slower, the number of accumulation operations is only counted in the accumulation calculation process of target parameters during the calculation process. Optionally, the total number of data involved in the calculation, that is, the total number of data included in the original data, N, can also be counted during the calculation. An element counter is added to the accumulation unit, and the element counter is used to count the total number of data included in the original data, that is, the number of global accumulations. When the accumulation unit is performing accumulation processing, the local counter counts the element counter once for each accumulation operation during the local accumulation processing. After the local accumulation processing is completed, if the remote local accumulation results of other layer normalization processing units need to be accumulated, the remote local accumulation results count the element counter while inputting the selector. When the accumulation processing is completely completed, the element counter outputs the total number of recorded data to the layer normalization parameter router, which is shared with other layer normalization processing units through the layer normalization parameter router.

It should be noted that for operations that do not require collaborative calculations of multiple layer normalization processing units, the calculation process within a single layer normalization processing unit is exactly the same, but after the calculation is completed, the data does not need to be shared, and the layer normalization processing unit does not need to wait for remote data. After the local accumulation processing is completed, the calculation results are directly output for the next step of parameter calculation.

Each layer normalization processing unit includes a layer normalization parameter router. For each layer normalization processing unit, the local accumulation result is sent to other layer normalization processing units for sharing through the layer normalization parameter router; for each layer normalization processing unit, the K-1 local accumulation results sent by other layer normalization processing units are received through the layer normalization parameter router. K layer normalization parameter router units can constitute a routing loop. Take one of the layer normalization parameter router units as an example for explanation. In a possible implementation, please refer to Figure 6, which shows an architectural schematic diagram of a layer normalization parameter router provided by an exemplary embodiment of the present disclosure, and the architecture can be implemented as a part of the layer normalization parameter router. The internal architecture of the layer normalization parameter router may include the following parts, namely, a remote parameter queue unit, a remote parameter register, a local parameter register, a selector, a local register and a remote register. According to actual needs, the internal units of the layer normalization parameter router can be increased or decreased, and the embodiment of the present disclosure is not limited to this.

When working, the local accumulation results (each of the K-1 local accumulation results) sent by the previous layer normalization calculation processing unit enter the remote parameter queue unit, receive the local accumulation results sent by other layer normalization processing units through the remote parameter queue unit, and store the local accumulation results in the remote parameter register. The index of the current layer normalization parameter router is marked by the broadcast counting unit. The index is used to uniquely identify the current layer normalization parameter router among the K layer normalization parameter routers, that is, the index is used to indicate which level of layer normalization calculation processing unit the layer normalization parameter router is. The minimum value can be 1 and the maximum value can be K. The local accumulation results are input to the layer normalization calculation processing unit through the remote parameter register through the target data channel. The target data channel is the data channel between the layer normalization parameter router and the layer normalization calculation processing unit.

Optionally, the local accumulation result of the normalization processing unit of the current layer is stored in a local parameter register; and the local accumulation result is sent to the layer normalization parameter router of other layer normalization processing units for sharing through a data selector.

The local register is used to indicate the state of the remote parameter register, and the remote register is used to indicate the state of the local parameter register. The initial values of the local register and the remote register are both the default second value, such as 0. When the local accumulation result of the remote parameter register is input into the layer normalization calculation processing unit, the value of the local register is set to the first value; when the local accumulation result of the local parameter register is sent to other layer normalization parameter routers through the data selector, the value of the remote register is set to the first value; when the values of the local register and the remote register are both the first value, the remote parameter register is cleared, and the first value is different from the second value, such as 1.

When data is transmitted between normalized parameter routers at different layers, the data link width can be configured according to resource requirements. Optionally, the data link width of the routing path between the pre-configured layer normalized parameter routers is smaller than a preset width threshold, such as the data link width is smaller than the data bit width of the local accumulation result to be transmitted, thereby reducing resource overhead.

If the data link width is smaller than the data bit width, the data can be shifted and output when outputting data, and one data can be converted into a data packet including several flits. In the disclosed embodiment, the local accumulation result to be transmitted can be converted into a data packet including several flits for transmission, that is, the local accumulation result is transmitted between layer normalization parameter routers in the form of a data packet (the data packet includes multiple flits). When receiving the shared local accumulation result, the data packet sent by other layer normalization processing units is received through the remote parameter queue unit, and the multiple flits in the data packet are spliced through the remote parameter queue unit to obtain the shared local accumulation result.

It should be noted that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

In the related art, when multiple computing nodes perform reduction calculations, it is necessary to move the original data together to perform reduction operations. The movement of original data will cause high latency and high power consumption, which is not conducive to improving the computing efficiency of the accelerator. In addition, for reduction calculations, multiple computing nodes are required to communicate and synchronize with the master controller, which will increase the computing delay and reduce the computing efficiency. In addition, data sharing between multiple computing nodes often uses the same data path, which will cause data transmission conflicts and reduce transmission efficiency. However, the embodiment of the present disclosure designs a special pre-processing accumulation unit for the reduction calculation of the layer normalization operator, and the pre-processing accumulation unit can efficiently perform the accumulation operation in the layer normalization operator. On the other hand, the accumulation operation in the pre-processing simultaneously supports the reduction calculation of a single node (i.e., a single layer normalization processing unit) and the collaborative reduction calculation of multiple nodes (i.e., multiple layer normalization processing units). When performing layer normalization calculations, the reduction calculations of multiple layer normalization processing units can work together, and the synchronization operation is automatically completed in the layer normalization calculation processing unit, without the need for the system controller to intervene in the synchronization operation. On the other hand, for the collaborative reduction calculation of the layer normalization operator, local data processing can be performed inside each layer normalization processing unit. When the local data processing is completed, only the local accumulation results are shared, and there is no need to share the intermediate data calculated during the local data processing process, thus achieving distributed computing acceleration. On the other hand, a dedicated data path is designed for data sharing, and the sharing of local accumulation results uses a dedicated data path (the routing path between the layer normalization parameter routers), which improves data sharing efficiency and reduces data transmission conflicts. In addition, the data sharing path can be configured with different link widths to save circuit overhead.

The following is a method embodiment of the present disclosure. For parts not described in detail in the method embodiment, reference may be made to the technical details disclosed in the above-mentioned device embodiment.

Please refer to Figure 7, which shows a flowchart of a method for co-processing layer normalization calculation provided by an exemplary embodiment of the present disclosure. This embodiment uses the method used in the above-mentioned co-processing device for layer normalization calculation as an example. The method includes the following steps.

Step 701: Each of the K layer normalization processing units obtains input data, where the input data is local data in the original data input to the layer normalization processing unit, and K is a positive integer greater than 1.

Step 702: Each layer normalization processing unit preprocesses the input data to obtain a local accumulation result, where the local accumulation result includes an accumulation result of parameters related to the input data.

Step 703: Each layer normalization processing unit shares the local accumulation result with other layer normalization processing units.

Step 704: Each layer normalization processing unit performs collaborative processing of layer normalization calculations based on the shared K local accumulation results.

In a possible implementation, each layer normalization processing unit includes a layer normalization parameter router, and each layer normalization processing unit shares the local accumulation result with other layer normalization processing units, including:

For each layer normalization processing unit, the local accumulation result is sent to other layer normalization processing units for sharing through the layer normalization parameter router;

The method further includes:

For each layer normalization processing unit, K-1 local accumulation results sent by other layer normalization processing units are received through the layer normalization parameter router.

In another possible implementation, the K layer normalization parameter routers are connected using a ring routing structure.

In another possible implementation, each layer normalization processing unit includes a memory unit, a direct memory access unit, and a layer normalization calculation processing unit; the method further includes:

For each layer normalization processing unit, the input data stored in the memory unit is obtained through the direct memory access unit;

dispatching input data to a layer normalization calculation processing unit via a direct memory access unit;

For each layer normalization processing unit, the scheduled input data is obtained through the layer normalization calculation processing unit.

In another possible implementation, each layer normalization calculation processing unit includes a calculation unit, a preprocessing accumulation unit, and a preprocessing calculation unit;

Each layer normalization processing unit preprocesses the input data to obtain local accumulation results, including:

For each layer normalization processing unit, the input data is preprocessed by the preprocessing accumulation unit to obtain a local accumulation result;

Each layer normalization processing unit performs collaborative processing of layer normalization calculation based on the shared K local accumulation results, including:

For each layer normalization processing unit, the shared K local accumulation results are accumulated through the preprocessing accumulation unit to obtain the global accumulation result;

After the accumulation is completed, the preprocessing calculation unit performs parameter calculation based on the global accumulation result to obtain the intermediate parameters of the layer normalization calculation;

The calculation unit uses a layer normalization operator to perform reduction calculations according to intermediate parameters and input data to obtain calculation results.

In another possible implementation, the preprocessing accumulation unit includes a selector, an accumulation unit, and a local counter, the local accumulation result includes a parameter accumulation sum and a local accumulation number, and the preprocessing accumulation unit preprocesses the input data to obtain the local accumulation result, including:

Send the input data to the accumulation unit through the selector;

The parameters related to the input data are accumulated by the accumulation unit to obtain the parameter accumulation sum, and the accumulation times of the parameters related to the input data are counted by the local counter to obtain the local accumulation times.

In another possible implementation, the preprocessing accumulation unit further includes an accumulation completion unit, and the method further includes:

When a local accumulation result is obtained by accumulation, a first valid signal is generated by the accumulation completion unit, and the first valid signal is used to instruct the accumulation unit to output the local accumulation result to the corresponding layer normalization parameter router.

In another possible implementation, the preprocessing accumulation unit further includes waiting for the remote unit, and the method further includes:

Waiting for the remote unit to generate a second valid signal through local counter control, where the second valid signal is used to instruct the selector to send the received local accumulation results shared by the normalization calculation processing units of other layers to the accumulation unit;

The shared K local accumulation results are accumulated through the preprocessing accumulation unit to obtain a global accumulation result, including:

The received K-1 local accumulation results are accumulated with the local local accumulation result by the accumulation unit to obtain a global accumulation result.

In another possible implementation, the preprocessing accumulation unit further includes a remote completion unit, and the method further includes:

When a global accumulation result is accumulated, the accumulation completion unit is controlled by the remote completion unit to generate a third valid signal, and the third valid signal is used to instruct the accumulation unit to transmit the global accumulation result to the preprocessing calculation unit for calculation;

The waiting remote unit, the local counter and the accumulation unit are reset and cleared by the accumulation completion unit.

In another possible implementation, the preprocessing accumulation unit further includes an element counter, and the method further includes:

The total number of data included in the original data is counted by the element counter.

In another possible implementation, the layer normalization parameter router includes a remote parameter queue unit and a remote parameter register; receiving K-1 local accumulation results sent by other layer normalization processing units through the layer normalization parameter router includes:

For each of the K-1 local accumulation results, the local accumulation results sent by the normalization processing units of other layers are received through the remote parameter queue unit;

The method further includes:

Store the local accumulation result in the remote parameter register;

The local accumulation result is input to the layer normalization calculation processing unit through the target data channel via the remote parameter register. The target data channel is the data channel between the layer normalization parameter router and the layer normalization calculation processing unit.

In another possible implementation, the layer normalization parameter router further includes a local parameter register and a data selector, and the local accumulation result is sent to other layer normalization processing units for sharing through the layer normalization parameter router, including:

The local accumulation result of the normalization processing unit of the current layer is stored in the local parameter register;

The local accumulation results are sent to the layer normalization parameter routers of other layer normalization processing units through the data selector for sharing.

In another possible implementation, the layer normalization parameter router further includes a data selector, a local register, and a remote register; and the method further includes:

When the local accumulation result is input into the layer normalization calculation processing unit, the value of the local register is set to the first value;

When the local accumulation result is sent to the normalization parameter routers of other layers through the data selector, the value of the remote register is set to the first value;

When the values of the local register and the remote register are both the first value, the remote parameter register is cleared.

In another possible implementation, the local accumulation result is transmitted between the layer normalization parameter routers in the form of a data packet, and the local accumulation result sent by other layer normalization processing units is received through the remote parameter queue unit, including:

receiving a data packet sent by a normalization processing unit of another layer through a remote parameter queue unit, wherein the data packet includes a plurality of micro slices;

Multiple micro-slices in the data packet are spliced together through the remote parameter queue unit to obtain a local accumulation result.

In another possible implementation, the input data of each layer normalization processing unit includes multiple data,

The parameters related to the input data include multiple target parameters, the local accumulation result includes the cumulative sum of the multiple target parameters and the number of local accumulations, the target parameter is the square of the data, and the number of local accumulations is the total number of accumulations of the multiple target parameters; or,

The parameters related to the input data include multiple data and multiple target parameters, and the local accumulation result includes the cumulative sum of the multiple data, the cumulative sum of the multiple target parameters, and the number of local accumulations.

In another possible implementation manner, the layer normalization operator includes a LayerNorm operator or a RMS-LayerNorm operator.

It should be noted that, regarding the method in the above embodiment, the specific manner of performing the operation in each step has been described in detail in the relevant device embodiment, and will not be elaborated here.

FIG8 is a block diagram of a collaborative processing device for layer normalization calculation according to an exemplary embodiment. For example, the device 10 may be provided as a server or a terminal device. Referring to FIG8 , the device 10 includes a processing component 822, which further includes one or more processors, and a memory resource represented by a memory 832 for storing instructions executable by the processing component 822, such as an application. The application stored in the memory 832 may include one or more modules, each of which corresponds to a set of instructions. In addition, the processing component 822 may include a plurality of acceleration units, each of which includes a layer normalization processing unit, and the processing component 822 is configured to execute instructions to perform the above method.

The device 10 may also include a power supply component 826 configured to perform power management of the device 10, a wired or wireless network interface 850 configured to connect the device 10 to a network, and an input/output interface 858 (I/O interface). The device 10 may operate based on an operating system stored in the memory 832, such as Windows Server ^TM , Mac OS X ^TM , Unix ^TM , Linux ^TM , FreeBSD ^TM , or the like.

The present disclosure may be a system, a method and/or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing/processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. The network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device.

The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

Various aspects of the present disclosure are described herein with reference to flowcharts and/or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each box in the flowchart and/or block diagram and the combination of boxes in the flowchart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions/actions specified in one or more boxes in the flowchart and/or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and/or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and/or flow chart, and the combination of the square boxes in the block diagram and/or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (15)

1. The cooperative processing device for layer normalization calculation is characterized by comprising K layer normalization processing units, wherein K is a positive integer greater than 1;

The K layer normalization processing units are used for sharing respective local accumulation results and carrying out cooperative processing of layer normalization calculation, the local accumulation results comprise accumulation results of parameters related to input data, and the input data is local data input to the layer normalization processing units in original data.

2. The apparatus of claim 1, wherein each of the layer normalization processing units further comprises a layer normalization calculation processing unit and a layer normalization parameter router;

the layer normalization calculation processing unit is used for preprocessing the input data to obtain the local accumulation result;

The layer normalization parameter router is used for sending the local accumulation results to other layer normalization processing units for sharing and receiving K-1 local accumulation results sent by the other layer normalization processing units;

the layer normalization calculation processing unit is further used for performing reduction calculation on the input data by adopting a layer normalization operator according to the shared K local accumulation results to obtain a calculation result.

3. The apparatus of claim 2, wherein K of the layer normalization parameter routers are connected by a ring routing fabric.

4. The apparatus of claim 2, wherein each of the layer normalization processing units further comprises a memory unit and a direct memory access unit;

The memory unit is used for storing the input data;

The direct memory access unit is used for acquiring the stored input data and scheduling the input data to the layer normalization calculation processing unit;

the layer normalization computing processing unit is further configured to obtain the scheduled input data.

5. The apparatus of claim 2, wherein each of the layer normalization computation processing units comprises a computation unit, a preprocessing accumulation unit, and a preprocessing computation unit;

the preprocessing accumulation unit is used for preprocessing the input data to obtain the local accumulation result;

The preprocessing accumulation unit is also used for accumulating the shared K local accumulation results to obtain a global accumulation result;

The preprocessing calculation unit is used for carrying out parameter calculation according to the global accumulation result to obtain intermediate parameters of the layer normalization calculation;

And the calculation unit is used for carrying out protocol calculation by adopting the layer normalization operator according to the intermediate parameter and the input data to obtain the calculation result.

6. The apparatus of claim 5, wherein the preprocessing accumulation unit comprises a selector, an accumulation unit and a local counter, the local accumulation result comprises a parameter accumulation sum and a local accumulation number,

The selector is used for sending the input data to the accumulation unit;

the accumulation unit is used for accumulating parameters related to the input data to obtain the parameter accumulation sum;

The local counter is used for counting the accumulation times of the parameters related to the input data to obtain the local accumulation times.

7. The apparatus of claim 6, wherein the preprocessing accumulation unit further comprises an accumulation completion unit,

The accumulation completion unit is used for generating a first effective signal when the local accumulation result is obtained through accumulation, and the first effective signal is used for indicating the accumulation unit to output the local accumulation result to a corresponding layer normalization parameter router.

8. The apparatus of claim 6, wherein the preprocessing accumulation unit further comprises a waiting remote unit,

The local counter is further configured to control the waiting remote unit to generate a second valid signal, where the second valid signal is used to instruct the selector to send the received local accumulation result shared by the other layer normalization computation processing units to the accumulation unit;

the accumulation unit is also used for accumulating the received K-1 local accumulation results and the local accumulation results to obtain the global accumulation result.

9. The apparatus of claim 7, wherein the preprocessing accumulation unit further comprises a remote completion unit,

The remote completion unit is used for controlling the accumulation completion unit to generate a third effective signal when the global accumulation result is obtained by accumulation, and the third effective signal is used for indicating the accumulation unit to transmit the global accumulation result to the preprocessing calculation unit for calculation;

The accumulation completion unit is also used for resetting and resetting the waiting remote unit, the local counter and the accumulation unit.

10. The apparatus of claim 2, wherein the layer normalized parameter router comprises a remote parameter queue element and a remote parameter register;

the remote parameter queue unit is used for receiving the local accumulation results sent by other layer normalization processing units;

The remote parameter register is used for registering the local accumulation results sent by other layer normalization processing units, and inputting the local accumulation results to the layer normalization calculation processing unit through a target data channel, wherein the target data channel is a data channel between the layer normalization parameter router and the layer normalization calculation processing unit.

11. The apparatus of claim 2, wherein the layer normalization parameter router further comprises a local parameter register and a data selector;

the local parameter register is used for registering the local accumulation result of the current layer normalization processing unit;

The data selector is used for sending the local accumulation result to the layer normalization parameter routers of other layer normalization processing units for sharing.

12. The apparatus of claim 10, wherein the layer normalization parameter router further comprises a data selector, a local register, and a remote register;

the local register is used for setting the value of the local register to be a first numerical value when the local accumulation result is input into the layer normalization calculation processing unit;

The remote register is used for setting the value of the remote register to be a first numerical value when the local accumulation result is sent to other layer normalization parameter routers through the data selector;

The remote parameter register is further configured to clear the remote parameter register when the values of the local register and the remote register are both the first value.

13. The apparatus of claim 10, wherein the local accumulation results are communicated in data packets between the layer normalized parameter routers, the remote parameter queue unit further to:

receiving the data packet sent by other layer normalization processing units, wherein the data packet comprises a plurality of flits;

and splicing the plurality of flits in the data packet to obtain the local accumulation result.

14. The apparatus according to any one of claims 1 to 13, wherein the input data of each of the layer normalization processing units comprises a plurality of data,

The parameters related to the input data comprise a plurality of target parameters, the local accumulation result comprises accumulation sums of a plurality of target parameters and local accumulation times, the target parameters are squares of the data, and the local accumulation times are accumulation total times of the plurality of target parameters; or alternatively

The parameters related to the input data include a plurality of the data and a plurality of the target parameters, and the local accumulation result includes an accumulation sum of the plurality of the data, an accumulation sum of the plurality of the target parameters, and the local accumulation number.

15. A co-processing method of layer normalization computation, for use in the apparatus of any one of claims 1 to 14, the method comprising:

Each layer normalization processing unit preprocesses the input data to obtain the local accumulation result;

Each layer normalization processing unit shares the local accumulation result to other layer normalization processing units;

And each layer normalization processing unit performs cooperative processing of layer normalization calculation according to the shared K local accumulation results.