KR20190114208A - In DRAM Bitwise Convolution Circuit for Low Power and Fast Computation - Google Patents

Abstract

The present invention relates to a bitwise convolution circuit for a DRAM for low-power and high-speed operations which can effectively perform bitwise convolution operations by using a processing-in-memory circuit in a DRAM. The bitwise convolution circuit for a DRAM for low-power and high-speed operations comprises a plurality of processing units to execute convolution operations for input data and partial results. The processing unit processes bitwise convolution operations using a ternary kernel with a processing-in-memory circuit structure in a DRAM, has a plurality of bit lines (BLs) crossing a single word line (WL), has pixels on a crossing region, and has a sense amplifier (SA) in response to each pixel. The processing unit includes: a first bank to which an accommodation field is assigned; a second bank assigned for a positive kernel and a negative kernel; and a feature map output means to output a feature map by convolution operations for the positive kernel and the negative kernel.

Description

In-DDRAM Bitwise Convolution Circuit for Low Power and Fast Computation

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for performing convolution operations, and more particularly, to efficiently perform bitwise convolution operations using a processing-in-memory circuit in a DRAM. A bitwise convolution circuit for DRAMs for low power and high speed operations.

Neural networks represent data structures that model the living brain. In neural networks, nodes may represent neurons that are interconnected and collectively processed to process input data.

Examples of different kinds of neural networks may include, but are not limited to, Convolutional Neural Networks, Recurrent Neural Networks, Deep Belief Network, and Restricted Boltzmann Machines. It is not limited.

Neural networks can be used to extract "features" from complex input data.

The neural network may include a plurality of layers. Each layer may receive input data and process the received input data to generate output data.

The output data may be a feature map of the input data, and the neural network may be used to generate a feature map of the input data by convolving the input image or feature map with the convolution kernel.

In particular, in the field of image recognition, CNN (Convolutional Neural Networks) models are generally used to determine the classification of recognized images.

Before the classification of the recognized image can be recognized through the CNN model, the CNN model needs to be trained first.

Training of the CNN model is generally implemented in the following way.

First, model parameters of the trained CNN model are initialized, where the model parameters are the initial convolution kernels of each convolution layer, the initial bias matrixes of each convolution layer, And an initial weight matrix and an initial bias vector of the fully connected layer.

 And an area to be processed having a fixed height and a fixed width is obtained from each of the preselected training images, where the fixed height and the fixed width match the classification of the recognized image, and the recognized image is subjected to the trained CNN model. As an image that can be processed by the user.

The region to be processed corresponding to each of the training images is input to the trained CNN model. Then, in each convolutional layer, a convolutional operation and a maximum pooling operation are performed in each region that is processed using the initial convolutional kernel and initial bias matrix of each convolutional layer, so that each convolutional layer Acquire a feature image of each region processed by.

Each feature image is then processed to obtain a classification probability of each region that is processed using the initial weight matrix and the initial bias vector of the fully connected layer.

The classification error is calculated according to the classification probability and the initial classification of each training image. The average of the classification errors is calculated according to the classification errors of all training images.

The model parameters of the trained CNN model are then adjusted using the average of the classification errors.

In the CNN (Convolutional Neural Networks), a type of deep learning, a convolution operation using a kernel is performed on an input image to extract features and extract images. To perform.

Therefore, in CNN, a huge amount of operations must be performed, and CNN hardware has to use a high speed / high capacity computing circuit such as a GPU in order to perform these operations at high speed / mass.

When the CPU and the memory of the related art are separated from each other and perform a complicated operation such as a convolution, the CPU must continuously write and read data to and from the memory.

In addition, due to the limitation of the data width between the CPU and memory, a large number of operation cycles are required for complex operations such as convolution, which inevitably consumes a large amount of energy. do.

Accordingly, there is a need for the development of a new technology that can be applied to neural networks such as the CNN (Convolutional Neural Networks) model by greatly reducing the power consumption and the number of clock cycles consumed in the convolution operation.

Korean Laid-Open Patent No. 10-2017-0099848 Korean Laid-Open Patent No. 10-2017-0091140 Korean Laid-Open Patent No. 10-2016-0143505

The present invention is to solve the problems of the prior art convolution operation, for the DRAM for low power and high speed operation that can greatly reduce the number of power consumption and clock cycles consumed in the convolution operation The purpose is to provide a bitwise convolution circuit.

The present invention provides a low power and high speed operation that enables a bitwise convolution operation using a ternary kernel to be efficiently performed using a processing-in-memory circuit inside a DRAM. It is an object of the present invention to provide a bitwise convolution circuit for a DRAM for the purpose.

The present invention provides a low power for bitwise operation that allows processing of convolutional operations such as convolutional operations to be more energy efficient and within fewer operating cycles by having a Processing-In-Memory circuit. And a bitwise convolution circuit for DRAM for high-speed operation.

The objects of the present invention are not limited to the above-mentioned objects, and other objects that are not mentioned will be clearly understood by those skilled in the art from the following description.

The bitwise convolution circuit for DRAM for low power and high speed operation according to the present invention for achieving the above object comprises a plurality of processing units for performing convolution operations on input data and partial results, A bitwise convolution operation using a ternary kernel is processed with a processing-in-memory circuit structure inside the DRAM and intersected in one word line (WL). It has a plurality of bit lines BL, has pixels in the cross region and has sense amplifiers (SA) corresponding to each pixel, and the processing unit is for bank # 1 to which an accommodating field is assigned, for a positive kernel and a negative kernel. And a feature map output means for outputting a feature map according to a convolution operation for the positive kernel and the negative kernel.

The method may further include a current mirror providing means for providing a current mirror by the pixel value and the kernel value during the convolution operation.

)과 음의 부분(

)으로 나누어지고,And to multiply the pixel value (X ₀ ) by the ternary kernel W ₀ , the kernel (W ₀ ) is a positive part (

) And the negative part (

Divided by)

W ₀ is characterized by being +1, -1 or 0.

In the convolution operation, if both the pixel value and the kernel value are 1, a current mirror is transmitted, and if one or all of them is 0, the current mirror is not transmitted.

으로 이루어지고, 여기서, X_k와 W_k는 각각 픽셀 값과 커널 값인 것을 특징으로 한다.If the result of the convolution operation is stored in the 16x16 feature map, the acceptance field and the kernel are given as 4x4, respectively, and the convolution calculation for the 4x4 acceptance field is

Here, X _k and W _k is characterized in that the pixel value and the kernel value, respectively.

And if the bank has a structure of 1024 cells per word line, one accommodating field is composed of 16 pixels, and if one pixel is represented with 8 bit precision, 256 DRAM cells to represent one accommodating field. And 128 bits of 256 cells are for the positive kernel and the other half are for the negative kernel.

The bitwise convolution circuit for DRAM for low power and high speed operation according to the present invention has the following effects.

First, it significantly reduces the power consumption and clock cycles consumed in convolution operations, enabling low power and high speed operations.

Second, bitwise convolution operations using the ternary kernel can be efficiently performed using processing-in-memory circuitry inside the DRAM.

Third, having processing-in-memory circuitry allows complex operations, such as convolution operations, to be completed more energy efficiently and in fewer operating cycles.

1 and 2 are schematic diagrams showing a convolution operation
3A and 3B are diagrams illustrating an inter-bank bitwise operation
4 is a block diagram of a bitwise convolution circuit for DRAM for low power and high speed operation according to the present invention;

Hereinafter, a preferred embodiment of a bitwise convolution circuit for DRAM for low power and high speed operation according to the present invention will be described in detail.

Features and advantages of the bitwise convolution circuit for DRAM for low power and high speed operation according to the present invention will be apparent from the detailed description of each embodiment below.

1 and 2 are diagrams illustrating a convolution operation, and FIGS. 3A and 3B are diagrams illustrating an inter-bank bitwise operation.

4 is a block diagram of a DRAM bitwise convolution circuit for low power and high speed operation according to the present invention.

The bitwise convolution circuit for DRAM for low-power and high-speed operation according to the present invention can significantly reduce the power consumption and the number of clock cycles consumed in the convolution operation, and the bitwise convolution using a ternary kernel Bitwise convolution operations can be efficiently performed using processing-in-memory circuitry inside the DRAM.

FIG. 1 shows that the MNIST image size is 19 × 19, the receptive field and the kernel are 4 × 4, and the convolution results are stored in the 16 × 16 feature map.

2 illustrates a bitwise process of DRAM in a convolution operation.

In-memory bitwise operations are very useful in many applications that require large amounts of computation and frequent access to memory, such as deep-learning neural networks.

Looking at the convolutional neural network (CNN), one of the most well-known deep learning algorithms for image recognition, in CNN, the image pixels in the acceptance field are multiplied by the kernel. This multiplication is repeated up to the last pixel shifted by one pixel at a time.

1 diagrammatically illustrates a convolution operation to process handwritten digits of a Modified National Institute of Standards and Technology (MNIST) image with 19 × 19 pixels.

으로 이루어진다.Receptive fields and kernels are given by 4 × 4, respectively, and the convolution calculation

Is done.

Where X _k and W _k represent pixel values and kernel values, respectively.

The convolution result is stored in the 16x16 feature map of FIG. 1 and requires 4096 multiplications, additions, and memory accesses to obtain a 16x16 feature map.

In order to perform such a large amount of computation and frequent memory access more efficiently, bitwise processing in DRAM as shown in FIG. 2 is required.

FIG. 3A shows an interbank bitwise processing circuit for a convolution operation, and FIG. 3B shows an operating waveform of an interbank bitwise processing circuit.

The present invention proposes a bitwise convolution circuit in DRAM based on the interbank operation in FIG. 3A. Figure 3b shows the operating waveform of the circuit of Figure 3a.

First, a row of A is activated in bank # 1, and then a row B is activated in bank # 2.

The bit operation can then be performed by the convolution circuit shown in detail in FIG.

The inter bank bitwise circuit can exhibit better power efficiency than the intra bank circuit. This is because convolution can be completed in two cycles as shown in FIG. 3B.

In particular, once the kernel is stored in bank # 2, convolution can be performed in one cycle so that the power consumption and convolution calculation time of FIG. 3A can be significantly reduced compared to intra bank circuitry.

및

의 연산 및 비교를 위한 인터 뱅크 비트와이즈 회로를 도식적으로 나타낸 것이다.4 shows a 16 × 16 feature map.

And

A schematic diagram of an inter-bank bitwise circuit for computing and comparing is shown.

In FIG. 4, RF denotes an acceptance field, and S denotes sense amplifiers (SAs).

The bitwise convolution circuit for DRAM for low power and high speed operation according to the present invention includes a plurality of processing units that execute convolution operations on input data and partial results, as shown in FIG.

The processing unit has a processing-in-memory circuit structure in which DRAM performs bitwise convolution operations using a ternary kernel and crosses one word line WL. Has a plurality of bit lines BL, has pixels in the intersection area, and has sense amplifiers (SAs) corresponding to each pixel.

The processing unit has bank # 1 40 to which the acceptance field is assigned, and bank # 2 50 to be allocated for the positive kernel and the negative kernel.

And feature map output means for outputting a feature map according to a convolution operation for the positive kernel and the negative kernel.

And a current mirror providing means for providing a current mirror by the pixel value and the kernel value in the convolution operation.

4 illustrates an inter bank bitwise processing circuit, in which a bank has a structure having 1024 cells per word line.

One accommodating field is composed of 16 pixels, and if one pixel is represented with 8 bit precision, 256 DRAM cells are allocated to represent one accommodating field.

Of the 256 cells, 128 bits are for the positive kernel and the other half for the negative kernel.

Here, the acceptance field is stored in bank # 1.

X ₀ in FIG. 1A is represented by X _0,0 , X _0,1 , X _0,2 , X _0,3 , X _0,4 , X _0,5 , X _0,6 and X _0,7 in FIG. However, where X _0,7 and X _0,0 respectively represent the most significant bit and the least significant bit in the binary form of X ₀ .

To multiply X ₀ by the ternary kernel W ₀ , the kernel must be divided into a positive part and a negative part.

와

는 각각 W₀의 양 및 음의 부분을 나타낸다.In Figure 4,

Wow

_Denotes the positive and negative portions of W ₀ , respectively.

W ₀ is +1, -1 or 0. This is because the ternary kernel is used in the present invention.

는 감지 및 증폭된다.To calculate the positive part of the convolution, we first use the 8-bit vectors (X _0,0 , X _0,1 , X _0,2 , X _0,3 , X _0,4 , X _0,5 , X _0,6 and X _0,7 )

Is detected and amplified.

가 모두 1이면 전류 미러(current mirror)는 ×128 전류를 R⁺에 전달할 수 있다.X _0,7 and

If all are 1, the current mirror can deliver x128 currents to R ⁺ .

가 1이면 ×64 전류는 R⁺에 전달된다. 이들 중 하나 또는 모두가 0이면 R⁺에 전류가 전달되지 않는다.Similarly, X _0,6 and

If 1, × 64 currents are delivered to R ⁺ . If one or both of these is zero, no current is delivered to R ⁺ .

의 양의 합으로 계산될 수 있고, R⁺의 전류는

의 음의 합계의 R^- 전류와 비교된다.R ⁺ current is

Can be calculated as the sum of the quantities of and the current in R ⁺

The negative sum of is compared with the R ^- current.

If the current of R ⁺ is greater than the current of R ⁻ , C ₀ , the first element of the feature map, becomes 1. Otherwise, C ₀ is 0.

에 의해 턴온되고, X_0,7과 M₂가 모두 '1'일 때만, ×128 전류가 M₄에서 M₃으로 복사된다.In FIG. 4, M ₁ is turned on by X _0,7 and M ₂ is

_X128 current is radiated from M ₄ to M ₃ only when X _0,7 and M ₂ are both '1'.

에 의해 제어된다.Similarly, M ₅ and M ₆ are each X _0,6 and

Controlled by

가 모두 '1'일 때, ×64 전류는 M₈에서 M₇로 복사될 수 있다.X _0,6 and

When both are '1', the x64 current can be radiated from M ₈ to M ₇ .

Also, current mirrors such as M ₄ , M _8, etc. can be shared between convolutional circuits for other receiving fields.

The current mirror with the reference current of FIG. 4 is shared across the entire chip, and only three transistors of M ₁ , M ₂ and M ₃ are mounted in one or two pitches of one or two DRAM cells.

Table 1 compares the intra bank and inter bank bitwise processing circuits.

In terms of computation time, the intra bank bitwise circuit requires 192 cycles of row activation to complete the convolution operation on the 16 × 16 feature map.

The interbank bitwise circuit can complete the same task within 65 cycles, so in terms of power consumption, the interbank circuit consumes 35% less power than the intrabank.

The bitwise convolution circuit for DRAM for low power and high speed operation according to the present invention as described above can significantly reduce the power consumption and the number of clock cycles consumed in the convolution operation, using a ternary kernel. Bitwise convolution operations can be efficiently performed using processing-in-memory circuitry inside the DRAM.

The present invention thus has a processing-in-memory circuitry that allows complex operations, such as convolutional operations, to be more energy efficient and complete in fewer operating cycles.

It will be understood that the present invention is implemented in a modified form without departing from the essential features of the present invention as described above.

Therefore, the described embodiments should be considered in descriptive sense only and not for purposes of limitation, and the scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope are included in the present invention. It should be interpreted.

40. Bank # 1
50. Bank # 2

Claims (6)

A plurality of processing units for executing a convolution operation on the input data and the partial results,
The processing unit processes a bitwise convolution operation using a ternary kernel with a processing-in-memory circuit structure inside the DRAM and processes one word line (WL). Has a plurality of bit lines (BL) intersected at, and has pixels in the intersection area, and has sense amplifiers (SAs) corresponding to each pixel,
The processing unit comprises bank # 1 to which the acceptance field is assigned, bank # 2 to be allocated for the positive kernel and the negative kernel,
A bitwise convolution circuit for DRAM for low power and high speed operation, comprising: feature map output means for outputting a feature map according to a convolution operation for a positive kernel and a negative kernel. 2. The DRAM of claim 1, further comprising: a current mirror providing means for providing a current mirror by a pixel value and a kernel value in a convolution operation. Bitwise convolution circuit. The method of claim 1, in order to multiply the pixel value X ₀ by the ternary kernel W ₀ , the kernel W ₀ is a positive portion (

) And the negative part (

Divided by)
Wise is a bitwise convolution circuit for DRAM for low power and high speed operation, characterized in that + ₀ , -1 or 0. 4. The low power and high speed operation of claim 3, wherein a current mirror is transmitted when both the pixel value and the kernel value are 1 during the convolution operation, and when one or all of them is not transmitted, the current mirror is not transmitted. Bitwise convolution circuit for DRAM. The method of claim 1, wherein when storing the convolution operation result in a 16x16 feature map,
Receptive fields and kernels are given by 4 × 4, respectively, and the convolution calculation for 4 × 4 accepting fields is

Made of
Here, X _k and W _k is a bitwise convolution circuit for DRAM for low power and high speed operation, characterized in that the pixel value and the kernel value, respectively. The method of claim 1, wherein the bank has a structure having 1024 cells per word line.
One accommodating field is composed of 16 pixels, if one pixel is represented with 8 bit precision, 256 DRAM cells are allocated to represent one accommodating field,
Bitwise convolution circuit for DRAM for low power and high speed computation, wherein 128 bits of 256 cells are for the positive kernel and the other half for the negative kernel.

Images