KR102578826B1 - Neural network device and operation method of the same - Google Patents

Abstract

A method of performing operations on a plurality of inputs and the same kernel using delay time with the same processor and a neural network device therefor are disclosed. A neural network device may include neuromorphic hardware and may perform CNN mapping.

Description

Neural network device and operation method thereof {Neural network device and operation method of the same}

This disclosure relates to a neural network device and a method of operating the same.

Neural network refers to a computational architecture that models the biological brain. As neural network technology has recently developed, research is being actively conducted to analyze input data and extract valid information using neural network devices in various types of electronic systems.

In particular, in devices implemented with low power and low performance, technology that can efficiently process neural network operations is required in order to extract desired information by analyzing large amounts of input data in real time using a neural network.

The object is to provide a neural network device using time delay and a method of operating the same. The technical challenges that this embodiment aims to achieve are not limited to the technical challenges described above, and other technical challenges can be inferred from the following embodiments.

A neural network device according to a first aspect includes a memory storing at least one program; and a processor that drives a neural network by executing the at least one program to perform an operation on input data including a first input and a second input, wherein the processor operates between the first input and a plurality of kernels. A first result is obtained by performing an operation, and a second result is obtained by performing an operation between the second input received at a time delayed by a first interval from the time the first input is received and the plurality of kernels. can be obtained, and output data for the input data can be obtained using the first result and the second result.

Additionally, the neural network device includes neuromorphic hardware, and the neuromorphic hardware can perform Convolution Neural Network (CNN) mapping using the first input and the second input.

Additionally, the input data may include image data, the first input may include data for a first region of the image data, and the second input may include data for a second region of the image data. there is.

Additionally, the first area and the second area may partially overlap and be adjacent to each other.

Additionally, the processor determines whether the second input is a valid input,

If the second input is a valid input, the second result can be obtained by performing an operation between the second input and the plurality of kernels.

Additionally, the processor may determine the second input as the valid input when the second input is pixel data constituting the second area.

In addition, the processor receives a plurality of data streams representing the image data with different delay times from a plurality of input terminals, receives the first input from the plurality of data streams received from the plurality of input terminals, and The step of obtaining a second result may include receiving the second input from the plurality of data streams received from the plurality of input terminals.

Additionally, the first input may be received during a first cycle, and the second input may be received during a second cycle delayed by the first interval from the first cycle.

In addition, the processor may obtain the first result by adding the operation results between the first input and the plurality of kernels, and obtain the second result by adding the operation results between the second input and the plurality of kernels. .

In addition, the processor receives a third input included in the input data at a time delayed by a second interval from the time the second input is received, and performs an operation between the third input and the plurality of kernels to obtain a first 3 results may be obtained, and the output data may be obtained using the first result, the second result, and the third result.

In addition, according to a second aspect, a method of performing an operation on input data including a first input and a second input by a neural network device uses a processor included in the neural network device, the first input and a plurality of inputs. Obtaining a first result by performing an operation between the kernels; Obtaining a second result by using the processor to perform an operation between the plurality of kernels and the second input received at a time delayed by a first interval from the time the first input is received; And it may include obtaining output data for the input data using the first result and the second result.

Additionally, a computer-readable non-transitory recording medium on which a program for implementing the method according to the second aspect is recorded on a computer may be provided.

According to the present disclosure, a neural network device can increase the amount of data that can be performed using the same processor by processing data using delay time.

Figure 1 is a diagram for explaining the architecture of a neural network according to an embodiment.
Figure 2 is a diagram for explaining the relationship between an input feature map and an output feature map in a neural network according to an embodiment.
Figure 3 is a block diagram showing the hardware configuration of a neural network device according to an embodiment.
FIG. 4 is a diagram illustrating an example in which a neural network device processes a plurality of inputs included in input data multiple times, according to an embodiment.
FIG. 5 is a diagram illustrating an example in which a neural network device generates a data stream according to an embodiment.
Figure 6 is a diagram illustrating an example of a plurality of data streams having different delay times.
FIG. 7 is a diagram illustrating an example in which a neural network device obtains results of operations with a kernel for a plurality of data streams having different delay times, according to an embodiment.
FIG. 8 is a diagram illustrating an example in which a neural network device performs an operation between an input and a kernel according to an embodiment.
FIG. 9 is a diagram illustrating an example in which a neural network device acquires a plurality of data streams with different delay times, according to an embodiment.
FIG. 10 is a diagram illustrating an example in which a neural network device acquires data for a plurality of areas from a plurality of data streams having different delay times, according to an embodiment.
FIG. 11 is a flowchart illustrating a method by which a neural network device obtains output data from a first input and a second input, according to an embodiment.
FIG. 12 is a flowchart illustrating a method by which a neural network device obtains output data from a plurality of data streams having different delay times, according to an embodiment.
Figure 13 is a flowchart showing a method by which a neural network device acquires output data using first to third inputs according to an embodiment.
Figure 14 is a flowchart showing a method by which a neural network device obtains output data from input data, according to an embodiment.
15A to 15B are diagrams for explaining a method of operating a neuromorphic device according to an embodiment.
16A to 16B are diagrams for comparing vector-matrix multiplication and operations performed in a neural network device according to an embodiment.
FIG. 17 is a diagram illustrating an example of a convolution operation being performed in a neural network device according to an embodiment.
FIG. 18 is a diagram illustrating an example of matching a sub-feature map and a core according to an embodiment.
Figure 19 is a diagram for explaining an example of a vector multiplication operation performed in a core according to an embodiment.
Figure 20 is a diagram for explaining a method of combining output values calculated from a plurality of cores according to an embodiment.
Figure 21 is a flowchart explaining a method of implementing a neural network in a neural network device according to an embodiment.
Figure 22 is a block diagram illustrating a neural network device and memory according to an embodiment.

The terms used in the present embodiments were selected from general terms that are currently widely used as much as possible while considering the functions in the present embodiments, but this may vary depending on the intention or precedent of a technician working in the technical field, the emergence of new technology, etc. . Additionally, in certain cases, there are terms that are arbitrarily selected, and in this case, their meanings will be described in detail in the description of the relevant embodiment. Therefore, the terms used in the present embodiments should not be defined simply as the names of the terms, but should be defined based on the meaning of the term and the overall content of the present embodiments.

In descriptions of embodiments, when a part is connected to another part, this includes not only the case where it is directly connected, but also the case where it is electrically connected with another component in between. Additionally, when it is said that a part includes a certain component, this does not mean that other components are excluded, but that it can further include other components, unless specifically stated to the contrary.

Terms such as “consists of” or “comprises” used in the present embodiments should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or parts thereof Steps may not be included, or may further include additional components or steps.

These embodiments relate to the field of neural network technology, and detailed descriptions of matters widely known to those skilled in the art to which the following embodiments belong will be omitted.

The description of the following examples should not be construed as limiting the scope of rights, and what a person skilled in the art can easily infer should be interpreted as falling within the scope of rights of the embodiments. Hereinafter, embodiments for illustrative purposes only will be described in detail with reference to the attached drawings.

Figure 1 is a diagram for explaining the architecture of a neural network according to an embodiment.

Referring to FIG. 1, the neural network 1 may be an architecture of a deep neural network (DNN) or an n-layer neural network. DNN or n-layer neural networks may correspond to Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Belief Networks, Restricted Boltzman Machines, etc. For example, the neural network 1 may be implemented as a convolutional neural network (CNN), but is not limited thereto. In Figure 1, some convolutional layers are shown in the convolutional neural network corresponding to the example of neural network 1, but in addition to the convolutional layers shown, the convolutional neural network includes a pooling layer and a fully connected ( may further include fully connected layers, etc.

Neural network 1 may be implemented as an architecture with multiple layers including an input image, feature maps, and output. In the neural network (1), a convolution operation is performed on the input image with a filter called a kernel, and as a result, feature maps are output. At this time, the generated output feature maps are input feature maps, and a convolution operation with the kernel is performed again, and new feature maps can be output. As a result of repeatedly performing this convolution operation, a result of recognizing the features of the input image through the neural network 1 can be finally output.

For example, when an image with a size of 24 × 24 pixels is input to the neural network 1 of FIG. 1, the input image can be output as 4-channel feature maps with a size of 20 × 20 through a convolution operation with a kernel. Afterwards, the size of the 20×20 feature maps is reduced through repeated convolution operations with the kernel, and ultimately features of 1×1 size can be output. The neural network (1) filters and outputs strong features that can represent the entire image from the input image by repeatedly performing convolution and subsampling (or pooling) operations in several layers, and outputs them through the output final features. Recognition results of the input image can be derived.

Figure 2 is a diagram for explaining the relationship between an input feature map and an output feature map in a neural network according to an embodiment.

Referring to FIG. 2, in a layer 2 of a neural network, the first feature map FM1 may correspond to an input feature map, and the second feature map FM2 may correspond to an output feature map. A feature map may refer to a data set in which various features of input data are expressed. The feature maps FM1 and FM2 may have elements of a two-dimensional matrix or elements of a three-dimensional matrix, and a pixel value may be defined for each element. The feature maps (FM1, FM2) have a width (W) (also called a column), a height (H) (also called a row), and a depth (C). At this time, depth (C) may correspond to the number of channels.

A convolution operation may be performed on the first feature map (FM1) and the weight map (WM) of the kernel, and as a result, the second feature map (FM2) may be generated. The weight map (WM) filters the features of the first feature map (FM1) by performing a convolution operation with the first feature map (FM1) using the weight defined for each element. The weight map WM performs a convolution operation with the windows (or tiles) of the first feature map FM1 while shifting the first feature map FM1 using a sliding window method. During each shift, each of the weights included in the weight map (WM) may be multiplied and added to each of the pixel values of the overlapping windows in the first feature map (FM1). As the first feature map (FM1) and the weight map (WM) are convolved, one channel of the second feature map (FM2) can be created. Figure 1 shows the weight map (WM) for one kernel, but in reality, the weight maps of a plurality of kernels are each convolved with the first feature map (FM1) to produce a second feature map (FM2) of a plurality of channels. This can be created.

Meanwhile, the second feature map FM2 may correspond to the input feature map of the next layer. For example, the second feature map FM2 may be an input feature map of a pooling (or subsampling) layer.

In FIGS. 1 and 2, only the schematic architecture of the neural network 1 is shown for convenience of explanation. However, unlike what is shown, the neural network 1 may be implemented with more or fewer layers, feature maps, kernels, etc., and its sizes may also be varied in various ways. Anyone skilled in the art can understand this.

FIG. 3 is a block diagram showing the hardware configuration of the neural network device 100 according to an embodiment.

The neural network device 100 can be implemented with various types of devices such as personal computers (PCs), server devices, mobile devices, and embedded devices, and specific examples include voice recognition, image recognition, image classification, etc. using neural networks. This may apply to smartphones, tablet devices, AR (Augmented Reality) devices, IoT (Internet of Things) devices, self-driving cars, robotics, medical devices, etc., but is not limited thereto. Furthermore, the neural network device 100 may correspond to a dedicated hardware accelerator (HW accelerator) mounted on the above device, and the neural network device 100 includes a neural processing unit (NPU), a dedicated module for running a neural network, It may be a hardware accelerator such as a TPU (Tensor Processing Unit), Neural Engine, etc., but is not limited thereto.

Referring to FIG. 3, the neural network device 100 includes a processor 120 and a memory 110. In the neural network device 100 shown in FIG. 3, only components related to the present embodiments are shown. Accordingly, it is obvious to those skilled in the art that the neural network device 100 may further include other general-purpose components in addition to the components shown in FIG. 3.

The processor 120 serves to control overall functions for executing the neural network device 100. For example, the processor 120 generally controls the neural network device 100 by executing programs stored in the memory 110 within the neural network device 100. The processor 120 may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc. provided in the neural network device 100, but is not limited thereto.

The memory 110 is hardware that stores various data processed within the neural network device 100. For example, the memory 110 can store data processed and data to be processed in the neural network device 100. there is. Additionally, the memory 110 may store applications, drivers, etc. to be run by the neural network device 100. The memory 110 includes random access memory (RAM) such as dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD- It may include ROM, Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or flash memory.

Alternatively, the memory 110 may be an on-chip memory. The neural network device 100 according to one embodiment has the memory 110 only in the form of an on-chip memory, and can perform calculations without access to external memory. For example, the memory 110 may be SRAM implemented as an on-chip memory. In this case, unlike what was described above, types of memory that are mainly used as external memories, such as DRAM, ROM, HDD, and SSD, may not be used as the memory 110.

The processor 120 reads/writes neural network data, such as image data, feature map data, kernel data, etc., from the memory 110, and executes the neural network using the read/written data. You can. When the neural network is executed, the processor 120 may repeatedly perform a convolution operation between the input feature map and the kernel to generate data related to the output feature map. At this time, the amount of convolution operation may be determined depending on various factors such as the number of channels of the input feature map, the number of channels of the kernel, the size of the input feature map, the size of the kernel, and the precision of the value. Unlike the neural network 1 shown in FIG. 1, an actual neural network running on the neural network device 100 may be implemented with a more complex architecture. Accordingly, the processor 120 performs convolution operations with a very large operation count ranging from hundreds of millions to tens of billions, and the frequency with which the processor 120 accesses the memory 110 for convolution operations also dramatically decreases. can be increased.

The neural network device 100 according to one embodiment may include neuromorphic hardware. Neuromorphic hardware can perform CNN mapping, etc. Neuromorphic hardware can perform calculations using only on-chip memory without using external memory. For example, neuromorphic hardware performs CNN mapping using only on-chip memory without using external memory (e.g., off-chip memory, etc.), allowing calculations to be performed without memory updates during image processing.

The processor 120 according to one embodiment may perform operations on a plurality of inputs. The processor 120 may perform an operation on input data including a first input and a second input. The first input or the second input may represent all or part of the input feature map or input image data. For example, the first input may represent data for a first region of the input feature map or input image data, and the second input may represent data for a second region of the input feature map or input image data.

The processor 120 according to one embodiment does not place each kernel reused in multiple locations in CNN mapping in physically different memories, but actually places them only in one location and uses input images from different locations over time for several times. Operations can be performed on output data (e.g., data about output images). The processor 120 according to one embodiment may obtain output data by performing operations on each area of the output image multiple times.

The processor 120 according to one embodiment may receive a first input, perform an operation between the first input and a plurality of kernels, and obtain a first result.

According to one embodiment, the first input may include data for the first area of the input feature map.

The processor 120 according to one embodiment may perform an operation between the first input and a plurality of kernels. For example, the processor 120 performs an operation between a first input and a first kernel (e.g., the kernel for red) to obtain a 1-1 result, and performs an operation between the first input and a second kernel (e.g., the kernel for green). The 1-2 result can be obtained by performing an operation between the first input and the third kernel (eg, the blue kernel), and the 1-3 result can be obtained by performing the operation between the first input and the third kernel (eg, the blue kernel). The processor 120 may obtain the first result using the 1-1 result, the 1-2 result, and the 1-3 result. For example, the processor 120 may obtain the first result by adding the 1-1 result, the 1-2 result, and the 1-3 result.

The processor 120 according to one embodiment receives the second input at a time delayed by a first interval from the time the first input is received, and obtains a second result by performing an operation between the second input and a plurality of kernels. can do.

According to one embodiment, the second input may include data for the second area of the input feature map.

The processor 120 according to one embodiment may receive the second input at a time delayed by the first interval from the time the first input is received. The processor 120 may perform an operation between the received second input and a plurality of kernels. For example, the processor 120 performs an operation between a second input and a first kernel (e.g., the kernel for red) to obtain a 2-1 result, and performs an operation between the second input and the second kernel (e.g., the kernel for green). The 2-2 result may be obtained by performing an operation between the second input and the third kernel (e.g., the blue kernel), and the 2-3 result may be obtained by performing an operation between the second input and the third kernel (e.g., the blue kernel). The processor 120 may obtain the first result using the 2-1 result, the 2-2 result, and the 2-3 result. For example, the processor 120 may obtain the second result by adding the 2-1 result, the 2-2 result, and the 2-3 result. Additionally, the plurality of kernels used when obtaining the first result and the plurality of kernels used when obtaining the second result may be the same.

The processor 120 according to one embodiment may obtain output data for input data using the first result and the second result.

Input data may include feature map image data. For example, the input data may be 2D image data. As another example, the input data may be 3D image data.

When the input data is image data, the processor 120 according to an embodiment generates output data for the input data using the first result, which is the result of processing for the first area, and the second result, which is the result of processing for the second area. can be obtained.

According to one embodiment, the first area and the second area may partially overlap. For example, when the first area and the second area are 2x2 in size, a 1x2 area may overlap between the first area and the second area.

According to one embodiment, the first area and the second area may be adjacent to each other. For example, the second area may be located to the right of the first area. As another example, the second area may be located below the first area.

FIG. 4 is a diagram illustrating an example in which the neural network device 100 processes a plurality of inputs included in input data multiple times according to an embodiment.

Input data 400 may include a first input 411, a second input 412, and a third input 413.

The plurality of kernels 421 and 422 may include a first kernel 421 and a second kernel 422.

The first results 431 and 432 may include the 1-1 result 431 and the 1-2 result 432, and the second results 441 and 442 may include the 2-1 result 441 and It may include the 2-2 result 442, the third result 451, 452, the 3-1 result 451, and the 3-2 result 452.

The neural network device 100 according to an embodiment performs an operation between a first input 411 and a plurality of kernels 421 and 422 to obtain first results 431 and 432, and obtains a second input 412. and a plurality of kernels 421, 422 to obtain second results (441, 442), and to perform an operation between the third input 413 and a plurality of kernels (421, 422) to obtain a third result ( 451, 452) can be obtained.

Input data 400 may be 3D image data. Additionally, the first input 411, the second input 412, the third input 413, etc. may be part of the input data 400. The first input 411 and the second input 412 may partially overlap. Alternatively, the first input 411 and the second input 412 may be adjacent to each other.

The 1-1 result 431 and the 1-2 result 432 included in the first results 431 and 432 may represent result data from different layers. Result data may refer to data representing an output image.

FIG. 5 is a diagram illustrating an example in which the neural network device 100 generates a data stream according to an embodiment. Referring to FIG. 5, FIG. 5 schematically shows an example in which the neural network device 100 acquires data streams 510, 520, 530, and 540 from input data 500, according to an embodiment.

In the neural network device 100 according to an embodiment, input data 500 including first data 510, second data 520, third data 530, and fourth data 540 is first By sequentially outputting data 510, second data 520, third data 530, and fourth data 540, first data 510, second data 520, and third data 530 and data streams 510, 520, 530, and 540 consisting of fourth data 540.

In addition, the generated data streams 510, 520, 530, and 540 are transmitted to the processor 120 included in the neural network device 100, and the processor 120 processes the data streams 510, 520, 530, and 540. Operations can be performed between multiple kernels.

Figure 6 is a diagram illustrating an example of a plurality of data streams having different delay times. Referring to FIG. 6, the neural network device 100 according to an embodiment acquires a plurality of data streams 610, 620, and 630 with different delay times generated from input data 500. An example is shown schematically.

The neural network device 100 according to an embodiment may generate data streams with different delay times. For example, the first data stream 610, the second data stream 620, and the third data stream 630 may have different delay times. When the delay time of the first data stream 610 is 0 cycle, the second data stream 620 may have a delay time of 1 cycle, and the third data stream 630 may have a delay time of 2 cycles.

Delay time according to one embodiment may include axonal delay time.

The neural network device 100 according to an embodiment can generate a plurality of data streams with different delay times by duplicating the data stream multiple times and rearranging it temporally using the axon delay time for each neuron and several synapses. there is.

FIG. 7 is a diagram illustrating an example in which the neural network device 100 obtains results of operations with a kernel for a plurality of data streams having different delay times, according to an embodiment. Referring to FIG. 7, the neural network device 100 according to an embodiment receives a plurality of inputs 710, 720, 730, and 740 from a plurality of data streams 610, 620, and 630 having different delay times. ) and obtain a plurality of results (715, 725, 735, 745) through operations between a plurality of inputs (710, 720, 730, 740) and a plurality of kernels.

The neural network device 100 according to an embodiment receives a first input 710, a second input 720, and a third input 730 from a plurality of data streams 610, 620, and 630 having different delay times. , the fourth input 740, etc. can be obtained.

The neural network device 100 according to one embodiment obtains a first result 715 by performing an operation between the first input 710 and a plurality of kernels, and performs an operation between the second input 720 and a plurality of kernels. A second result 725 is obtained by performing an operation between the third input 730 and a plurality of kernels, and a third result 735 is obtained by performing an operation between the fourth input 740 and a plurality of kernels. The fourth result 745 can be obtained by performing the process. Each result (715, 725, 735, 745) may represent data about the output image. For example, each result 715, 725, 735, 745 may include data for one or more pixels. Additionally, each result (715, 725, 735, 745) can be used as input to the next layer. For example, each result (715, 725, 735, 745) can be used as input for the next layer by being transmitted in an overlapping manner.

FIG. 8 is a diagram illustrating an example in which the neural network device 100 performs an operation between an input and a kernel according to an embodiment.

According to one embodiment, the input data 810 may be 4x4x3 image data, several kernels 820 may be composed of 2x2x3 kernels, and the output data 830 may be 3x3x3 image data. The first layer of output data 830 is the 1-1 result (a), the 2-1 result (b), the 3-1 result (c), the 4-1 result (d), and the 5-1 result. (e), 6-1 result (f), 7-1 result (g), 8-1 result (h), and 9-1 result (i). In a similar manner, the second layer of output data 830 will be composed of results 1-2 to 9-2, and the third layer of output data 830 will be composed of results 1-3 to 9-3. You can.

FIG. 9 is a diagram illustrating an example in which the neural network device 100 acquires a plurality of data streams with different delay times according to an embodiment.

The neural network device 100 according to one embodiment may generate data streams 900 and 901 from input data. The neural network device 100 may generate a plurality of data streams 910, 920, 930, and 940 with different delay times from the generated data streams 900 and 901. For example, the neural network device 100 includes a first data stream 940 with a delay time of 0, a second data stream 930 with a delay time of 1, a third data stream 920 with a delay time of 4, A fourth data stream 910 with a delay time of 5 can be created using the data stream 900 for channel 0.

The neural network device 100 according to one embodiment may obtain a plurality of inputs from a plurality of data streams having different delay times. For example, the neural network device 100 uses data obtained in the same cycle from the first data stream 940 to the fourth data stream 910 to input the first input 950 and the second input 960. , the third input 970, the fourth input 990, etc. can be obtained. The first to fourth inputs 950 to 990 may represent an input feature map or a portion of an input image. For example, the first input 950 represents data for the first region (pixels 1, 2, 5, and 6) of the input feature map, and the second input 960 represents data for the second region (pixels 1, 2, 5, and 6) of the input feature map. 2, 3, 6, 7), the third input 970 represents data for the third area (pixels 3, 4, 7, and 8) of the input feature map, and the fourth input 990 may represent data for the fourth area (pixels 5, 6, 9, and 10) of the input feature map.

The neural network device 100 according to an embodiment may generate output data using some of the inputs 950, 960, 970, 980, and 990 obtained from a plurality of data streams having different delay times. For example, the neural network device 100 includes a first input 950, a second input 960, and Output data may be generated using only the third input 970 and the fourth input 990. In this case, the fifth input 980 may not be used to generate output data.

FIG. 10 is a diagram illustrating an example in which the neural network device 100 acquires data for a plurality of areas from a plurality of data streams having different delay times, according to an embodiment.

When the input data 1000 is 4x4 image data or a feature map, the neural network device 100 according to an embodiment generates a plurality of data streams 1054, 1053, and 1052 with different delay times from the input data 1000. , 1051), and a plurality of inputs (1015, 1025, 1035) can be generated from the generated plurality of data streams (1054, 1053, 1052, 1051). For example, the neural network device 100 includes a first data stream 1054 with a delay time of 0, a second data stream 1053 with a delay time of 1, a third data stream 1052 with a delay time of 4, A fourth data stream 1051 with a delay time of 5 is generated from the input data 1000, and a first input 1015 and a second input are generated from the generated first data streams 1054 to 4 data streams 1051. (1025), third input (1035), etc. can be generated.

A plurality of inputs 1015, 1025, and 1035 according to an embodiment may represent part of the input data 1000. For example, the first input 1015 represents data for the first area 1010 of the input data 1000, and the second input 1025 represents data for the second area 1020 of the input data 1000. Represents data, and the third input 1035 may represent data for the third area 1030 of the input data 1000.

FIG. 11 is a flowchart illustrating a method by which the neural network device 100 obtains output data from a first input and a second input according to an embodiment.

In step S1110, the neural network device 100 according to an embodiment obtains a first result by performing an operation between the first input and a plurality of kernels.

According to one embodiment, the first input may include an input feature map or data for a first area of input image data.

The neural network device 100 according to one embodiment may perform an operation between a first input and a plurality of kernels. For example, the neural network device 100 performs an operation between a first input and a first kernel (e.g., a kernel for red) to obtain a 1-1 result, and performs an operation between the first input and a second kernel (e.g., a kernel for red). The 1-2 result may be obtained by performing an operation between the first input and the third kernel (e.g., the kernel for blue), and the 1-3 result may be obtained by performing the operation between the first input and the third kernel (e.g., the kernel for blue). The neural network device 100 may obtain the first result using the 1-1 result, the 1-2 result, and the 1-3 result. For example, the neural network device 100 may obtain the first result by adding the 1-1 result, the 1-2 result, and the 1-3 result.

In step S1120, the neural network device 100 according to an embodiment obtains a second result by performing an operation between the second input received and a plurality of kernels at a time when the first input is delayed by the first interval. do. The plurality of kernels used to obtain the second result may be the same as the plurality of kernels used in step S1110.

According to one embodiment, the second input may include an input feature map or data for a second area of the input image data.

The neural network device 100 according to one embodiment may receive the second input at a time delayed by the first interval from the time the first input is received. The neural network device 100 may perform an operation between the received second input and a plurality of kernels. For example, the neural network device 100 obtains a 2-1 result by performing an operation between the second input and the first kernel (e.g., the kernel for red), and performs an operation between the second input and the second kernel (e.g., the kernel for red). The 2-2 result may be obtained by performing an operation between the second input and the third kernel (e.g., the kernel for blue), and the 2-3 result may be obtained by performing the operation between the second input and the third kernel (e.g., the kernel for blue). The neural network device 100 may obtain the first result using the 2-1 result, the 2-2 result, and the 2-3 result. For example, the neural network device 100 may obtain the second result by adding the 2-1 result, the 2-2 result, and the 2-3 result.

In step S1130, the neural network device 100 according to an embodiment acquires output data for input data using the first result and the second result.

Input data may include feature maps or image data. For example, the input data may be 2D image data. As another example, the input data may be 3D image data.

When the input data is image data, the processor 120 according to an embodiment generates output data for the input data using the first result, which is the result of processing for the first area, and the second result, which is the result of processing for the second area. can be obtained.

According to one embodiment, the first area and the second area may partially overlap. For example, when the first area and the second area are 2x2 in size, a 1x2 area may overlap between the first area and the second area, but this is not limited to this embodiment and the first area and the second area can be of various sizes. This can be implemented.

According to one embodiment, the first area and the second area may be adjacent to each other. For example, the second area may be located to the right of the first area. As another example, the second area may be located below the first area, but this is not limited to this embodiment, and the first area and the second area may be implemented in various mutual positions.

FIG. 12 is a flowchart illustrating a method by which the neural network device 100 obtains output data from a plurality of data streams having different delay times, according to an embodiment.

In step S1210, the neural network device 100 according to an embodiment receives a plurality of data streams representing image data with different delay times from a plurality of input terminals.

The neural network device 100 according to an embodiment may acquire a plurality of data streams representing image data with different delay times. The neural network device 100 may acquire a data stream from input data representing image data and transmit the obtained data stream multiple times, thereby obtaining a plurality of data streams with different delay times.

The neural network device 100 according to an embodiment may obtain a plurality of data streams with different delay times by receiving the plurality of generated data streams through a plurality of input terminals.

In step S1220, the neural network device 100 according to an embodiment receives a first input representing a first area from a plurality of data streams received from a plurality of input terminals.

For example, the neural network device 100 may obtain data obtained at a specific time from a plurality of input terminals as the first input. As an example, the neural network device 100 may acquire data obtained from a plurality of input terminals in the sixth cycle as the first input. (See FIGS. 9 and 10) The first input may represent a first region of image data.

In step S1230, the neural network device 100 according to an embodiment obtains a first result by performing an operation between the first input and a plurality of kernels.

For example, the 1-1 operation result is the result of the operation between the 1-1 input constituting the first input and the first kernel, and the 1-1 operation result is the operation result between the 1-2 input constituting the first input and the second kernel. The first result can be obtained using the 1-2 operation result, which is the result of the 1-3 operation between the 1-3 input constituting the first input and the third kernel. As an example, the neural network device 100 may generate a first result by adding the 1-1 operation result, the 1-2 operation result, and the 1-3 operation result.

In step S1240, the neural network device 100 according to an embodiment receives a second input from a plurality of data streams received from a plurality of input terminals at a time delayed by the first interval from the time the first input is received.

For example, the neural network device 100 may obtain data obtained at a specific time from a plurality of input terminals as a second input. As an example, the neural network device 100 may acquire data obtained from a plurality of input terminals in the seventh cycle as a second input. (See Figures 9 and 10)

In step S1250, the neural network device 100 according to an embodiment determines the second input as a valid input when the second input is data representing the second area.

The second area may be determined based on its relative position with respect to the first area.

For example, the first area and the second area may partially overlap. For example, when the first area and the second area are 2x2 in size, a 1x2 area may overlap between the first area and the second area.

As another example, the first area and the second area may be adjacent to each other. For example, the second area may be located to the right of the first area. As another example, the second area may be located below the first area.

The second area may be determined according to a preset method among various methods, and when the second input is data representing the second area, the neural network device 100 according to one embodiment may determine the second input as a valid input. there is.

Additionally, in a manner similar to that described above in steps S1210 to S1250, as shown in FIG. 10, the neural network device 100 according to one embodiment converts data obtained from a plurality of input terminals in 7 cycles into a valid third input. In cycle 8, data obtained from a plurality of input terminals is acquired as a valid fourth input, in cycle 10, data obtained from a plurality of input terminals are acquired as a valid fifth input, and in cycle 11, data obtained from a plurality of input terminals are acquired as a valid fifth input. Data is acquired as a valid sixth input, data acquired from a plurality of input terminals is acquired as a valid seventh input in cycle 12, data acquired from a plurality of input terminals is acquired as a valid eighth input in cycle 14, and data obtained from a plurality of input terminals is acquired as a valid eighth input in cycle 15. Data obtained from a plurality of input terminals can be obtained as a valid ninth input.

In step S1260, when the second input is a valid input, the neural network device 100 according to an embodiment performs an operation between the second input and a plurality of kernels to obtain a second result.

For example, the 2-1 operation result is the result of the operation between the 2-1 input constituting the second input and the first kernel, and the 2-1 operation result is the operation result between the 2-2 input constituting the second input and the second kernel. The second result can be obtained using the 2-2 operation result, which is the result of the operation between the 2-3 input constituting the second input and the third kernel. As an example, the neural network device 100 may generate a second result by adding the 2-1st operation result, the 2-2nd operation result, and the 2-3th operation result.

In step S1270, the neural network device 100 according to an embodiment acquires output data for input data using the first result obtained in step S1230 and the second result obtained in step S1260.

Output data may include input data and results of operations with a plurality of kernels, and output data that is the result of operations on image data may be output at once (e.g., in one cycle) or sequentially.

FIG. 13 is a flowchart illustrating a method by which the neural network device 100 acquires output data using first to third inputs according to an embodiment.

Since step S1310 and step S1320 correspond to step S1110 and step S1120, respectively, detailed description is omitted to simplify the overall description. For step S1310 and step S1320, refer to the description of step S1110 and step S1120 described above.

In step S1330, the neural network device 100 according to an embodiment receives the third input included in the input data at a time delayed by the second interval from the time the second input is received.

The second interval may be different from the first interval. For example, the first interval may correspond to the first cycle and the second interval may correspond to the second cycle.

In step S1340, the neural network device 100 according to an embodiment obtains a third result by performing an operation between the third input and a plurality of kernels.

The plurality of kernels used to obtain the third result may be the same as the plurality of kernels used in step S1310.

According to one embodiment, the third input may include an input feature map or data for a third area of input image data.

The neural network device 100 according to one embodiment may perform an operation between the received third input and a plurality of kernels. For example, the neural network device 100 performs an operation between a third input and a first kernel (e.g., a kernel for red) to obtain a 3-1 result, and performs an operation between the third input and a second kernel (e.g., a kernel for red). The 3-2 result may be obtained by performing an operation between the third input and the third kernel (e.g., the kernel for blue), and the 3-3 result may be obtained by performing the operation between the third input and the third kernel (e.g., the kernel for blue). The neural network device 100 may obtain the third result using the 3-1st result, the 3-2nd result, and the 3-3rd result. For example, the neural network device 100 may obtain the third result by adding the 3-1 result, the 3-2 result, and the 3-3 result.

In step S1350, the neural network device 100 according to an embodiment acquires output data using the first result, the second result, and the third result.

Output data may include input data and results of operations with a plurality of kernels, and output data that is the result of operations on image data may be output at once (e.g., in one cycle) or sequentially. For example, the first result, the second result, and the third result may be output sequentially.

FIG. 14 is a flowchart illustrating a method by which the neural network device 100 obtains output data from input data according to an embodiment.

In step S1410, the neural network device 100 according to an embodiment generates a data stream from input data. Step S1410 may be performed on the first layer.

In step S1420, the neural network device 100 according to an embodiment acquires a plurality of data streams with different delay times by duplicating and temporally rearranging the data stream generated in step S1410.

In step S1430, the neural network device 100 according to an embodiment uses a plurality of data streams with different delay times as input to a memory array existing in a specific kernel to perform an operation on the output stream. Steps S1420 and S1430 may be performed in the middle layer and may be performed repeatedly multiple times.

In step S1440, the neural network device 100 according to an embodiment acquires output data from an output stream generated over several cycles. Additionally, an image to be output can be created using the acquired output data. For example, the neural network device 100 can generate a 3D image. Additionally, step S1440 may be performed in the last layer.

15A to 15B are diagrams for explaining a method of operating a neuromorphic device according to an embodiment.

Referring to FIG. 15A, a neural network device may include a plurality of cores, and each core may be implemented as RCA (Resistive Crossbar Memory Arrays). Specifically, each core includes a plurality of presynaptic neurons (1510), a plurality of postsynaptic neurons (1520), and a plurality of presynaptic neurons (1510) and a plurality of post synaptic neurons (1520). It may include a synapse 1530 that provides each connection.

In one embodiment, the core of the neural network device includes 4 pre-synaptic neurons 1510, 4 post-synaptic neurons 1520, and 16 synapses 1530, but these numbers may vary. The number of pre-synaptic neurons 1510 is N (where N is a natural number of 2 or more), and the number of post-synaptic neurons 1520 is M (where M is a natural number of 2 or more and may be equal to or different from N). In this case, N*M synapses 1530 may be arranged in a matrix form.

Specifically, a wiring 1512 connected to each of a plurality of pre-synaptic neurons 1510 and extending in a first direction (e.g., horizontal direction), and a wiring 1512 connected to each of a plurality of post-synaptic neurons 1520 and extending in the first direction. A wiring 1522 extending in a second direction (eg, vertical direction) that intersects may be provided. Hereinafter, for convenience of explanation, the wiring 1512 extending in the first direction will be referred to as a row line, and the wiring 1522 extending in the second direction will be referred to as a column line. A plurality of synapses 1530 may be disposed at each intersection of the row wiring 1512 and the column wiring 1522 to connect the corresponding row wiring 1512 and the corresponding column wiring 1522 to each other.

The pre-synaptic neuron 1510 generates a signal, for example, a signal corresponding to specific data and sends it to the row wiring 1512, and the post-synaptic neuron 1520 transmits the synaptic signal that has passed through the synaptic element 1530 to the column wiring. It can perform the role of receiving and processing through (1522). The pre-synaptic neuron 1510 may correspond to an axon, and the post-synaptic neuron 1520 may correspond to a neuron. However, whether a neuron is pre-synaptic or postsynaptic can be determined by its relative relationship to other neurons. For example, when the pre-synaptic neuron 1510 receives synaptic signals in a relationship with another neuron, it may function as a post-synaptic neuron. Similarly, a post-synaptic neuron 1520 may function as a pre-synaptic neuron when it sends signals in relationship to other neurons. The pre-synaptic neuron 1510 and the post-synaptic neuron 1520 may be implemented with various circuits such as CMOS.

The connection between the pre-synaptic neuron 1510 and the post-synaptic neuron 1520 may be made through the synapse 1530. Here, the synapse 1530 is an element whose electrical conductance or weight changes depending on electrical pulses, such as voltage or current, applied to both ends.

The synapse 1530 may include, for example, a variable resistance element. A variable resistance element is an element that can switch between different resistance states depending on the voltage or current applied to both ends, and is made of various materials that can have multiple resistance states, such as transition metal oxides and perovskite systems. It may have a single-layer structure or a multi-layer structure including metal oxides such as metal oxides, phase change materials such as chalcogenide-based materials, ferroelectric materials, and ferromagnetic materials. The operation in which the variable resistance element and/or synapse 1530 changes from a high resistance state to a low resistance state is called a set operation, and the operation in which it changes from a low resistance state to a high resistance state can be called a reset operation. .

However, unlike the variable resistance elements used in memory devices such as RRAM, PRAM, FRAM, and MRAM, the synapse 1530 of the core does not have an abrupt change in resistance during set and reset operations and does not change the resistance of the input electrical pulse. It can be implemented to have various characteristics that are distinct from variable resistance elements in memory, such as showing analog behavior in which conductivity gradually changes depending on the number. This is because the characteristics required for the variable resistance element in the memory and the characteristics required for the synapse 1530 in the core of the neural network device are different from each other.

The operation of the above neural network device is explained with reference to FIG. 15B as follows. For convenience of explanation, the row wiring 1512 may be referred to as a first row wiring 1512A, a second row wiring 1512B, a third row wiring 1512C, and a fourth row wiring 1512D in order from the top. , the column wiring 1522 may be referred to as a first column wiring 1522A, a second column wiring 1522B, a third column wiring 1522C, and a fourth column wiring 1522D in that order from the left.

Referring to FIG. 15B, in the initial state, all of the synapses 1530 may be in a state of relatively low conductivity, that is, a high resistance state. If at least some of the plurality of synapses 1530 are in a low-resistance state, an additional initialization operation may be required to bring them into a high-resistance state. Each of the plurality of synapses 1530 may have a predetermined threshold required for change in resistance and/or conductance. More specifically, when a voltage or current smaller than a predetermined threshold is applied to both ends of each synapse 1530, the conductivity of the synapse 1530 does not change, and a voltage or current larger than the predetermined threshold is applied to the synapse 1530. When this happens, the conductivity of the synapse 1530 may change.

In this state, in order to perform an operation of outputting specific data as a result of the specific column wiring 1522, an input signal corresponding to specific data is input to the row wiring 1512 in response to the output of the pre-synaptic circuit 1510. You can. At this time, the input signal may appear as an electrical pulse applied to each row wiring 1512. For example, when an input signal corresponding to data of '0011' is input to the row wiring 1512, an electrical pulse is generated in the row wiring 1512 corresponding to '0', for example, the first and second row wirings 1512A and 1512B. is not applied, and an electrical pulse may be applied only to the row wiring 1512 corresponding to '1', for example, the third and fourth row wirings 1512C and 1512D. At this time, the column wire 1522 may be driven with an appropriate voltage or current for output.

As an example, when the column wire 1522 to output specific data has already been determined, the synapse 1530 located at the intersection of this column wire 1522 with the row wire 1512 corresponding to '1' performs the set operation. It may be driven to receive a voltage greater than the required voltage (hereinafter referred to as set voltage), and the remaining column wiring 1522 may be driven to allow the remaining synapses 1530 to receive a voltage smaller than the set voltage. For example, if the size of the set voltage is Vset and the column wire 1522 to output data of '0011' is set to the third column wire 1522C, the third column wire 1522C and the third and fourth rows The size of the electrical pulse applied to the third and fourth row wires 1512C and 1512D so that the first and second synapses 1530A and 1530B located at the intersection with the wires 1512C and 1512D receive a voltage equal to or higher than Vset. may be greater than Vset and the voltage applied to the third column wiring 1522C may be 0V. Accordingly, the first and second synapses 1530A and 1530B may be in a low resistance state. The conductivity of the first and second synapses 1530A and 1530B in a low-resistance state may gradually increase as the number of electrical pulses increases. The size and width of the applied electrical pulse may be substantially constant. Except for the first and second synapses 1530A and 1530B, the remaining synapses 1530 are connected to the remaining column wirings, that is, the first, second and fourth column wirings 1522A, 1522B and 1522D, so that a voltage smaller than Vset is applied. The applied voltage may have a value between 0V and Vset, for example, 1/2Vset. Accordingly, the resistance state of the remaining synapses 1530 except for the first and second synapses 1530A and 1530B may not change. The current or electron flow in this case is indicated by a dotted arrow.

As another example, the column wiring 1522 to output specific data may not be determined. In this case, while applying an electrical pulse corresponding to specific data to the row wiring 1512, the current flowing through each of the column wirings 1522 is measured to determine the column wiring 1522 that reaches a predetermined threshold current first, for example, the third column wiring. (1522C) may be the column wiring 1522 that outputs this specific data.

Using the method described above, different data can be output to different column wires 1522, respectively.

16A to 16B are diagrams for comparing vector-matrix multiplication and operations performed in a neural network device according to an embodiment.

16A to 16B are diagrams for comparing vector-matrix multiplication and operations performed in a neural network device according to an embodiment.

First, referring to FIG. 16A, the convolution operation between the input feature map and the kernel can be performed using vector-matrix multiplication. For example, pixel data of the input feature map can be expressed as a matrix X (1610), and kernel values can be expressed as a matrix W (1611). Pixel data of the output feature map can be expressed as matrix Y (1612), which is the result of a multiplication operation between matrix X (1610) and matrix W (1611).

Referring to FIG. 16b, a vector multiplication operation can be performed using the core of a neural network device. In comparison with FIG. 16A, pixel data of the input feature map may be received as an input value of the core, and the input value may be a voltage 1620. Additionally, kernel values may be stored in the synapse of the core, that is, a memory cell, and the kernel values stored in the memory cell may be conductance 1621. Accordingly, the output value of the core can be expressed as a current 1622, which is the result of a multiplication operation between the voltage 1620 and the conductance 1621.

FIG. 17 is a diagram for explaining an example of a convolution operation being performed in a neural network device according to an embodiment.

The neural network device can receive pixel data of the input feature map 1710, and the core 1700 of the neural network device can be implemented with RCA (Resistive Crossbar Memory Arrays).

In one embodiment, when the core 1700 is a matrix of size NxM (N and M are natural numbers of 2 or more), the number of pixel data of the input feature map 1710 is less than the number of columns (M) of the core 1700. It may be the same. Pixel data of the input feature map 1710 may be parameters in floating point format or fixed point format. Meanwhile, in another embodiment, the number of pixel data of the input feature map 1710 may be larger than the number of columns (M) of the core 1700, which will be described in detail in FIG. 18.

A neural network device can receive pixel data in the form of a digital signal and, using a DAC (Digital Analog Converter, 1720), convert the received pixel data into a voltage in the form of an analog signal. Pixel data of the input feature map 1710 may have various bit resolution values, such as 1-bit, 4-bit, and 8-bit resolution. In one embodiment, the neural network device may convert pixel data into voltage using the DAC 1720 and then receive the voltage as the input value 1701 of the core 1700.

Additionally, learned kernel values may be stored in the core 1700 of the neural network device. Kernel values may be stored in a memory cell of the core, and the kernel values stored in the memory cell may be conductance 1702. At this time, the neural network device can calculate the output value by performing a vector multiplication operation between the voltage 1701 and the conductance 1702, and the output value can be expressed as a current 1703. That is, the neural network device can use the core 1700 to output the same result as the result of the convolution operation between the input feature map and the kernel.

Since the current 1703 output from the core 1700 is an analog signal, the neural network device can use an analog digital converter (ADC) 1730 to use the current 1703 as input data for another core. The neural network device can convert the current 1703, which is an analog signal, into a digital signal using the ADC 1730. In one embodiment, the neural network device may use the ADC 1730 to convert the current 1703 into a digital signal to have the same bit resolution as the pixel data of the input feature map 1710. For example, if the pixel data of the input feature map 1710 has 1-bit resolution, the neural network device can convert the current 1703 into a digital signal with 1-bit resolution using the ADC 1730.

The neural network device can apply an activation function to the digital signal converted by the ADC 1730 using the activation unit 1740. The Sigmoid function, Tanh function, and ReLU (Rectified Linear Unit) function can be used as activation functions, but the activation function applicable to digital signals is not limited to these.

The digital signal to which the activation function is applied can be used as an input value for another core 1750. When a digital signal to which an activation function is applied is used as an input value for another core 1750, the above-described process can be applied equally to the other core 1750.

Figure 18 is a diagram for explaining an example of matching a sub-feature map and a core according to an embodiment.

The input feature map 1810 used for learning and inference may have various sizes, but since the size of the core 1800 of the neural network device is limited, the number of pixel data of the single input feature map 1810 is limited to the core 1800. There may be more than the number of input values that can be received.

Referring to FIG. 18, the size of the input feature map 1810 is 8x8, and the size of the core 1800 is 16x16. In this case, the number of pixel data of the 8x8 input feature map 1810 is 64 (=8x8), so it has a value greater than 16, the number of input values that can be received from the core 1800.

If the number of pixel data of the input feature map 1810 is greater than the number of input values of the core 1800, that is, the number of columns (M), the neural network device converts the input feature map 1810 into a sub-feature map 1811. It can be divided. The neural network device may divide the input feature map 1810 into a sub-feature map 1811 based on size information of the core 1800.

Specifically, when the size of the input feature map 1810 is 8x8 and the size of the core 1800 is 16x16, the neural network device divides the input feature map 1810 into 4 so that the number of pixel data in each of the sub-feature maps is 16. It can be divided into sub-feature maps. The neural network device can match each of the divided sub-feature maps to separate cores. For example, the neural network device may receive 'aa' of the sub-feature map 1810 as the first input value 'V1' of the core 1800, and receive 'ab' of the sub-feature map 1810 as the core ( It can be received as the second input value 'V2' of the sub-feature map 1810, and 'dd' of the sub-feature map 1810 can be received as the sixteenth input value 'V16' of the core 1800.

Meanwhile, as described above in FIG. 17, the pixel data of the sub-feature map 1810 may be digital signals (e.g., 1 bit, 4 bits, etc.), and the neural network device uses a DAC (Digital Analog Converter). After converting the pixel data into an analog signal, the converted value (voltage V) can be received as an input value of the core 1800.

Figure 19 is a diagram for explaining an example of a vector multiplication operation performed in a core according to an embodiment.

Figure 19 is a diagram for explaining an example of a vector multiplication operation performed in a core according to an embodiment. The sizes of the input feature map 1910, kernel 1920, sub-feature map 1930, and core 1900 shown in FIG. 19 are exemplary and are not limited to the sizes shown in FIG. 19.

Referring to FIG. 19, as in FIG. 18, the size of the input feature map 1910 is 8x8, and the size of the core 1900 is 16x16. Since the number of pixel data of the 8x8 input feature map 1910 is 64 (=8x8), it has a value greater than 16, the number of input values that can be received from the core 1900. The neural network device may divide the input feature map 1910 into four sub-feature maps so that the number of pixel data in each of the sub-feature maps is 16.

The kernel 1920 having a size of 2x2 slides on the input feature map 1910 in units of windows (or tiles) having a size of 2x2 pixels, and a convolution operation is performed between the kernel 1920 and the input feature map 1910. The convolution operation adds up all the values obtained by multiplying each pixel data of a window of the input feature map 1910 and the weight of each element at the corresponding position in the original kernel 1920, and calculates each pixel data of the output feature map. This refers to the operation to find .

Even when the input feature map 1910 is divided into sub-feature maps as shown in FIG. 19, a convolution operation is performed between the sub-feature map 1930 and the kernel 1920. Specifically, the kernel 1920 first performs a convolution operation with the first window 1911a of the sub-feature map 1930. That is, each pixel data (aa, ab, ba, and bb) of the first window 1911a is multiplied by the element weights (W1, W2, W3, and W4) of the kernel 1920, and the multiplied result values are all added up. As a result, pixel data of the output feature map is calculated. In the same way, the kernel 1920 may perform a convolution operation with the second window 1911b to the ninth window 1911c of the sub-feature map 1930. Hereinafter, the element weight of the kernel will be referred to as the kernel value.

The above-described convolution operation process can be performed in the core 1900 as follows.

The neural network device may receive 16 pixel data (aa, ab, ac, ... dd) included in the sub-feature map 1930 as an input to the core 1900 having a size of 16x16. In addition, the neural network device converts pixel data in the form of a digital signal into voltages (V1 to V16) in the form of an analog signal using a DAC (Digital Analog Converter), and then converts the voltages (V1 to V16) into the input of the core 1900. It can be received by . In one embodiment, when the core 1900 receives pixel data with 4-bit resolution as input, it may be desirable in terms of DAC power consumption.

Additionally, kernel values (W1, W2, W3, and W4) may be stored in the core 1900 of the neural network device. Here, the kernel values (W1, W2, W3, and W4) may be values that have been learned in the neural network. Specifically, the kernel values W1, W2, W3, and W4 may be stored in a synapse of the core 1900, that is, a memory cell. The kernel values (W1, W2, W3 and W4) may be stored in the memory cell as conductance values (G1, G2, G3 and G4), with each of the kernel values W1 to W4 corresponding to the conductance values G1 to G4 stored in the memory cell. do. Meanwhile, among the memory cells of the core 1900, the conductance values of memory cells in which conductance values (G1, G2, G3, and G4) are not stored may be '0'.

In one embodiment, the neural network device may initialize kernel values. The initialized kernel values may be stored as conductance values in the memory cell of the core 1900. Initialization methods include, but are not limited to, the Gaussian standard normal distribution method, Xavier initialization method, and He initialization method.

Additionally, the neural network device can divide the initialized kernel values by the square root of the number of divided sub-feature maps. In one embodiment, when the input feature map 1910 is divided into four sub-feature maps, the neural network device uses the initialized kernel values. It can be divided into The neural network device may store the kernel values on which the division operation was performed as conductance values in the memory cell of the core 1900. Prediction accuracy can be improved if the kernel values are initialized using the He initialization method, divided by the square root of the number of sub-feature maps, and the resulting value is used as the conductance value of the memory cell.

Hereinafter, for convenience of explanation, the horizontal row wires will be referred to as first row wires 1901a to 16th row wires 1901p in order from the top, and the column wires will be referred to as first column wires 1902a to 16th row wires in order from the left. It will be referred to as the 16th column wiring 1902p.

Looking at the first column wiring 1902a, the memory cells that intersect the first row wiring 1901a, the second row wiring 1901b, the fifth row wiring 1901e, and the sixth row wiring 1901f have a conductance value. G1, G2, G3 and G4 are stored respectively. In addition, the input values of the core 1900 corresponding to each of the first row wiring 1901a, the second row wiring 1901b, the fifth row wiring 1901e, and the sixth row wiring 1901f are V1, V2, and V5. and V6. Since Ohm's law is applied between voltage and conductance, as a result of the vector multiplication operation between the conductance value stored in the memory cell and the input voltage value of the core, the first output value I1 of the core 1900 can be calculated as Equation 1 below: there is.

Each of the kernel values W1 to W4 corresponds to conductance values G1 to G4 stored in the memory cell, and each of the input values V1, V2, V5, and V6 of the core 1900 corresponds to pixel data aa, ab, ba, and bb. That is, the first output value I1 of the core 1900 corresponds to the result of the convolution operation between the kernel 1920 and the first window 1911a.

Additionally, looking at the second column wiring 1902b, memory cells that intersect the second row wiring 1901b, the third row wiring 1901c, the sixth row wiring 1901f, and the seventh row wiring 1901e have Conductance values G1, G2, G3 and G4 are stored respectively. In the same manner as when calculating the first output value I1 of the core 1900, the second output value I2 of the core 1900 can be calculated as shown in Equation 2 below. The second output value I2 of the core 1900 corresponds to the result of the convolution operation between the kernel 1920 and the second window 1911b.

According to the above-described process, the neural network device calculates the first output value I1 to the sixteenth output value I16 by performing a vector multiplication operation between the input value of the core 1900 and the conductance value stored in the memory cell. You can.

Meanwhile, in FIG. 19, since there are 9 windows 1911a, 1911b, and 1911c where the kernel 1920 and the vector multiplication operation are performed, the conductance value is only displayed on the memory cells on the first to ninth column wires 1902a to 1902i. G1, G2, G3 and G4 can be stored. That is, the conductance values stored in the memory cells on the 10th column wiring 1902j to 16th column wiring 1902p may all be '0', and in this case, the 10th output value I10 to the 16th output value of the core 1900 I16 becomes 0.

Since the output values (I1 to I16) calculated by the core 1900 are analog signals, the neural network device can convert the output values (I1 to I16) into digital signals using an analog digital converter (ADC). Additionally, the neural network device can apply an activation function to the digital signal converted from the ADC and then use it to calculate the input value of another core.

Figure 20 is a diagram for explaining a method of combining output values calculated from a plurality of cores according to an embodiment.

Referring to FIG. 20, the size of the input feature map 2010 is 4x4, and the size of the first to fifth cores 2001 to 2005 is 4x4. Since the number of pixel data of the 4x4 input feature map 2010 is 16 (=4x4), it has a value greater than 4, the number of input values that can be received from the first to fifth cores 2001 to 2005. The neural network device may divide the input feature map 2010 into four sub-feature maps so that the number of pixel data in each of the first internal fourth sub-feature maps 2011 to 2014 is four.

Pixel data 'aa, ab, ba, bb' of the first sub-feature map 2011 may be received as an input to the first core 2001. The neural network device converts the pixel data 'aa, ab, ba, bb' in the form of a digital signal into a voltage in the form of an analog signal using a DAC (Digital Analog Converter), and then converts the voltage to the input of the first core (2001). It can be received by . In the same way, the neural network device can convert the pixel data of each of the second to fourth sub-feature maps (2012 to 2014) into an analog signal and receive it as an input of the second to fourth cores (2002 to 2004). .

Meanwhile, as described above with reference to FIG. 19 , initialized kernel values may be stored as conductance values in the memory cells of the first to fifth cores 2001 to 2005. In one embodiment, the He initialization method may be used as an initialization method for kernel values. The neural network device can initialize kernel values using the He initialization method, divide them by the square root of the number of sub-feature maps, and store the resulting value as the conductance value of the memory cell.

The neural network device performs a vector multiplication operation between the conductance value stored in the memory cells of the first to fourth cores 2001 to 2004 and the input voltage value. As a result of performing the vector multiplication operation, output values of the first to fourth cores 2001 to 2004 are calculated. At this time, since each of the first to fourth cores (2001 to 2004) receives pixel data of the first to fourth sub-feature maps (2011 to 2014) as input, the neural network device receives the pixel data of the first to fourth cores (2001 to 2004) as input. (2001 to 2004) The output values calculated from each can be merged. A neural network device can transmit synthesized output values as input values of a new core.

In one embodiment, the neural network device may synthesize output values of column wiring having the same order in each core among output values calculated from each of the first to fourth cores 2001 to 2004. For example, the neural network device includes the output value I1 of the first column wiring of the first core (2001), the output value I2 of the first column wiring of the second core (2002), and the output value I2 of the first column wiring of the third core (2003). The output value I3 of the column wiring and the output value I4 of the first column wiring of the fourth core (2004) can be combined. The neural network device may synthesize output values I1 to I4 and then transmit the synthesized output values as the input value V1 of the fifth core (2005).

Additionally, the neural network device may multiply each of the output values I1 to I4 calculated from the first to fourth cores 2001 to 2004 by the weight values W1 to W4, and then synthesize the output values multiplied by the weight values. That is, the input value V1 of the fifth core (2005) can be calculated as in Equation 3 below.

Here, the weight values W1 to W4 may be different from the kernel value and may be values determined through learning in a neural network. In one embodiment, the weight values W1 to W4 may be '1', but are not limited thereto.

In the same manner, the neural network device may synthesize the remaining output values calculated from the first to fourth cores 2001 to 2004 and then transmit the synthesized output values as input values of the fifth core 2005.

Meanwhile, since the output values calculated from the first to fourth cores (2001 to 2004) (or the result values of the calculated output values multiplied by the weight value) are in the form of analog signals (current values), the neural network device uses an ADC (Analog The output values can be converted into digital signals using a digital converter. Additionally, the neural network device can apply the ReLU function to the output values converted to digital signals from the ADC. The neural network device may synthesize output values to which an activation function is applied and then transmit the synthesized output values as input values of the fifth core (2005).

As described above, if the number of pixel data of the input feature map is greater than the number of input values of the core, the input feature map can be divided into sub-feature maps and each of the divided sub-feature maps can be matched to separate cores. there is. By dividing the input feature map as described above, the neural network device can reduce the consumption of DAC (Digital Analog Converter) power, ADC (Analog Digital Converter) power, and chip power, and further reduce chip area. there is.

Figure 21 is a flowchart explaining a method of implementing a neural network in a neural network device according to an embodiment.

Figure 21 is a flowchart explaining a method of implementing a neural network in a neural network device according to an embodiment. The method of implementing a neural network in a neural network device shown in FIG. 21 is related to the embodiments described in the previously described drawings, so even if the contents are omitted below, the contents described in the previous drawings are included in FIG. 21 It can also be applied to the method of .

Referring to FIG. 21, in step s2110, the neural network device may divide the input feature map into sub-feature maps based on size information of the core constituting the neural network device. The size of the input feature map and core can be expressed as a matrix, and the number of pixel data constituting the input feature map may be larger than the number of input values (number of columns) of the core. In this case, the neural network device may divide the input feature map into a plurality of sub-feature maps so that the number of pixel data of the divided sub-feature map is equal to or smaller than the number of input values of the core.

In step s2120, the neural network device may receive pixel data of the sub-feature map as an input value of the core.

Since the pixel data of the sub-feature map is a digital signal, in order to receive the pixel data as the input value of the core, the neural network device can convert the pixel data into an analog signal (voltage) using a DAC (Digital Analog Converter). Meanwhile, in one embodiment, the pixel data of the sub-feature map may be a digital signal with 4-bit resolution.

In step s2130, the neural network device may store kernel values to be applied to the sub-feature map in memory cells constituting the core.

Kernel values may be stored in a memory cell of the core, and the kernel values stored in the memory cell may be conductance. Kernel values may be values learned in a separate neural network, and when learning a separate neural network, a segmented sub-feature map may be used as input data instead of an input feature map.

Meanwhile, the neural network device can initialize the learned kernel values using the He initialization method, divide them by the square root of the number of sub-feature maps, and store the resulting value as the conductance value of the memory cell.

In step s2140, the neural network device may calculate the output value of the core by performing a vector multiplication operation between the input value and the kernel values stored in the memory cells.

Since Ohm's law applies between voltage and conductance, the neural network device calculates the output value (current) of the core by performing a vector multiplication operation between the kernel value (conductance) stored in the memory cell and the input value (voltage) of the core. You can.

In step s2150, the neural network device may merge output values calculated from the core corresponding to each divided sub-feature map.

The neural network device may synthesize the output values of the column wiring having the same order in each core among the output values calculated from each of the plurality of cores. Additionally, the neural network device may multiply each output value calculated from each of the plurality of cores by a waiter value and then synthesize the output values multiplied by the weight value. Here, the weight values W1 to W4 may be different from the kernel value.

In step s2160, the neural network device may transmit the synthesized output values as input values of a new core.

Since the output values calculated from the core (or the results obtained by multiplying the calculated output value by the weight value) are in the form of an analog signal (current), the neural network device converts the output values into digital signals using an ADC (Analog Digital Converter). can do. Additionally, the neural network device can apply the ReLU function to the output values converted to digital signals from the ADC. The neural network device can synthesize output values to which an activation function is applied and then transmit the synthesized output values as input values of a new core.

The neural network device may transmit the synthesized output values as input values of a new core and then perform steps s2120 to s2150.

Figure 22 is a block diagram illustrating a neural network device and memory according to an embodiment.

Referring to FIG. 22, the neural network device 100 may include a processor 120 and an on-chip memory (on-chip memory, 2210). In the neural network device 100 shown in FIG. 22, only components related to the present embodiments are shown. Accordingly, it is obvious to those skilled in the art that the neural network device 100 may further include other general-purpose components in addition to the components shown in FIG. 22.

The neural network device 100 will be installed in digital systems that require low-power neural network operation, such as smartphones, drones, tablet devices, AR (Augmented Reality) devices, IoT (Internet of Things) devices, self-driving cars, robotics, and medical devices. may, but is not limited to this.

The neural network device 100 may include a plurality of on-chip memories 2210, and each on-chip memory 2210 may be comprised of a plurality of cores. The core may include a plurality of presynaptic neurons, a plurality of postsynaptic neurons, and synapses that provide respective connections between the plurality of presynaptic neurons and the plurality of postsynaptic neurons, that is, memory cells. You can. In one embodiment, the core may be implemented with Resistive Crossbar Memory Arrays (RCA).

The external memory 2220 is hardware that stores various data processed by the neural network device 100. The external memory 2220 can store data processed by the neural network device 100 and data to be processed. Additionally, the external memory 2220 can store applications, drivers, etc. to be run by the neural network device 100. The external memory 2220 includes random access memory (RAM) such as dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and CD. -Can include ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

The processor 120 serves to control overall functions for driving the neural network device 100. For example, the processor 120 generally controls the neural network device 100 by executing programs stored in the on-chip memory 2210 within the neural network device 100. The processor 120 may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc. provided in the neural network device 100, but is not limited thereto. The processor 120 reads/writes various data from the external memory 2220 and executes the neural network device 100 using the read/written data.

The processor 120 may divide the input feature map into sub-feature maps based on the size information of the core, and receive pixel data of the divided sub-feature map as an input value of the core. The processor 120 can convert pixel data into an analog signal (voltage) using a digital analog converter (DAC).

The processor 120 may store kernel values to be applied to the sub-feature map in memory cells constituting the core. Kernel values may be stored in a memory cell of the core, and the kernel values stored in the memory cell may be conductance. Additionally, the processor 120 may calculate the output value of the core by performing a vector multiplication operation between the input value and kernel values stored in memory cells.

The processor 120 may merge output values calculated from cores corresponding to each divided sub-feature map. Specifically, the neural network device may multiply each output value calculated from each of the plurality of cores by a waiter value and then synthesize the output values multiplied by the weight value. Meanwhile, since the output values calculated from the core (or the resulting values obtained by multiplying the calculated output value by the weight value) are in the form of an analog signal (current), the processor 120 converts the output values into digital using an ADC (Analog Digital Converter). It can be converted into a signal. Additionally, the processor 120 may apply the ReLU function to output values converted into digital signals from the ADC.

The processor 120 may synthesize output values to which an activation function is applied and then transmit the synthesized output values as input values of a new core.

Devices according to the present embodiments include a processor, memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with an external device, and a user such as a touch panel, keys, buttons, etc. It may include an interface device, etc. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable codes or program instructions executable on the processor. Here, computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disk, hard disk, etc.) and optical read media (e.g., CD-ROM). ), DVD (Digital Versatile Disc), etc. The computer-readable recording medium is distributed among computer systems connected to a network, so that computer-readable code can be stored and executed in a distributed manner. The media may be readable by a computer, stored in memory, and executed by a processor.

The specific implementations described in this embodiment are examples and do not limit the technical scope in any way. For the sake of brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connections or connection members of lines between components shown in the drawings exemplify functional connections and/or physical or circuit connections, and in actual devices, various functional connections or physical connections may be replaced or added. Can be represented as connections, or circuit connections.

Claims (21)

In a neural network device,
a memory in which at least one program is stored; and
A processor configured to drive a neural network by executing the at least one program to perform an operation on input data including a first input and a second input,
The processor,
Receiving a plurality of data streams representing image data with different delay times from a plurality of input terminals,
Receiving the first input and the second input from the plurality of data streams,
A first result is obtained by performing an operation between the first input and a plurality of kernels, and the second input and the plurality of kernels are received at a time delayed by a first interval from the time the first input is received. A device that obtains a second result by performing an inter-kernel operation and obtains output data for the input data using the first result and the second result. According to claim 1,
The neural network device includes neuromorphic hardware,
The neuromorphic hardware is a device that performs CNN (Convolution Neural Network) mapping using the first input and the second input. According to claim 1,
The input data includes image data,
The first input includes data for a first region of the image data, and the second input includes data for a second region of the image data. According to claim 3,
The device wherein the first area and the second area partially overlap and are adjacent to each other. According to claim 4,
The processor is
determine whether the second input is a valid input;
When the second input is a valid input, a device that obtains the second result by performing an operation between the second input and the plurality of kernels. According to claim 5,
The processor is
A device for determining the second input as the valid input when the second input is pixel data constituting the second area. delete According to claim 1,
the first input is received during a first cycle,
The device wherein the second input is received during a second cycle delayed from the first cycle by the first interval. According to claim 1,
The processor is
A device for obtaining the first result by adding the operation results between the first input and the plurality of kernels, and obtaining the second result by adding the operation results between the second input and the plurality of kernels. According to claim 1,
The processor receives a third input included in the input data at a time delayed by a second interval from the time the second input is received,
Obtaining a third result by performing an operation between the third input and the plurality of kernels,
An apparatus for obtaining the output data using the first result, the second result, and the third result. In a method for a neural network device to perform an operation on input data including a first input and a second input,
Receiving a plurality of data streams representing image data with different delay times from a plurality of input terminals using a processor included in the neural network device;
receiving the first input and the second input from the plurality of data streams;
Obtaining a first result by performing an operation between the first input and a plurality of kernels;
Obtaining a second result by performing an operation between the second input received at a time delayed by a first interval from the time the first input is received and the plurality of kernels; and
A method comprising obtaining output data for the input data using the first result and the second result. According to claim 11,
The neural network device includes neuromorphic hardware,
A method in which the neuromorphic hardware performs CNN (Convolution Neural Network) mapping using the first input and the second input. According to claim 11,
The input data includes image data,
The method of claim 1, wherein the first input includes data for a first region of the image data, and the second input includes data for a second region of the image data. According to claim 13,
The first area and the second area partially overlap and are adjacent to each other. According to claim 14,
The step of obtaining the second result is
determining whether the second input is a valid input; and
When the second input is a valid input, performing an operation between the second input and the plurality of kernels to obtain the second result. According to claim 15,
The step of determining whether the second input is the valid input is
A method comprising determining the second input as the valid input when the second input is pixel data constituting the second area. delete According to claim 11,
the first input is received during a first cycle,
The method wherein the second input is received during a second cycle delayed from the first cycle by the first interval. According to claim 11,
The step of obtaining the first result includes obtaining the first result by adding operation results between the first input and the plurality of kernels,
In the step of obtaining the second result, the second result is obtained by adding operation results between the second input and the plurality of kernels. According to claim 11,
Using the processor, receiving a third input included in the input data at a time delayed by a second interval from the time the second input is received;
Obtaining a third result by performing an operation between the third input and the plurality of kernels; and
The method further includes obtaining the output data using the first result, the second result, and the third result. A computer program stored in a recording medium for implementing the method of any one of claims 11 to 16 and 18 to 20.

Images