KR20220149414A - Npu implemented for artificial neural networks to process fusion of heterogeneous data received from heterogeneous sensors - Google Patents

Abstract

According to one disclosure of the present invention, a neural processing unit (NPU) is provided. The NPU includes: a control unit configured to receive a machine code of a compiled fusion-artificial neural network; an input unit (or sensor) configured to receive a plurality of input signals corresponding to the fusion-artificial neural network; an array of processing elements (PEs) configured to perform a fusion-artificial neural network operation; a special function unit (SFU) configured to perform a special function of the fusion-artificial neural network operation; and an on-chip memory configured to store fusion-artificial neural network operation data. The control unit can include a scheduler configured to control the PE array, the SPU, and the on-chip memory so that all operation sequences of the fusion-artificial neural network are processed in a preset order according to artificial neural network data locality information included in the machine code.

Description

NPU IMPLEMENTED FOR ARTIFICIAL NEURAL NETWORKS TO PROCESS FUSION OF HETEROGENEOUS DATA RECEIVED FROM HETEROGENEOUS SENSORS}

The present disclosure relates to an artificial neural network.

Humans are equipped with intelligence capable of recognition, classification, inference, prediction, and control/decision making. Artificial intelligence (AI) refers to artificially mimicking human intelligence.

The human brain consists of numerous neurons called neurons, and each neuron is connected to hundreds to thousands of other neurons through connections called synapses. In order to imitate human intelligence, the modeling of the operating principle of biological neurons and the connection relationship between neurons is called an artificial neural network (ANN) model. In other words, an artificial neural network is a system in which nodes imitating neurons are connected in a layer structure.

These artificial neural network models are divided into 'single-layer neural network' and 'multi-layer neural network' according to the number of layers. A typical multilayer neural network consists of an input layer, a hidden layer, and an output layer. Here, (1) the input layer is a layer that receives external data, and the number of neurons in the input layer is the same as the number of input variables. (2) The hidden layer is located between the input layer and the output layer, receives a signal from the input layer, extracts characteristics, and transmits it to the output layer. (3) The output layer receives the signal from the hidden layer and outputs it to the outside. The input signal between neurons is multiplied by each connection strength with a value between 0 and 1, and then summed.

Meanwhile, in order to implement higher artificial intelligence, an increase in the number of hidden layers of an artificial neural network is called a deep neural network (DNN).

On the other hand, for autonomous driving of the vehicle, various sensors, for example, LiDAR (Light Detection And Ranging, LiDAR), radar (RADAR), camera, GPS, ultrasonic sensor, NPU, etc. may be mounted on the vehicle. Since the data provided from such various sensors is large in size, there is a disadvantage in that processing time is considerably long.

However, since a vast amount of data must be processed in near real time for autonomous driving, artificial neural networks are emerging as a solution recently.

However, implementing a dedicated artificial neural network for each of heterogeneous sensor data is very inefficient.

Therefore, the inventor of this patent has studied a neural processing unit (NPU) for effectively processing different data provided from heterogeneous sensors through a fusion neural network.

According to one disclosure herein, a neural processing unit (NPU) is provided. The NPU includes a compiled fusion-control unit configured to receive the machine code of the artificial neural network; an input unit (or sensor) configured to receive a plurality of input signals corresponding to the fusion-artificial neural network; an array of processing elements (PE) configured to perform the fusion-artificial neural network operation; a special function unit (SFU) configured to perform a special function of the fusion-artificial neural network operation; and an on-chip memory configured to store the fusion-artificial neural network computation data. The control unit is configured to: the processing element array, the special function unit, and the on-chip memory so that all operation orders of the fusion-artificial neural network are processed in a preset order according to the artificial neural network data locality information included in the machine-code It may include a scheduler configured to control the.

According to another disclosure of the present specification, a neural processing unit (NPU) is provided. The NPU includes: a fusion-control unit configured to receive the machine code of the artificial neural network; an array of processing elements (PE) configured to perform computation of the fusion-artificial neural network based on the machine code; and a special function unit (SFU) configured to receive the convolution operation value processed in the PE array and calculate a corresponding special function. The SFU may include a plurality of functional units. The SFU may selectively control at least some of the plurality of functional units according to the artificial neural network data locality information included in the machine-code.

According to another disclosure of the present specification, a system is provided. The system includes a control unit configured to receive machine code of a fusion-artificial neural network, an input unit configured to receive at least one input signal, an array of processing elements configured to perform a convolution operation, and an on-chip configured to store a result of the convolution operation. at least one neural processing unit (NPU) comprising a memory; and receive neural network data locality information of the fusion-artificial neural network capable of predicting successive memory operation requests of the at least one neural processing unit, and based on the neural network data locality information, the corresponding at least one and a memory controller, comprising a memory configured to pre-caching the next memory operation request to be requested by a neural processing unit of

By utilizing the NPU presented according to the disclosures of the present specification, it is possible to improve the performance of a fusion artificial neural network for processing different data provided from heterogeneous sensors.

According to the disclosures of the present specification, through a CONCATANATION operation and a SKIP-CONNECTION operation, the fusion artificial neural network can effectively process different data provided from heterogeneous sensors. To this end, the NPU includes a special function unit (SFU) in which a plurality of functional units are connected by a pipeline, and the plurality of functional units are selectively turned off, thereby reducing power consumption.

According to one disclosure of the present specification, by turning on and turning off the NIR light source, by detecting the near-infrared reflected light reflected from the signs having a retro-reflector characteristic through the near-infrared sensor, effectively detecting a road sign can do.

1 is a schematic conceptual diagram illustrating a neural processing unit according to the present disclosure.
2 is a schematic conceptual diagram illustrating one processing element of an array of processing elements that may be applied to the present disclosure.
3 is an exemplary view showing a modified example of the NPU 100 shown in FIG.
4 is a schematic conceptual diagram illustrating an exemplary artificial neural network model.
5 is a diagram for explaining the basic structure of a convolutional neural network.
6 is a diagram for explaining input data of a convolution layer and a kernel used for a convolution operation.
7 is a diagram for explaining an operation of a convolutional neural network that generates an activation map using a kernel.
8 is a general diagram illustrating the operation of a convolutional neural network in an easy to understand manner.
9A shows an example of an autonomous vehicle to which the disclosures of the present specification are applied.
9B shows an autonomous driving level determined by the International Association of Automobile Engineers.
10 is an exemplary diagram illustrating a fusion algorithm.
11A is an exemplary diagram illustrating an example of recognizing an object, and FIG. 11B is an exemplary diagram illustrating a structure of an SSD.
12A shows an example of an artificial neural network using a radar mounted on a vehicle.
12B shows an example of a fusion processing method utilizing a radar and a camera.
13 shows an example of a fusion artificial neural network using a lidar and a camera.
14 is an exemplary diagram illustrating late fusion, early fusion, and deep fusion.
15 is an exemplary diagram illustrating a system including an NPU architecture according to the first example.
16A is an exemplary diagram illustrating a model of an artificial neural network including skip-connection.
16B is an exemplary diagram illustrating data locality information of an artificial neural network including skip-connection.
17 is an exemplary diagram illustrating a system including an NPU architecture according to a second example.
18 is an exemplary diagram illustrating a system including an NPU architecture according to a third example.
19 is an exemplary diagram illustrating a system including an NPU architecture according to a fourth example.
FIG. 20 shows an example in which the fusion artificial neural network shown in FIG. 13 is divided into threads according to the fourth example shown in FIG. 19 .
21 is an exemplary diagram illustrating a system including an NPU architecture according to a fifth example.
22 is an exemplary diagram illustrating a first example of the pipeline structure of the SFU shown in FIG. 21 .
23A is an exemplary diagram illustrating a second example of the pipeline structure of the SFU shown in FIG. 21 .
23B is an exemplary diagram illustrating a third example of the pipeline structure of the SFU shown in FIG. 21 .
24 is an exemplary diagram illustrating a system including an NPU architecture according to a sixth example.
25 is an exemplary diagram illustrating an example of utilizing a plurality of NPUs according to the seventh example.
26 is an exemplary diagram illustrating an example of processing the fusion artificial neural network shown in FIG. 13 through a plurality of NPUs shown in FIG. 25 .
27A to 27C show examples of application of a fusion artificial neural network using a near-infrared sensor and a camera.
28 shows an example of utilizing a polarizer according to the eighth example.
29A and 29B are exemplary views showing the performance of the polarizer.

Specific structural or step-by-step descriptions for the embodiments according to the concept of the present disclosure disclosed in this specification or the application are merely illustrative for the purpose of describing the embodiments according to the concept of the present disclosure.

Embodiments according to the concept of the present disclosure may be embodied in various forms, and embodiments according to the concept of the present disclosure may be embodied in various forms, and should not be construed as being limited to the embodiments described in the present specification or application. .

Since the embodiment according to the concept of the present disclosure may have various changes and may have various forms, specific embodiments will be illustrated in the drawings and described in detail in the present specification or application. However, this is not intended to limit the embodiment according to the concept of the present disclosure with respect to the specific disclosure form, and should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present disclosure.

Terms such as first and/or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one element from another element, for example, without departing from the scope of rights according to the concept of the present disclosure, a first element may be named a second element, and similarly The second component may also be referred to as the first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Other expressions describing the relationship between elements, such as "between" and "immediately between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The terms used herein are used only to describe specific embodiments, and are not intended to limit the present disclosure. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the stated feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers. , it is to be understood that it does not preclude the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

In describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present disclosure pertains and are not directly related to the present disclosure will be omitted. This is to more clearly convey the gist of the present disclosure without obscuring the gist of the present disclosure by omitting unnecessary description.

<Definition of Terms>

Hereinafter, in order to help understanding of the disclosures presented in the present specification, terms used in the present specification will be briefly summarized.

NPU: An abbreviation of Neural Processing Unit (NPU), which may refer to a processor specialized for computation of an artificial neural network model separately from a central processing unit (CPU).

ANN: An abbreviation of artificial neural network. In order to imitate human intelligence, by mimicking how neurons in the human brain are connected through synapses, nodes are layered (Layer) It can mean a network connected by a structure.

For example, the artificial neural network model can be a model such as Bisenet, Shelfnet, Alexnet, Densenet, Efficientnet, EfficientDet, Googlenet, Mnasnet, Mobilenet, Resnet, Shufflenet, Squeezenet, VGG, Yolo, RNN, CNN, DBN, RBM, LSTM, etc. have. However, the present disclosure is not limited thereto, and a new artificial neural network model to operate in the NPU 100 has been continuously released.

Information on the structure of the artificial neural network: Information including information on the number of layers, the number of nodes in a layer, the value of each node, information on a calculation processing method, information on a weight matrix applied to each node, and the like.

Information on data locality of artificial neural network: Information including an artificial neural network and a data access request sequence for requesting a memory determined based on a structure of a neural processing unit that processes the artificial neural network.

DNN: An abbreviation of Deep Neural Network, which may mean that the number of hidden layers of the artificial neural network is increased in order to implement higher artificial intelligence.

CNN: Abbreviation for Convolutional Neural Network, a neural network that functions similar to image processing in the visual cortex of the human brain. Convolutional neural networks are known to be suitable for image processing, and are known to be easy to extract features from input data and to identify patterns of features.

Kernel: It may mean a weight matrix applied to CNN.

Out-of-Chip Memory: Memory size may be limited inside the NPU. Accordingly, a memory may be disposed outside the chip for storing large-capacity data. The off-chip memory may include one of memories such as ROM, SRAM, DRAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, and the like. The off-chip memory may consist of at least one memory unit. The off-chip memory may be configured as a homogeneous memory unit or a heterogeneous memory unit.

In-Chip Memory: The NPU may include in-Chip memory. In-chip memory may include volatile memory and/or non-volatile memory. For example, the chip-internal memory may include one of memories such as ROM, SRAM, DRAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, and the like. The in-chip memory may consist of at least one memory unit. The in-chip memory may be composed of a single (homogeneous) memory unit or a heterogeneous (heterogeneous) memory unit.

Hereinafter, the present disclosure will be described in detail by describing preferred embodiments of the present disclosure with reference to the accompanying drawings. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a schematic conceptual diagram illustrating a neural processing unit according to the present disclosure.

The neural processing unit (NPU) 100 shown in FIG. 1 is a processor specialized to perform an operation for an artificial neural network.

An artificial neural network refers to a network of artificial neurons that multiplies and adds weights when multiple inputs or stimuli are received, and transforms and transmits the value obtained by adding an additional deviation through an activation function. The artificial neural network trained in this way can be used to output inference results from input data.

The NPU 100 may be a semiconductor implemented as an electric/electronic circuit. The electric/electronic circuit may mean including a number of electronic devices (eg, a transistor, a capacitor). The NPU 100 may include a processing element (PE) array 110 , an NPU internal memory 120 , an NPU scheduler 130 , and an NPU interface 140 . Each of the processing element array 110 , the NPU internal memory 120 , the NPU scheduler 130 , and the NPU interface 140 may be a semiconductor circuit to which numerous transistors are connected. Accordingly, some of them may be difficult to identify and distinguish with the naked eye, and may be identified only by operation. For example, any circuit may operate as the processing element array 110 , or may operate as the NPU scheduler 130 . NPU scheduler 130 may be configured to perform the function of the control unit configured to control the artificial neural network reasoning operation of the NPU (100).

The NPU 100 is a processing element array 110, an NPU internal memory 120 configured to store an artificial neural network model that can be inferred from the processing element array 110, and data locality information or structure of the artificial neural network model. and an NPU scheduler 130 configured to control the processing element array 110 and the NPU internal memory 120 based on the information. Here, the artificial neural network model may include data locality information or information about the structure of the artificial neural network model. The artificial neural network model may refer to an AI recognition model trained to perform a specific reasoning function.

The processing element array 110 may perform an operation for an artificial neural network.

The NPU interface 140 may communicate with various components connected to the NPU 100 through a system bus, for example, a memory.

The NPU scheduler 130 is configured to control the operation of the processing element array 100 for the reasoning operation of the neural processing unit 100 and the read and write order of the NPU internal memory 120 .

The NPU scheduler 130 may be configured to control the processing element array 100 and the NPU internal memory 120 based on the data locality information or information about the structure of the artificial neural network model.

The NPU scheduler 130 may analyze the structure of an artificial neural network model to be operated in the processing element array 100 or may be provided with already analyzed information. For example, the data of the artificial neural network that the artificial neural network model may include is node data of each layer (ie, feature map), arrangement data of layers, locality information or information on structure, and nodes of each layer are connected. may include at least a portion of weight data (ie, weight kernel) of each connection network. The data of the artificial neural network may be stored in a memory provided inside the NPU scheduler 130 or in the NPU internal memory 120 .

The NPU scheduler 130 may schedule the operation sequence of the artificial neural network model to be performed by the NPU 100 based on data locality information or structure information of the artificial neural network model.

The NPU scheduler 130 may obtain a memory address value in which the feature map and weight data of the layer of the artificial neural network model are stored based on the data locality information or the structure information of the artificial neural network model. For example, the NPU scheduler 130 may obtain a memory address value in which the feature map and weight data of the layer of the artificial neural network model stored in the memory are stored. Therefore, the NPU scheduler 130 may bring the feature map and weight data of the layer of the artificial neural network model to be driven from the memory 200 and store it in the NPU internal memory 120 .

The feature map of each layer may have a corresponding memory address value.

Each weight data may have a corresponding respective memory address value.

The NPU scheduler 130 is based on information on the data locality information or structure of the artificial neural network model, for example, the arrangement data locality information of the layers of the artificial neural network of the artificial neural network model, or information on the structure of the processing element array 110. You can schedule the operation order of

Since the NPU scheduler 130 performs scheduling based on data locality information or structure information of an artificial neural network model, it may operate differently from a general CPU scheduling concept. Scheduling of a general CPU operates to achieve the best efficiency by considering fairness, efficiency, stability, and response time. That is, it is scheduled to perform the most processing within the same time in consideration of priority and operation time.

Conventional CPUs use an algorithm for scheduling tasks in consideration of data such as priority order of each processing and operation processing time.

Alternatively, the NPU scheduler 130 may control the NPU 100 in the processing order of the NPU 100 determined based on information on the data locality or structure of the artificial neural network model.

Further, the NPU scheduler 130 NPU in the processing order determined based on the information on the data locality information or structure of the artificial neural network model and/or the data locality information or the structure of the neural processing unit 100 to be used. (100) can be driven.

However, the present disclosure is not limited to information on data locality information or structure of the NPU 100 .

NPU scheduler 130 may be configured to store information about the data locality information or structure of the artificial neural network.

That is, the NPU scheduler 130 may determine the processing order even if only information on the data locality information or structure of the artificial neural network of the artificial neural network model is utilized at least.

Furthermore, the NPU scheduler 130 may determine the processing order of the NPU 100 in consideration of the information on the data locality information or structure of the artificial neural network model and the data locality information or information on the structure of the NPU 100 . In addition, processing optimization of the NPU 100 in the determined processing order is also possible.

The processing element array 110 refers to a configuration in which a plurality of processing elements PE1 to PE12 configured to calculate the feature map and weight data of the artificial neural network are disposed. Each processing element may include a multiply and accumulate (MAC) operator and/or an Arithmetic Logic Unit (ALU) operator. However, examples according to the present disclosure are not limited thereto.

Although a plurality of processing elements are illustrated in FIG. 1 by way of example, by replacing the MAC within one processing element, operators implemented as a plurality of multipliers and adder trees are arranged and configured in parallel. It is possible. In this case, the processing element array 110 may be referred to as at least one processing element including a plurality of operators.

The processing element array 110 is configured to include a plurality of processing elements PE1 to PE12. The plurality of processing elements PE1 to PE12 illustrated in FIG. 1 is merely an example for convenience of description, and the number of the plurality of processing elements PE1 to PE12 is not limited. The size or number of the processing element array 110 may be determined by the number of the plurality of processing elements PE1 to PE12 . The size of the processing element array 110 may be implemented in the form of an N x M matrix. where N and M are integers greater than zero. The processing element array 110 may include N x M processing elements. That is, there may be more than one processing element.

The size of the processing element array 110 may be designed in consideration of the characteristics of the artificial neural network model in which the NPU 100 operates.

The processing element array 110 is configured to perform functions such as addition, multiplication, and accumulation required for artificial neural network operation. In other words, the processing element array 110 may be configured to perform a multiplication and accumulation (MAC) operation.

Hereinafter, the first processing element PE1 of the processing element array 110 will be described as an example.

2 is a schematic conceptual diagram illustrating one processing element of an array of processing elements that may be applied to the present disclosure.

NPU 100 according to an example of the present disclosure is a processing element array 110, an NPU internal memory 120 configured to store an artificial neural network model that can be inferred from the processing element array 110, and data locality of the artificial neural network model an NPU scheduler 130 configured to control the processing element array 110 and the NPU internal memory 120 based on the information or information about the structure, wherein the processing element array 110 is configured to perform a MAC operation, The processing element array 110 may be configured to quantize and output a MAC operation result. However, examples of the present disclosure are not limited thereto.

The NPU internal memory 120 may store all or part of the artificial neural network model according to the memory size and the data size of the artificial neural network model.

The first processing element PE1 may include a multiplier 111 , an adder 112 , an accumulator 113 , and a bit quantization unit 114 . However, examples according to the present disclosure are not limited thereto, and the processing element array 110 may be modified in consideration of the computational characteristics of the artificial neural network.

The multiplier 111 multiplies the received (N)bit data and (M)bit data. The operation value of the multiplier 111 is output as (N+M)bit data.

The multiplier 111 may be configured to receive one variable and one constant input.

The accumulator 113 accumulates the operation value of the multiplier 111 and the operation value of the accumulator 113 by using the adder 112 as many times as (L) loops. Accordingly, the bit width of the data of the output unit and the input unit of the accumulator 113 may be output as (N+M+log2(L)) bits. where L is an integer greater than 0.

When the accumulation is finished, the accumulator 113 may receive an initialization signal (initialization reset) to initialize the data stored in the accumulator 113 to 0. However, examples according to the present disclosure are not limited thereto.

The bit quantization unit 114 may reduce the bit width of data output from the accumulator 113 . The bit quantization unit 114 may be controlled by the NPU scheduler 130 . The bit width of the quantized data may be output as (X) bits. where X is an integer greater than 0. According to the above configuration, the processing element array 110 is configured to perform a MAC operation, and the processing element array 110 has an effect of quantizing and outputting the MAC operation result. In particular, such quantization has the effect of further reducing power consumption as (L)loops increases. In addition, when power consumption is reduced, there is an effect of reducing heat generation. In particular, when the heat generation is reduced, there is an effect that can reduce the possibility of malfunction due to the high temperature of the NPU 100 .

The output data (X) bit of the bit quantization unit 114 may be node data of a next layer or input data of convolution. If the artificial neural network model has been quantized, the bit quantization unit 114 may be configured to receive quantized information from the artificial neural network model. However, the present invention is not limited thereto, and the NPU scheduler 130 may be configured to extract quantized information by analyzing the artificial neural network model. Therefore, the output data (X)bit may be converted into a quantized bit width to correspond to the quantized data size and output. The output data (X) bit of the bit quantization unit 114 may be stored in the NPU internal memory 120 with a quantized bit width.

The processing element array 110 of the NPU 100 according to an example of the present disclosure includes a multiplier 111 , an adder 112 , an accumulator 113 , and a bit quantization unit 114 .

3 is an exemplary view showing a modified example of the NPU 100 shown in FIG.

Since the NPU 100 shown in FIG. 3 is substantially the same as the processing unit 100 exemplarily shown in FIG. 1 , except for the processing element array 110 , the following is duplicated for convenience of description only. A description may be omitted.

The processing element array 110 exemplarily illustrated in FIG. 3 includes register files RF1 to RF12 corresponding to respective processing elements PE1 to PE12 in addition to the plurality of processing elements PE1 to PE12. may further include.

The plurality of processing elements PE1 to PE12 and the plurality of register files RF1 to RF12 illustrated in FIG. 3 are merely examples for convenience of description, and the plurality of processing elements PE1 to PE12 and the plurality of registers The number of files RF1 to RF12 is not limited.

The size or number of the processing element array 110 may be determined by the number of the plurality of processing elements PE1 to PE12 and the plurality of register files RF1 to RF12 . The size of the processing element array 110 and the plurality of register files RF1 to RF12 may be implemented in the form of an N×M matrix. where N and M are integers greater than zero.

The array size of the processing element array 110 may be designed in consideration of the characteristics of the artificial neural network model in which the NPU 100 operates. In other words, the memory size of the register file may be determined in consideration of the data size of the artificial neural network model to be operated, the required operating speed, the required power consumption, and the like.

The register files RF1 to RF12 of the NPU 100 are static memory units directly connected to the processing elements PE1 to PE12. The register files RF1 to RF12 may include, for example, flip-flops and/or latches. The register files RF1 to RF12 may be configured to store MAC operation values of the corresponding processing elements PE1 to PE12. The register files RF1 to RF12 may be configured to provide or receive the NPU internal memory 120 and weight data and/or node data.

It is also possible that the register files RF1 to RF12 are configured to perform the function of a temporary memory of the accumulator during MAC operation.

4 is a schematic conceptual diagram illustrating an exemplary artificial neural network model.

Hereinafter, the operation of the exemplary artificial neural network model 110-10 that can be operated in the NPU 100 will be described.

The exemplary artificial neural network model 110-10 of FIG. 4 may be an artificial neural network learned by the NPU 100 shown in FIG. 1 or FIG. 3 or by a separate machine learning device. The artificial neural network model may be an artificial neural network trained to perform various inference functions, such as object recognition and speech recognition.

The artificial neural network model 110 - 10 may be a deep neural network (DNN).

However, the artificial neural network model 110-10 according to examples of the present disclosure is not limited to a deep neural network.

For example, an artificial neural network model is trained to perform inference such as Object Detection, Object Segmentation, Image/Video Reconstruction, Image/Video Enhancement, Object Tracking, Event Recognition, Event Prediction, Anomaly Detection, Density Estimation, Event Search, Measurement, etc. It could be a model to be

For example, the artificial neural network model can be a model such as Bisenet, Shelfnet, Alexnet, Densenet, Efficientnet, EfficientDet, Googlenet, Mnasnet, Mobilenet, Resnet, Shufflenet, Squeezenet, VGG, Yolo, RNN, CNN, DBN, RBM, LSTM, etc. have. However, the present disclosure is not limited thereto, and new artificial neural network models to operate in the NPU are being continuously released.

However, the present disclosure is not limited to the above-described models. Also, the artificial neural network model 110-10 may be an ensemble model based on at least two different models.

The artificial neural network model 110 - 10 may be stored in the NPU internal memory 120 of the NPU 100 .

Hereinafter, with reference to FIG. 4 , it will be described that an inference inference process is performed by the NPU 100 in the exemplary artificial neural network model 110 - 10 .

The artificial neural network model 110-10 includes an input layer 110-11, a first connection network 110-12, a first hidden layer 110-13, a second connection network 110-14, and a second hidden layer ( 110-15), a third connection network 110-16, and an exemplary deep neural network model including an output layer 110-17. However, the present disclosure is not limited to the artificial neural network model shown in FIG. 4 . The first hidden layer 110-13 and the second hidden layer 110-15 may be referred to as a plurality of hidden layers.

The input layer 110 - 11 may include, for example, x1 and x2 input nodes. That is, the input layer 110-11 may include information on two input values. The NPU scheduler 130 shown in FIG. 1 or 3 sets a memory address in which information about an input value from the input layer 110-11 is stored in the NPU internal memory 120 shown in FIG. 1 or 3 . can

The first connection network 110-12 exemplarily provides information on six weight values for connecting each node of the input layer 110-11 to each node of the first hidden layer 110-13. may include The NPU scheduler 130 shown in FIG. 1 or 3 may set a memory address in which information about the weight value of the first connection network 110 - 12 is stored in the NPU internal memory 120 . Each weight value is multiplied by an input node value, and an accumulated value of the multiplied values is stored in the first hidden layer 110 - 13 . Here, the nodes may be referred to as a feature map.

The first hidden layer 110-13 may include nodes a1, a2, and a3 for example. That is, the first hidden layer 110-13 may include information about three node values. The NPU scheduler 130 shown in FIG. 1 or FIG. 3 may set a memory address for storing information about the node value of the first hidden layer 110-13 in the NPU internal memory 120 .

The NPU scheduler 130 may be configured to schedule the operation order so that the first processing element PE1 performs the MAC operation of the a1 node of the first hidden layer 110-13. The NPU scheduler 130 may be configured to schedule an operation order so that the second processing element PE2 performs the MAC operation of the a2 node of the first hidden layer 110-13. The NPU scheduler 130 may be configured to schedule an operation order so that the third processing element PE3 performs the MAC operation of the a3 node of the first hidden layer 110-13. Here, the NPU scheduler 130 may pre-schedule an operation order so that the three processing elements simultaneously perform MAC operations in parallel.

The second connection network 110-14 exemplarily relates to nine weight values for connecting each node of the first hidden layer 110-13 to each node of the second hidden layer 110-15. may contain information. The NPU scheduler 130 shown in FIG. 1 or 3 may set a memory address for storing information on the weight value of the second connection network 110-14 in the NPU internal memory 120 . The weight value of the second connection network 110-14 is multiplied by the node value input from the first hidden layer 110-13, and the accumulated value of the multiplied values is transmitted to the second hidden layer 110-15. is saved

The second hidden layer 110 - 15 may include nodes b1, b2, and b3 for example. That is, the second hidden layer 110 - 15 may include information about three node values. The NPU scheduler 130 may set a memory address for storing information about the node value of the second hidden layer 110 - 15 in the NPU internal memory 120 .

The NPU scheduler 130 may be configured to schedule the operation order so that the fourth processing element PE4 performs the MAC operation of the b1 node of the second hidden layer 110-15. The NPU scheduler 130 may be configured to schedule an operation order so that the fifth processing element PE5 performs the MAC operation of the b2 node of the second hidden layer 110-15. The NPU scheduler 130 may be configured to schedule the operation order so that the sixth processing element PE6 performs the MAC operation of the b3 node of the second hidden layer 110-15.

Here, the NPU scheduler 130 may pre-schedule an operation order so that the three processing elements simultaneously perform MAC operations in parallel.

Here, the NPU scheduler 130 may determine the scheduling so that the operation of the second hidden layer 110-15 is performed after the MAC operation of the first hidden layer 110-13 of the artificial neural network model.

That is, the NPU scheduler 130 may be configured to control the processing element array 100 and the NPU internal memory 120 based on the data locality information or structure information of the artificial neural network model.

The third connection network 110-16 includes, for example, information on six weight values connecting each node of the second hidden layer 110-15 and each node of the output layer 110-17. can do. The NPU scheduler 130 may set a memory address for storing information about the weight value of the third connection network 110 - 16 in the NPU internal memory 120 . The weight value of the third connection network 110 - 16 is multiplied by the node value input from the second hidden layer 110 - 15 , and the accumulated value of the multiplied values is stored in the output layer 110 - 17 .

The output layer 110-17 may include, for example, y1 and y2 nodes. That is, the output layer 110-17 may include information about two node values. The NPU scheduler 130 may set a memory address in the NPU internal memory 120 to store information about the node value of the output layer 110-17.

The NPU scheduler 130 may be configured to schedule the operation order so that the seventh processing element PE7 performs the MAC operation of the y1 node of the output layer 110-17. The NPU scheduler 130 may be configured to schedule the operation order so that the eighth processing element PE8 performs the MAC operation of the y2 node of the output layer 110-15.

Here, the NPU scheduler 130 may pre-schedule an operation order so that the two processing elements simultaneously perform MAC operations in parallel.

Here, the NPU scheduler 130 may determine the scheduling so that the operation of the output layer 110-17 is performed after the MAC operation of the second hidden layer 110-15 of the artificial neural network model.

That is, the NPU scheduler 130 may be configured to control the processing element array 100 and the NPU internal memory 120 based on the data locality information or structure information of the artificial neural network model.

That is, the NPU scheduler 130 may analyze the structure of the artificial neural network model to operate in the processing element array 100 or may receive the analyzed information. The artificial neural network information that can be included in the artificial neural network model includes information on node values of each layer, information on locality information or structure of arrangement data of layers, and weight values of each connection network connecting nodes of each layer. may contain information.

Since the NPU scheduler 130 has been provided with information on the data locality information or structure of the exemplary neural network model 110-10, the NPU scheduler 130 from the input to the output of the neural network model 110-10. You can figure out the order of operations.

Accordingly, the NPU scheduler 130 may set the memory address in which the MAC operation values of each layer are stored in the NPU internal memory 120 in consideration of the scheduling order.

The NPU internal memory 120 may be configured to preserve the weight data of the connections stored in the NPU internal memory 120 while the reasoning operation of the NPU 100 is continued. Accordingly, there is an effect of reducing memory read/write operations.

That is, the NPU internal memory 120 may be configured to reuse the MAC operation value stored in the NPU internal memory 120 while the speculation operation is continued.

5 is a diagram for explaining the basic structure of a convolutional neural network.

Referring to FIG. 5 , a convolutional neural network may be a combination of one or several convolutional layers, a pooling layer, and fully connected layers. The convolutional neural network has a structure suitable for learning and inference of two-dimensional data, and can be learned through a backpropagation algorithm.

In the example of the present disclosure, in the convolutional neural network, there is a kernel for extracting the features of the input image of the channel for each channel. The kernel may be composed of a two-dimensional matrix, and convolution operation is performed while traversing input data. The size of the kernel may be arbitrarily determined, and the stride at which the kernel traverses the input data may also be arbitrarily determined. A result of convolution of all input data per one kernel may be referred to as a feature map or an activation map. Hereinafter, the kernel may include a set of weight values or a plurality of sets of weight values. The number of kernels for each layer may be referred to as the number of channels.

As such, since the convolution operation is an operation formed by combining input data and a kernel, an activation function for adding non-linearity may be applied thereafter. When an activation function is applied to a feature map that is a result of a convolution operation, it may be referred to as an activation map.

Specifically, referring to FIG. 5 , the convolutional neural network includes at least one convolutional layer, at least one pooling layer, and at least one fully connected layer.

For example, convolution (convolution) is defined by two main parameters: the size of the input data (typically a 1Х1, 3Х3 or 5Х5 matrix) and the depth of the output feature map (the number of kernels). can be defined. These key parameters can be computed by convolution. These convolutions may start at depth 32, continue to depth 64, and end at depth 128 or 256. The convolution operation may refer to an operation of sliding a kernel having a size of 3Х3 or 5Х5 over an input image matrix that is input data, multiplying each element of the input image matrix overlapping with each weight of the kernel, and then adding them all.

An activation function may be applied to the output feature map generated as described above to finally output the activation map. Also, the weight used in the current layer may be transmitted to the next layer through convolution. The pooling layer may perform a pooling operation to reduce the size of the feature map by downsampling the output data (ie, the activation map). For example, the pooling operation may include, but is not limited to, max pooling and/or average pooling.

The maximum pooling operation uses the kernel, and outputs the maximum value in the area of the feature map overlapping the kernel by sliding the feature map and the kernel. The average pooling operation outputs the average value within the area of the feature map overlapping the kernel by sliding the feature map and the kernel. As such, since the size of the feature map is reduced by the pooling operation, the number of weights of the feature map is also reduced.

The fully connected layer may classify data output through the pooling layer into a plurality of classes (ie, estimated values), and output the classified class and a score thereof. Data output through the pooling layer forms a three-dimensional feature map, and this three-dimensional feature map can be converted into a one-dimensional vector and input as a fully connected layer.

6 is a diagram for explaining input data of a convolution layer and a kernel used for a convolution operation.

The input data 300 may be an image or an image displayed as a two-dimensional matrix composed of a row 310 of a specific size and a column 320 of a specific size. The input data 300 may be referred to as a feature map. The input data 300 may have a plurality of channels 330 , where the channel 330 may represent a color RGB channel of the input data image.

Meanwhile, the kernel 340 may be a weight parameter used for convolution for extracting features of a certain portion of the input data 300 while scanning it. Like the input data image, the kernel 340 may be configured to have a specific size of a row 350 , a specific size of a column 360 , and a specific number of channels 370 . In general, the size of the row 350 and the column 360 of the kernel 340 is set to be the same, and the number of channels 370 may be the same as the number of channels 330 of the input data image.

7 is a diagram for explaining an operation of a convolutional neural network that generates a feature map using a kernel.

The kernel 410 may finally generate the feature map 430 by traversing the input data 420 at specified intervals and performing convolution. Convolution, when the kernel 410 is applied to a part of the input data 420, multiplies the input data values of a specific position of the part and the values of the corresponding position of the kernel 410, and then adds all the generated values. can be executed

Through this convolution process, calculated values of the feature map are generated, and whenever the kernel 410 traverses the input data 420 , the result values of the convolution are generated to configure the feature map 430 . .

Each component value of the feature map may be converted into the activation map 430 through the activation function of the convolution layer.

In FIG. 7 , the input data 420 input to the convolution layer is represented by a two-dimensional matrix having a size of 4×4, and the kernel 410 is represented by a two-dimensional matrix having a size of 3×3. However, the sizes of the input data 420 and the kernel 410 of the convolution layer are not limited thereto, and may be variously changed according to the performance and requirements of the convolutional neural network including the convolution layer. have.

As shown, when the input data 420 is input to the convolution layer, the kernel 410 traverses the input data 420 at a predetermined interval (eg, stride = 1), and the input data 420 ) and the values of the same location of the kernel 410 are multiplied, respectively, and the MAC operation of adding the respective values may be performed.

Specifically, the kernel 410 assigns the MAC operation value "15" calculated at a specific location 421 of the input data 420 to the corresponding element 431 of the feature map 430 . The kernel 410 assigns the MAC operation value "16" calculated at the next position 422 of the input data 420 to the corresponding element 432 of the feature map 430 . The kernel 410 assigns the MAC operation value "6" calculated at the next position 423 of the input data 420 to the corresponding element 433 of the feature map 430 . Next, the kernel 410 assigns the MAC operation value "15" calculated at the next position 424 of the input data 420 to the corresponding element 434 of the feature map 430 .

In this way, when the kernel 410 allocates all MAC operation values calculated while traversing the input data 420 to the feature map 430 , the feature map 430 having a size of 2 x 2 can be completed.

At this time, if the input data 510 is composed of, for example, three channels (R channel, G channel, and B channel), the same kernel or different channels for each channel are traversed over the data phase for each channel of the input data 420 , respectively. In addition, it is possible to generate a feature map for each channel through convolution with multiple multiplication and summation.

For the MAC operation, the scheduler 130 allocates processing elements PE1 to PE12 to perform each MAC operation based on a preset operation order, and determines a memory address in which MAC operation values are stored in consideration of the scheduling order. It can be set in the NPU internal memory 120 .

8 is a general diagram illustrating the operation of a convolutional neural network in an easy to understand manner.

Referring to FIG. 8 , for example, an input image is shown as a two-dimensional matrix having a size of 5×5. In addition, FIG. 9 illustrates that three nodes, ie, channel 1, channel 2, and channel 3, are used by way of example.

First, the convolution operation of layer 1 will be described.

The input image is convolutioned with kernel 1 for channel 1 at the first node of layer 1, and as a result, feature map 1 is output. Also, the input image is convolutioned with kernel 2 for channel 2 at the second node of layer 1, and as a result, feature map 2 is output. In addition, the input image is convolutioned with kernel 3 for channel 3 at the third node, and as a result, feature map 3 is output.

Next, the layer 2 polling operation will be described.

The feature map 1, the feature map 2, and the feature map 3 output from the layer 1 are input to the three nodes of the layer 2. Layer 2 may receive feature maps output from layer 1 as input and perform polling. The polling may reduce the size or emphasize a specific value in a matrix. Polling methods include maximum polling, average polling, and minimum polling. Maximum polling is used to collect the maximum values of values within a specific region of a matrix, and average polling can be used to find the average within a specific region of a matrix.

In order to process each convolution, the processing elements PE1 to PE12 of the NPU 100 are configured to perform a MAC operation.

In the example of FIG. 8 , the size of the feature map of a 5×5 matrix is reduced to a 4×4 matrix by polling.

Specifically, the first node of the layer 2 receives the feature map 1 for channel 1 as an input, performs polling, and outputs it as, for example, a 4x4 matrix. The second node of the layer 2 receives the feature map 2 for the channel 2 as an input, performs polling, and outputs, for example, a 4x4 matrix. The third node of layer 2 receives the feature map 3 for channel 3 as an input, performs polling, and outputs, for example, a 4x4 matrix.

Next, the layer 3 convolution operation will be described.

The first node of layer 3 receives the output from the first node of layer 2 as input, performs convolution with kernel 4, and outputs the result. The second node of layer 3 receives the output from the second node of layer 2 as input, performs convolution with kernel 5 for channel 2, and outputs the result. Similarly, the third node of layer 3 receives the output from the third node of layer 2 as input, performs convolution with kernel 6 for channel 3, and outputs the result.

In this way, convolution and polling are repeated, and finally, as shown in FIG. 7 , it may be input to a fully connected layer.

The aforementioned CNN is also widely used in the field of autonomous driving.

9A shows an example of an autonomous driving vehicle to which the disclosures of the present specification are applied, and FIG. 9B shows an autonomous driving level determined by the International Association of Automobile Engineers.

Referring to FIG. 9A , an autonomous vehicle (vehicle) may be equipped with a light detection and ranging (LiDAR), a radar (RADAR), a camera, a GPS, an ultrasonic sensor, an NPU, and the like.

The inventor of this patent studied an NPU that can assist autonomous driving by using deep learning.

For autonomous driving, NPUs must satisfy four key technical requirements.

1. Perception: NPUs should be able to use sensors to sense, understand and interpret their surroundings, including static and dynamic obstacles such as other vehicles, pedestrians, road signs, traffic signals, and road curbs.

2. Localization & Mapping: The NPU should be able to locate a vehicle, create a map around the vehicle, and continuously track the location of the vehicle with respect to that map.

3. Path planning: The NPU should be able to utilize the output of the previous two tasks to adopt the optimal, safe and feasible path for the vehicle to reach its destination, taking into account obstacles in the road.

4. Control: Based on the NPU selected path, the control element should be able to output the acceleration, torque and steering angle values required for the vehicle to follow the selected path.

Meanwhile, autonomous driving technology requires an advanced driver assistance system (ADAS) and/or a driver's status monitoring (DSM). ADAS & DSM includes the following technologies, etc.

- Smart Cruise Control (SCC)

- Autonomous Emergency Braking (AEB)

- Smart Parking Assistance System (SPAS)

- Lane Departure Warning System (LDWS)

- Lane Keeping Assist System (LKAS)

- Drowsiness detection, alcohol detection, heat and cold detection, carelessness detection, infant neglect detection, etc.

Various sensors are used in the ADAS technology, and the corresponding sensors can be used as input signals for deep learning.

- RGB CAMERA SENSOR (380nm~680nm)

- RGB CAMERA with Polarizer

- DEPTH CAMERA SENSOR

- NIR CAMERA SENSOR (850nm~940nm)

- THERMAL CAMERA SENSOR (9,000nm-14,000nm)

- RGB+IR HYBRID SENSOR (380nm~940nm)

- RADAR SENSOR

- LIDAR SENSOR

- ULTRASOUND SENSOR

Meanwhile, with reference to FIG. 9B , each level will be described based on the autonomous driving level determined by the International Association of Automobile Engineers.

In the non-automation stage, which is stage 0, the system simply warns and temporarily intervenes for safety while driving for manually driven vehicles that do not provide V2X (Vehicle to everything) communication function (Forward Collision-Avoidance Assist, FCA). ) and Blind-Spot Collision Warning (BCW). Therefore, in step 0, the driver must perform all vehicle control.

In the first stage, the driver assistance stage, a manually driven vehicle in which the system performs either steering or deceleration/acceleration in a specific driving mode, Lane Following Assist (LFA), and Smart Cruise Control (SCC) etc. are provided. Therefore, in step 1, the driver must be aware of the speed and the like.

In the second stage, the partial automation stage, an autonomous vehicle in which the system performs both steering and deceleration/acceleration in a specific driving mode provides Highway Driving Assist (HDA). Therefore, in step 2, the driver must be aware of the object or the like.

Up to stage 2, the system assists with some driving of the vehicle (assist), but from stage 3 onwards the system can perform the entire driving (pilot). That is, the vehicle can change lanes on its own or overtake the vehicle in front and avoid obstacles.

The third stage, the conditional automation stage, controls the vehicle and recognizes the driving environment at the same time, but may require the driver to transfer driving control in case of an emergency. Accordingly, in step 3, the driver must be aware of a specific road condition or the like.

The advanced automation stage, which is stage 4, means the stage in which the system performs the entire driving as in stage 3 and can respond safely even in the event of a dangerous situation. Therefore, in step 4, the driver must be aware of the weather, disasters, and accidents.

The 5th stage, the fully automated stage, means a stage in which there are no restrictions on the areas where autonomous driving can be performed, unlike the 4th stage. In step 5, driver recognition is unnecessary.

<Processing of different data signals from heterogeneous sensors>

In order to improve autonomous driving performance, the need for a fusion algorithm to process different data provided from heterogeneous sensors is emerging. Hereinafter, fusion algorithms are introduced.

10 is an exemplary diagram illustrating a fusion algorithm.

As shown in FIG. 10 , a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN) may be used for example to process different data provided from heterogeneous sensors. CNN can be used to detect an object in an image, and RNN can be used to predict an object using the concept of time. In addition, R-CCN (Region-based CNN), SPP-Net (Spatial Pyramid Pooling network), YOLO (You only look once), SSD (Single-Shot Multibox Detector), DSSD (Deconvolutional Single-Shot Multibox Detector), LTSM (Long-Short Term Memory), GRU (Gated Recurrent Unit), etc. may be used.

11A is an exemplary diagram illustrating an example of recognizing an object, and FIG. 11B is an exemplary diagram illustrating a structure of an SSD.

As can be seen with reference to FIG. 11A , a plurality of objects can be detected in an image by using the SSD artificial neural network model. Referring to FIG. 11B , the SSD may detect an object in the feature map for each step. For example, the SSD may be combined with a backbone of a VGG structure or a Mobilenet structure.

12A shows an example of an artificial neural network using a radar mounted on a vehicle, and FIG. 12B shows an example of a fusion processing method using a radar and a camera.

In order to process the signal provided from the radar, the artificial neural network shown in FIG. 12A may include convolution, pooling, ResNet, and the like.

In order to process the signal provided from the radar and the RGB signal provided from the camera, the fusion artificial neural network shown in FIG. 12B may be used.

13 shows an example of a fusion artificial neural network using a lidar and a camera.

Referring to FIG. 13 , an example of processing an RGB signal provided from a camera and a signal provided from a lidar through parallel processing is shown. During parallel processing, different information can be exchanged through transformers. The method may be the deep fusion method shown in FIG. 14 .

Meanwhile, although not shown, in order to process different data provided from heterogeneous sensors, the artificial neural network may include a concatenation operation and a skip-connection operation. The concatenation operation means merging output results of a specific layer with each other, and the skip and concatenation operation means skipping the output result of a specific layer and transferring the output result of a specific layer to another layer.

Such a concatenating operation and skipping and connecting operation may increase the control difficulty and usage of the internal memory 120 of the NPU 100 .

Up to now, an artificial neural network for fusion processing of different data provided from heterogeneous sensors has been described, but only the above-described contents have a weakness in that the performance of the artificial neural network cannot be improved. Therefore, the optimized artificial neural network and NPU structure will be described below.

<Fusion artificial neural network and NPU structure optimized to process different data from heterogeneous sensors>

First, the inventor of this patent studied NPU for processing different data from heterogeneous sensors.

In the design of the NPU, the following configuration should be considered:

i. It is necessary to have an NPU structure suitable for processing heterogeneous data signals (eg, RGB camera + radar).

ii. NPU memory control suitable for heterogeneous input signal processing (eg RGB camera + radar) is required.

iii. It is necessary to have an NPU structure suitable for multiple input channels (ADAS and DSM).

iv. NPU memory control suitable for multiple input channels (ADAS & DSM) is required.

v. It is necessary to have an NPU structure suitable for fusion artificial neural network model computation.

vi. For real-time application, a fast processing speed of 16ms or less is required.

vii. It is necessary to achieve low power consumption for battery operation.

An NPU to implement a fusion artificial neural network must support the following functions. The expected requirements are:

i. Support CNN function: It should be able to control the PE array and memory optimized for convolution.

ii. It should be able to process depthwise-separable convolution efficiently. It should have a structure that improves PE utilization and performance (throughput).

iii. Batch mode function support: Memory configuration is required to simultaneously process multiple channels (cameras 1 to 6) and heterogeneous sensors. (The size of PE array and memory size must be in an appropriate ratio)

iv. Concatenation function support: An NPU for a fusion artificial neural network must be able to process heterogeneous input data signals with a concatenation function.

v. Skip and connect (Skip) connection function support: The NPU for a fusion artificial neural network may include a SFU (Special Function Unit) that can provide a skip function.

vi. Support for deep learning image preprocessing function: An NPU for a fusion artificial neural network should be able to provide a function to preprocess different data signals.

vii. A compiler capable of efficiently compiling fusion neural networks should be provided.

The inventor of this patent proposes an NPU having the following characteristics.

i. The NPU may include a compiler that analyzes ANN data locality information of an artificial neural network, such as late fusion, early fusion, and deep fusion.

ii. The NPU may be configured to control the PE array to process heterogeneous sensor data based on an artificial neural network data locality controller (ADC). That is, the fused artificial neural network is fused into various structures depending on the sensor, and the PE utilization rate can be improved by providing the NPU 100 corresponding to the structure.

iii. It may be configured to appropriately set the size of the chip-internal memory 120 to process heterogeneous sensor data based on the ANN data locality information. That is, the memory bandwidth of the NPU that processes the fusion artificial neural network can be improved by analyzing the locality information of the ANN data.

iv. NPU can include SFU (Special Function Unit) that can efficiently process bilinear interpolation, concatenation, and skip-connection (skip-connection, etc.) required in fusion artificial neural networks. have.

14 is an exemplary diagram illustrating late fusion, early fusion, and deep fusion.

Referring to FIG. 14 , F" means a fusion operation, and each block means each layer. As can be seen with reference to FIG. 14 , in the late fusion operation, after performing an operation for each layer, the last It means fusion of the operation result in the process, Early fusion means fusion of different data early and then performing the operation for each layer Deep fusion means fusion of different data and then operation on different layers , and after fusion of the operation result again, it means that the operation is performed for each layer.

15 is an exemplary diagram illustrating a system including an NPU architecture according to the first example.

15, the NPU 100 is a PE array 110 for a fusion artificial neural network, and a chip-internal (On-chip) memory 120, an NPU scheduler 130 and, It may include a special function unit (SFU) 160 . In the description of FIG. 15 , overlapping descriptions may be omitted for convenience of description only.

The PE array 110 for the fusion neural network may refer to the PE array 110 configured to process the convolution of a multi-layered neural network model having at least one fusion layer. That is, the fusion layer may be configured to output a feature map in which data of heterogeneous sensors is fused. In more detail, the SFU 160 of the NPU 100 may be configured to receive multiple sensors and provide a function of fusion of each sensor input. The PE array 110 for the fusion artificial neural network may be configured to receive fusion data from the SFU 160 and process convolution.

The NPU 100 may receive different data from the M heterogeneous sensors 311 and 312 . The heterogeneous sensors may include a camera, radar, lidar, ultrasound, thermal imaging camera, and the like.

The NPU 100 may obtain fusion artificial neural network (ANN) data locality information from the compiler 200 .

At least one layer of the fusion artificial neural network may be a layer in which input data of a plurality of sensors are fused.

The NPU 100 may be configured to provide a concatenation function to at least one layer for fusion of heterogeneous sensor input data. In order to connect each feature map of the heterogeneous sensors of the concatenated layer to each other, the size of at least one axis may be processed to be the same. For example, in order to concatenate the heterogeneous sensor data along the X-axis, the size of the X-axis of each of the heterogeneous sensor data may be the same. For example, in order to concatenate heterogeneous sensor data along the Y-axis, the Y-axis size of each of the heterogeneous sensor data may be the same. For example, in order to concatenate heterogeneous sensor data along the Z-axis, the Z-axis sizes of the different types of sensor data may be the same.

In order to receive and process different data from the heterogeneous sensors 311 and 312 , the NPU scheduler 130 may process inference of a fusion artificial neural network model.

The NPU scheduler 130 may be included in the control unit as shown.

The NPU scheduler 130 obtains and analyzes data locality information of the fusion artificial neural network from the compiler 200 , and the chip-can control the operation of the internal memory 120 .

Specifically, it is as follows. The compiler 200 may generate data locality information of a fusion artificial neural network to be processed by the NPU 100 .

The NPU scheduler 130 may generate a list for a special function (special function) required for the fusion (fusion) artificial neural network. The special function may mean various functions required for artificial neural network operation other than convolution.

If the fusion artificial neural network data locality information is well utilized, fusion such as non-maximum suppression (NMS), skip and connect (SKIP-CONNECTION), bottleneck, and bilinear interpolation It is possible to efficiently control the problem of increasing memory access, which frequently occurs in artificial neural networks.

If the fusion artificial neural network data locality information is used, data to be stored (eg, the first Since the size, storage period, etc. of the output feature map) can be known at the compilation stage, it is possible to efficiently set the memory map for the on-chip memory 120 in advance.

The SFU 160 may perform skip-connection and concatenation necessary for a fusion artificial neural network. In other words, concatenation can be utilized to fuse heterogeneous sensor data. For concatenation, the size of each sensor data can be readjusted. For example, the NPU 100 may be configured to handle the concatenation of the fusion artificial artificial network by providing functions such as resizing, interpolation, and the like.

The chip-internal memory 120 of the NPU 100 may selectively preserve specific data according to the PE array 110 or the SFU 160 for a specific period based on the artificial neural network data locality information. Whether or not to preserve the selective preservation may be controlled by the controller 130 .

Also, the PE array 110 may be configured to have the number of threads corresponding to the number of heterogeneous sensors. That is, the array 110 of the NPU 100 configured to receive two sensor data may be configured to have two threads. That is, if one thread is configured with N×M processing elements, two threads may be configured with N×M×2 processing elements. For example, each thread of the PE array 110 may be configured to process a feature map of each heterogeneous sensor.

The NPU 100 may output the operation result of the fusion artificial neural network through an output unit.

The NPU architecture according to the first example described above may be variously modified.

16A is an exemplary diagram illustrating a model of an artificial neural network including skip-connection, and FIG. 16B is a data locality information of an artificial neural network including skip-connection. It is an example diagram shown.

As shown in FIG. 16A , in order to calculate five layers including a Skip-connection operation, for example, as shown in FIG. 16B , the compiler 200 executes a sequence of 16 steps. It is possible to generate artificial neural network data locality information having

The NPU 100 requests a data operation to the on-chip memory 120 in the order of the artificial neural network data locality information.

In the case of a skip-connection operation, the output feature map OFMAP of the first layer may be added to the output feature map OFMAP of the fourth layer.

For such a skip-connection operation, the output feature map of the first layer must be preserved until the fifth layer operation. However, other data may be deleted after operation in order to utilize memory space.

In the deleted memory area, data to be calculated later based on the sequence of locality information of the artificial neural network data may be stored. Accordingly, since necessary data can be sequentially brought into the chip-internal memory 120 according to the artificial neural network data area information sequence, and data that is not reused can be deleted, the on-chip memory 120 . Even if the size is small, the operating efficiency of the chip-internal memory 120 may be improved.

Therefore, the NPU 100 may selectively preserve or delete specific data of the chip-internal memory 120 for a predetermined period based on the artificial neural network data locality information.

Such a mechanism may be applied not only to a skip-connection operation, but also to various operations such as concatenation, non-maximum suppression (NMS), and bilinear interpolation.

For example, the NPU 100 performs a convolution operation of the second layer for efficient control of the chip-internal memory 120 , and then the data of the first layer excluding the output feature map OFMAP of the first layer can be deleted. As another example, the NPU 100 performs the operation of the third layer for efficient control of the chip-internal memory 120 , and then the data of the second layer excluding the output feature map OFMAP of the first layer can be deleted. As another example, the NPU 100 performs the operation of the fourth layer for efficient control of the chip-internal memory 120 , and then the data of the third layer except for the output feature map OFMAP of the first layer can be deleted. In addition, the NPU 100 performs the operation of the fifth layer for efficient control of the chip-internal memory 120 , and then deletes the data of the fourth layer including the output feature map OFMAP of the first layer. can

The artificial neural network data locality information refers to a data processing order to be generated by the compiler 200 and performed by the NPU 100 in consideration of the conditions listed below.

1. Structure of ANN model (fusion artificial neural networks such as Resnet, YOLO, SSD, etc. designed to receive heterogeneous sensor data)

2. Processor architecture (eg, CPU, GPU, NPU, etc. architecture)

In the case of NPU, the number of PEs, the structure of the PE (eg, input stationary, output stationary, weight stationary, etc.), the SFU structure configured to operate organically with the PE array, etc.

3. Chip-internal memory 120 size (eg, when the cache is smaller than the data, need to apply a tiling algorithm, etc.)

4. Data size of each layer of the fusion-artificial neural network model to be processed

5. Processing Policy. That is, the NPU 100 determines the order of whether the input feature map (IFMAP) first read request or the kernel (Kernel) first read request. This may vary depending on the processor or compiler.

17 is an exemplary diagram illustrating a system including an NPU architecture according to a second example.

As can be seen with reference to FIG. 17 , the NPU 100 is a PE array 110 for a fusion artificial neural network, and an on-chip memory 120 and an NPU scheduler 130 . ) and may include a special function unit (SFU) 160 . In the description of FIG. 17 , overlapping descriptions may be omitted for convenience of description only.

The NPU scheduler 130 may be included in the control unit as shown.

The NPU 100 may receive different data from the M heterogeneous sensors 311 and 312 . The heterogeneous sensors may include a camera, radar, lidar, ultrasound, thermal imaging camera, and the like.

The NPU 100 may obtain fusion artificial neural network (ANN) data locality information from the compiler 200 .

The NPU 100 may output N results (eg, heterogeneous inference results) through N output units. The heterogeneous data output from the NPU 100 may be classification, semantic segmentation, object detection, prediction, or the like.

18 is an exemplary diagram illustrating a system including an NPU architecture according to a third example.

As can be seen with reference to FIG. 18 , the NPU 100 has a PE array 110 for a fusion artificial neural network, and a chip-internal (On-chip) memory 120 , and an NPU scheduler 130 . ) and may include a special function unit (SFU) 160 . In the description of FIG. 18 , overlapping descriptions may be omitted for convenience of description only.

The NPU scheduler 130 may be included in the control unit as shown.

The NPU 100 may receive different data from the M heterogeneous sensors 311 and 312 . The heterogeneous sensors may include a camera, radar, lidar, ultrasound, thermal imaging camera, and the like.

The NPU 100 may obtain fusion artificial neural network (ANN) data locality information from the compiler 200 .

The NPU 100 may receive data necessary for artificial neural network operation from the chip-external memory 500 through an artificial neural network data locality controller (ADC) 400 .

The ADC 400 may manage data in advance based on artificial neural network data locality information provided from the compiler 200 .

Specifically, the ADC 400 receives and analyzes artificial neural network data locality information of a fusion artificial neural network from the compiler 200 or receives the analyzed information from the compiler 200, the chip-external memory The operation of 500 can be controlled.

The ADC 400 may read data stored in the chip-external memory 500 according to the fusion neural network data locality information and cache the data in advance in an internal buffer memory. The chip-external memory 500 may store all the weight kernels of the fusion artificial neural network, and the chip-internal memory 120 stores the artificial neural network data locality information among all the weight kernels stored in the chip-external memory 500 . We can only store at least some of the weight kernels we need. The memory capacity of the on-chip memory 500 may be greater than the memory capacity of the on-chip memory 120 .

The ADC 400 interworks with the NPU 100 or independently prepares the data required for the NPU 100 from the chip-external memory 500 based on the artificial neural network data locality information in advance, and the inference of the NPU 100 It may be configured to reduce the latency of the operation or to improve the speed of the operation.

The NPU 100 may output N results (eg, heterogeneous inference results) through N output units.

19 is an exemplary diagram illustrating a system including an NPU architecture according to a fourth example, and FIG. 20 is an example in which the fusion artificial neural network shown in FIG. 13 is divided into threads according to the fourth example shown in FIG. indicates.

As can be seen with reference to FIG. 19 , the NPU 100 is a PE array 110 for a fusion artificial neural network, and a chip-internal (On-chip) memory 120 , and an NPU scheduler 130 . ) and may include a special function unit (SFU) 160 .

The NPU scheduler 130 may be included in the control unit as shown.

The NPU 100 may receive different data from the M heterogeneous sensors 311 and 312 . The heterogeneous sensors may include a camera, radar, lidar, ultrasound, thermal imaging camera, and the like.

The NPU 100 may obtain fusion artificial neural network (ANN) data locality information from the compiler 200 .

The NPU 100 may output N pieces of heterogeneous data (eg, heterogeneous inference results). The heterogeneous data output from the NPU 100 may be classification, semantic segmentation, object detection, prediction, or the like.

The PE array 110 can process multiple threads. As shown in FIG. 20 , RGB image data obtained from the camera may be processed through thread #1, conversion may be processed through thread #2, and data obtained from the rider may be processed through thread #3.

To this end, the compiler 200 may analyze the artificial neural network model and classify the threads based on the parallel operation flow.

The PE array 110 of the NPU 100 may improve computational efficiency through multiple threads of a layer capable of parallel processing of a fusion artificial neural network.

The PE array 110 of the NPU 100 may include a preset thread.

The NPU 100 may control each thread in the PE array 110 to communicate with the on-chip memory 120 .

The NPU 100 may selectively allocate an internal space of the on-chip memory 120 for each thread.

The NPU 100 may allocate an appropriate on-chip memory 120 for each thread. The memory allocation of the chip-internal memory 120 may be determined by the controller based on the neural network data locality information of the fusion neural network.

The NPU 100 may set a thread in the PE array 110 based on a fusion artificial neural network.

The NPU 100 may output N results (eg, heterogeneous inference results) through N output units.

21 is an exemplary diagram illustrating a system including an NPU architecture according to a fifth example, and FIG. 22 is an exemplary diagram illustrating a first example of the pipeline structure of the SFU shown in FIG.

As can be seen with reference to FIG. 21 , the NPU 100 has a PE array 110 for a fusion artificial neural network, and a chip-internal (On-chip) memory 120 , and an NPU scheduler 130 . ) and may include a special function unit (SFU) 160 .

The NPU 100 may receive different data from the M heterogeneous sensors 311 and 312 . The heterogeneous sensors may include a camera, radar, lidar, ultrasound, thermal imaging camera, and the like.

The NPU 100 may obtain fusion artificial neural network (ANN) data locality information from the compiler 200 .

The NPU 100 may output N pieces of heterogeneous data (eg, heterogeneous inference results). The heterogeneous data output from the NPU 100 may be classification, semantic segmentation, object detection, prediction, or the like.

22, the SFU 160 includes several functional units. Each functional unit can be selectively operated. Each functional unit can be selectively turned on or off. That is, each functional unit is configurable.

In other words, the SFU 160 may include various functional units required for fusion artificial neural network inference.

For example, the functional units of the SFU 160 are a functional unit for a skip-connection operation, a functional unit for an activation function operation, a functional unit for a pooling operation, Functional unit for quantization operation, function unit for non-maximum suppression (NMS) operation, function unit for integer and floating-point conversion (INT to FP32) operation, function for batch-normalization operation It may include a unit, a functional unit for an interpolation operation, a functional unit for a concatenation operation, and a functional unit for a bias operation.

The functional units of the SFU 160 may be selectively turned on or turned off by artificial neural network data locality information. The artificial neural network data locality information may include turn-off or turn-off-related control information of a corresponding functional unit when an operation for a specific layer is performed.

23A is an exemplary diagram illustrating an example of the SFU shown in FIG. 21 , and FIG. 23B is an exemplary diagram illustrating another example of the SFU illustrated in FIG. 21 .

23A and 23B , an activated unit among functional units of the SFU 160 may be turned on.

Specifically, as shown in FIG. 23A , the SFU 160 may selectively activate a skip-connection operation and a concatenation operation. Illustratively, each activated functional unit may be marked with hatching.

For example, the SFU 160 may concatenate heterogeneous sensor data for a fusion operation. For example, in order to skip and connect the SFU 160 , the controller may control the chip-internal memory 120 and the SFU 160 .

Specifically, as shown in FIG. 23B , a quantization operation and a bias operation may be selectively activated. For example, in order to reduce the size of the feature map data output from the PE array 110 , the quantization function unit of the SFU 160 receives the feature map output from the PE array 110 and converts the feature map to a specific bit width. can be quantized. In addition, the quantized feature map may be stored in the chip-internal memory 120 . A series of operations may be sequentially performed through the control unit, and the NPU scheduler 130 may be configured to control the operation sequence of the operations.

In this way, when selectively turning off some functional units of the SFU 160, it is possible to reduce the power consumption of the NPU (100). Meanwhile, in order to turn off some functional units, power gating may be used. Alternatively, clock gating may be performed to turn off some functional units.

24 is an exemplary diagram illustrating a system including an NPU architecture according to a sixth example.

As shown in Figure 24, NPU batch mode (Batch mode) may be applied.

The NPU 100 is a PE array 110 for a fusion artificial neural network, and an on-chip memory 120 , an NPU scheduler 130 , and a special function unit (SFU) 160 . ) may be included.

The NPU scheduler 130 may be included in the control unit as shown.

The NPU 100 may obtain fusion artificial neural network (ANN) data locality information from the compiler 200 .

The batch mode disclosed in this example is a mode configured to achieve low power by sequentially processing a plurality of identical sensors with one artificial neural network model and reusing the weights of the one artificial neural network model as much as the number of the plurality of identical sensors. it means.

For batch mode operation, the controller of the NPU 100 may be configured to control the NPU scheduler 130 so that the weight stored in the chip-internal memory is reused as much as the number of sensors input to each batch channel. That is, by way of example, the NPU 100 may be configured to operate in batch mode with M sensors. In this case, the batch mode operation of the NPU 100 may be configured to operate in a fusion-artificial neural network model.

For the operation of the fusion-artificial neural network, the NPU 100 may be configured to have a plurality of deployment channels (BATCH CH#1, BATCH CH#2) for fusion. Each deployment channel may be configured to include the same plurality of sensors. The first batch channel BATCH CH#1 may include a plurality of first sensors. In this case, the number of first sensors may be M. The K-th batch channel BATCH CH#K may include a plurality of second sensors. In this case, the number of second sensors may be M.

The NPU 100 may reuse and process the weight corresponding to the input from the sensors 311 and 312 in the chip-internal memory 120 through the first placement channel. In addition, the NPU 100 may reuse and process the weight corresponding to the input from the sensors 321 and 322 in the chip-internal memory 120 through the second arrangement channel.

As such, the NPU 100 may receive inputs from various sensors through a plurality of deployment channels, reuse weights, and process the fusion-artificial neural network in a deployment mode. A sensor of at least one channel among the plurality of arrangement channels and a sensor of at least one other channel may be different from each other.

The on-chip memory 120 in the NPU 100 may be configured to have a storage space corresponding to a plurality of placement channels.

The NPU scheduler 130 in the NPU 100 may operate the PE array 110 according to the batch mode.

The SFU 160 in the NPU 100 may provide a special function for processing at least one fusion operation.

The NPU 100 may deliver each output through a plurality of deployment channels.

At least one of the plurality of deployment channels may be inferred data of a fusion artificial neural network.

25 is an exemplary diagram illustrating an example of using a plurality of NPUs according to the seventh example, and FIG. 26 is an example of processing the fusion artificial neural network shown in FIG. 13 through the plurality of NPUs shown in FIG. It is an example diagram shown.

As shown in FIG. 25 , a plurality of, for example, M NPUs may be used for autonomous driving. Among the M NPUs, the first NPU 100-1 may process data provided from, for example, the sensor #1 311, and the M-th NPU 100-M may, for example, process data provided from the sensor #M 312. can be processed. The plurality of NPUs (eg, 100-1, 100-2) may access the chip-external memory 500 through the ADC/DMA (Direct Memory Access) 400 .

The plurality of NPUs (eg, 100-1, 100-2) may obtain fusion neural network data locality information from the compiler 200 .

Each NPU may process a fusion artificial neural network and transfer an operation for fusion to different NPUs through the ADC/DMA 400 .

The ADC/DMA 400 may obtain data locality information for a fusion artificial neural network from the compiler 200 .

The compiler 200 divides the neural network data locality information into data locality information #1 and data locality information #M so that operations to be processed in parallel among operations according to the artificial neural network data locality information can be processed in each NPU. can create

The chip-external memory 500 may store data that can be shared by a plurality of NPUs, and may be transmitted to each NPU.

As shown in FIG. 26 , NPU#1 may be in charge of the first artificial neural network for processing data provided from the camera, and NPU#2 may be in charge of the second artificial neural network for processing data provided from LiDAR. can do. In addition, the NPU #2 may be in charge of conversion for the fusion of the first artificial neural network and the second artificial neural network.

27A to 27C show examples of application of a fusion artificial neural network using a near-infrared sensor and a camera.

As shown in FIG. 27A , in general, in a vehicle, a general headlight is installed so as to irradiate visible light at an angle less than or equal to the horizontal line. However, the inventor of the present patent proposes to additionally install a light source irradiating near-infrared (NIR) in the forward direction, and to install the NIR sensor in the vehicle.

A typical camera can generally shoot RGB images with a wavelength of 380 nm to 680 nm. On the other hand, the NIR sensor may take an image having a wavelength of 850 nm to 940 nm.

In this way, when the NIR light source and the NIR sensor are added, a high-quality image can be obtained without obstructing the view of a driver driving an oncoming vehicle at night.

The NIR sensor may be synchronized with a corresponding NIR light source and driven according to pulse width modulation (PWM). Accordingly, power consumption and SNR can be improved.

Meanwhile, the NIR light source may be turned on or turned off every frame. As shown in FIG. 27B , when the NIR light source is turned on or turned off, signs having a retro-reflector characteristic may be distinguished within the entire image. 27C shows the characteristics of retroreflection.

By turning the NIR light source on and off in this way, it is possible to distinguish signs having retro-reflector characteristics. In other words, when the NIR light source and the NIR sensor are adjacent to each other, the amount of light reflected by the NIR light source on the retroreflective plate may be detected to be 300 or more brighter than the amount of light reflected by a general object. Therefore, when on-off, retroreflective objects can be detected.

The NIR sensor can detect the NIR reflected light, but the general traffic light light is not detected, so the fusion artificial neural network can be trained to distinguish the light from the NIR reflected light.

As described above, by mixing the RGB image and the NIR image, it is possible to enable autonomous driving at night.

These applications can be extended in other ways.

For example, an NIR light source may be additionally installed in a vehicle headlamp, and a camera including an image sensor capable of detecting a wavelength of 380 nm to 680 nm of visible light and a wavelength of 850 nm to 940 nm of near infrared light may be installed. The fusion artificial neural network can distinguish front and rear approaching vehicles, traffic lights, obstacles, road surface conditions, and pedestrians in an image.

As another example, in order to photograph the interior of the vehicle at night, the NIR light source and the NIR sensor may be installed in the interior of the vehicle. For example, a plurality of NIR light sources may be installed at different optimal positions to photograph the driver and passenger states. Through this, it is possible to monitor the health status of the driver and passengers.

28 shows an example of using a polarizer according to the eighth example, and FIGS. 29A and 29B are exemplary views showing the performance of the polarizer.

As shown in FIG. 28 , a polarizer is additionally connected to the image sensor #1 ( 311 ), and an output from the polarizer is input to the NPU #1 ( 100 ).

When a polarizer is added to the image sensor #1 ( 311 ), reflection of sunlight can be reduced. 29A and 29B, if a polarizer is used, light reflected from vehicle paint, glass, water, direct light, etc. can be filtered. However, if a polarizer is used, the brightness of the image will be darkened by 25%. can Accordingly, the artificial neural network driven by the NPU 100 may be trained to compensate for the reduced brightness due to the polarizer.

In various examples of the present disclosure, in order to minimize AI operation speed and power consumption, the PE array 110 may be configured as an inference-only PE array. An inference-only PE array can be configured to exclude the learning function of an artificial neural network. That is, a speculation-only PE array can be configured to exclude floating-point operators. Therefore, for artificial neural network learning, a separate dedicated hardware for learning may be provided. For example, the PE array 110 according to various examples of the present disclosure may be configured as an inference-only PE array that will be configured to process only 8-bit integers. According to the above-described configuration, the PE array 110 has the effect of significantly reducing power consumption compared to the floating point. At this time, the SFU 160 may be configured to utilize a functional unit for integer and floating-point conversion (INT to FP32) operations for some special functions requiring floating-point arithmetic.

That is, according to some examples, the PE array 110 may be configured to enable only integer arithmetic, and may be configured to enable floating point arithmetic in the SFU 160 .

That is, according to some examples, the control unit of the NPU 100 for efficient operation of the chip-internal memory 120 may control all data stored in the chip-internal memory 120 from the SFU 160 to be integers. .

<Brief summary of the disclosures of the present specification>

According to one disclosure herein, a neural processing unit (NPU) is provided. The NPU includes a compiled fusion-control unit configured to receive the machine code of the artificial neural network; an input unit (or sensor) configured to receive a plurality of input signals corresponding to the fusion-artificial neural network; an array of processing elements (PE) configured to perform the fusion-artificial neural network operation; a special function unit (SFU) configured to perform a special function of the fusion-artificial neural network operation; and an on-chip memory configured to store the fusion-artificial neural network computation data. The control unit is configured to: the processing element array, the special function unit, and the on-chip memory so that all operation orders of the fusion-artificial neural network are processed in a preset order according to the artificial neural network data locality information included in the machine-code It may include a scheduler configured to control the.

The input unit may be configured to receive a signal sensed from at least one of a camera, a polarization camera, a 3D camera, a near-infrared camera, a thermal imaging camera, a radar, a lidar, and an ultrasonic sensor.

The fusion-artificial neural network is trained to infer smart cruise control, automatic emergency braking system, parking steering assistance system, lane departure warning system, lane keeping assistance system, drowsiness detection, drinking detection, heat and cold detection, carelessness detection, etc. It may be a fusion-artificial neural network.

The input unit may be set to receive different types of data.

The SFU may perform at least one function of skip-connection and concatenation for artificial neural network fusion.

The scheduler may control the on-chip memory to protect the specific data stored in the on-chip memory up to a specific operation stage of the fusion-artificial neural network based on the artificial neural network data locality information.

The NPU further comprises an output unit configured to output at least one inference result of the fusion-artificial neural network trained to process at least one inference operation of Classification, Semantic segmentation, Object detection, and Prediction by the PE array. can

The PE array may further include a plurality of threads. The controller may control the plurality of threads to process the parallel section of the fusion-artificial neural network based on the locality of the artificial neural network data.

According to another disclosure of the present specification, a neural processing unit (NPU) is provided. The NPU includes: a fusion-control unit configured to receive the machine code of the artificial neural network; an array of processing elements (PE) configured to perform operations of the fusion-artificial neural network based on the machine code; and a special function unit (SFU) configured to receive the convolution operation value processed in the PE array and calculate a corresponding special function. The SFU may include a plurality of functional units. The SFU may selectively control at least some of the plurality of functional units according to the artificial neural network data locality information included in the machine-code.

The plurality of functional units may be connected in a pipeline structure.

The plurality of functional units may be selectively activated by the control unit.

The plurality of functional units may be selectively deactivated by the control unit.

Each of the plurality of functional units may be selectively clock-gated and/or power-gated for each specific operation step by the controller.

The NPU includes: an input unit configured to receive a plurality of input signals corresponding to the fusion-artificial neural network; and an on-chip memory configured to store the fusion-artificial neural network computation data.

The NPU may include: a batch input unit configured to receive a plurality of input signals corresponding to the fusion-artificial neural network in a batch mode; an on-chip memory configured to store the fusion-artificial neural network computation data in a batch mode; and an output unit configured to output a speculation result obtained by processing at least one speculation operation of Classification, Semantic segmentation, Object detection, and Pridection in a batch mode by the processing element array.

According to another disclosure of the present specification, a system is provided. The system includes a control unit configured to receive machine code of a fusion-artificial neural network, an input unit configured to receive at least one input signal, an array of processing elements configured to perform a convolution operation, and an on-chip configured to store a result of the convolution operation. at least one neural processing unit (NPU) comprising a memory; and receive neural network data locality information of the fusion-artificial neural network capable of predicting successive memory operation requests of the at least one neural processing unit, and based on the neural network data locality information, the corresponding at least one and a memory control unit, comprising a memory configured to pre-caching the next memory operation request to be requested by a neural processing unit of .

The NPU is plural, and the fusion-machine code of the artificial neural network input to the control unit of each NPU may be processed in parallel in the plurality of NPUs.

The plurality of NPUs, the memory control unit may directly control the parallel processing of the plurality of NPUs.

The number of NPUs may be plural, and the machine code may be compiled to be processed in parallel in the plurality of NPUs.

The system may further include an infrared light source and a visible light source. The input unit may be set to receive a visible ray image and an infrared image. The machine code may be configured to fuse the visible light image and the infrared image.

The infrared light source may be set to be PWM driven, and the infrared image may be synchronized with the infrared light source.

The irradiation angle of the infrared light source and the irradiation angle of the visible light source may partially overlap each other.

Examples of the present disclosure published in the present specification and drawings are merely specific examples to easily explain the technical content of the present disclosure and help the understanding of the present disclosure, and are not intended to limit the scope of the present disclosure. It will be apparent to those of ordinary skill in the art to which the present disclosure pertains that other modified examples based on the technical spirit of the invention can be implemented in addition to the examples posted here.

100: neural processing unit (NPU)
110: processing element (PE)
120: NPU internal memory
130: NPU scheduler
140: NPU interface
150: kernel generator

Claims (22)

a controller configured to receive the compiled fusion-artificial neural network machine code;
an input unit configured to receive a plurality of input signals corresponding to the fusion-artificial neural network;
an array of processing elements configured to perform the fusion-artificial neural network operation;
a special function unit configured to perform a special function of the fusion-artificial neural network operation; and
an on-chip memory configured to store the fusion-artificial neural network computation data;
The control unit is configured to: the processing element array, the special function unit, and the on-chip memory so that all operation orders of the fusion-artificial neural network are processed in a preset order according to the artificial neural network data locality information included in the machine-code comprising a scheduler configured to control
neural processing unit. According to claim 1,
The input unit is configured to receive a signal sensed from at least one of a camera, a polarization camera, a 3D camera, a near-infrared camera, a thermal imaging camera, a radar, a lidar, and an ultrasonic sensor. According to claim 1,
The fusion-artificial neural network is trained to infer smart cruise control, automatic emergency braking system, parking steering assistance system, lane departure warning system, lane keeping assistance system, drowsiness detection, drinking detection, heat and cold detection, carelessness detection, etc. A neural processing unit, characterized in that it is a fusion-artificial neural network. According to claim 1,
The input unit, a neural processing unit configured to receive heterogeneous data. According to claim 1,
The special function unit further comprises at least one function of Skip-connection and Concatenation for artificial neural network fusion. According to claim 1,
The scheduler is configured to control the on-chip memory to protect the specific data stored in the on-chip memory up to a specific operation step of the fusion-AI, based on the artificial neural network data locality information. According to claim 1,
and an output unit configured to output at least one inference result of the fusion-artificial neural network trained to process the inference operation of at least one of Classification, Semantic segmentation, Object detection, and Pridection by the array of processing elements. unit. According to claim 1,
the processing element array further comprises a plurality of threads;
The control unit is configured to control the plurality of threads to process the parallel section of the fusion-artificial neural network based on the artificial neural network data locality. a control unit configured to receive a machine code of a fusion-artificial neural network;
an array of processing elements configured to perform calculation of the fusion-artificial neural network based on the machine code; and
a special function unit configured to receive a convolution operation value processed in the processing element array and calculate a corresponding special function;
The special function unit includes a plurality of functional units,
and the special function unit is configured to selectively control at least some of the plurality of functional units according to the artificial neural network data locality information included in the machine-code. 10. The method of claim 9,
wherein the plurality of functional units are configured in a pipeline structure. 10. The method of claim 9,
wherein the plurality of functional units are configured to be selectively activated by a control unit. 10. The method of claim 9,
wherein the plurality of functional units are configured to be selectively deactivated by a control unit. 10. The method of claim 9,
Each of the plurality of functional units is configured to be selectively clock-gated and/or power-gated for each specific operation step by a control unit. 10. The method of claim 9,
an input unit configured to receive a plurality of input signals corresponding to the fusion-artificial neural network; and
and an on-chip memory configured to store the fusion-artificial neural network computation data. 10. The method of claim 9,
a batch input unit configured to receive a plurality of input signals corresponding to the fusion-artificial neural network in a batch mode;
an on-chip memory configured to store the fusion-artificial neural network computation data in a batch mode; and
and an output unit configured to output a speculation result obtained by processing in a batch mode a speculation operation of at least one of Classification, Semantic segmentation, Object detection, and Pridection by the processing element array. a control unit configured to receive machine code of the fusion-artificial neural network, an input unit configured to receive at least one input signal, an array of processing elements configured to perform a convolution operation, and an on-chip memory configured to store a result of the convolution operation at least one neural processing unit; and
configured to receive neural network data locality information of the fusion-artificial neural network capable of predicting successive memory operation requests of the at least one neural processing unit, and based on the neural network data locality information, the corresponding at least one a memory controller, comprising a memory configured to pre-caching the next memory operation request to be requested by a neural processing unit. 17. The method of claim 16,
wherein the neural processing unit is plural, and the machine code of the fusion-artificial neural network input to the control unit of each neural processing unit is configured to be processed in parallel in the plurality of neural processing units. 17. The method of claim 16,
The neural processing unit is a plurality,
and the memory controller is configured to directly control the parallel processing of the plurality of neural processing units. 17. The method of claim 16,
The neural processing unit is a plurality,
wherein the machine code is compiled for parallel processing in the plurality of neural processing units. 17. The method of claim 16,
Further comprising an infrared light source and a visible light source,
The input unit is configured to receive a visible light image and an infrared image,
wherein the machine code is machine code of a fusion-artificial neural network configured to fuse the visible light image and the infrared image. 21. The method of claim 20,
The infrared light source is configured to be PWM driven,
and the infrared image is synchronized with the infrared light source. 21. The method of claim 20,
The system, wherein the irradiation angle of the infrared light source and the irradiation angle of the visible light source are configured to partially overlap each other.

Images