KR102651559B1 - Neural processing unit and artificial neural network system for image fusion - Google Patents

Abstract

According to one disclosure of the present disclosure, a neural processing unit for an image fusion artificial neural network model is provided. The neural processing unit includes: a control unit configured to receive a machine code of an image fusion artificial neural network model learned to input a first image and a second image having different resolutions and image characteristics and output a new third image; an input circuit configured to receive a plurality of input signals corresponding to the image fusion artificial neural network model; a processing element array configured to perform a main operation of the image fusion artificial neural network model operation; a special function unit circuit configured to perform special function calculation of the image fusion artificial neural network model; and an on-chip memory configured to store data of the main operation and/or the special function operation of the image fusion artificial neural network model; and the control unit, the processing element array, the special processing element array, so that the operation order of the image fusion artificial neural network model is processed in a preset order according to the data locality information of the image fusion artificial neural network model included in the machine code. configured to control a functional unit circuit and the on-chip memory, wherein the third resolution of the third image has a value between the first resolution of the first image and the second resolution of the second image, The third image characteristic of the three images may be at least partially the same as the first image characteristic of the first image or the second image characteristic of the second image.

Description

Neural processing unit and artificial neural network system for image fusion {NEURAL PROCESSING UNIT AND ARTIFICIAL NEURAL NETWORK SYSTEM FOR IMAGE FUSION}

This disclosure relates to a neural processing unit and artificial neural network system for image fusion.

A thermal imaging sensor provides thermal imaging by collecting and visualizing radiant energy emitted by an object even when there is no separate external light source when shooting.

Thermal imaging is largely divided into NIR (near-infrared), SWIR (short-wave infrared), MWIR (medium-wave infrared), and LWIR (long-wave infrared) depending on the infrared frequency band. In the past, LWIR was mainly used in special fields such as military and medical purposes. Recently, the field of use of thermal imaging has increased, and it is also used for night-time object recognition of objects and cars.

In particular, the image quality of thermal images is very important in object recognition technology at night. However, because the price of thermal image sensors varies greatly depending on the resolution, high-resolution infrared image sensors are economically burdensome.

Accordingly, super-resolution (SR) technology has recently been launched, which upscales the resolution of images acquired from low-resolution thermal image sensors through artificial intelligence-based learning algorithms.

However, when thermal images are used as single input data, there are limitations in expressing the texture information of images that can be obtained from a general visible light image sensor at high resolution. Accordingly, technology is being developed to use heterogeneous images as input data and fuse them. However, the conventional image fusion technology has a problem in that the amount of calculation becomes excessively large in the process of performing synchronization and matching on a frame-by-frame basis.

The technology behind the present disclosure has been prepared to facilitate easier understanding of the present disclosure. It should not be understood as an admission that matters described in the background technology of this disclosure exist as prior art.

Accordingly, a neural processing unit (NPU) and an artificial neural network system including the same are designed to generate one image by fusing two features based on images acquired from heterogeneous image sensors, while minimizing the amount of calculation. It is required.

The inventors of the present disclosure have developed a neural processing unit and artificial neural network that can effectively process an artificial neural network model to generate an image that satisfies higher resolution by combining images with different resolutions and image characteristics for one object. We wanted to develop a system.

In particular, the inventors of the present disclosure have developed a neural processing unit and artificial neural network system that can quickly generate high-resolution images by performing concatenation and skip-connection operations that can effectively process different data. led to the development of

The tasks of the present disclosure are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

To solve the above-mentioned problems, a neural processing unit for an image fusion artificial neural network model is provided. The neural processing unit for the image fusion artificial neural network model inputs a first image and a second image having different resolutions and image characteristics, and generates a machine code of the image fusion artificial neural network model learned to output a new third image. A control unit configured to receive input; an input circuit configured to receive a plurality of input signals corresponding to the image fusion artificial neural network model; a processing element array configured to perform main operations of the image fusion artificial neural network model; a special function unit circuit configured to perform special function calculation of the image fusion artificial neural network model calculation; and an on-chip memory configured to store data of the main operation and/or the special function operation of the image fusion artificial neural network model; and the control unit, the processing element array, the special processing element array, so that the operation order of the image fusion artificial neural network model is processed in a preset order according to the data locality information of the image fusion artificial neural network model included in the machine code. configured to control a functional unit circuit and the on-chip memory, wherein the third resolution of the third image has a value between the first resolution of the first image and the second resolution of the second image, The third image characteristic of the three images may be at least partially the same as the first image characteristic of the first image or the second image characteristic of the second image.

According to another disclosure of the present disclosure, a system for an image fusion artificial neural network model is provided. The system includes a first sensor that acquires a first image having a first resolution and first image characteristics; a second sensor that acquires a second image having a second resolution smaller than the first resolution and second image characteristics different from the first image characteristics; and a neural processing unit configured to input a first image and a second image having different resolutions and image characteristics and process a learned image fusion artificial neural network model to output a new third image. Includes, the third resolution of the third image has a value between the first resolution of the first image and the second resolution of the second image, and the third image characteristic of the third image is, It may be at least partially the same as the first image characteristic of the first image or the second image characteristic of the second image.

Other example details are included in the detailed description and drawings.

The present disclosure can generate high-resolution thermal images through an artificial neural network model. In particular, the present disclosure can prevent the problem of personal information being exposed by generating and storing only high-resolution thermal images in a device for photographing an unspecified number of people, such as a surveillance camera. In addition, the present disclosure can protect personal information by fusing thermal images only in the face area of a person.

In addition, high-resolution thermal images can be generated by utilizing a high-resolution general visible light image sensor and a low-resolution thermal image sensor built into a general device, rather than a specialized device. Accordingly, the present disclosure can generate high-resolution thermal images at low cost. In addition, the present disclosure can improve night-time identification of images not only in devices designed for night-time identification, but also in, for example, devices owned by the user or vehicle black boxes.

Additionally, the present disclosure can generate high-resolution thermal images for observing an object even on days with poor weather conditions. In addition, the present disclosure can generate high-resolution thermal images of an object even on days when the sensitivity of laser sensors, electromagnetic wave sensors, and ultrasonic sensors used to detect the object in addition to the image sensor is low. Accordingly, the neural processing unit of the present disclosure can be installed in a vehicle to prevent safety accidents in the vehicle.

Additionally, the present disclosure can estimate the user's movements (skeleton detection or pose estimation) using high-resolution thermal images. For example, the present disclosure can estimate a user's abnormal movement or estimate a user's fall through a device installed in a specific space.

Additionally, the present disclosure can control a neural processing unit for implementing an image fusion artificial neural network model that generates a new image based on images acquired from heterogeneous image sensors to operate more efficiently. Accordingly, the present disclosure can reduce power consumption even when processing a huge amount of data. Accordingly, the present disclosure is not limited by battery capacity and can implement an image fusion artificial neural network model in various devices.

Additionally, the present disclosure can effectively process heterogeneous sensing data through concatenation and skip-connection operations. Accordingly, the present disclosure can quickly generate high-resolution thermal images while reducing the amount of computation.

Additionally, the present disclosure can minimize power consumed while acquiring data required to fuse high-resolution images from an external memory by maximally reusing data stored in the on-chip memory.

The effects according to the present disclosure are not limited to the content exemplified above, and further various effects are included within the present disclosure.

1 and 2 are schematic conceptual diagrams illustrating an image fusion artificial neural network model according to an example of the present disclosure.
3 and 4 are schematic conceptual diagrams illustrating images generated through an image fusion artificial neural network model according to an example of the present disclosure.
Figure 5 is a schematic conceptual diagram explaining a neural processing unit according to an example of the present disclosure.
FIG. 6 is a schematic conceptual diagram explaining one processing element among the processing element array shown in FIG. 5.
FIG. 7 is a conceptual diagram showing a modified example of the neural processing unit shown in FIG. 5.
Figure 8 is a conceptual diagram illustrating an image fusion artificial neural network model according to an example of the present disclosure.
FIG. 9 is a diagram for explaining a partial structure of a GAN neural network constituting an image fusion artificial neural network model according to an example of the present disclosure.
FIG. 10 is a diagram for explaining the input data of the convolution layer shown in FIG. 9 and the kernel used for the convolution operation.
FIG. 11 is a diagram for explaining the operation of a convolutional neural network that generates a feature map using the kernel shown in FIG. 10.
Figure 12 is a conceptual diagram illustrating an image fusion artificial neural network model according to an example of the present disclosure.
Figure 13 is an example diagram showing an NPU fusion method according to an example of the present disclosure.
Figure 14 is a conceptual diagram illustrating a system including an NPU architecture according to the first example of the present disclosure.
FIG. 15A is an example diagram illustrating skip-connection included in the image fusion artificial neural network model according to the first example of the present disclosure.
FIG. 15B is an example diagram showing artificial neural network data locality information of the image fusion artificial neural network model shown in FIG. 15A.
Figure 16 is a conceptual diagram illustrating a system including an NPU architecture according to a second example of the present disclosure.
Figure 17 is a conceptual diagram illustrating a system including an NPU architecture according to the third example of the present disclosure.
Figure 18 is a conceptual diagram illustrating a system including an NPU architecture according to the fourth example of the present disclosure.
FIG. 19 shows an example of dividing the image fusion artificial neural network model shown in FIG. 12 into threads according to the fourth example shown in FIG. 18.
Figure 20 is a conceptual diagram illustrating a system including an NPU architecture according to the fifth example of the present disclosure.
FIG. 21 is an exemplary diagram showing a first example of the pipeline structure of the SFU shown in FIG. 20.
FIG. 22A is an exemplary diagram showing an example of the SFU shown in FIG. 20.
FIG. 22b is an exemplary diagram showing another example of the SFU shown in FIG. 20.
Figure 23 is a conceptual diagram illustrating a system including an NPU architecture according to the sixth example of the present disclosure.
Figure 24 is an example diagram showing an example of utilizing a plurality of NPUs according to the seventh example of the present disclosure.
FIG. 25 is an example diagram showing an example of processing the fusion artificial neural network shown in FIG. 12 through a plurality of NPUs shown in FIG. 24.

The advantages and features of the present disclosure and methods for achieving them will become clear by referring to the examples described in detail below along with the accompanying drawings. However, the present disclosure is not limited to the examples disclosed below and will be implemented in various different forms. These examples are provided to ensure that the disclosure of the present disclosure is complete and to fully inform those skilled in the art of the scope of the invention to which the present disclosure pertains. The present disclosure is defined solely by the scope of the claims. In connection with the description of the drawings, similar reference numbers may be used for similar components.

In the present disclosure, expressions such as “have,” “may have,” “includes,” or “may include” refer to the presence of the corresponding feature (e.g., component such as numerical value, function, operation, or part). , and does not rule out the existence of additional features.

In the present disclosure, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” includes (1) at least one A, (2) at least one B, or (3) it may refer to all cases including both at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” used in the present disclosure can modify various components regardless of order and/or importance, and can refer to one component. It is only used to distinguish from other components and does not limit the components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, the first component may be renamed to the second component without departing from the scope of rights described in the present disclosure, and similarly, the second component may also be renamed to the first component.

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When referred to as being “connected to,” it should be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is said to be “directly connected” or “directly connected” to another component (e.g., a second component), It may be understood that no other component (e.g., a third component) exists between other components.

The expression “configured to” used in the present disclosure may mean, for example, “suitable for,” “having the capacity to,” depending on the situation. ," can be used interchangeably with "designed to," "adapted to," "made to," or "capable of." The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

The terms used in this disclosure are merely used to describe specific examples and may not be intended to limit the scope of other examples. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary skill in the technical field described in this disclosure. Among the terms used in this disclosure, terms defined in general dictionaries may be interpreted to have the same or similar meaning as the meaning they have in the context of related technology. Additionally, terms defined in general dictionaries are not to be interpreted in an idealized or overly formal sense unless clearly defined in the present disclosure. In some cases, even terms defined in this disclosure cannot be interpreted to exclude examples of this disclosure.

Each feature of the various examples of the present disclosure can be partially or entirely combined or combined with each other. As can be fully understood by those skilled in the art, the features of the various examples of the present disclosure are technically capable of various interconnections and operations, and each example may be implemented independently of each other or may be implemented together in a related relationship.

For clarity of interpretation of the present disclosure, terms used in the present disclosure will be defined below.

NPU is an abbreviation for Neural Processing Unit, and can refer to a processor specialized for calculating artificial neural network models, separate from the CPU (Central processing unit).

ANN is an abbreviation for artificial neural network. In order to imitate human intelligence, neurons in the human brain are connected through synapses, forming nodes into layers. It can mean a network connected in a structure.

An artificial neural network model is a model for fusing images and may be a model trained to perform inferences such as Image/Video Reconstruction and Image/Video Enhancement.

In addition, other artificial neural network models that use fused images as input are trained to perform inferences such as Object Classification, Object Detection, Object Segmentation, Object Tracking, Event Recognition, Event Prediction, Anomaly Detection, Density Estimation, Event Search, and Measurement. It could be a model.

For example, artificial neural network models include Transformer, Bisenet, Shelfnet, Alexnet, Densenet, Efficientnet, EfficientDet, Googlenet, Mnasnet, Mobilenet, Resnet, Shufflenet, Squeezenet, VGG, Yolo, RNN, CNN, DBN, RBM, LSTM, etc. It can be. However, the present disclosure is not limited to this, and may be a new artificial neural network model that can operate on the NPU 100.

Information about the structure of the artificial neural network includes information about the number of layers, the number of nodes within the layer, the value of each node, information about the calculation processing method, and information about the weight matrix applied to each node.

Information about data locality of the image fusion artificial neural network model is information including the order of data access requests to memory determined based on the structure of the artificial neural network and the neural processing unit that processes the artificial neural network.

DNN is an abbreviation for Deep Neural Network, and can mean increasing the number of hidden layers of an artificial neural network to implement higher artificial intelligence.

CNN is an abbreviation for Convolutional Neural Network, and is a neural network that functions similar to image processing in the visual cortex of the human brain. Convolutional neural networks are known to be suitable for image processing and are known to be easy to extract features of input data and identify patterns of features.

Kernel may refer to a weight matrix applied to CNN.

The chip-external memory may be memory placed considering that the memory size inside the NPU is limited. That is, the memory may be placed outside the chip to store large-capacity data. Off-chip memory may include one of the following memories: ROM, SRAM, DRAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, etc. The off-chip memory may consist of at least one memory unit. Off-chip memory may be comprised of a single (homogeneous) memory unit or a heterogeneous (heterogeneous) memory unit.

Chip-internal memory The NPU may include chip-internal memory. On-chip memory may include volatile memory and/or non-volatile memory. For example, the chip-internal memory may include one of memories such as ROM, SRAM, DRAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, etc. The chip-internal memory may be composed of at least one memory unit. The chip-internal memory may be composed of a single (homogeneous) memory unit or a heterogeneous (heterogeneous) memory unit.

Hereinafter, an example of the present disclosure will be described in detail with reference to the attached drawings.

1 and 2 are schematic conceptual diagrams illustrating an image fusion artificial neural network model according to an example of the present disclosure.

Referring to FIG. 1 , the neural processing unit 100 includes a processing element 102 configured to perform calculations of an image fusion artificial neural network model 101 . The neural processing unit 100 may generate a new image (third image) based on two images (first image, second image). The first image and the second image are different images of one object and may be images obtained from different types of sensors. For example, the heterogeneous sensor may be an image sensor for visible light photography, an image sensor for infrared photography, etc.

The image sensor for visible light photography acquires a color image (first image) in the visible light region through R (Red), G (Green), and B (Blue) pixels. An image sensor for infrared photography can acquire a thermal image (thermal color map) (second image) through pixels. In general, image sensors for infrared photography detect energy with a larger wavelength than visible light, so even if the sensor is the same size, the number of pixels, or resolution, is inevitably low. In the present disclosure, a third image that satisfies the resolution of the image sensor for visible light photography can be generated through the image fusion artificial neural network model 101 without using a high-resolution image sensor for infrared photography.

Accordingly, the neural processing unit 100 may be a model learned to input a first image and a second image having different resolutions and image characteristics and output a new third image. The third resolution of the third image may have a value between the first resolution of the first image and the second resolution of the second image. For example, the third resolution of the third image may be the same as the first resolution of the first image. Additionally, the third image characteristics of the third image may be at least partially the same as the image characteristics of the first image or the second image characteristics of the second image. For example, when the first resolution of each RGB channel of the first image is 1024 x 786, and the second resolution of the thermal image channel of the second image is 100 x 60, the third resolution of the third image is 1024 x 786. You can. That is, the resolution of the third image is the same as the size and first resolution of the first image, and the thermal image corresponding to the second image characteristic can be applied to the third image characteristic.

In another example, the neural processing unit 100 may use different input data for the image fusion artificial neural network model 101. Specifically, when the first image is a color image, the processing speed of the neural processing unit 100 may increase. Accordingly, the image fusion artificial neural network model 101 takes only the brightness value of each pixel of the first image as input, and generates a third image identical to that of the color image as input, while reducing the processing speed of the neural processing unit 100. You can do it. That is, the image fusion artificial neural network model 101 may be a model configured to input only the RGB values (3 channels) of the first image or the brightness value (1 channel) of each pixel of the first image.

In this way, the weight parameters of the image fusion artificial neural network model 101, which can generate new images by combining the characteristics of different images, can be learned based on the GAN (generative adversarial networks) structure. The GAN structure consists of a generator that generates virtual images or real images, and a verifier (discriminator) that determines the authenticity of the images generated by the generator. GAN can be a model in which generators and verifiers compete adversarially to improve each other's performance. Specifically, the generator provides real images (real data) to the verifier so that the verifier learns to determine the image as real, and secondly, it inputs virtual images (fake data) generated by the generator so that the verifier determines the image to be fake. You can learn to discriminate. In addition, the generator learns to create virtual images to deceive the verifier, so that it can evolve to generate harmonious images according to competition between them.

That is, in the learning stage, the image fusion artificial neural network model 101 may be a model in which the generator and verifier constituting the GAN compete with each other to update weights to increase the third resolution of the third image.

In the present disclosure, in order to minimize memory usage of the neural processing unit 100, image generation learning through a classifier (generator) can be performed in a separate device/server (not shown). In addition, the image fusion artificial neural network model 101 operated by the neural processing unit 100 takes different images of one object (e.g. 1. high-resolution visible light image, 2. low-resolution thermal image) as input and creates a new image ( Yes, it can correspond to a generator configured to generate high-resolution thermal images.

Meanwhile, the image fusion artificial neural network model 101 may be learned based on a set of learning data that is substantially similar to the first image and the second image. That is, the image used for learning and the image received later to create a new image may be different.

Hereinafter, an image generated through an image fusion artificial neural network model will be described as an example.

3 and 4 are schematic conceptual diagrams illustrating images generated through an image fusion artificial neural network model according to an example of the present disclosure.

Referring to FIG. 3, the image fusion artificial neural network model can use images acquired from heterogeneous sensors as input. For example, the heterogeneous sensor may be an image sensor for visible light photography, an image sensor for infrared photography, etc. The image fusion artificial neural network model may use as input the first image and the second image acquired by different image sensors for one object. Additionally, the image fusion artificial neural network model can generate an image in which image characteristics of each image sensor are fused based on images acquired from heterogeneous image sensors. That is, the image fusion artificial neural network model can generate a third image in which temperature information (second image characteristics) of the second image is reflected while maintaining the size and resolution of the first image.

In another example, the image fusion artificial neural network model may be a model to which weights are applied to emphasize at least one feature that can be determined in the first image and at least one feature that can be determined in the second image. Specifically, the neural processing unit 100 may generate features of the first image and the second image. As an example, the neural processing unit 100 may generate a feature map by inferring features of high-resolution edge content from a first image, which is a color image. A feature map may be referred to as a heatmap, activation map, or parameter.

Additionally, the neural processing unit 100 may generate a feature map by inferring segmentation according to temperature from the second image, which is a thermal image. Subsequently, image fusion of high-resolution thermal images can be processed based on the high-resolution edge feature map and the low-resolution temperature segmentation line feature map.

As some examples, the neural processing unit 100 may further detect the presence of an object using features of the first image and determine an area where the temperature is greater than a threshold value using features of the second image. Accordingly, the image fusion artificial neural network model can generate a new image by applying weights to determinable features in each image. For example, as shown in FIG. 4 , the neural processing unit 100 may detect an object (person) in the first image and thermally image only an area with a temperature higher than a certain temperature within the area where the object is detected. That is, the neural processing unit 100 may generate a third image in which at least one feature that can be determined from the second image is applied to at least a partial area of the first image.

Meanwhile, in order to generate high-resolution thermal images limited to some areas, the processing element array of the neural processing unit 100 may be configured to process at least one of Dilated Convolution, Transposed Convolution, and Bilinear Interpolation operations. .

In another example, the neural processing unit 100 may extract a first partial image and a second partial image corresponding to a face area from a first image and a second image of one object. Accordingly, the neural processing unit 100 may generate a fused image of image features only for the face using an artificial neural network unit configured to input only the first partial image and the second partial image.

In this way, personal information can be protected by the neural processing unit 100 fusing and generating images to which different image features are applied only to the face area of the person.

Hereinafter, the neural processing unit 100 that performs calculations of an image fusion artificial neural network model will be described.

5 is a schematic conceptual diagram illustrating a neural processing unit according to an example of the present disclosure.

The NPU 100 shown in FIG. 5 is a processor specialized to perform operations for an image fusion artificial neural network model.

An artificial neural network refers to a network of artificial neurons that, when multiple inputs or stimuli come in, multiply each by its weight and add it, and additionally transform and transmit the added value through an activation function. The artificial neural network learned in this way can be used to output inference results from input data.

The NPU 100 may be a semiconductor implemented as an electrical/electronic circuit. An electrical/electronic circuit may mean including numerous electronic elements (eg, transistors, capacitors).

Referring to FIG. 5, the NPU 100 may include a processing element (PE) array 110, an NPU internal memory 120, an NPU scheduler 130, and an NPU interface 140. Each of the processing element array 110, NPU internal memory 120, NPU scheduler 130, and NPU interface 140 may be a semiconductor circuit to which numerous transistors are connected. Therefore, some of them may be difficult to identify and distinguish with the naked eye, and can only be identified through movement. For example, an arbitrary circuit may operate as the processing element array 110 or may operate as the NPU scheduler 130. The NPU scheduler 130 may be configured to perform the function of a control unit configured to control the artificial neural network inference operation of the NPU 100. To elaborate, a part of the control unit may be referred to as the scheduler 130. The NPU scheduler 130 may be part of the control unit. The NPU scheduler 130 may also be referred to as a control unit. The control unit may include an NPU scheduler 130. The control unit may be a general name for a circuit that performs various control functions of the NPU 100, such as direct memory access (DMA). It is also possible for the control unit to be defined by the function of the circuit. To elaborate, the control unit may define a circuit that controls the processing element array 110 according to the order of each operation step of the artificial neural network model based on the artificial neural network data locality of the artificial neural network model as the NPU scheduler 130.

The NPU 100 includes a processing element array 110, an NPU internal memory 120 configured to store an image fusion artificial neural network model that can be inferred from the processing element array 110, and artificial neural network data locality of the image fusion artificial neural network model. It may include an NPU scheduler 130 configured to control the processing element array 110 and the NPU internal memory 120 based on information or structure information. Here, the NPU internal memory 120 may store information about the artificial neural network data locality or structure of the image fusion artificial neural network model. In other words, an image fusion artificial neural network model may refer to an AI recognition model learned to perform a specific inference function (e.g., image fusion, object movement, object posture, motion tracking, etc.).

The processing element array 110 can perform operations for an artificial neural network.

NPU interface 140 may communicate with various components connected to NPU 100, such as memory, through a system bus (e.g., one or more communication buses or signal lines).

The NPU scheduler 130 is configured to control the operation of the processing element array 100 for the inference operation of the neural processing unit 100 and the read and write order of the NPU internal memory 120.

The NPU scheduler 130 may be configured to control the processing element array 100 and the NPU internal memory 120 based on information about the artificial neural network data locality or structure of the image fusion artificial neural network model.

The NPU scheduler 130 may analyze the structure of an image fusion artificial neural network model to be operated in the processing element array 100 or may receive information that has already been analyzed. The analyzed information may be included in machine code. For example, the artificial neural network data that the image fusion artificial neural network model can include is node data (i.e. feature map) of each layer, arrangement data of the layers, locality information or structure information, and node of each layer. It may include at least some of the weight data (i.e., weight kernel) of each network connecting the . Data of the artificial neural network may be stored in the memory provided inside the NPU scheduler 130 or in the NPU internal memory 120. The NPU scheduler 130 may be operated by machine code.

The NPU scheduler 130 may schedule the operation order of the image fusion artificial neural network model to be performed by the NPU 100 based on information about the artificial neural network data locality or structure of the image fusion artificial neural network model. Machine code may include scheduling data. The NPU scheduler 130 may operate according to scheduling included in machine code. That is, the NPU scheduler 130 may be configured to operate by machine code.

The NPU scheduler 130 may obtain a memory address value where the feature map and weight data of the layer of the image fusion artificial neural network model are stored based on information about the locality information or structure of the artificial neural network data of the image fusion artificial neural network model. For example, the NPU scheduler 130 may obtain a memory address value where the feature map and weight data of the layer of the image fusion artificial neural network model stored in the memory are stored. Therefore, the NPU scheduler 130 can retrieve the feature map and weight data of the layer of the image fusion artificial neural network model to be run from the memory 200 and store them in the NPU internal memory 120.

The feature map of each layer may have a corresponding memory address value.

Each weight data may have a respective memory address value of the corresponding NPU internal memory 120.

The NPU scheduler 130 is based on information about the locality information or structure of the artificial neural network data of the image fusion artificial neural network model, for example, information about the locality information or structure of the artificial neural network layers of the image fusion artificial neural network model. The operation order of the processing element array 110 can be scheduled.

Since the NPU scheduler 130 performs scheduling based on information about the locality or structure of artificial neural network data of the image fusion artificial neural network model, it may operate differently from the scheduling concept of a general CPU. Typical CPU scheduling takes into account fairness, efficiency, stability, response time, etc. and operates to achieve the best efficiency. In other words, scheduling is performed to perform the most processing within the same amount of time, taking into account priority, computation time, etc.

Conventional CPUs used an algorithm to schedule tasks by considering data such as priority order of each processing and operation processing time.

Differently, the NPU scheduler 130 may control the NPU 100 in the processing order of the NPU 100 determined based on information about the artificial neural network data locality or structure of the image fusion artificial neural network model.

Furthermore, the NPU scheduler 130 determines the information about the artificial neural network data locality information or structure of the image fusion artificial neural network model and/or the information about the data locality information or structure of the neural processing unit 100 to be used. The NPU (100) can be driven in processing order.

However, the present disclosure is not limited to information on data locality or structure of the NPU 100.

The NPU scheduler 130 may be configured to store data locality information or information on the structure of the artificial neural network.

In other words, the NPU scheduler 130 can determine the processing order even if only using at least information about the artificial neural network data locality information or structure of the image fusion artificial neural network model.

Furthermore, the NPU scheduler 130 determines the processing order of the NPU 100 by considering the information about the artificial neural network data locality information or structure of the image fusion artificial neural network model and the information about the data locality information or structure of the NPU 100. You can decide. Additionally, it is possible to optimize processing of the NPU 100 in the determined processing order.

The processing element array 110 refers to a configuration in which a plurality of processing elements (PE1 to PE12) configured to calculate feature maps and weight data of an artificial neural network are arranged. Each processing element may include a multiply and accumulate (MAC) operator and/or an Arithmetic Logic Unit (ALU) operator. However, examples according to the present disclosure are not limited thereto.

Although a plurality of processing elements are exemplarily shown in FIG. 5, instead of a MAC, inside one processing element, operators implemented as a plurality of multipliers and adder trees are arranged in parallel. possible. In this case, the processing element array 110 may also be referred to as at least one processing element including a plurality of operators.

The processing element array 110 is configured to include a plurality of processing elements (PE1 to PE12). The plurality of processing elements (PE1 to PE12) shown in FIG. 3 are merely examples for convenience of explanation, and the number of processing elements (PE1 to PE12) is not limited. The size or number of the processing element array 110 may be determined by the number of the plurality of processing elements (PE1 to PE12). The size of the processing element array 110 may be implemented in the form of an N x M matrix. Here N and M are integers greater than 0. The processing element array 110 may include N x M processing elements. That is, there may be one or more processing elements.

The size of the processing element array 110 can be designed considering the characteristics of the image fusion artificial neural network model on which the NPU 100 operates. Accordingly, the utilization rate % of the processing element array 110 can be improved.

The processing element array 110 is configured to perform functions such as addition, multiplication, and accumulation required for artificial neural network operations. Stated differently, the processing element array 110 may be configured to perform a multiplication and accumulation (MAC) operation.

The processing element array 110 may be configured to quantize and output the MAC operation result. However, the examples of the present disclosure are not limited thereto.

The NPU internal memory 120 may store all or part of the image fusion artificial neural network model depending on the memory size and the data size of the image fusion artificial neural network model.

Hereinafter, the first processing element PE1 of the processing element array 110 will be described as an example.

FIG. 6 is a schematic conceptual diagram explaining one processing element among the processing element array shown in FIG. 5.

Referring to FIG. 6 , the first processing element PE1 may include a multiplier 111, an adder 112, an accumulator 113, and a bit quantization unit 114. However, examples according to the present disclosure are not limited thereto, and the processing element array 110 may be modified and implemented in consideration of the computational characteristics of an artificial neural network.

The multiplier 111 multiplies the input (N) bit data and (M) bit data. The operation value of the multiplier 111 is output as (N+M) bit data.

The multiplier 111 may be configured to receive one variable and one constant as input.

The accumulator 113 accumulates the operation value of the multiplier 111 and the operation value of the accumulator 113 using the adder 112 the number of (L) loops. Therefore, the bit width of the data of the output and input circuits of the accumulator 113 can be output as (N+M+log2(L)) bits. Here, L is an integer greater than 0.

When accumulation is completed, the accumulator 113 can receive an initialization reset signal and initialize the data stored inside the accumulator 113 to 0. However, examples according to the present disclosure are not limited thereto.

The bit quantization unit 114 can reduce the bit width of data output from the accumulator 113. The bit quantization unit 114 may be controlled by the NPU scheduler 130. The bit width of quantized data can be output as (X)bit. Here, X is an integer greater than 0. According to the above-described configuration, the processing element array 110 is configured to perform a MAC operation, and the processing element array 110 has the effect of quantizing and outputting the MAC operation result. In particular, this quantization has the effect of further reducing power consumption as (L)loops increase. Additionally, reducing power consumption has the effect of reducing heat generation. In particular, reducing heat generation has the effect of reducing the possibility of malfunction due to high temperature of the NPU (100).

The output data (X) bit of the bit quantization unit 114 may be node data of the next layer or input data of convolution. If the image fusion artificial neural network model is quantized, the bit quantization unit 114 may be configured to receive quantized feature maps and/or weights from the image fusion artificial neural network model. However, it is not limited to this, and the NPU scheduler 130 may be configured to extract quantized information by analyzing an image fusion artificial neural network model. Therefore, the output data (X) bit can be converted to the quantized bit width and output to correspond to the quantized data size. The output data (X) bit of the bit quantization unit 114 may be stored in the NPU internal memory 120 as a quantized bit width.

That is, the processing element array 110 of the NPU 100 according to an example of the present disclosure may include a multiplier 111, an adder 112, an accumulator 113, and a bit quantization unit 114.

Hereinafter, another example of the NPU 100 of the present disclosure will be described.

FIG. 7 is a conceptual diagram showing a modified example of the neural processing unit shown in FIG. 5.

Before describing FIG. 7, since the NPU 100 shown in FIG. 7 is substantially the same as the processing unit 100 shown as an example in FIG. 5, except for the processing element array 110, For convenience of explanation below, redundant description will be omitted.

Referring to FIG. 7, in addition to a plurality of processing elements (PE1 to PE12), the element array 110 further includes register files (RF1 to RF12) corresponding to each processing element (PE1 to PE12). can do.

However, the plurality of processing elements (PE1 to PE12) and the plurality of register files (RF1 to RF12) shown in FIG. 7 are merely examples for convenience of explanation, and the plurality of processing elements (PE1 to PE12) and the plurality of register files (RF1 to RF12) are only examples for convenience of explanation. The number of register files (RF1 to RF12) is not limited.

The size or number of the processing element array 110 may be determined by the number of processing elements (PE1 to PE12) and the number of register files (RF1 to RF12). The size of the processing element array 110 and the plurality of register files (RF1 to RF12) may be implemented in the form of an N x M matrix. Here N and M are integers greater than 0.

The array size of the processing element array 110 can be designed considering the characteristics of the image fusion artificial neural network model on which the NPU 100 operates. To elaborate, the memory size of the register file can be determined by considering the data size of the image fusion artificial neural network model to be operated, required operation speed, required power consumption, etc.

The register files (RF1 to RF12) of the NPU (100) are static memory units directly connected to the processing elements (PE1 to PE12). Register files (RF1 to RF12) may be composed of, for example, flip-flops and/or latches. Register files (RF1 to RF12) may be configured to store MAC operation values of corresponding processing elements (PE1 to PE12). Register files (RF1 to RF12) may be configured to provide or receive NPU internal memory 120, weight data, and/or node data.

The register files (RF1 to RF12) can also be configured to perform the function of the accumulator's temporary memory during MAC operation.

Register files (RF1 to RF12) can temporarily store feature maps that have been computed, and then reuse the feature maps in the next operation to reduce power consumption.

Hereinafter, the operation of the exemplary image fusion artificial neural network model 110-10 that can be operated on the NPU 100 will be described.

Figure 8 is a conceptual diagram illustrating an image fusion artificial neural network model according to an example of the present disclosure.

The image fusion artificial neural network model 110-10 of FIG. 8 may be an artificial neural network learned in the NPU 100 shown in FIG. 5 or 7 or in a separate machine learning device. The image fusion artificial neural network model 110-10 may be an artificial neural network learned to perform various inference functions, such as motion and posture estimation of objects in the image.

The image fusion artificial neural network model 110-10 may be a deep neural network (DNN). However, the image fusion artificial neural network model 110-10 according to examples of the present disclosure is not limited to deep neural networks.

For example, an image fusion artificial neural network model may be a model trained to perform inference such as Image/Video Reconstruction or Image/Video Enhancement.

In addition, other artificial neural network models that use fused images as input include Super-resolution, Upscaling, Image fusion, Object Classification, Object Detection, Object Segmentation, Object Tracking, Event Recognition, Event Prediction, Anomaly Detection, Density Estimation, Event Search, It may be a model trained to perform inference such as measurement.

For example, image fusion artificial neural network models include Transformer, Bisenet, Shelfnet, Alexnet, Densenet, Efficientnet, EfficientDet, Googlenet, Mnasnet, Mobilenet, Resnet, Shufflenet, Squeezenet, VGG, Yolo, RNN, CNN, DBN, RBM, LSTM, etc. It may be a model of However, the present disclosure is not limited to this, and may be a new artificial neural network model that can be operated on the NPU 100.

In various examples, the image fusion artificial neural network model 110-10 may be an ensemble model based on at least two different models.

At least some of the parameters such as weight values, node values, accumulated values, feature maps, and weights of each layer of the image fusion artificial neural network model 110-10 may be stored in the NPU internal memory 120 of the NPU 100. there is.

Specifically, referring to FIG. 8, the inference process using the image fusion artificial neural network model 110-10 may be performed by the NPU 100.

The image fusion artificial neural network model (110-10) includes an input layer (110-11), a first connection network (110-12), a first hidden layer (110-13), a second connection network (110-14), and a second hidden layer. An exemplary deep neural network model includes a layer 110-15, a third network 110-16, and an output layer 110-17. However, the present disclosure is not limited to the image fusion artificial neural network model shown in FIG. 8. The first hidden layer 110-13 and the second hidden layer 110-15 may also be referred to as a plurality of hidden layers.

The input layer 110-11 may illustratively include x1 and x2 input nodes. That is, the input layer 110-11 may include information about two input values. The NPU scheduler 130 shown in FIG. 5 or 7 sets the memory address where information about the input value from the input layer 110-11 is stored in the NPU internal memory 120 shown in FIG. 5 or 7. You can.

The first connection network 110-12 illustratively includes information on six weight values for connecting each node of the input layer 110-11 to each node of the first hidden layer 110-13. It can be included. The NPU scheduler 130 shown in FIG. 5 or 7 may set a memory address where information about the weight value of the first connection network 110-12 is stored in the NPU internal memory 120. Each weight value is multiplied by the input node value, and the accumulated value of the multiplied values is stored in the first hidden layer 110-13. Here, nodes with accumulated values may be referred to as feature maps.

The first hidden layer 110-13 may exemplarily include nodes a1, a2, and a3. That is, the first hidden layer 110-13 may include information about three node values. The NPU scheduler 130 shown in FIG. 5 or 7 may set a memory address for storing information about the node value of the first hidden layer 110-13 in the NPU internal memory 120.

The NPU scheduler 130 may be configured to schedule an operation order so that the first processing element (PE1) performs the MAC operation of the a1 node of the first hidden layer (110-13). The NPU scheduler 130 may be configured to schedule an operation order so that the second processing element (PE2) performs the MAC operation of the a2 node of the first hidden layer (110-13). The NPU scheduler 130 may be configured to schedule an operation order so that the third processing element (PE3) performs the MAC operation of the a3 node of the first hidden layer 110-13. Here, the NPU scheduler 130 may pre-schedule the operation order so that the three processing elements each perform the MAC operation simultaneously in parallel. The scheduling information may be included in machine code. Therefore, the NPU scheduler 130 may operate according to scheduling information included in the machine code.

The second connection network 110-14 illustratively includes nine weight values for connecting each node of the first hidden layer 110-13 to each node of the second hidden layer 110-15. May contain information. The NPU scheduler 130 shown in FIG. 5 or 7 may set a memory address for storing information about the weight value of the second connection network 110-14 in the NPU internal memory 120. The weight value of the second network 110-14 is multiplied by the node value input from the first hidden layer 110-13, and the accumulated value of the multiplied values is sent to the second hidden layer 110-15. It is saved.

The second hidden layer 110-15 may exemplarily include nodes b1, b2, and b3. That is, the second hidden layer 110-15 may include information about three node values. The NPU scheduler 130 may set a memory address for storing information about the node value of the second hidden layer 110-15 in the NPU internal memory 120.

The NPU scheduler 130 may be configured to schedule an operation order so that the fourth processing element (PE4) performs the MAC operation of the b1 node of the second hidden layer 110-15. The NPU scheduler 130 may be configured to schedule an operation order so that the fifth processing element (PE5) performs the MAC operation of the b2 node of the second hidden layer 110-15. The NPU scheduler 130 may be configured to schedule an operation order so that the sixth processing element (PE6) performs the MAC operation of the b3 node of the second hidden layer 110-15. The scheduling information may be included in machine code.

Here, the NPU scheduler 130 may pre-schedule the operation order so that the three processing elements each perform the MAC operation simultaneously in parallel.

Here, the NPU scheduler 130 may determine scheduling so that the operation of the second hidden layer 110-15 is performed after the MAC operation of the first hidden layer 110-13 of the image fusion artificial neural network model.

That is, the NPU scheduler 130 may be configured to control the processing element array 100 and the NPU internal memory 120 based on information about the artificial neural network data locality or structure of the image fusion artificial neural network model.

The third network 110-16 illustratively includes information on six weight values connecting each node of the second hidden layer 110-15 and each node of the output layer 110-17. can do. The NPU scheduler 130 may set a memory address for storing information about the weight value of the third connection network 110-16 in the NPU internal memory 120. The weight value of the third network 110-16 is multiplied by the node value input from the second hidden layer 110-15, and the accumulated value of the multiplied values is stored in the output layer 110-17.

The output layer 110-17 may illustratively include y1 and y2 nodes. That is, the output layer 110-17 may include information about two node values. The NPU scheduler 130 may set a memory address in the NPU internal memory 120 to store information about the node value of the output layer 110-17.

The NPU scheduler 130 may be configured to schedule an operation order so that the seventh processing element (PE7) performs the MAC operation of the y1 node of the output layer (110-17). The NPU scheduler 130 may be configured to schedule an operation order so that the eighth processing element (PE8) performs the MAC operation of the y2 node of the output layer 110-15. The scheduling information may be included in machine code.

Here, the NPU scheduler 130 may pre-schedule the operation order so that the two processing elements each perform the MAC operation simultaneously in parallel.

Here, the NPU scheduler 130 may determine scheduling so that the operation of the output layer 110-17 is performed after the MAC operation of the second hidden layer 110-15 of the image fusion artificial neural network model.

That is, the NPU scheduler 130 may be configured to control the processing element array 100 and the NPU internal memory 120 based on information about the artificial neural network data locality or structure of the image fusion artificial neural network model.

That is, the NPU scheduler 130 may analyze the structure of an image fusion artificial neural network model to operate in the processing element array 100 or receive the analyzed information. The artificial neural network information that the image fusion artificial neural network model can include is information about the node values of each layer, information about the locality or structure of the arrangement data of the layers, and each weight value of the network connecting the nodes of each layer. It may include information about.

Because the NPU scheduler 130 has been provided with information about the artificial neural network data locality information or structure of the example image fusion artificial neural network model 110-10, the NPU scheduler 130 uses the image fusion artificial neural network model 110-10. You can understand the operation sequence from input to output.

Accordingly, the NPU scheduler 130 may set the memory address where the MAC operation values of each layer are stored in the NPU internal memory 120 in consideration of the scheduling order.

The NPU internal memory 120 may be configured to preserve weight data of connection networks stored in the NPU internal memory 120 while the inference operation of the NPU 100 continues. Therefore, there is an effect of reducing memory read and write operations.

That is, the NPU internal memory 120 may be configured to reuse the MAC operation value stored in the NPU internal memory 120 while the inference operation continues.

Hereinafter, the structure of the image fusion artificial neural network model of the present disclosure will be described through FIGS. 9 to 12.

FIG. 9 is a diagram illustrating a partial structure of a GAN constituting an image fusion artificial neural network model according to an example of the present disclosure.

Referring to FIG. 9, the GAN neural network structure that constitutes the image fusion artificial neural network model has a structure corresponding to a generator for generating high-resolution thermal images. That is, the scheduler 130 of the neural processing unit 100 may be configured to receive machine code compiled with an image fusion artificial neural network model excluding a verifier (discriminator) as input and process an inference operation.

In one example, the image fusion artificial neural network model corresponding to the generator can use 3-channel RGB visible light images and 1-channel thermal image as input data, respectively, and performs a convolution operation to which an activation function (ELU) is applied. By performing the process, a feature map and/or an activation map can be output. For example, the input data of a visible light image can be calculated by sliding 64 filters with a size of 3 x 3 for each channel, and the input data of a thermal image can be calculated by sliding 64 filters with a size of 3 x 3. In other words, the size of the feature map of the input data of the visible light image can be reduced to the same size as the size of the feature map output from the input data of the thermal image before image fusion. The output feature maps output through each operation can be merged into one filter with a size of 1 × 1. The output results of the merged feature maps like this can be transferred to other layers through a skip-connection operation, and a high-resolution thermal image can be finally generated through multiple layers. Figure 9 is just an example for configuring a generator in a GAN, and is not limited to this, and various model configurations can be adopted.

FIG. 10 is a diagram for explaining the input data of the convolution layer shown in FIG. 9 and the kernel used for convolution operation or matrix multiplication.

Referring to FIG. 10, input data 300 may be an image or video displayed as a two-dimensional matrix consisting of rows 310 of a specific size and columns 320 of a specific size. Input data 300 may be referred to as a feature map. The input data 300 may have a plurality of channels 330, where the channels 330 may represent color RGB channels of the input data image.

Meanwhile, the kernel 340 may be a weight parameter used in convolution to extract features of a certain portion of the input data 300 while scanning it. The kernel 340 may be configured to have rows 350 of a specific size, columns 360 of a specific size, and a specific number of channels 370, similar to the input data image. In general, the sizes of the rows 350 and columns 360 of the kernel 340 are set to be the same, and the number of channels 370 may be the same as the number of channels 330 of the input data image.

FIG. 11 is a diagram for explaining the operation of a convolutional neural network that generates a feature map using the kernel shown in FIG. 10.

Referring to FIG. 11, the kernel 410 traverses the input data 420 at designated intervals and performs convolution, thereby finally generating the feature map 430. Convolution is performed by multiplying the input data values at a specific location of the portion and the values at the corresponding location of the kernel 410, when applying the kernel 410 to a portion of the input data 420, and then adding all the values generated. It can be executed.

Through this convolution process, the calculated values of the feature map are generated, and each time the kernel 410 traverses the input data 420, the result values of this convolution are generated to form the feature map 430. .

Each component value of the feature map can be converted into an activation map 430 through the activation function of the convolution layer.

In FIG. 11, the input data 420 input to the convolution layer is displayed as a two-dimensional matrix with a size of 4 x 4, and the kernel 410 is displayed as a two-dimensional matrix with a size of 3 x 3. However, the sizes of the input data 420 and the kernel 410 of the convolution layer are not limited to this and may vary depending on the performance and requirements of the convolutional neural network including the convolution layer.

As shown, when input data 420 is input to the convolutional layer, the kernel 410 traverses the input data 420 at a predetermined interval (e.g., stride = 1), and the input data 420 A MAC operation can be performed by multiplying the values at the same position of the kernel 410 and adding the values.

Specifically, the kernel 410 assigns the MAC operation value “15” calculated at a specific location 421 of the input data 420 to the corresponding element 431 of the feature map 430. The kernel 410 assigns the MAC operation value “16” calculated at the next position 422 of the input data 420 to the corresponding element 432 of the feature map 430. The kernel 410 assigns the MAC operation value “6” calculated at the next position 423 of the input data 420 to the corresponding element 433 of the feature map 430. Next, the kernel 410 assigns the MAC operation value “15” calculated at the next position 424 of the input data 420 to the corresponding element 434 of the feature map 430.

In this way, if the kernel 410 allocates all the MAC operation values calculated while traversing the input data 420 to the feature map 430, a feature map 430 with a size of 2 x 2 can be completed.

At this time, if the input data 510 consists of, for example, three channels (R channel, G channel, B channel), the same kernel or a different channel for each channel is traversed over the data for each channel of the input data 420. A feature map for each channel can be created through convolution, which involves multiple multiplication and summation.

For the MAC operation, the scheduler 130 allocates processing elements (PE1 to PE12) to perform each MAC operation based on a preset operation order, and determines the memory address where the MAC operation values are stored by considering the scheduling order. It can be set in the NPU internal memory (120).

Figure 12 is a conceptual diagram illustrating an image fusion artificial neural network model according to an example of the present disclosure.

Referring to FIG. 12, an example is shown in which signals provided from an RGB camera and a thermal image sensor are processed through parallel processing. During parallel processing, different information can be exchanged through transformers. This method may be the deep fusion method shown in FIG. 14, which will be described later.

Meanwhile, although not shown, an artificial neural network may include a concatenation operation and a skip-connection operation to process different data provided from heterogeneous sensors. The concatenation operation means merging the output results of a specific layer, and the skip and concatenation operation means passing the output results of a specific layer to another layer, skipping the subsequent layer.

These concatenation operations and skip-connect operations may increase the control difficulty and usage of the internal memory 120 of the NPU 100.

So far, we have explained the artificial neural network for processing different data provided by heterogeneous sensors by fusion, but there is a weakness in that the performance of the artificial neural network cannot be improved with the above explanation alone. Accordingly, the optimized artificial neural network and NPU structure will be described below.

<Fusion artificial neural network and NPU structure optimized to process different data from heterogeneous sensors>

First, the inventor of this patent studied NPU for processing different data from heterogeneous sensors.

The following configuration should be considered in the design of the NPU:

i. It is necessary to have an NPU structure suitable for heterogeneous data signal processing (e.g., RGB camera + thermal image sensor).

ii. NPU memory control suitable for heterogeneous input signal processing (e.g., RGB camera + thermal image sensor) is required.

iii. It is necessary to have an NPU structure suitable for multiple input channels.

iv. NPU memory control suitable for multiple input channels is required.

v. It is necessary to have an NPU structure suitable for calculating an image fusion artificial neural network model (fusion artificial neural network model).

vi. For real-time application, a fast processing speed of 16ms or less is required.

vii. For battery operation, it is necessary to achieve low power consumption.

The NPU for implementing the image fusion artificial neural network model (fusion artificial neural network model) must support the following functions. The expected requirements are as follows:

i. CNN function support: Must be able to control PE array and memory optimized for convolution.

ii. Depthwise-separable convolution must be able to be processed efficiently. It must have a structure that improves PE utilization and performance (throughput).

iii. Batch mode function supported: Memory configuration is required to process multiple channels (cameras 1 to 6) and heterogeneous sensors simultaneously. (PE array size and memory size must be in an appropriate ratio)

iv. Concatenation function support: NPU for image fusion artificial neural network model (fusion artificial neural network model) must be able to process heterogeneous input data signals with the concatenation function.

v. Skip-connection function support: NPU for image fusion artificial neural network model (fusion artificial neural network model) has SFU (Special Function Unit) that can provide skip-connection function. It can be included.

vi. Support for deep learning image preprocessing function: NPU for image fusion artificial neural network model (fusion artificial neural network model) must be able to provide the function of preprocessing different data signals.

vii. A compiler that can efficiently compile image fusion artificial neural network models (fusion artificial neural network models) must be provided.

In one embodiment of the present disclosure, an NPU 100 with the following characteristics is proposed.

i. NPU (100) is a machine that analyzes ANN data locality information of image fusion artificial neural network models (fusion artificial neural network models) such as late fusion, early fusion, and deep fusion. Code can be processed.

ii. The NPU 100 may be configured to control the PE array to process heterogeneous sensor data based on an artificial neural network data locality controller (ADC). In other words, the image fusion artificial neural network model (fusion artificial neural network model) is fused into various structures depending on the sensor, and the PE utilization rate can be improved by providing an NPU (100) corresponding to the structure. .

iii. The size of the chip-internal memory 120 may be appropriately set to process heterogeneous sensor data based on ANN data locality information. In other words, by analyzing the locality information of the artificial neural network data of the image fusion artificial neural network model (fusion artificial neural network model), the memory bandwidth of the NPU 100 that processes the fusion artificial neural network can be improved.

iv. The NPU (100) can efficiently process the bilinear interpolation, concatenation, and skip-connection required in the image fusion artificial neural network model (fusion artificial neural network model). It may include a special function unit (SFU).

Figure 13 is an example diagram showing an NPU fusion method according to an example of the present disclosure.

Referring to Figure 13, “F” stands for fusion operation, and each block represents each layer. NPU (100) can perform late fusion, early fusion, and deep fusion. Late fusion means performing calculations for each layer and then fusing the calculation results in the final process. Early fusion means fusion of different data early and then performing calculations for each layer. Deep fusion means fusing different data, performing calculations on different layers, fusing the calculation results again, and then performing calculations for each layer. In the present disclosure, through early fusion operation, two different images can be merged at the beginning of the operation of a plurality of layers, and the operation of subsequent layers can be performed. Differently, through late fusion, it is possible to perform calculations on two different images for each layer assigned to each, merge the calculation results, and then perform calculations on subsequent layers. For example, the two different images may be an image acquired through a visible light image sensor and an image acquired through a thermal image sensor, but are not limited thereto.

Hereinafter, the structure of the NPU 100 that can exhibit the above features will be described.

Figure 14 is a conceptual diagram illustrating a system including an NPU architecture according to the first example of the present disclosure.

Referring to FIG. 14, the NPU 100 includes a PE array 110, an on-chip memory 120, an NPU scheduler 130, and a special function unit (SFU) for an image fusion artificial neural network model. It may include (160). When describing FIG. 14, overlapping descriptions may be omitted simply for convenience of description.

The PE array 110 for an image fusion artificial neural network model may refer to a PE array 110 configured to process convolution of a multi-layered image fusion artificial neural network model having at least one fusion layer. In other words, the fusion layer can be configured to output a feature map in which data from heterogeneous sensors is fused. To elaborate, the SFU 160 of the NPU 100 may be configured to receive sensor data from multiple sensors and provide a function to fuse each sensor input data. The PE array 110 for the image fusion artificial neural network model may be configured to receive fused data from the SFU 160 and process convolution.

The NPU 100 may receive different data from M heterogeneous sensors 311 and 312. The heterogeneous sensors may include image sensors with different image characteristics and resolutions.

The NPU 100 may obtain artificial neural network data locality information of an image fusion artificial neural network model (fusion artificial neural network (ANN)) from the compiler 200.

At least one layer of the image fusion artificial neural network model may be a layer in which input data from a plurality of sensors is fused.

The NPU 100 may be configured to provide a concatenation function to at least one layer for fusion of heterogeneous sensor input data. In order to be concatenated, each feature map of heterogeneous sensors in a concatenated layer may be processed to have the size of at least one axis the same. For example, to concatenate heterogeneous sensor data on the X-axis, the size of the X-axis of each of the heterogeneous sensor data may be the same. For example, to concatenate heterogeneous sensor data on the Y-axis, the size of the Y-axis of each heterogeneous sensor data may be the same. For example, in order to concatenate heterogeneous sensor data along the Z-axis, the size of the Z-axis of each heterogeneous sensor data may be the same. To improve the processing efficiency of the NPU 100, the size of one piece of heterogeneous sensor data can be scaled up or scaled down. Therefore, it is possible for the size of one axis of the fused data of heterogeneous sensor data to be the same. To elaborate, since the processing element array 100 is in the form of an N x M matrix, the PE utilization rate of the processing element array 100 may vary depending on the size of at least one axis of sensor data.

In order to receive and process different data from the heterogeneous sensors 311 and 312, the NPU scheduler 130 may process inference of an image fusion artificial neural network model (fusion artificial neural network model).

The NPU scheduler 130 may be included in the control unit as shown.

The NPU scheduler 130 obtains and analyzes artificial neural network data locality information of the image fusion artificial neural network model (fusion artificial neural network) from the compiler 200, and controls the operation of the chip-internal memory 120. there is.

To be specific, it is as follows. The compiler 200 may generate artificial neural network data locality information of a fusion artificial neural network to be processed by the NPU 100.

The NPU scheduler 130 may generate a list of special functions required for the image fusion artificial neural network model (fusion artificial neural network). Special functions may refer to various functions required for artificial neural network calculations other than convolution.

Using the artificial neural network data locality information of the image fusion artificial neural network model (fusion artificial neural network), non-maximum suppression (NMS), skip-connection (SKIP-CONNECTION), bottleneck, and bilinear interpolation The problem of increased memory access, which frequently occurs in fusion artificial neural networks such as bilinear interpolation, can be efficiently controlled.

When using the artificial neural network data locality information of the image fusion artificial neural network model (fusion artificial neural network), the first output feature map information calculated first and the second output feature map information processed later are fused. Since the size and storage period of the data to be stored (e.g., the first output feature map) and the storage period can be known at the compilation stage, the memory map for the on-chip memory 120 can be efficiently created in advance. You can set it.

The SFU 160 can perform skip-connection and concatenation required for an image fusion artificial neural network model (fusion artificial neural network). To elaborate, concatenation can be used to fuse heterogeneous sensor data. For concatenation, the size of each sensor data can be readjusted. For example, the NPU 100 may be configured to process concatenation of a fusion artificial intelligence network by providing functions such as resize and interpolation.

The chip-internal memory 120 of the NPU 100 can selectively preserve specific data according to the PE array 110 or SFU 160 for a specific period based on artificial neural network data locality information of the image fusion artificial neural network model. You can. Whether or not the selective preservation is performed can be controlled by a control unit.

Additionally, the PE array 110 may be configured to have a number of threads corresponding to the number of heterogeneous sensors. That is, the array 110 of the NPU 100 configured to receive two sensor data may be configured to have two threads. That is, if one thread is composed of N x M processing elements, two threads may be composed of N x M x 2 processing elements. For example, each thread of the PE array 110 may be configured to process the feature map of each heterogeneous sensor. Multiple threads of an NPU may be referred to as multi-cores of the NPU.

The NPU 100 may output the calculation result of the image fusion artificial neural network model through an output unit.

The NPU architecture according to the first example described above may be modified in various ways.

FIG. 15A is an example diagram for explaining skip-connection included in the image fusion artificial neural network model according to the first example of the present disclosure, and FIG. 15B is an illustration of the image fusion artificial neural network model shown in FIG. 15A. This is an example diagram showing artificial neural network data locality information.

Referring to FIG. 15A, in order to calculate five layers including a skip-connection operation, the compiler 200, as shown in FIG. 15B, generates, for example, an image with a sequence of 16 steps. Artificial neural network data locality information of the fusion artificial neural network model can be generated.

The NPU (100) requests data operations from the on-chip memory (120) in the order of artificial neural network data locality information of the image fusion artificial neural network model.

In the case of a skip-connection operation, the output feature map (OFMAP) of the first layer may be added to the output feature map (OFMAP) of the fourth layer.

For this skip-connection operation, the output feature map of the first layer must be preserved until the fifth layer operation. However, other data may be deleted after operation to utilize memory space.

Data to be calculated later based on the order of artificial neural network data locality information of the image fusion artificial neural network model may be stored in the deleted memory area. Therefore, necessary data can be sequentially imported into the chip-internal memory 120 according to the data local information order of the image fusion artificial neural network model, and unused data can be deleted, so that the on-chip memory ( Even if the memory size of the chip 120 is small, the operating efficiency of the chip internal memory 120 can be improved.

Accordingly, the NPU 100 can selectively preserve or delete specific data in the chip-internal memory 120 for a certain period of time based on artificial neural network data locality information of the image fusion artificial neural network model.

This mechanism can be applied not only to skip-connection operations, but also to various operations such as concatenation, non-maximum suppression (NMS), and bilinear interpolation.

For example, the NPU 100 performs a convolution operation of the second layer for efficient control of the chip-internal memory 120, and then performs the data of the first layer excluding the output feature map (OFMAP) of the first layer. can be deleted. For another example, the NPU 100 performs the operation of the third layer for efficient control of the chip-internal memory 120 and then data of the second layer excluding the output feature map (OFMAP) of the first layer. can be deleted. For another example, the NPU 100 performs the operation of the fourth layer for efficient control of the chip-internal memory 120 and then data of the third layer excluding the output feature map (OFMAP) of the first layer. can be deleted. In addition, for efficient control of the chip-internal memory 120, the NPU 100 performs the operation of the fifth layer and then deletes the data of the fourth layer including the output feature map (OFMAP) of the first layer. You can.

The artificial neural network data locality information of the image fusion artificial neural network model refers to the data processing order generated by the compiler 200 and performed by the NPU 100 in consideration of the conditions listed below.

1. Structure of ANN model (fusion artificial neural networks such as Resnet, YOLO, SSD, etc. designed to receive heterogeneous sensor data).

2. Structure of the processor (e.g., architecture of CPU, GPU, NPU, etc.).

In the case of the NPU 100, the number of PEs, the structure of the PE (e.g., input stationary, output stationary, weight stationary, etc.), the SFU structure configured to operate organically with the PE array, etc.

3. Chip-internal memory 120 size (e.g., when the cache is smaller than the data, tiling algorithm needs to be applied, etc.).

4. Data size of each layer of the image fusion artificial neural network model to be processed.

5. Processing policy. That is, the NPU 100 determines the order of whether to request to read the input feature map (IFMAP) first or to request to read the kernel first. This may vary depending on the processor or compiler 200.

Figure 16 is a conceptual diagram illustrating a system including an NPU architecture according to a second example of the present disclosure.

Referring to FIG. 16, the NPU 100 includes a PE array 110, an on-chip memory 120, an NPU scheduler 130, and a special function unit (SFU) for an image fusion artificial neural network model. It may include (160). When describing FIG. 16, overlapping descriptions may be omitted for convenience of explanation.

The NPU scheduler 130 may be included in the control unit as shown.

The NPU 100 may receive different data from M heterogeneous sensors 311 and 312. The heterogeneous sensors may include a microphone, a touch screen, a camera, an altimeter, a barometer, a photoplethysmography sensor, an electrocardiogram sensor, an inertial measurement sensor, a geopositioning system, an optical sensor, a thermometer, an electromyography measuring device, an electrode measuring device, etc.

The NPU 100 may obtain artificial neural network data locality information of the image fusion artificial neural network model from the compiler 200.

The NPU 100 can output N results (eg, heterogeneous inference results) through N output units. Heterogeneous data output from the NPU (100) may include image fusion, classification, semantic segmentation, object detection, prediction, etc.

Figure 17 is a conceptual diagram illustrating a system including an NPU architecture according to the third example of the present disclosure.

Referring to FIG. 17, the NPU 100 includes a PE array 110, an on-chip memory 120, an NPU scheduler 130, and a special function unit (SFU) for an image fusion artificial neural network model. It may include (160). When describing FIG. 17, overlapping descriptions may be omitted simply for convenience of description.

The NPU scheduler 130 may be included in the control unit as shown.

The NPU 100 may receive different data from M heterogeneous sensors 311 and 312. The heterogeneous sensors may include image sensors with different image characteristics and resolutions.

The NPU 100 may obtain artificial neural network data locality information of the image fusion artificial neural network model from the compiler 200.

The NPU 100 may receive data required for calculating an image fusion artificial neural network model from the chip-external memory 500 through an artificial neural network data locality controller (ADC) 400.

The ADC 400 may prefetch data from chip-external memory to chip-internal memory based on artificial neural network data locality information of the image fusion artificial neural network model provided by the compiler 200.

Specifically, the ADC 400 receives and analyzes artificial neural network data locality information of the image fusion artificial neural network model from the compiler 200, or receives the analyzed information from the compiler 200, and receives the chip-external memory ( 500) operation can be controlled.

The ADC 400 may read data stored in the chip-external memory 500 and cache it in the chip-internal memory in advance according to the artificial neural network data locality information of the image fusion artificial neural network model. The chip-external memory 500 may store all weight kernels of the image fusion artificial neural network model, and the chip-internal memory 120 may store the image fusion artificial neural network model among all weight kernels stored in the chip-external memory 500. Depending on the locality information of the artificial neural network data, only at least some of the necessary weight kernels can be stored. The memory capacity of the chip-external memory 500 may be larger than that of the chip-internal memory 120.

The ADC (400) can prepare data required for the NPU (100) in advance from the chip-external memory (500) in conjunction with the NPU (100) or independently based on the artificial neural network data locality information of the image fusion artificial neural network model. there is. Accordingly, the latency of the inference operation of the NPU 100 may be reduced or the operation speed may be improved.

The NPU 100 can output N results (eg, heterogeneous inference results) through N output units.

FIG. 18 is a conceptual diagram illustrating a system including an NPU architecture according to the fourth example of the present disclosure, and FIG. 19 is a diagram showing the image fusion artificial neural network model shown in FIG. 12 as a thread according to the fourth example shown in FIG. 18. Shows an example divided by .

Referring to FIG. 18, the NPU 100 includes a PE array 110, an on-chip memory 120, an NPU scheduler 130, and a special function unit (SFU) for an image fusion artificial neural network model. It may include (160).

The NPU scheduler 130 may be included in the control unit as shown.

The NPU 100 may receive different data from M heterogeneous sensors 311 and 312. The heterogeneous sensors may include image sensors having different image characteristics and resolutions.

The NPU 100 may obtain artificial neural network data locality information of the image fusion artificial neural network model from the compiler 200.

The NPU 100 may output N pieces of heterogeneous data (eg, heterogeneous inference results). Heterogeneous data output from the NPU (100) may include image fusion, classification, semantic segmentation, object detection, prediction, etc.

The PE array 110 can process multiple threads. As shown in Figure 19, RGB image data obtained from the camera can be processed through thread #1, Transformer model processing can be processed through thread #2, and data obtained from the thermal image sensor can be processed through thread #3. there is. Multiple threads of the PE array 110 may be referred to as multi-cores of the NPU. That is, each thread can refer to an independent PE array.

To this end, the compiler 200 may analyze the image fusion artificial neural network model and distinguish threads based on parallel computation flows.

The PE array 110 of the NPU 100 can improve computational efficiency through multiple threads of layers capable of parallel processing operations of an image fusion artificial neural network model.

Each thread may be configured to include the same or different numbers of processing elements.

The NPU 100 can control each thread in the PE array 110 to communicate with the on-chip memory 120.

The NPU 100 can selectively allocate space inside the on-chip memory 120 for each thread.

The NPU 100 can allocate appropriate on-chip memory 120 for each thread. Memory allocation of the chip-internal memory 120 may be determined by the control unit based on artificial neural network data locality information of the image fusion artificial neural network model.

The NPU 100 can set up a thread in the PE array 110 based on a fusion artificial neural network.

The NPU 100 can output N results (eg, heterogeneous inference results) through N output units.

FIG. 20 is a conceptual diagram illustrating a system including an NPU architecture according to the fifth example of the present disclosure, and FIG. 21 is an exemplary diagram showing a first example of the pipeline structure of the SFU shown in FIG. 20.

Referring to FIG. 20, the NPU 100 includes a PE array 110, an on-chip memory 120, an NPU scheduler 130, and a special function unit (SFU) for an image fusion artificial neural network model. It may include (160).

The NPU 100 may receive different data from M heterogeneous sensors 311 and 312. The heterogeneous sensors may include image sensors with different image characteristics and resolutions.

The NPU 100 may obtain artificial neural network data locality information of an image fusion artificial neural network model (fusion artificial neural network (ANN)) from the compiler 200.

The NPU 100 may output N pieces of heterogeneous data (eg, heterogeneous inference results). Heterogeneous data output from the NPU (100) may include image fusion, classification, semantic segmentation, object detection, prediction, etc.

As shown in Figure 21, the SFU 160 includes several functional units. Each functional unit can be operated selectively. Each functional unit can be selectively turned on or off. That is, each functional unit can be set.

The processing element array may refer to a circuit unit configured to perform the main operation of the image fusion artificial neural network model. The main operation may refer to convolution or matrix multiplication. In other words, the main operation may refer to most operations in an artificial neural network (ANN) (eg, Fusion artificial neural network).

A special function unit (SFU) may refer to a set of a plurality of special function circuit units configured to selectively perform special function calculations of an image fusion artificial neural network model. That is, the special function unit (SFU) can additionally calculate special functions, and the special function calculation may refer to additional calculations in various artificial neural networks (ANNs) (e.g., Fusion artificial neural networks).

The computation amount of the main computation of the image fusion artificial neural network model may be relatively larger than that of the special function computation.

In other words, the SFU 160 may include various functional units required for inference of an image fusion artificial neural network model.

For example, the functional units of the SFU 160 include a functional unit for a skip-connection operation, a functional unit for an activation function operation, a functional unit for a pooling operation, Functional unit for quantization operation, functional unit for NMS (non-maximum suppression) operation, functional unit for integer and floating point conversion (INT to FP32) operation, function for batch-normalization operation It may include a unit, a functional unit for an interpolation operation, a functional unit for a concatenation operation, and a functional unit for a bias operation.

The functional units of the SFU 160 can be selectively turned on or turned off based on artificial neural network data locality information of the image fusion artificial neural network model. To elaborate, the types of special function operations required by each layer of the image fusion artificial neural network model may be different for each layer. Artificial neural network data locality information included in machine code may include control information related to the turn-on or turn-off of the corresponding functional unit when an operation for a specific layer is performed.

FIG. 22A is an exemplary diagram showing an example of the SFU shown in FIG. 20, and FIG. 22B is an exemplary diagram showing another example of the SFU shown in FIG. 20.

Referring to FIGS. 22A and 22B, an activated unit among the functional units of the SFU 160 may be turned on.

Specifically, as shown in FIG. 22A, the SFU 160 can selectively activate a skip-connection operation and a concatenation operation. By way of example, each activated functional unit may be indicated by hatching.

For example, the SFU 160 can concatenate heterogeneous sensor data for a fusion operation. For example, for the skip and connect operation of the SFU 160, the control unit may control the chip-internal memory 120 and the SFU 160.

Specifically, as shown in FIG. 22b, quantization operation and bias operation can be selectively activated. For example, in order to reduce the size of the feature map data output from the PE array 110, the quantization function unit of the SFU 160 receives the feature map output from the PE array 110 and converts the feature map into a specific bit width. It can be quantized. And the quantized feature map can be stored in the chip's internal memory 120. A series of operations can be performed sequentially through the control unit, and the NPU scheduler 130 can be configured to control the operation order of the operations.

In this way, when some functional units of the SFU 160 are selectively turned off, power consumption of the NPU 100 can be reduced. Meanwhile, power gating can be used to turn off some functional units. Alternatively, clock gating may be performed to turn off some functional units.

Figure 23 is a conceptual diagram illustrating a system including an NPU architecture according to the sixth example of the present disclosure.

Referring to FIG. 23, NPU batch mode may be applied. The NPU (100) to which the batch mode is applied includes a PE array (110), an on-chip memory (120), an NPU scheduler (130), and a special function (SFU) for an image fusion artificial neural network model. unit) (160).

The NPU scheduler 130 may be included in the control unit as shown.

The NPU 100 may obtain artificial neural network data locality information of the image fusion artificial neural network model from the compiler 200.

The batch mode disclosed in this example achieves low power by sequentially processing multiple identical sensors with one image fusion artificial neural network model and reusing the weights of the single image fusion artificial neural network model as many as the number of identical sensors. This means a mode configured to do so.

For batch mode operation, the control unit of the NPU 100 may be configured to control the NPU scheduler 130 so that the weights stored in the chip's internal memory are reused by the number of sensors input to each batch channel. That is, as an example, the NPU 100 may be configured to operate in batch mode with M sensors. At this time, the batch mode operation of the NPU 100 may be configured to operate with an image fusion artificial neural network model.

To operate the image fusion artificial neural network model, the NPU 100 may be configured to have a plurality of batch channels (BATCH CH#1, BATCH CH#2) for fusion. Each deployment channel can be configured to include the same plurality of sensors. The first batch channel (BATCH CH#1) may be composed of a plurality of first sensors. At this time, there may be M number of first sensors. The Kth batch channel (BATCH CH#K) may be composed of a plurality of second sensors. At this time, there may be M number of second sensors.

The NPU 100 may reuse and process weights corresponding to inputs from the sensors 311 and 312 in the chip internal memory 120 through the first batch channel. In addition, the NPU 100 can reuse and process weights corresponding to inputs from the sensors 321 and 322 in the chip internal memory 120 through the second batch channel.

In this way, the NPU 100 can receive input from multiple sensors through a plurality of batch channels, reuse weights, and process an image fusion artificial neural network model in batch mode. A sensor of at least one channel among the plurality of deployment channels may be different from a sensor of at least one other channel.

The on-chip memory 120 within the NPU 100 may be set to have a storage space corresponding to a plurality of batch channels.

The NPU scheduler 130 within the NPU 100 may operate the PE array 110 according to the batch mode.

The SFU 160 within the NPU 100 may provide special functions for processing at least one fusion operation.

The NPU 100 may deliver each output through a plurality of deployment channels.

At least one channel among the plurality of placement channels may be inference data of an image fusion artificial neural network model network.

FIG. 24 is an example diagram showing an example of utilizing a plurality of NPUs according to the seventh example of the present disclosure, and FIG. 25 is a diagram illustrating the fusion artificial neural network shown in FIG. 12 through the plurality of NPUs shown in FIG. 24. This is an example diagram showing an example.

Referring to FIG. 24, a plurality of M NPUs, for example, may be used to generate a fusion image. Among the M NPUs, the first NPU (100-1) can process data provided from, for example, sensor #1 (311), and the M-th NPU (100-M) can process data provided by, for example, sensor #M (312). can be processed. The plurality of NPUs (eg, 100-1, 100-2) can access the chip-external memory 500 through ADC/DMA (Direct Memory Access) 400.

The plurality of NPUs (eg, 100-1 and 100-2) may obtain artificial neural network data locality information of the image fusion artificial neural network model from the compiler 200.

Each NPU can process an image fusion artificial neural network model and transfer operations for fusion to different NPUs through the ADC/DMA (400).

The ADC/DMA 400 may obtain data locality information for an artificial neural network of a fusion image fusion artificial neural network model from the compiler 200.

The compiler 200 converts the artificial neural network data locality information into data locality information #1 and data locality information so that operations that must be processed in parallel among the operations according to the artificial neural network data locality information of the image fusion artificial neural network model can be processed in each NPU. It can be created separately by information #M.

The chip-external memory 500 can store data that can be shared by a plurality of NPUs and transmit it to each NPU.

Referring to FIG. 25, NPU #1 may be in charge of the first artificial neural network for processing data provided from the camera, and NPU #2 may be in charge of the second artificial neural network for processing data provided from the thermal image sensor. You can. Additionally, the NPU #2 may be responsible for conversion for fusion of the first artificial neural network and the second artificial neural network.

So far, the NPU 100 for an image fusion artificial neural network model according to various examples of the present disclosure has been described. According to the present disclosure, a high-resolution thermal image can be generated by utilizing a high-resolution general visible light image sensor and a low-resolution thermal image sensor built into a general device, not a professional device. Accordingly, the present disclosure can generate high-resolution thermal images at low cost. In addition, the present disclosure can improve night-time identification of images not only in devices designed for night-time identification, but also in, for example, devices owned by the user or vehicle black boxes.

<A brief summary of the disclosures of this disclosure>

According to one disclosure of the present disclosure, a neural processing unit for an image fusion artificial neural network model is provided. The neural processing unit for the image fusion artificial neural network model inputs a first image and a second image having different resolutions and image characteristics, and generates a machine code of the image fusion artificial neural network model learned to output a new third image. A control unit configured to receive input; an input circuit configured to receive a plurality of input signals corresponding to the image fusion artificial neural network model; a processing element array configured to perform a main operation of the image fusion artificial neural network model operation; a special function unit circuit configured to perform special function calculation of the image fusion artificial neural network model; and an on-chip memory configured to store data of the main operation and/or the special function operation of the image fusion artificial neural network model; and the control unit, the processing element array, the special processing element array, so that the operation order of the image fusion artificial neural network model is processed in a preset order according to the data locality information of the image fusion artificial neural network model included in the machine code. configured to control a functional unit circuit and the on-chip memory, wherein the third resolution of the third image has a value between the first resolution of the first image and the second resolution of the second image, The third image characteristic of the three images may be at least partially the same as the first image characteristic of the first image or the second image characteristic of the second image.

The first image may be an image acquired through a visible light image sensor.

The second image may be an image acquired through a thermal image sensor.

The first image and the second image include different images of one object, and the image characteristics may be determined by the type of image sensor that acquires the first image and the second image.

The image fusion artificial neural network model may be an artificial neural network model configured to input only the first partial image and the second partial image corresponding to the face area from the object extracted from the first image and the second image.

The third image may be an image in which at least one feature that can be determined from the second image is applied to at least a partial area of the first image.

The image fusion artificial neural network model may be a model to which weights are applied to emphasize at least one feature that can be determined in the first image and at least one feature that can be determined in the second image.

The image fusion artificial neural network model may be an artificial neural network model configured to input only RGB values of the first image or brightness values for each pixel of the first image.

The third resolution of the third image may be the same as the first resolution of the first image.

The image fusion artificial neural network model is learned based on the GAN (generative adversarial networks) structure, and may correspond to a generator configured to generate a new image by inputting different images of one object.

The image fusion artificial neural network model is configured so that the generator constituting the GAN and a verifier (discriminator) that verifies the image generated by the generator compete with each other to update weights to increase the third resolution of the third image. It may be an artificial neural name model.

The image fusion artificial neural network model may be learned based on a learning data set of a format substantially similar to the first image and the second image.

The processing element array may be configured to process convolution and activation function operations.

The processing element array may be configured to process at least one of Dilated Convolution, Transposed Convolution, and Bilinear Interpolation operations to increase the third resolution of the third image.

It may further include an output unit configured to output the result of the at least one inference operation of the image fusion artificial neural network model learned to process at least one inference operation among classification, semantic segmentation, object detection, and prediction by the processing element array. You can.

The special function unit circuit may further include at least one of Skip-connection and Concatenation functions for artificial neural network fusion.

The control unit further includes a scheduler, wherein the scheduler preserves specific data stored in the on-chip memory based on data locality information of the image fusion artificial neural network model until a specific operation step of the image fusion artificial neural network model, It may be configured to control the on-chip memory.

The processing element array may further include a plurality of threads, and the controller may be configured to control the plurality of threads to process a parallel section of the image fusion artificial neural network model based on data locality of the image fusion artificial neural network model. there is.

According to another disclosure of the present disclosure, a system for an image fusion artificial neural network model is provided. The system includes a first sensor that acquires a first image having a first resolution and first image characteristics; a second sensor that acquires a second image having a second resolution smaller than the first resolution and second image characteristics different from the first image characteristics; and a neural processing unit configured to input a first image and a second image having different resolutions and image characteristics and process a learned image fusion artificial neural network model to output a new third image. Includes, the third resolution of the third image has a value between the first resolution of the first image and the second resolution of the second image, and the third image characteristic of the third image is, It may be at least partially the same as the first image characteristic of the first image or the second image characteristic of the second image.

Although examples of the present disclosure have been described in more detail with reference to the attached drawings, the present disclosure is not necessarily limited to these examples and may be modified and implemented in various ways without departing from the technical spirit of the present disclosure. Accordingly, the examples disclosed in the present disclosure are not intended to limit the technical idea of the present disclosure but are for illustrative purposes, and the scope of the technical idea of the present disclosure is not limited by these examples. Therefore, the examples described above should be understood in all respects as illustrative and not restrictive. The scope of protection of this disclosure should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this disclosure.

100: Neural Processing Unit (NPU)
110: Processing element (PE) 120: NPU internal memory
130: NPU scheduler 140: NPU interface
150: Kernel creation unit
200: Compiler
400: ADC 500: On-chip external memory

Claims (19)

A control unit configured to receive the machine code of an image fusion artificial neural network model learned to input a first image and a second image with different resolutions and different image characteristics for an object and output a new third image. ;
an input circuit configured to receive a plurality of input signals corresponding to the image fusion artificial neural network model;
a processing element array configured to perform main operations of the image fusion artificial neural network model;
a special function unit circuit configured to perform special function calculation of the image fusion artificial neural network model; and
an on-chip memory configured to store data of the main operation and/or the special function operation of the image fusion artificial neural network model; Including,
The control unit is configured to process the processing element array, the special function unit circuit, and the operation order of the image fusion artificial neural network model in a preset order according to data locality information of the image fusion artificial neural network model included in the machine code. Configured to control on-chip memory,
The third resolution of the third image is the same as or smaller than the first resolution of the first image, but has a higher value than the second resolution of the second image,
A neural processing unit for image fusion, wherein the third image characteristic of the third image is at least partially the same as the second image characteristic of the second image. According to paragraph 1,
The first image is an image acquired through a visible light image sensor. A neural processing unit for image fusion. According to paragraph 1,
The second image is an image acquired through a thermal image sensor. A neural processing unit for image fusion. According to paragraph 1,
The first image and the second image include different images of one object,
The image characteristics are determined by the type of image sensor that acquires the first image and the second image. According to clause 4,
The image fusion artificial neural network model is,
A neural processing unit for image fusion, characterized in that it is an artificial neural name model configured to input only the first partial image and the second partial image corresponding to the face region from the object extracted from the first image and the second image. According to paragraph 1,
The third image is an image in which at least one feature determinable from the second image is applied to at least a partial region of the first image. According to paragraph 1,
The image fusion artificial neural network model is,
A neural processing unit for image fusion, characterized in that it is a model to which weights are applied to emphasize at least one feature that can be determined in the first image and at least one feature that can be determined in the second image. According to paragraph 1,
The image fusion artificial neural network model is,
A neural processing unit for image fusion, characterized in that it is an artificial neural network model configured to input only the RGB value of the first image or the brightness value for each pixel of the first image. According to paragraph 1,
The third resolution of the third image is,
A neural processing unit for image fusion, wherein the first resolution is equal to the first resolution of the first image. According to paragraph 1,
The image fusion artificial neural network model is,
It is learned based on the GAN (generative adversarial networks) structure,
A neural processing unit for image fusion that corresponds to a generator configured to generate a new image by inputting different images of one object. According to clause 10,
The image fusion artificial neural network model is,
Characterized in that the generator constituting the GAN and a verifier (discriminator) that verifies the image generated by the generator compete with each other to update the weight to increase the third resolution of the third image. Characterized in that it is an artificial neural name model configured to update , Neural processing unit for image fusion. According to paragraph 1,
The image fusion artificial neural network model is,
A neural processing unit for image fusion that is learned based on a learning data set of a format substantially similar to the first image and the second image. According to paragraph 1,
The processing element array is,
A neural processing unit for image fusion, configured to process convolution, and activation function operations. According to paragraph 1,
The processing element array is,
A neural processing unit for image fusion, configured to process at least one of matrix multiplication, dilated convolution, transposed convolution, and bilinear interpolation operations to increase the third resolution of the third image. According to paragraph 1,
An output unit configured to output the result of the at least one inference operation of the image fusion artificial neural network model learned to process at least one inference operation among classification, semantic segmentation, object detection, pose estimation, and prediction by the processing element array. Further comprising: a neural processing unit for image fusion. According to paragraph 1,
The special function unit circuit further includes at least one function of skip-connection and concatenation for artificial neural network fusion. According to paragraph 1,
The control unit further includes a scheduler,
The scheduler is configured to control the on-chip memory to preserve specific data stored in the on-chip memory until a specific operation step of the image fusion artificial neural network model based on data locality information of the image fusion artificial neural network model. , Neural processing unit for image fusion. According to paragraph 1,
The processing element array further includes a plurality of threads,
The control unit is configured to control the plurality of threads to process parallel sections of the image fusion artificial neural network model based on data locality of the image fusion artificial neural network model. A first sensor that acquires a first image with first resolution and first image characteristics for one object;
a second sensor for acquiring a second image of the object having a second resolution smaller than the first resolution and second image characteristics different from the first image characteristics; and
a neural processing unit configured to input a first image and a second image having different resolutions and image characteristics and process an image fusion artificial neural network model learned to output a new third image; Including,
The third resolution of the third image is the same as or smaller than the first resolution of the first image, but has a higher value than the second resolution of the second image,
An artificial neural network system for image fusion, wherein the third image characteristics of the third image are at least partially the same as the second image characteristics of the second image.

Images