WO2023106723A1

WO2023106723A1 - Neural processing unit for image fusion, and artificial neural network system

Info

Publication number: WO2023106723A1
Application number: PCT/KR2022/019243
Authority: WO
Inventors: 김녹원
Original assignee: 주식회사 딥엑스
Priority date: 2021-12-08
Filing date: 2022-11-30
Publication date: 2023-06-15
Also published as: KR20240043735A; US20240265503A1

Abstract

According to one embodiment of the present disclosure, an artificial neural network system for image fusion is provided. The artificial neural network system comprises: a first sensor for acquiring a first image having a first resolution and a first image feature; a second sensor for acquiring a second image having a second resolution that is lower than the first resolution and a second image feature differing from the first image feature; and a neural processing unit, which inputs the first image and the second image having different resolutions and image features from each other, so as to process an image fusion artificial neural network model trained to output a new third image, wherein a third resolution of the third image has a value between the first resolution of the first image and the second resolution of the second image, and a third image feature of the third image can at least partially be the same as the first image feature of the first image or the second image feature of the second image.

Description

Neural processing unit and artificial neural network system for image fusion

The present disclosure relates to a neural processing unit and an artificial neural network system for image fusion.

A thermal image sensor provides a thermal image (thermal image) by collecting radiant energy emitted from an object and visualizing it even without an externally supplied light source during photographing.

Thermal imaging is largely divided into near-infrared (NIR), short-wave infrared (SWIR), medium-wave infrared (MWIR), and long-wave infrared (LWIR) according to infrared frequency bands. LWIR has been mainly used in special fields such as military and medical applications in the past. In recent years, the field of application of thermal imaging has increased, and it is also used for night object recognition of objects and automobiles.

In particular, the quality of thermal images is very important in object recognition technology at night. However, since the price of the thermal image sensor varies greatly depending on the resolution, the high-resolution infrared image sensor is economically burdensome.

Accordingly, a super-resolution (SR) technology for upscaling the resolution of an image obtained from a low-resolution thermal image sensor through an artificial intelligence-based learning algorithm has recently been disclosed.

However, when a thermal image is used as a single input data, there is a limit to expressing texture information of an image obtainable by a general visible light image sensor in high resolution. Accordingly, a technique for merging different kinds of images as input data is being developed. However, the conventional image convergence technology has a problem in that the amount of calculation is excessively large in the process of performing synchronization and matching in units of frames.

The background technology of this disclosure was created to facilitate understanding of this disclosure. It should not be construed as an admission that matters described in the art that form the background of this disclosure exist as prior art.

Accordingly, a Neural Processing Unit (NPU) and an artificial neural network system including the same are created to generate an image in which two features are fused based on images obtained from heterogeneous image sensors, but to minimize the amount of computation. It is required.

The inventors of the present disclosure have proposed a neural processing unit and an artificial neural network capable of effectively processing an artificial neural network model for generating an image satisfying a higher resolution by combining images having different resolutions and image characteristics for one object. I was trying to develop a system.

In particular, the inventors of the present disclosure performed a concatenation operation and a skip-connection operation capable of effectively processing different data, and a neural processing unit and an artificial neural network system capable of quickly generating high-resolution images. has come to develop

The tasks of the present disclosure are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, a neural processing unit for an image fusion artificial neural network model is provided. The neural processing unit for the image fusion artificial neural network model inputs a first image and a second image having different resolutions and image characteristics, and uses machine code of the image fusion artificial neural network model learned to output a new third image. a controller configured to receive an input; an input circuit configured to receive a plurality of input signals corresponding to the image fusion artificial neural network model; a processing element array configured to perform a main operation of the image fusion artificial neural network model; a special function unit circuit configured to perform a special function operation of the image fusion artificial neural network model operation; and an on-chip memory configured to store data of the main operation and/or the special function operation of the image fusion artificial neural network model. and the control unit, according to data locality information of the image fusion artificial neural network model included in the machine-code, so that the operation order of the image fusion artificial neural network model is processed in a preset order, the processing element array, the special configured to control a functional unit circuit and the on-chip memory, wherein a third resolution of the third image has a value between a first resolution of the first image and a second resolution of the second image; The third image characteristic of the 3 images may be at least partially the same as the first image characteristic of the first image or the second image characteristic of the second image.

According to another disclosure of the present disclosure, a system for an image fusion artificial neural network model is provided. The system may include: a first sensor for acquiring a first image having a first resolution and a first image characteristic; a second sensor configured to acquire a second image having a second resolution smaller than the first resolution and a second image characteristic different from the first image characteristic; and a neural processing unit configured to process an image convergence artificial neural network model trained to output a new third image by inputting the first image and the second image having different resolutions and image characteristics; wherein the third resolution of the third image has a value between the first resolution of the first image and the second resolution of the second image, and the third image characteristic of the third image comprises: The first image characteristic of 1 image or the second image characteristic of the second image may be at least partially the same.

Other example specifics are included in the detailed description and drawings.

The present disclosure can generate a high-resolution thermal image through an artificial neural network model. In particular, the present disclosure can prevent the problem of personal information exposure by generating and storing only high-resolution thermal images in a device for photographing an unspecified number of people, such as a surveillance camera. In addition, the present disclosure can protect personal information by converging thermal images only in the face region of a person.

In addition, a high-resolution thermal image may be generated by using a high-resolution general visible light image sensor and a low-resolution thermal image sensor built into a general device, not a specialized device. Accordingly, the present disclosure can generate a high-resolution thermal image at low cost. In addition, the present disclosure can improve the night vision of an image not in a device designed for night vision, but in a device owned by a user or a black box of a vehicle, for example.

In addition, according to the present disclosure, it is possible to generate a high-resolution thermal image for observing an object even on days when weather conditions are bad. In addition, according to the present disclosure, a high-resolution thermal image of an object can be generated even when the sensitivity of a laser sensor, an electromagnetic wave sensor, and an ultrasonic sensor used to sense an object other than an image sensor is low. Thus, the neural processing unit of the present disclosure may be installed in a vehicle to prevent a vehicle safety accident.

In addition, according to the present disclosure, a user's motion may be estimated (skeleton detection or pose estimation) using a high-resolution thermal image. For example, the present disclosure may estimate a user's abnormal movement or a user's fall through a device installed in a specific space.

In addition, according to the present disclosure, a neural processing unit for implementing an image convergence artificial neural network model that generates a new image based on images obtained from heterogeneous image sensors can be controlled to operate more efficiently. Thus, the present disclosure can reduce power consumption even when processing a huge amount of data. Accordingly, the present disclosure can implement an image convergence artificial neural network model in various devices without being limited by battery capacity.

In addition, the present disclosure can effectively process heterogeneous sensing data through a concatenation operation and a skip-connection operation. Therefore, according to the present disclosure, a high-resolution thermal image can be quickly generated while reducing the amount of computation.

In addition, according to the present disclosure, data stored in an on-chip memory can be maximally reused to minimize power consumption while obtaining data necessary for fusing high-resolution images from an external memory.

Effects according to the present disclosure are not limited by the contents exemplified above, and more diverse effects are included within the present disclosure.

1 and 2 are schematic conceptual diagrams illustrating an image convergence artificial neural network model according to an example of the present disclosure.

3 and 4 are schematic conceptual views illustrating an image generated through an image convergence artificial neural network model according to an example of the present disclosure.

5 is a schematic conceptual diagram illustrating a neural processing unit according to an example of the present disclosure.

FIG. 6 is a schematic conceptual diagram illustrating one processing element of the array of processing elements shown in FIG. 5 .

FIG. 7 is a conceptual diagram illustrating a modified example of the neural processing unit shown in FIG. 5 .

8 is a conceptual diagram illustrating an image convergence artificial neural network model according to an example of the present disclosure.

9 is a diagram for explaining a partial structure of a GAN neural network constituting an image convergence artificial neural network model according to an example of the present disclosure.

FIG. 10 is a diagram for explaining input data of the convolution layer shown in FIG. 9 and a kernel used for a convolution operation.

FIG. 11 is a diagram for explaining the operation of a convolutional neural network that generates a feature map using the kernel shown in FIG. 10 .

12 is a conceptual diagram illustrating an image convergence artificial neural network model according to an example of the present disclosure by way of example.

13 is an exemplary diagram illustrating a fusion method of an NPU according to an example of the present disclosure.

14 is a conceptual diagram illustrating a system including an NPU architecture according to a first example of the present disclosure by way of example.

15A is an exemplary diagram for explaining skip-connection included in the image convergence artificial neural network model according to the first example of the present disclosure.

FIG. 15B is an exemplary diagram illustrating locality information of artificial neural network data of the image convergence artificial neural network model shown in FIG. 15A.

16 is a conceptual diagram illustrating a system including an NPU architecture according to a second example of the present disclosure by way of example.

17 is a conceptual diagram illustrating a system including an NPU architecture according to a third example of the present disclosure by way of example.

18 is a conceptual diagram illustrating a system including an NPU architecture according to a fourth example of the present disclosure by way of example.

FIG. 19 shows an example in which the image convergence artificial neural network model shown in FIG. 12 is divided into threads according to the fourth example shown in FIG. 18 .

20 is a conceptual diagram illustrating a system including an NPU architecture according to a fifth example of the present disclosure by way of example.

21 is an exemplary view showing a first example of a pipeline structure of the SFU shown in FIG. 20 .

22a is an exemplary diagram illustrating an example of the SFU shown in FIG. 20;

22b is an exemplary diagram illustrating another example of the SFU shown in FIG. 20;

23 is a conceptual diagram illustrating a system including an NPU architecture according to a sixth example of the present disclosure by way of example.

24 is an exemplary diagram illustrating an example of utilizing a plurality of NPUs according to a seventh example of the present disclosure.

FIG. 25 is an exemplary diagram illustrating an example of processing the fusion artificial neural network shown in FIG. 12 through a plurality of NPUs shown in FIG. 24 .

Advantages and features of the present disclosure, and methods of achieving them, will become clear with reference to the examples described below in detail in conjunction with the accompanying drawings. However, the present disclosure is not limited to the examples disclosed below and will be implemented in various different forms. These examples are provided to make the disclosure of the present disclosure complete, and to fully inform those skilled in the art of the scope of the invention to which the present disclosure belongs. The disclosure is only defined by the scope of the claims. In connection with the description of the drawings, like reference numerals may be used for like elements.

In the present disclosure, expressions such as “has,” “can have,” “includes,” or “can include” indicate the presence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

In this disclosure, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, Or (3) may refer to all cases including at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” as used in the present disclosure may modify various components regardless of order and/or importance, and refer to one component as It is used only to distinguish it from other components and does not limit the corresponding components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, a first component may be named a second component without departing from the scope of rights described in the present disclosure, and similarly, the second component may also be renamed to the first component.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or connected through another component (eg, a third component). On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), the element and the above It may be understood that other components (eg, third components) do not exist between the other components.

The expression “configured to (or configured to)” as used in this disclosure means, depending on the situation, for example, “suitable for,” “having the capacity to.” ," "designed to," "adapted to," "made to," or "capable of." The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware. Instead, in some contexts, the phrase "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured (or configured) to perform A, B, and C" may include a dedicated processor (e.g., embedded processor) to perform those operations, or by executing one or more software programs stored in a memory device. , may mean a general-purpose processor (eg, CPU or application processor) capable of performing corresponding operations.

Terms used in this disclosure are used only to describe specific examples, and may not be intended to limit the scope of other examples. Singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the art described in this disclosure. Among the terms used in the present disclosure, terms defined in general dictionaries may be interpreted as the same as or similar to the meaning in the context of related art. In addition, terms defined in general dictionaries are not to be construed in an ideal or overly formal sense unless explicitly defined in this disclosure. In some cases, even terms defined in this disclosure cannot be interpreted to exclude examples of this disclosure.

Each feature of the various examples of the present disclosure may be partially or wholly combined or combined with each other. Each feature of the various examples of the present disclosure can be technically variously interlocked and driven as can be fully understood by those skilled in the art, and each example may be implemented independently of each other or together in an association relationship.

For clarity of interpretation of the present disclosure, terms used in the present disclosure will be defined below.

NPU is an abbreviation of Neural Processing Unit, and may mean a specialized processor for computation of an artificial neural network model, separate from a central processing unit (CPU).

ANN is an abbreviation of artificial neural network. In order to imitate human intelligence, neurons in the human brain are connected through synapses, and nodes are layered. It can mean a network connected by a structure.

The artificial neural network model is a model for image convergence and may be a model trained to perform inference such as Image/Video Reconstruction and Image/Video Enhancement.

In addition, other artificial neural network models that take fused images as input are trained to perform inference such as Object Classification, Object Detection, Object Segmentation, Object Tracking, Event Recognition, Event Prediction, Anomaly Detection, Density Estimation, Event Search, and Measurement. can be a model

For example, artificial neural network models include Transformer, Bisenet, Shelfnet, Alexnet, Densenet, Efficientnet, EfficientDet, Googlenet, Mnasnet, Mobilenet, Resnet, Shufflenet, Squeezenet, VGG, Yolo, RNN, CNN, DBN, RBM, LSTM, etc. can be However, the present disclosure is not limited thereto, and may be a new artificial neural network model other than those operable in the NPU 100.

Information on the structure of an artificial neural network includes information on the number of layers, the number of nodes in a layer, the value of each node, information on the calculation method, and information on the weight matrix applied to each node.

The information on the data locality of the image fusion artificial neural network model is information including an order of data access requests to the memory determined based on the artificial neural network and the structure of a neural processing unit processing the artificial neural network.

DNN is an abbreviation of deep neural network, and may mean that the number of hidden layers of an artificial neural network is increased in order to implement higher artificial intelligence.

CNN is an abbreviation of Convolutional Neural Network, which is a neural network that functions similarly to image processing in the visual cortex of the human brain. Convolutional neural networks are known to be suitable for image processing, and are known to be easy to extract features of input data and identify patterns of features.

A kernel may mean a weight matrix applied to CNN.

The off-chip memory may be a memory arranged in consideration of a limited memory size inside the NPU. That is, a memory may be disposed outside the chip to store large-capacity data. The off-chip memory may include one of memories such as ROM, SRAM, DRAM, resistive RAM, magneto-resistive RAM, phase-change RAM, ferroelectric RAM, flash memory, HBM, and the like. The off-chip memory may consist of at least one memory unit. The off-chip memory may consist of a single (homogeneous) memory unit or a heterogeneous (heterogeneous) memory unit.

On-Chip Memory The NPU may include on-chip memory. On-chip memory may include volatile memory and/or non-volatile memory. For example, the on-chip memory may include one of memories such as ROM, SRAM, DRAM, resistive RAM, magneto-resistive RAM, phase-change RAM, ferroelectric RAM, flash memory, HBM, and the like. The on-chip memory may consist of at least one memory unit. The on-chip memory may be composed of a single (homogeneous) memory unit or a heterogeneous (heterogeneous) memory unit.

Hereinafter, an example of the present disclosure will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1 , a neural processing unit 100 includes a processing element 102 configured to perform an operation of an image fusion artificial neural network model 101 . The neural processing unit 100 may generate a new image (third image) based on the two images (first image and second image). The first image and the second image are different images of one object, and may be obtained from different types of sensors. For example, the heterogeneous sensor may be an image sensor for capturing visible light, an image sensor for capturing infrared light, and the like.

An image sensor for capturing visible light acquires a color image (first image) in the visible light region through R (Red), G (Green), and B (Blue) pixels. The image sensor for infrared imaging may acquire a thermal color map (second image) through pixels. In general, since an infrared image sensor detects energy having a wavelength greater than that of visible light, the number of pixels, that is, the resolution, is inevitably low even if the sensor has the same size. In the present disclosure, it is possible to generate a third image that satisfies the resolution of a visible ray imaging image sensor through the image convergence artificial neural network model 101 without using a high-resolution infrared imaging image sensor.

Accordingly, the neural processing unit 100 may be a model learned to output a new third image by inputting the first image and the second image having different resolutions and image characteristics. The third resolution of the third image may have a value between the first resolution of the first image and the second resolution of the second image. For example, the third resolution of the third image may be the same as the first resolution of the first image. Also, the third image characteristic of the third image may be at least partially the same as the image characteristic of the first image or the second image characteristic of the second image. For example, when the first resolution of each RGB channel of the first image is 1024 x 786 and the second resolution of the thermal image channel of the second image is 100 x 60, the third resolution of the third image is 1024 x 786 can That is, the resolution of the third image is the same as the size and the first resolution of the first image, and a thermal image corresponding to the second image characteristic may be applied to the third image characteristic.

In another example, the neural processing unit 100 may use different input data from the image fusion artificial neural network model 101 . Specifically, when the first image is a color image, the processing speed of the neural processing unit 100 may increase. Accordingly, the image convergence artificial neural network model 101 takes only the brightness value of each pixel of the first image as an input, and generates a third image identical to that of the color image as an input, while reducing the processing speed of the neural processing unit 100 can make it That is, the image convergence artificial neural network model 101 may be a model configured to input only RGB values (3 channels) of the first image or brightness values (1 channel) for each pixel of the first image.

In this way, the weight parameters of the image convergence artificial neural network model 101 capable of generating a new image by combining characteristics of different images may be learned based on a generative adversarial networks (GAN) structure. The GAN structure is composed of a generator that generates virtual or real images and a discriminator that determines whether the image generated by the generator is authentic or not. A GAN can be a model in which generators and verifiers compete against each other to improve each other's performance. Specifically, the generator provides real data to the verifier so that the verifier learns to determine the video as real, and secondly inputs the fake data generated by the generator so that the verifier can fake the video. can be learned to discriminate with Then, the generator can then develop to create a harmonious image according to competition with each other by learning to create a virtual image to deceive the verifier.

That is, in the learning step, the image convergence artificial neural network model 101 may be a model configured such that generators and verifiers constituting the GAN compete with each other to update weights for increasing the third resolution of the third image.

In the present disclosure, in order to minimize memory usage of the neural processing unit 100, image generation learning through a generator may be performed in a separate device/server (not shown). In addition, the image convergence artificial neural network model 101 calculated by the neural processing unit 100 takes different images of one object (eg, 1. high-resolution visible light image, 2. low-resolution thermal image) as inputs to create a new image ( eg high-resolution thermal image) may correspond to a generator configured to generate.

Meanwhile, the image fusion artificial neural network model 101 may be learned based on a set of learning data substantially similar to the first image and the second image. That is, an image used for learning may be different from an image received to generate a new image thereafter.

Hereinafter, an image generated through an image convergence artificial neural network model will be described as an example.

Referring to FIG. 3 , the image fusion artificial neural network model may take as input images acquired from different types of sensors. For example, the heterogeneous sensor may be an image sensor for capturing visible light, an image sensor for capturing infrared light, and the like. The image convergence artificial neural network model may take as inputs a first image and a second image obtained by different image sensors for one object. Also, the image convergence artificial neural network model may generate an image in which image characteristics of each image sensor are fused based on images acquired from different types of image sensors. That is, the image convergence artificial neural network model can generate a third image in which temperature information (second image characteristics) of the second image is reflected while maintaining the size and resolution of the first image.

In another example, the image fusion artificial neural network model may be a model to which weights are applied to emphasize at least one feature determinable from the first image and at least one feature determinable from the second image. Specifically, the neural processing unit 100 may generate features of the first image and the second image. For example, the neural processing unit 100 may generate a feature map by inferring features of high-resolution edge content from a first image, which is a color image. A feature map may be referred to as a heat map, an activation map, or a parameter.

In addition, the neural processing unit 100 may generate a feature map by inferring segmentation according to temperature in the second image, which is a thermal image. Subsequently, image fusion of the high-resolution thermal image may be processed based on the high-resolution edge feature map and the low-resolution temperature segmentation feature map.

For some examples, the neural processing unit 100 may further detect the presence of an object as a feature of the first image, and may determine a region having a temperature greater than or equal to a threshold value as a feature of the second image. Accordingly, the image convergence artificial neural network model may generate a new image by applying weights to determinable features in each image. For example, as shown in FIG. 4 , the neural processing unit 100 may detect an object (human) in the first image and thermally image only a region having a specific temperature or higher within the region where the object is detected. That is, the neural processing unit 100 may generate a third image to which at least one feature that can be determined from the second image is applied to at least a partial region of the first image.

Meanwhile, in order to generate a high-resolution thermal image limited to a partial region as described above, the processing element array of the neural processing unit 100 may be configured to process at least one operation among dilated convolution, transposed convolution, and bilinear interpolation operations. .

In another example, the neural processing unit 100 may extract a first partial image and a second partial image corresponding to the face region from the first image and the second image obtained by capturing one object. Accordingly, the neural processing unit 100 may generate a fused image of image features only on the face using an artificial neural network unit configured to input only the first partial image and the second partial image.

In this way, the neural processing unit 100 fusion-generates an image in which different image characteristics are applied only to the human face region, thereby protecting personal information.

Hereinafter, the neural processing unit 100 that performs an operation of an image convergence artificial neural network model will be described.

As shown in FIG. 5, NPU 100 is a processor specialized to perform an operation for an image convergence artificial neural network model.

An artificial neural network refers to a network of artificial neurons that multiply and add weights when various inputs or stimuli come in, and transform and transmit the value obtained by adding additional deviations through an activation function. The artificial neural network trained in this way can be used to output an inference result from input data.

The NPU 100 may be a semiconductor implemented as an electric/electronic circuit. An electric/electronic circuit may mean including a number of electronic devices (eg, transistors and capacitors).

Referring to FIG. 5 , the NPU 100 may include a processing element (PE) ray 110, an NPU internal memory 120, an NPU scheduler 130, and an NPU interface 140. Each of the processing element array 110 , NPU internal memory 120 , NPU scheduler 130 , and NPU interface 140 may be a semiconductor circuit to which numerous transistors are connected. Accordingly, some of them may be difficult to discern and distinguish with the naked eye, and may only be identified by motion. For example, an arbitrary circuit may operate as the processing element array 110 or as the NPU scheduler 130 . The NPU scheduler 130 may be configured to perform a function of a control unit configured to control an artificial neural network inference operation of the NPU 100 . To elaborate, a part of the control unit may be referred to as the scheduler 130 . NPU scheduler 130 may be part of the control unit. The NPU scheduler 130 may also be referred to as a control unit. The control unit may include the NPU scheduler 130. The controller may be a common name for circuits that perform various control functions of the NPU 100, such as direct memory access (DMA). It is also possible that the controller is defined by the function of the circuit. In other words, a circuit for controlling the processing element array 110 according to the order of each operation step of the artificial neural network model based on the locality of the artificial neural network data of the artificial neural network model by the controller may be defined as the NPU scheduler 130.

The NPU 100 includes a processing element array 110, an NPU internal memory 120 configured to store an image fusion artificial neural network model that can be inferred from the processing element array 110, and artificial neural network data locality of the image fusion artificial neural network model. It may include an NPU scheduler 130 configured to control the processing element array 110 and the NPU internal memory 120 based on information or structure information. Here, the NPU internal memory 120 may store information on locality or structure of artificial neural network data of an image convergence artificial neural network model. That is, the image convergence artificial neural network model may refer to an AI recognition model learned to perform a specific reasoning function (eg, image convergence, object movement, object posture, motion tracking, etc.).

The processing element array 110 may perform operations for an artificial neural network.

The NPU interface 140 may communicate with various components, such as memory, connected to the NPU 100 through a system bus (eg, one or more communication buses or signal lines).

The NPU scheduler 130 is configured to control an operation of the processing element array 100 for an inference operation of the neural processing unit 100 and a reading and writing order of the NPU internal memory 120 .

The NPU scheduler 130 may be configured to control the processing element array 100 and the NPU internal memory 120 based on locality information or structure information of artificial neural network data of an image convergence artificial neural network model.

The NPU scheduler 130 may analyze the structure of the image fusion artificial neural network model to be operated in the processing element array 100 or may receive previously analyzed information. The analyzed information may be included in machine code. For example, artificial neural network data that can be included in an image convergence artificial neural network model include node data (i.e., feature map) of each layer, arrangement data of layers, locality information or structure information, and node data of each layer. It may include at least some of the weight data (ie, weight kernel) of each network connecting the . Data of the artificial neural network may be stored in memory provided inside the NPU scheduler 130 or in the NPU internal memory 120 . NPU scheduler 130 may be operated by machine code.

The NPU scheduler 130 may schedule an operation order of the image fusion artificial neural network model to be performed by the NPU 100 based on artificial neural network data locality information or structure information of the image fusion artificial neural network model. Machine code may include scheduling data. NPU scheduler 130 may operate according to scheduling included in machine code. That is, the NPU scheduler 130 may be configured to operate by machine code.

The NPU scheduler 130 may obtain memory address values at which feature maps and weight data of layers of the image fusion artificial neural network model are stored based on locality information or structure information of artificial neural network data of the image fusion artificial neural network model. For example, the NPU scheduler 130 may obtain a memory address value at which a feature map and weight data of a layer of an image convergence artificial neural network model stored in a memory are stored. Accordingly, the NPU scheduler 130 may retrieve feature maps and weight data of layers of an image convergence artificial neural network model to be driven from the memory 200 and store them in the NPU internal memory 120 .

A feature map of each layer may have a corresponding memory address value.

Each weight data may have a corresponding memory address value of the NPU internal memory 120 .

The NPU scheduler 130 is based on locality information or structure information of the artificial neural network data of the image convergence artificial neural network model, for example, arrangement data locality information or structure information of the layers of the artificial neural network of the image convergence artificial neural network model. The order of operations of the processing element array 110 may be scheduled.

Since the NPU scheduler 130 performs scheduling based on locality information or structure information of artificial neural network data of an image convergence artificial neural network model, it may operate differently from a general CPU scheduling concept. General CPU scheduling takes into account fairness, efficiency, stability, response time, etc., and operates to achieve the best efficiency. That is, it is scheduled to perform the most processing within the same time considering priority, operation time, and the like.

A conventional CPU uses an algorithm for scheduling tasks in consideration of data such as the priority order of each processing and operation processing time.

Unlike this, the NPU scheduler 130 may control the NPU 100 in the processing order of the NPU 100 determined based on information on the locality or structure of the artificial neural network data of the image convergence artificial neural network model.

Furthermore, the NPU scheduler 130 determines based on the information on the artificial neural network data locality information or structure of the image fusion artificial neural network model and / or the information on the data locality information or structure of the neural processing unit 100 to be used. The NPU 100 may be driven in the processing order.

However, the present disclosure is not limited to data locality information or structure information of the NPU 100 .

The NPU scheduler 130 may be configured to store data locality information or structure information of an artificial neural network.

That is, the NPU scheduler 130 may determine the processing order even when at least using only the locality information or structure information of the artificial neural network data of the image convergence artificial neural network model.

Furthermore, the NPU scheduler 130 determines the processing order of the NPU 100 in consideration of the artificial neural network data locality information or structure information of the image convergence artificial neural network model and the data locality information or structure information of the NPU 100. can decide In addition, processing optimization of the NPU 100 may be performed according to the determined processing order.

The processing element array 110 means a configuration in which a plurality of processing elements PE1 to PE12 configured to calculate feature maps and weight data of an artificial neural network are disposed. Each processing element may include a multiply and accumulate (MAC) operator and/or an arithmetic logic unit (ALU) operator. However, examples according to the present disclosure are not limited thereto.

Although a plurality of processing elements are illustratively shown in FIG. 5, it is also possible that operators implemented as a plurality of multipliers and adder trees are arranged in parallel by replacing the MAC inside one processing element. possible. In this case, the processing element array 110 may also be referred to as at least one processing element including a plurality of operators.

The processing element array 110 is configured to include a plurality of processing elements PE1 to PE12. The plurality of processing elements PE1 to PE12 shown in FIG. 3 are merely examples for convenience of explanation, and the number of the plurality of processing elements PE1 to PE12 is not limited. The size or number of the processing element array 110 may be determined by the number of the plurality of processing elements PE1 to PE12 . The size of the processing element array 110 may be implemented in the form of an N x M matrix. where N and M are integers greater than zero. Processing element array 110 may include N x M processing elements. That is, there may be one or more processing elements.

The size of the processing element array 110 may be designed in consideration of the characteristics of an image convergence artificial neural network model in which the NPU 100 operates. Accordingly, a utilization rate % of the processing element array 110 may be improved.

The processing element array 110 is configured to perform functions such as addition, multiplication, and accumulation required for artificial neural network operations. Stated differently, the processing element array 110 may be configured to perform multiplication and accumulation (MAC) operations.

The processing element array 110 may be configured to quantize and output MAC operation results. However, examples of the present disclosure are not limited thereto.

The NPU internal memory 120 may store all or part of the image fusion artificial neural network model according to the memory size and the data size of the image fusion artificial neural network model.

Hereinafter, the first processing element PE1 of the processing element array 110 will be described as an example.

Referring to FIG. 6 , the first processing element PE1 may include a multiplier 111 , an adder 112 , an accumulator 113 , and a bit quantization unit 114 . However, examples according to the present disclosure are not limited thereto, and the processing element array 110 may be modified in consideration of the computational characteristics of an artificial neural network.

The multiplier 111 multiplies the received (N) bit data and (M) bit data. The operation value of the multiplier 111 is output as (N+M) bit data.

The multiplier 111 may be configured to receive input of one variable and one constant.

The accumulator 113 accumulates the operation value of the multiplier 111 and the operation value of the accumulator 113 by using the adder 112 as many times as the number of (L) loops. Accordingly, the bit width of the data of the output unit of the accumulator 113 and the input circuit may be output as (N+M+log2(L)) bits. where L is an integer greater than zero.

When the accumulation is completed, the accumulator 113 may receive an initialization reset to initialize data stored in the accumulator 113 to 0. However, examples according to the present disclosure are not limited thereto.

The bit quantization unit 114 may reduce the bit width of data output from the accumulator 113 . The bit quantization unit 114 may be controlled by the NPU scheduler 130. The bit width of the quantized data may be output as (X) bits. where X is an integer greater than zero. According to the configuration described above, the processing element array 110 is configured to perform a MAC operation, and the processing element array 110 has an effect of quantizing and outputting a result of the MAC operation. In particular, such quantization has an effect of further reducing power consumption as (L) loops increase. In addition, when power consumption is reduced, there is an effect of reducing heat generation. In particular, reducing heat generation has an effect of reducing the possibility of malfunction due to high temperature of the NPU 100 .

The output data (X) bits of the bit quantization unit 114 may be node data of the next layer or input data of convolution. If the image fusion artificial neural network model is quantized, the bit quantization unit 114 may be configured to receive quantized feature maps and/or weights from the image fusion artificial neural network model. However, it is not limited thereto, and the NPU scheduler 130 may also be configured to extract quantized information by analyzing the image convergence artificial neural network model. Therefore, to correspond to the size of the quantized data, the output data (X) bits may be converted into a quantized bit width and then output. The output data (X) bits of the bit quantization unit 114 may be stored in the NPU internal memory 120 with a quantized bit width.

That is, the processing element array 110 of the NPU 100 according to an example of the present disclosure may include a multiplier 111, an adder 112, an accumulator 113, and a bit quantization unit 114.

Hereinafter, another example of the NPU 100 of the present disclosure will be described.

Prior to describing FIG. 7, since the NPU 100 shown in FIG. 7 is substantially the same as the processing unit 100 exemplarily shown in FIG. 5, except for the processing element array 110, For convenience of description below, redundant descriptions are omitted.

Referring to FIG. 7 , the element array 110 further includes, in addition to the plurality of processing elements PE1 to PE12, respective register files RF1 to RF12 corresponding to each of the processing elements PE1 to PE12. can do.

However, the plurality of processing elements PE1 to PE12 and the plurality of register files RF1 to RF12 shown in FIG. 7 are merely examples for convenience of description, and the plurality of processing elements PE1 to PE12 and the plurality of register files RF1 to RF12 The number of register files (RF1 to RF12) of is not limited.

The size or number of the processing element array 110 may be determined by the number of the plurality of processing elements PE1 to PE12 and the plurality of register files RF1 to RF12. The size of the processing element array 110 and the plurality of register files RF1 to RF12 may be implemented in the form of an NxM matrix. where N and M are integers greater than zero.

The array size of the processing element array 110 may be designed in consideration of the characteristics of an image convergence artificial neural network model in which the NPU 100 operates. To elaborate, the memory size of the register file may be determined in consideration of the data size of the image convergence artificial neural network model to operate, the required operation speed, and the required power consumption.

The register files RF1 to RF12 of the NPU 100 are static memory units directly connected to the processing elements PE1 to PE12. The register files RF1 to RF12 may be composed of, for example, flip-flops and/or latches. The register files RF1 to RF12 may be configured to store MAC operation values of corresponding processing elements PE1 to PE12. The register files RF1 to RF12 may be configured to provide or receive weight data and/or node data from the NPU internal memory 120 .

The register files RF1 to RF12 may also be configured to perform the function of the temporary memory of the accumulator during MAC operation.

In the register files RF1 to RF12, power consumption may be reduced by temporarily storing the calculated feature map and then reusing the feature map in the next calculation.

Hereinafter, calculation of an exemplary image fusion artificial neural network model 110-10 that can be operated in the NPU 100 will be described.

The image convergence artificial neural network model 110-10 of FIG. 8 may be an artificial neural network learned in the NPU 100 shown in FIG. 5 or 7 or learned in a separate machine learning device. The image convergence artificial neural network model 110 - 10 may be an artificial neural network trained to perform various inference functions such as motion and posture estimation of an object in an image.

The image fusion artificial neural network model 110 - 10 may be a deep neural network (DNN). However, the image fusion artificial neural network model 110 - 10 according to examples of the present disclosure is not limited to a deep neural network.

For example, the image convergence artificial neural network model may be a model trained to perform inference such as Image/Video Reconstruction and Image/Video Enhancement.

In addition, other artificial neural network models that take fused images as input are Super-resolution, Upscaling, Image fusion, Object Classification, Object Detection, Object Segmentation, Object Tracking, Event Recognition, Event Prediction, Anomaly Detection, Density Estimation, Event Search, It may be a model trained to perform inference such as measurement.

For example, image fusion artificial neural network models include Transformer, Bisenet, Shelfnet, Alexnet, Densenet, Efficientnet, EfficientDet, Googlenet, Mnasnet, Mobilenet, Resnet, Shufflenet, Squeezenet, VGG, Yolo, RNN, CNN, DBN, RBM, LSTM, etc. can be a model of However, the present disclosure is not limited thereto, and may be a new artificial neural network model other than those operable in the NPU 100.

In various examples, the image fusion artificial neural network model 110 - 10 may be an ensemble model based on at least two different models.

At least some of parameters such as weight values, node values, accumulated values, feature maps, and weights of each layer of the image convergence artificial neural network model 110-10 may be stored in the NPU internal memory 120 of the NPU 100. there is.

Specifically, referring to FIG. 8 , the inference process by the image convergence artificial neural network model 110-10 may be performed by the NPU 100.

The image fusion artificial neural network model 110-10 includes an input layer 110-11, a first connection network 110-12, a first hidden layer 110-13, a second connection network 110-14, and a second hidden layer. An exemplary deep neural network model including a layer 110-15, a third network 110-16, and an output layer 110-17. However, the present disclosure is not limited to the image fusion artificial neural network model shown in FIG. 8 . The first hidden layer 110-13 and the second hidden layer 110-15 may also be referred to as a plurality of hidden layers.

The input layer 110 - 11 may illustratively include x1 and x2 input nodes. That is, the input layer 110-11 may include information on two input values. The NPU scheduler 130 shown in FIG. 5 or 7 sets the memory address where the information on the input value from the input layer 110-11 is stored in the NPU internal memory 120 shown in FIG. 5 or 7. can

The first connection network 110-12 exemplarily provides information about six weight values for connecting each node of the input layer 110-11 to each node of the first hidden layer 110-13. can include The NPU scheduler 130 shown in FIG. 5 or 7 may set a memory address in which information about weight values of the first connection network 110-12 is stored in the NPU internal memory 120. Each weight value is multiplied with the input node value, and the accumulated value of the multiplied values is stored in the first hidden layer 110-13. Here, nodes having accumulated values may be referred to as feature maps.

The first hidden layer 110 - 13 may illustratively include nodes a1 , a2 , and a3 . That is, the first hidden layers 110-13 may include information about three node values. The NPU scheduler 130 shown in FIG. 5 or 7 may set a memory address for storing information about node values of the first hidden layers 110-13 in the NPU internal memory 120.

The NPU scheduler 130 may be configured to schedule an operation order such that the first processing element PE1 performs the MAC operation of the a1 node of the first hidden layer 110 - 13 . The NPU scheduler 130 may be configured to schedule an operation sequence so that the second processing element PE2 performs the MAC operation of the a2 node of the first hidden layer 110 - 13 . The NPU scheduler 130 may be configured to schedule an operation sequence so that the third processing element PE3 performs the MAC operation of the a3 node of the first hidden layer 110 - 13 . Here, the NPU scheduler 130 may pre-schedule an operation sequence such that three processing elements perform MAC operations in parallel and simultaneously. The scheduling information may be included in machine code. Accordingly, the NPU scheduler 130 may operate according to scheduling information included in machine code.

The second network 110-14 is illustratively, for nine weight values for connecting each node of the first hidden layer 110-13 to each node of the second hidden layer 110-15 information may be included. The NPU scheduler 130 shown in FIG. 5 or 7 may set a memory address for storing information on the weight value of the second connection network 110-14 in the NPU internal memory 120. The weight value of the second network 110-14 is multiplied with the node value input from the first hidden layer 110-13, and the accumulated value of the multiplied values is applied to the second hidden layer 110-15. Saved.

The second hidden layer 110 - 15 may illustratively include b1, b2, and b3 nodes. That is, the second hidden layers 110-15 may include information about three node values. The NPU scheduler 130 may set a memory address for storing information about node values of the second hidden layer 110-15 in the NPU internal memory 120.

The NPU scheduler 130 may be configured to schedule an operation sequence so that the fourth processing element PE4 performs the MAC operation of the b1 node of the second hidden layer 110 - 15 . The NPU scheduler 130 may be configured to schedule an operation sequence so that the fifth processing element PE5 performs the MAC operation of the node b2 of the second hidden layer 110-15. The NPU scheduler 130 may be configured to schedule an operation sequence so that the sixth processing element PE6 performs the MAC operation of the b3 node of the second hidden layer 110 - 15 . The scheduling information may be included in machine code.

Here, the NPU scheduler 130 may pre-schedule an operation sequence such that three processing elements perform MAC operations in parallel and simultaneously.

Here, the NPU scheduler 130 may determine scheduling such that the operation of the second hidden layer 110-15 is performed after the MAC operation of the first hidden layer 110-13 of the image fusion artificial neural network model.

That is, the NPU scheduler 130 may be configured to control the processing element array 100 and the NPU internal memory 120 based on locality information or structure information of the artificial neural network data of the image convergence artificial neural network model.

The third network 110-16 illustratively includes information about six weight values connecting each node of the second hidden layer 110-15 and each node of the output layer 110-17. can do. The NPU scheduler 130 may set a memory address for storing information about weight values of the third connection networks 110-16 in the NPU internal memory 120. The weight value of the third network 110-16 is multiplied with the node value input from the second hidden layer 110-15, and the accumulated value of the multiplied values is stored in the output layer 110-17.

The output layer 110-17 may illustratively include nodes y1 and y2. That is, the output layers 110-17 may include information about two node values. The NPU scheduler 130 may set a memory address in the NPU internal memory 120 to store information on node values of the output layers 110-17.

The NPU scheduler 130 may be configured to schedule an operation order such that the seventh processing element PE7 performs the MAC operation of the node y1 of the output layer 110-17. The NPU scheduler 130 may be configured to schedule an operation order such that the eighth processing element PE8 performs the MAC operation of the y2 node of the output layer 110-15. The scheduling information may be included in machine code.

Here, the NPU scheduler 130 may pre-schedule an operation sequence such that two processing elements perform MAC operations in parallel and simultaneously.

Here, the NPU scheduler 130 may determine scheduling such that the operation of the output layer 110-17 is performed after the MAC operation of the second hidden layer 110-15 of the image convergence artificial neural network model.

That is, the NPU scheduler 130 may analyze the structure of the image convergence artificial neural network model to be operated in the processing element array 100 or may receive analyzed information. The artificial neural network information that can be included in the image convergence artificial neural network model is information on the node value of each layer, information on the locality or structure of the arrangement data of the layers, and the weight value of each network connecting the nodes of each layer. may contain information about

Since the NPU scheduler 130 is provided with information on the artificial neural network data locality information or structure of the exemplary image fusion artificial neural network model 110-10, the NPU scheduler 130 generates the image fusion artificial neural network model 110-10. You can understand the operation sequence from input to output.

Accordingly, the NPU scheduler 130 may set a memory address at which MAC calculation values of each layer are stored in the NPU internal memory 120 in consideration of a scheduling order.

The NPU internal memory 120 may be configured to preserve weight data of networks stored in the NPU internal memory 120 while the inference operation of the NPU 100 continues. Accordingly, there is an effect of reducing a memory read/write operation.

That is, the NPU internal memory 120 may be configured to reuse the MAC operation value stored in the NPU internal memory 120 while the inference operation continues.

Hereinafter, the structure of the image convergence artificial neural network model of the present disclosure will be described with reference to FIGS. 9 to 12.

9 is a diagram for explaining a partial structure of a GAN constituting an image convergence artificial neural network model according to an example of the present disclosure.

Referring to FIG. 9 , the GAN neural network structure constituting the image convergence artificial neural network model has a structure corresponding to a generator for generating high-resolution thermal images. That is, the scheduler 130 of the neural processing unit 100 may be configured to process an inference operation by receiving a machine code compiled from an image convergence artificial neural network model excluding a discriminator.

In one example, the image fusion artificial neural network model corresponding to the generator may use a visible light image of 3 RGB channels and a thermal image of 1 channel as input data, and perform a convolution (convolution) operation to which an activation function (ELU) is applied. performed to output a feature map and/or an activation map. For example, input data of a visible light image may be calculated by sliding 64 3x3 filters for each channel, and input data of a thermal image may be calculated by sliding 64 3x3 filters. That is, the size of the feature map of the input data of the visible light image may be reduced to the same size as that of the feature map output from the input data of the thermal image before image fusion. The output feature maps output through each operation may be merged into one filter having a size of 1 × 1. The feature maps merged in this way can transfer output results to other layers through a skip-connection operation, and finally generate a high-resolution thermal image through a plurality of layers. 9 is just one example for configuring a generator in a GAN, and is not limited thereto, and configurations of various models may be employed.

FIG. 10 is a diagram for explaining input data of the convolution layer shown in FIG. 9 and a kernel used for a convolution operation or matrix multiplication.

Referring to FIG. 10 , input data 300 may be an image or video displayed in a two-dimensional matrix composed of rows 310 of a specific size and columns 320 of a specific size. The input data 300 may be referred to as a feature map. The input data 300 may have a plurality of channels 330, where the channels 330 may represent color RGB channels of the input data image.

Meanwhile, the kernel 340 may be a weight parameter used in convolution for extracting a feature of a certain portion of the input data 300 while scanning it. Like the input data image, the kernel 340 may be configured to have rows 350 of a specific size, columns 360 of a specific size, and a specific number of channels 370 . In general, the sizes of rows 350 and columns 360 of the kernel 340 are set to be the same, and the number of channels 370 may be the same as the number of channels 330 of the input data image.

Referring to FIG. 11 , the kernel 410 may traverse the input data 420 at designated intervals and perform convolution, thereby finally generating a feature map 430 . In convolution, when the kernel 410 is applied to a portion of the input data 420, the input data values at a specific location of the portion are multiplied by the values at the corresponding location of the kernel 410, and then the generated values are added together. can be executed

Through this convolution process, calculated values of feature maps are generated, and whenever the kernel 410 traverses the input data 420, these convolution result values are generated to configure the feature map 430. .

Each component value of the feature map may be converted into an activation map 430 through an activation function of a convolution layer.

In FIG. 11 , the input data 420 input to the convolution layer is displayed as a 2-dimensional matrix having a size of 4×4, and the kernel 410 is displayed as a 2-dimensional matrix having a size of 3×3. However, the sizes of the input data 420 and the kernel 410 of the convolution layer are not limited thereto, and may be variously changed according to the performance and requirements of the convolution neural network including the convolution layer.

As shown, when input data 420 is input to the convolution layer, the kernel 410 traverses the input data 420 at a predetermined interval (eg, stride = 1), and the input data 420 It is possible to perform a MAC operation that multiplies and values at the same location of the kernel 410 and adds the respective values.

Specifically, the kernel 410 assigns the MAC operation value “15” calculated at the specific position 421 of the input data 420 to the corresponding element 431 of the feature map 430. The kernel 410 assigns the MAC operation value “16” calculated at the next position 422 of the input data 420 to the corresponding element 432 of the feature map 430. The kernel 410 assigns the MAC operation value “6” calculated at the next position 423 of the input data 420 to the corresponding element 433 of the feature map 430. Next, the kernel 410 allocates the MAC operation value “15” calculated at the next position 424 of the input data 420 to the corresponding element 434 of the feature map 430.

In this way, if the kernel 410 traverses the input data 420 and assigns all MAC calculation values to the feature map 430, the feature map 430 having a size of 2x2 can be completed.

At this time, if the input data 510 is composed of, for example, three channels (R channel, G channel, B channel), the same kernel or a different channel for each channel traverses the data for each channel of the input data 420, respectively. and a feature map for each channel can be generated through convolution that proceeds with multiple multiplication and summing.

For the MAC operation, the scheduler 130 allocates processing elements (PE1 to PE12) to perform each MAC operation based on a predetermined operation sequence, and assigns a memory address where MAC operation values are stored in consideration of the scheduling sequence. It can be set in the NPU internal memory 120.

Referring to FIG. 12 , an example of processing signals provided from an RGB camera and a thermal image sensor through parallel processing is illustrated. During parallel processing, different information can be exchanged through transformers. The method may be a deep fusion method shown in FIG. 14 to be described later.

Meanwhile, although not shown, the artificial neural network may include a concatenation operation and a skip-connection operation in order to process different data provided from heterogeneous sensors. The concatenation operation means to combine the output results of a specific layer with each other, and the skip-and-connect operation means to pass the output result of a specific layer to another layer while skipping subsequent layers.

Such a concatenation operation and a skip-and-connect operation may increase control difficulty and usage of the internal memory 120 of the NPU 100 .

So far, artificial neural networks for fusion and processing of different data provided from heterogeneous sensors have been described. Accordingly, the optimized artificial neural network and NPU structure will be described below.

First, the inventors of this patent studied NPUs for processing different data from heterogeneous sensors.

In the design of the NPU, the following configuration should be considered:

i. It is necessary to have an NPU structure suitable for heterogeneous data signal processing (eg, RGB camera + thermal image sensor).

ii. NPU memory control suitable for heterogeneous input signal processing (eg, RGB camera + thermal image sensor) is required.

iii. It is necessary to have an NPU structure suitable for multiple input channels.

iv. NPU memory control suitable for multiple input channels is required.

v. It is necessary to have an NPU structure suitable for image fusion artificial neural network model (fusion artificial neural network model) calculation.

vi. Fast processing speed of less than 16 ms is required for real-time application

vii. It is necessary to achieve low power consumption for battery operation.

An NPU for implementing an image fusion artificial neural network model (fusion artificial neural network model) must support the following functions. Expected requirements include:

i. CNN function support: Must be able to control PE array and memory optimized for convolution.

ii. It should be able to efficiently handle depthwise-separable convolutions. It should have a structure that improves PE utilization and performance.

iii. Batch mode function support: Memory configuration is required to process multiple channels (cameras 1 to 6) and heterogeneous sensors at the same time. (PE array size and memory size must be in an appropriate ratio)

iv. Concatenation function support: The NPU for image convergence artificial neural network model (fusion artificial neural network model) must be able to process heterogeneous input data signals with concatenation function.

v. Skip-connection function support: The NPU for the image fusion artificial neural network model (fusion artificial neural network model) has a special function unit (SFU) that can provide the skip function. can include

vi. Support for deep learning image preprocessing function: The NPU for image fusion artificial neural network model (fusion artificial neural network model) must be able to provide the function of preprocessing different data signals.

vii. A compiler capable of efficiently compiling an image fusion artificial neural network model (fusion artificial neural network model) must be provided.

In one embodiment of the present disclosure, the NPU 100 having the following characteristics is proposed.

i. The NPU 100 is a machine that analyzes ANN data locality information of an image fusion artificial neural network model (fusion artificial neural network model) such as late fusion, early fusion, and deep fusion. code can be processed.

ii. The NPU 100 may be configured to control the PE array to process heterogeneous sensor data based on an artificial neural network data locality controller (ADC). That is, the image fusion artificial neural network model (fusion artificial neural network model) is fused into various structures according to the sensor, and PE utilization rate can be improved by providing the NPU 100 corresponding to the structure. .

iii. It may be configured to appropriately set the size of the on-chip memory 120 to process heterogeneous sensor data based on ANN data locality information. That is, the memory bandwidth of the NPU 100 processing the fusion artificial neural network can be improved by analyzing the artificial neural network data locality information of the image fusion artificial neural network model (fusion artificial neural network model).

iv. The NPU 100 can efficiently process bilinear interpolation, concatenation, and skip-connection (skip-connection, etc.) required in an image fusion artificial neural network model (fusion artificial neural network model). A special function unit (SFU) may be included.

Referring to FIG. 13, “F” means fusion operation, and each block means each layer. The NPU 100 may perform late fusion, early fusion, and deep fusion. Late fusion means performing calculations for each layer and then fusion of the calculation results in the final process. Early fusion means performing operations on each layer after fusion of different data at an early stage. Deep fusion means that after fusion of different data, calculations are performed in different layers, and calculation results are fused again, and then calculations are performed for each layer. In the present disclosure, through an early fusion operation, two different images may be merged at the beginning of a plurality of layer operation, and operation of a subsequent layer may be performed. Alternatively, through late fusion, operations may be performed on two different images for each layer allocated to each of the two images, and then, after merging the operation results, operations on a subsequent layer may be performed. For example, the two different images may be an image obtained through a visible ray image sensor and an image obtained through a thermal image sensor, but are not limited thereto.

Hereinafter, the structure of the NPU 100 capable of disclosing the above features will be described.

Referring to FIG. 14, the NPU 100 includes a PE array 110 for an image fusion artificial neural network model, an on-chip memory 120, an NPU scheduler 130, and a special function unit (SFU). (160). In describing FIG. 14 , redundant descriptions may be omitted for convenience of explanation.

The PE array 110 for the image fusion artificial neural network model may refer to a PE array 110 configured to process convolution of a multi-layered image fusion artificial neural network model having at least one fusion layer. That is, the fusion layer may be configured to output a feature map in which data from different types of sensors are fused. To explain further, the SFU 160 of the NPU 100 may be configured to receive sensor data from multiple sensors and provide a function of fusion of each sensor input data. The PE array 110 for the image fusion artificial neural network model may be configured to receive fusion data from the SFU 160 and process convolution.

The NPU 100 may receive different data from the M

heterogeneous sensors

311 and 312 . The heterogeneous sensors may include image sensors having different image characteristics and resolutions.

The NPU 100 may obtain artificial neural network data locality information of an image fusion artificial neural network model (fusion artificial neural network (ANN)) from the compiler 200 .

At least one layer of the image fusion artificial neural network model may be a layer in which input data of a plurality of sensors are fused.

The NPU 100 may be configured to provide a concatenation function to at least one layer for fusion of heterogeneous sensor input data. Each feature map of the heterogeneous sensors of the concatenated layer may be processed to have the same size as at least one axis in order to be concatenated with each other. For example, in order to connect heterogeneous sensor data on the X-axis, the X-axis size of each of the heterogeneous sensor data may be the same. For example, in order to concatenate heterogeneous sensor data on the Y-axis, the Y-axis size of each of the heterogeneous sensor data may be the same. For example, in order to concatenate the heterogeneous sensor data in the Z-axis, the size of the Z-axis of each of the heterogeneous sensor data may be the same. In order to improve the processing efficiency of the NPU 100, the size of one of the heterogeneous sensor data may be scaled up or scaled down. Accordingly, it is also possible that the sizes of one axis of the fused data of heterogeneous sensor data are the same. In other words, since the processing element array 100 is in the form of an NxM matrix, the PE utilization rate of the processing element array 100 may vary according to the size of at least one axis of sensor data.

In order to receive and process different data from the

heterogeneous sensors

311 and 312, the NPU scheduler 130 may process inference of an image fusion artificial neural network model (fusion artificial neural network model).

The NPU scheduler 130 may be included in the controller as shown.

The NPU scheduler 130 acquires and analyzes artificial neural network data locality information of an image fusion artificial neural network model (fusion artificial neural network) from the compiler 200, and controls the operation of the on-chip memory 120. there is.

Specifically, it is as follows. The compiler 200 may generate artificial neural network data locality information of a fusion artificial neural network to be processed by the NPU 100 .

The NPU scheduler 130 may generate a list of special functions required for the image fusion artificial neural network model (fusion artificial neural network). The special function may mean various functions required for artificial neural network operations other than convolution.

Using the artificial neural network data locality information of the image fusion artificial neural network model (fusion artificial neural network), non-maximum suppression (NMS), skip and connection (SKIP-CONNECTION), bottleneck, bilinear interpolation It is possible to efficiently control the memory access increase problem that frequently occurs in fusion artificial neural networks such as bilinear interpolation.

If the artificial neural network data locality information of the image fusion artificial neural network model (fusion artificial neural network) is used, the first output feature map information calculated earlier and the second output feature map information processed later are fused. , Since the size and storage period of data to be stored (eg, the first output feature map) can be known in the compilation step, a memory map for the on-chip memory 120 can be efficiently prepared in advance. can be set

The SFU 160 may perform skip-connection and concatenation necessary for an image convergence artificial neural network model (fusion artificial neural network). To elaborate, junctions can be used to fuse heterogeneous sensor data. For concatenation, the size of each sensor data may be readjusted. For example, the NPU 100 may be configured to process concatenation of fusion artificial artificial networks by providing functions such as resize and interpolation.

The chip-internal memory 120 of the NPU 100 selectively preserves specific data according to the PE array 110 or the SFU 160 for a specific period of time based on the artificial neural network data locality information of the image convergence artificial neural network model. can Whether or not to selectively preserve may be controlled by a control unit.

Also, the PE array 110 may be configured to have the number of threads corresponding to the number of heterogeneous sensors. That is, the array 110 of the NPU 100 configured to receive two sensor data may be configured to have two threads. That is, if one thread is composed of NxM processing elements, two threads may be composed of NxMx2 processing elements. For example, each thread of the PE array 110 may be configured to process feature maps of each heterogeneous sensor. A plurality of threads of an NPU may be referred to as a multi-core of the NPU.

The NPU 100 may output an operation result of the image convergence artificial neural network model through an output unit.

The NPU architecture according to the above-described first example may be variously modified.

15A is an exemplary diagram for explaining skip-connection included in the image convergence artificial neural network model according to the first example of the present disclosure, and FIG. 15B is an example of the image convergence artificial neural network model shown in FIG. 15A. It is an example diagram showing artificial neural network data locality information.

Referring to FIG. 15A, in order to calculate 5 layers including a skip-connection operation, as shown in FIG. 15B, the compiler 200 uses, for example, an image having a sequence of 16 steps. Artificial neural network data locality information of the convergence artificial neural network model can be created.

The NPU 100 requests data operations from the on-chip memory 120 in the order of artificial neural network data locality information of the image fusion artificial neural network model.

In the case of a skip-connection operation, the output feature map (OFMAP) of the first layer may be added to the output feature map (OFMAP) of the fourth layer.

For such a skip-connection operation, the output feature map of the first layer must be preserved until the fifth layer operation. However, other data may be deleted after operation to utilize memory space.

In the deleted memory area, data to be calculated later based on the order of artificial neural network data locality information of the image convergence artificial neural network model may be stored. Therefore, necessary data can be sequentially brought to the chip-internal memory 120 according to the order of data region information of the image convergence artificial neural network model, and data that is not reused can be deleted, so that the on-chip memory ( Operation efficiency of the on-chip memory 120 may be improved even when the memory size of the memory 120 is small.

Accordingly, the NPU 100 may selectively preserve or delete specific data of the chip-internal memory 120 for a certain period of time based on the artificial neural network data locality information of the image convergence artificial neural network model.

This mechanism may be applied to various operations such as concatenation, non-maximum suppression (NMS), and bilinear interpolation as well as skip-connection operations.

For example, the NPU 100 performs a convolution operation on the second layer for efficient control of the on-chip memory 120, and then data of the first layer excluding the output feature map (OFMAP) of the first layer. can be deleted. For another example, the NPU 100 performs the operation of the third layer for efficient control of the on-chip memory 120, and then data of the second layer excluding the output feature map (OFMAP) of the first layer. can be deleted. As another example, the NPU 100 performs the operation of the fourth layer for efficient control of the on-chip memory 120, and then data of the third layer excluding the output feature map (OFMAP) of the first layer. can be deleted. In addition, the NPU 100 may delete data of the fourth layer including the output feature map (OFMAP) of the first layer after performing the operation of the fifth layer for efficient control of the on-chip memory 120. can

The artificial neural network data locality information of the image convergence artificial neural network model refers to a data processing sequence generated by the compiler 200 and performed by the NPU 100 in consideration of the conditions listed below.

1. Structure of ANN model (fusion artificial neural networks such as Resnet, YOLO, SSD, etc. designed to receive heterogeneous sensor data).

2. The structure of the processor (eg architecture of CPU, GPU, NPU, etc.).

In the case of the NPU 100, the number of PEs, the structure of the PEs (eg, input stationary, output stationary, weight stationary, etc.), SFU structure configured to operate organically with the PE array, and the like.

3. On-chip memory 120 size (eg, when cache is smaller than data, tiling algorithm needs to be applied, etc.).

4. Data size of each layer of the image convergence artificial neural network model to be processed.

5. Processing Policy. That is, the NPU 100 determines the order of requesting to read the input feature map (IFMAP) first or request to read the kernel first. This may vary depending on the processor or compiler 200 .

Referring to FIG. 16, the NPU 100 includes a PE array 110 for an image fusion artificial neural network model, an on-chip memory 120, an NPU scheduler 130, and a special function unit (SFU). (160). In describing FIG. 16 , overlapping descriptions may be omitted for convenience of description.

The NPU scheduler 130 may be included in the controller as shown.

The NPU 100 may receive different data from the M

heterogeneous sensors

311 and 312 . The heterogeneous sensors may include a microphone, a touch screen, a camera, an altimeter, a barometer, an optical blood flow measurement sensor, an electrocardiogram measurement sensor, an inertial measurement sensor, a geo-positioning system, an optical sensor, a thermometer, an electromyograph, an electrode measurement device, and the like.

The NPU 100 may obtain artificial neural network data locality information of an image fusion artificial neural network model from the compiler 200 .

The NPU 100 may output N results (eg, heterogeneous inference results) through N output units. The heterogeneous data output from the NPU 100 may include image fusion, classification, semantic segmentation, object detection, and prediction.

Referring to FIG. 17, the NPU 100 includes a PE array 110 for an image convergence artificial neural network model, an on-chip memory 120, an NPU scheduler 130, and a special function unit (SFU) (160). In describing FIG. 17 , redundant descriptions may be omitted for convenience of explanation.

The NPU scheduler 130 may be included in the controller as shown.

The NPU 100 may receive different data from the M

heterogeneous sensors

The NPU 100 may receive data required for operation of an image convergence artificial neural network model from the chip-external memory 500 through an artificial neural network data locality controller (ADC) 400 .

The ADC 400 may prefetch data from an on-chip memory to an on-chip memory based on the artificial neural network data locality information of the image convergence artificial neural network model provided from the compiler 200.

Specifically, the ADC 400 receives and analyzes artificial neural network data locality information of an image convergence artificial neural network model from the compiler 200 or receives analyzed information from the compiler 200 and uses the chip-external memory ( 500) can be controlled.

The ADC 400 may read the data stored in the off-chip memory 500 according to the artificial neural network data locality information of the image convergence artificial neural network model and cache it in the on-chip memory in advance. The off-chip memory 500 may store all weight kernels of the image convergence artificial neural network model, and the on-chip memory 120 may store the image convergence artificial neural network model among all weight kernels stored in the off-chip memory 500. According to the artificial neural network data locality information of , only at least some weight kernels necessary can be stored. The memory capacity of the off-chip memory 500 may be greater than that of the on-chip memory 120 .

The ADC 400 may prepare data necessary for the NPU 100 in advance from the chip-external memory 500 independently or in conjunction with the NPU 100 based on the artificial neural network data locality information of the image convergence artificial neural network model. there is. Therefore, the latency of the reasoning operation of the NPU 100 may be reduced or the operation speed may be improved.

The NPU 100 may output N results (eg, heterogeneous inference results) through N output units.

18 is a conceptual diagram exemplarily illustrating a system including an NPU architecture according to a fourth example of the present disclosure, and FIG. 19 is a threaded image fusion artificial neural network model shown in FIG. 12 according to the fourth example shown in FIG. Indicates an example separated by .

Referring to FIG. 18, the NPU 100 includes a PE array 110 for an image convergence artificial neural network model, an on-chip memory 120, an NPU scheduler 130, and a special function unit (SFU) (160).

The NPU scheduler 130 may be included in the controller as shown.

The NPU 100 may receive different data from the M

heterogeneous sensors

The NPU 100 may output N pieces of heterogeneous data (eg, heterogeneous inference results). The heterogeneous data output from the NPU 100 may include image fusion, classification, semantic segmentation, object detection, and prediction.

The PE array 110 can process multiple threads. As shown in FIG. 19, RGB image data obtained from the camera can be processed through thread #1, Transformer model processing can be processed through thread #2, and data obtained from the thermal image sensor can be processed through thread #3. there is. Multiple threads of the PE array 110 may be referred to as a multi-core of the NPU. That is, each thread may refer to an independent PE array.

To this end, the compiler 200 may analyze the image convergence artificial neural network model and classify threads based on a parallel operation flow.

The PE array 110 of the NPU 100 can improve computation efficiency through multiple threads of a layer capable of parallel processing computation of an image convergence artificial neural network model.

Each thread may be configured to include the same or different numbers of processing elements.

The NPU 100 may control each thread in the PE array 110 to communicate with the on-chip memory 120 .

The NPU 100 may selectively allocate an internal space of the on-chip memory 120 for each thread.

The NPU 100 may allocate an appropriate on-chip memory 120 for each thread. Memory allocation of the chip-internal memory 120 may be determined by a controller based on artificial neural network data locality information of an image convergence artificial neural network model.

The NPU 100 may set a thread in the PE array 110 based on a fusion artificial neural network.

20 is a conceptual diagram exemplarily illustrating a system including an NPU architecture according to a fifth example of the present disclosure, and FIG. 21 is an exemplary diagram illustrating a first example of a pipeline structure of the SFU shown in FIG. 20 .

Referring to FIG. 20, the NPU 100 includes a PE array 110 for an image convergence artificial neural network model, an on-chip memory 120, an NPU scheduler 130, and a special function unit (SFU) (160).

The NPU 100 may receive different data from the M

heterogeneous sensors

As shown in FIG. 21, the SFU 160 includes several functional units. Each functional unit can be operated selectively. Each functional unit can be selectively turned on or turned off. That is, each functional unit can be set.

The processing element array may refer to circuitry configured to perform a main operation of an image fusion artificial neural network model. The main operation may refer to convolution or matrix multiplication. That is, the main operation may refer to most operations in an artificial neural network (ANN) (eg, a fusion artificial neural network).

A special function unit (SFU) may refer to a set of a plurality of special function circuits configured to selectively perform a special function operation of an image convergence artificial neural network model. That is, the special function unit (SFU) may additionally calculate a special function, and the special function operation may refer to an additional operation in various artificial neural networks (ANNs) (eg, a fusion artificial neural network).

The amount of calculation of the main operation of the image convergence artificial neural network model may be relatively greater than the amount of calculation of the special function calculation.

In other words, the SFU 160 may include various functional units required for reasoning of an image convergence artificial neural network model.

For example, the functional units of the SFU 160 include a functional unit for a skip-connection operation, a functional unit for an activation function operation, a functional unit for a pooling operation, Functional units for quantization operations, functional units for non-maximum suppression (NMS) operations, functional units for integer and floating point conversion (INT to FP32) operations, and functions for batch-normalization operations unit, a functional unit for an interpolation operation, a functional unit for a concatenation operation, and a functional unit for a bias operation.

Functional units of the SFU 160 may be selectively turned on or off according to artificial neural network data locality information of an image convergence artificial neural network model. To elaborate, the type of special function operations required by each layer of the image convergence artificial neural network model may be different for each layer. The artificial neural network data locality information included in the machine code may include control information related to turn-on or turn-off of a corresponding functional unit when an operation for a specific layer is performed.

22a is an exemplary view showing an example of the SFU shown in FIG. 20, and FIG. 22b is an example view showing another example of the SFU shown in FIG.

Referring to FIGS. 22A and 22B , activated units among functional units of the SFU 160 may be turned on.

Specifically, as shown in FIG. 22A, the SFU 160 may selectively activate a skip-connection operation and a concatenation operation. For example, each activated functional unit may be marked with hatching.

For example, the SFU 160 may concatenate heterogeneous sensor data for a fusion operation. For example, the controller may control the chip-internal memory 120 and the SFU 160 for the skip and connect operation of the SFU 160 .

Specifically, as shown in FIG. 22B, a quantization operation and a bias operation may be selectively activated. For example, in order to reduce the size of feature map data output from the PE array 110, the quantization function unit of the SFU 160 receives the feature map output from the PE array 110 and converts the feature map to a specific bit width. can be quantized. The quantized feature map may be stored in the chip-internal memory 120 . A series of operations may be performed sequentially through the controller, and the NPU scheduler 130 may be configured to control the sequence of operations.

In this way, when some functional units of the SFU 160 are selectively turned off, power consumption of the NPU 100 can be reduced. Meanwhile, in order to turn off some functional units, power gating may be used. Alternatively, clock gating may be performed to turn off some functional units.

Referring to FIG. 23 , NPU batch mode may be applied. The NPU 100 to which the batch mode is applied includes a PE array 110 for an image convergence artificial neural network model, an on-chip memory 120, an NPU scheduler 130, and a special function (SFU). unit) (160).

The NPU scheduler 130 may be included in the controller as shown.

The batch mode disclosed in this example sequentially processes a plurality of identical sensors with one image fusion artificial neural network model, and reuses the weights of the one image fusion artificial neural network model by the number of the plurality of identical sensors to achieve low power consumption mode configured to do so.

For the batch mode operation, the controller of the NPU 100 may be configured to control the NPU scheduler 130 so that the weights stored in the on-chip memory are reused as many as the number of sensors input to each batch channel. That is, illustratively, the NPU 100 may be configured to operate in batch mode with M sensors. At this time, the arrangement mode operation of the NPU 100 may be configured to operate as an image fusion artificial neural network model.

For the operation of the image fusion artificial neural network model, the NPU 100 may be configured to have a plurality of batch channels (BATCH CH#1, BATCH CH#2) for fusion. Each deployment channel may be configured to include a plurality of identical sensors. The first batch channel (BATCH CH#1) may include a plurality of first sensors. At this time, the number of first sensors may be M. The Kth batch channel (BATCH CH#K) may be composed of a plurality of second sensors. At this time, the number of second sensors may be M.

The NPU 100 reuses and processes corresponding weights in the chip-internal memory 120 for inputs from the

sensors

311 and 312 through the first configuration channel. And, the NPU 100 reuses and processes corresponding weights in the chip-internal memory 120 for inputs from the

sensors

321 and 322 through the second configuration channel.

As such, the NPU 100 may receive inputs from various sensors through a plurality of batch channels, reuse weights, and process an image fusion artificial neural network model in batch mode. A sensor of at least one channel among the plurality of arrangement channels may be different from a sensor of at least one other channel.

The on-chip memory 120 in the NPU 100 may be configured to have a storage space corresponding to a plurality of deployment channels.

The NPU scheduler 130 in the NPU 100 may operate the PE array 110 according to a batch mode.

The SFU 160 in the NPU 100 may provide a special function for processing at least one fusion operation.

The NPU 100 may deliver each output through a plurality of batch channels.

At least one of the plurality of batch channels may be inference data of an image convergence artificial neural network model network.

24 is an exemplary diagram illustrating an example of utilizing a plurality of NPUs according to the seventh example of the present disclosure, and FIG. 25 processes the fusion artificial neural network shown in FIG. 12 through the plurality of NPUs shown in FIG. It is an example diagram showing an example of doing.

Referring to FIG. 24 , a plurality of illustratively M NPUs may be used to generate a fusion image. Among the M NPUs, the first NPU 100-1 may process data provided from, for example, sensor #1 311, and the M-th NPU 100-M may process data provided from, for example, sensor #M 312. can handle The plurality of NPUs (eg, 100-1 and 100-2) may access the off-chip memory 500 through ADC/DMA (Direct Memory Access) 400.

The plurality of NPUs (eg, 100-1 and 100-2) may obtain artificial neural network data locality information of an image convergence artificial neural network model from the compiler 200.

Each NPU may process an image fusion artificial neural network model and transfer an operation for fusion to different NPUs through the ADC/DMA 400.

The ADC/DMA 400 may obtain data locality information for an artificial neural network of a fusion image fusion artificial neural network model from the compiler 200 .

The compiler 200 converts the artificial neural network data locality information into data locality information #1 and data locality information so that operations to be processed in parallel among operations according to the artificial neural network data locality information of the image fusion artificial neural network model can be processed in each NPU. It can be created by separating it with information #M.

The off-chip memory 500 may store data that can be shared by a plurality of NPUs and transfer it to each NPU.

Referring to FIG. 25 , NPU#1 may be in charge of a first artificial neural network for processing data provided from a camera, and NPU#2 may be in charge of a second artificial neural network for processing data provided from a thermal image sensor. can In addition, the NPU#2 may be in charge of conversion for fusion between the first artificial neural network and the second artificial neural network.

So far, the NPU 100 for an image convergence artificial neural network model according to various examples of the present disclosure has been described. According to the present disclosure, a high-resolution thermal image may be generated by using a high-resolution general visible light image sensor and a low-resolution thermal image sensor built into a general device, not a specialized device. Accordingly, the present disclosure can generate a high-resolution thermal image at low cost. In addition, the present disclosure can improve the night vision of an image not in a device designed for night vision, but in a device owned by a user or a black box of a vehicle, for example.

According to one disclosure of the present disclosure, a neural processing unit for an image fusion artificial neural network model is provided. The neural processing unit for the image fusion artificial neural network model inputs a first image and a second image having different resolutions and image characteristics, and uses machine code of the image fusion artificial neural network model learned to output a new third image. a controller configured to receive an input; an input circuit configured to receive a plurality of input signals corresponding to the image fusion artificial neural network model; a processing element array configured to perform a main operation of the image fusion artificial neural network model operation; a special function unit circuit configured to perform a special function operation of the image convergence artificial neural network model; and an on-chip memory configured to store data of the main operation and/or the special function operation of the image fusion artificial neural network model. and the control unit, according to data locality information of the image fusion artificial neural network model included in the machine-code, so that the operation order of the image fusion artificial neural network model is processed in a preset order, the processing element array, the special configured to control a functional unit circuit and the on-chip memory, wherein a third resolution of the third image has a value between a first resolution of the first image and a second resolution of the second image; The third image characteristic of the 3 images may be at least partially the same as the first image characteristic of the first image or the second image characteristic of the second image.

The first image may be an image obtained through a visible ray image sensor.

The second image may be an image acquired through a thermal image sensor.

The first image and the second image include different images of one object, and the image characteristics may be determined by the type of image sensor that acquires the first image and the second image.

The image convergence artificial neural network model may be an artificial neural name model configured to input only a first partial image and a second partial image corresponding to a face region from objects extracted from the first image and the second image.

The third image may be an image to which at least one feature that can be determined from the second image is applied to at least a partial region of the first image.

The image fusion artificial neural network model may be a model to which weights are applied to emphasize at least one feature that can be determined from the first image and at least one feature that can be determined from the second image.

The image fusion artificial neural network model may be an artificial neural network model configured to input only RGB values of the first image or brightness values of each pixel of the first image.

The third resolution of the third image may be the same as the first resolution of the first image.

The image convergence artificial neural network model is learned based on a generative adversarial networks (GAN) structure, and may correspond to a generator configured to generate a new image by using different images of one object as inputs.

The image fusion artificial neural network model is configured such that the generator constituting the GAN and a discriminator verifying an image generated by the generator compete with each other to update a weight for increasing the third resolution of the third image. It may be an artificial nerve name model.

The image fusion artificial neural network model may be learned based on a learning data set having a format substantially similar to that of the first image and the second image.

The array of processing elements may be configured to handle convolutions and activation function operations.

The processing element array may be configured to process at least one operation among dilated convolution, transposed convolution, and bilinear interpolation operations for increasing the third resolution of the third image.

Further comprising an output unit configured to output a result of the at least one inference operation of the image fusion artificial neural network model learned to process at least one inference operation among classification, semantic segmentation, object detection, and prediction by the processing element array. can

The special function unit circuit may further include at least one function of skip-connection and concatenation for artificial neural network fusion.

The controller further comprises a scheduler, wherein the scheduler preserves specific data stored in the on-chip memory based on data locality information of the image fusion artificial neural network model until a specific operation step of the image fusion artificial neural network model, It may be configured to control the on-chip memory.

The processing element array may further include a plurality of threads, and the controller may be configured to control the plurality of threads to process parallel sections of the image fusion artificial neural network model based on data locality of the image fusion artificial neural network model. there is.

According to another disclosure of the present disclosure, a system for an image fusion artificial neural network model is provided. The system may include: a first sensor for acquiring a first image having a first resolution and a first image characteristic; a second sensor configured to acquire a second image having a second resolution smaller than the first resolution and a second image characteristic different from the first image characteristic; and a neural processing unit configured to process an image convergence artificial neural network model trained to output a new third image by inputting the first image and the second image having different resolutions and image characteristics; wherein the third resolution of the third image has a value between the first resolution of the first image and the second resolution of the second image, and the third image characteristic of the third image comprises: The first image characteristic of one image or the second image characteristic of the second image may be at least partially the same.

Although one example of the present disclosure has been described in more detail with reference to the accompanying drawings, the present disclosure is not necessarily limited to these examples, and may be variously modified and implemented without departing from the technical spirit of the present disclosure. Therefore, the examples disclosed in this disclosure are not intended to limit the technical spirit of the present disclosure, but to explain, and the scope of the technical spirit of the present disclosure is not limited by these examples. Therefore, the examples described above should be understood as illustrative in all respects and not limiting. The protection scope of the present disclosure should be construed by the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present disclosure.

[National research and development project supporting this invention]

[Assignment identification number] 1711175834

[Assignment number] R-20210401-010439

[Name of department] Ministry of Science and ICT

[Task management (professional) institution name] National IT Industry Promotion Agency

[Research Project Name] Intensive Fostering of Innovative AI Semiconductor Companies

[Research Project Title] Compiler and Runtime for Artificial Neural Network Processor for Edge

SW technology development

[Contribution rate] 1/1

[Name of project performing organization] DeepX Co., Ltd.

[Research Period] 2022.06.01 ~ 2022.12.31

Claims

a control unit configured to receive machine code of an image convergence artificial neural network model learned to output a new third image by inputting first and second images having different resolutions and image characteristics;

an input circuit configured to receive a plurality of input signals corresponding to the image fusion artificial neural network model;

a processing element array configured to perform a main operation of the image fusion artificial neural network model;

a special function unit circuit configured to perform a special function operation of the image convergence artificial neural network model; and

an on-chip memory configured to store data of the main operation and/or the special function operation of the image fusion artificial neural network model; including,

The control unit is configured to process the processing element array, the special function unit circuit, and the special function unit circuit so that the operation order of the image fusion artificial neural network model is processed in a preset order according to the data locality information of the image fusion artificial neural network model included in the machine-code. configured to control the on-chip memory;

The third resolution of the third image has a value between the first resolution of the first image and the second resolution of the second image,

The neural processing unit for image fusion, wherein the third image characteristic of the third image is at least partially the same as the first image characteristic of the first image or the second image characteristic of the second image.
According to claim 1,

The first image is an image acquired through a visible ray image sensor, the neural processing unit for image fusion.
According to claim 1,

The second image is an image acquired through a thermal image sensor, the neural processing unit for image fusion.
According to claim 1,

The first image and the second image include different images of one object,

Wherein the image characteristics are determined by types of image sensors that acquire the first image and the second image, the neural processing unit for image fusion.
According to claim 4,

The image fusion artificial neural network model,

Characterized in that the neural processing unit for image fusion, characterized in that the artificial neural name model configured to input only the first partial image and the second partial image corresponding to the face region from the object extracted from the first image and the second image.
According to claim 1,

The third image is an image to which at least one feature that can be determined from the second image is applied to at least a partial region of the first image, the neural processing unit for image fusion.
According to claim 1,

The image fusion artificial neural network model,

A neural processing unit for image fusion, characterized in that the model is a weighted model for emphasizing at least one feature determinable from the first image and at least one feature determinable from the second image.
According to claim 1,

The image fusion artificial neural network model,

Characterized in that the artificial neural network model configured to input only the RGB values of the first image or the brightness values of each pixel of the first image, the neural processing unit for image fusion.
According to claim 1,

The third resolution of the third image,

A neural processing unit for image fusion equal to the first resolution of the first image.
According to claim 1,

The image fusion artificial neural network model,

It is learned based on the structure of generative adversarial networks (GAN),

A neural processing unit for image fusion, corresponding to a generator configured to generate a new image by taking different images of one object as inputs.
According to claim 10,

The image fusion artificial neural network model,

Characterized in that the generator constituting the GAN and a discriminator verifying the image generated by the generator compete with each other to update a weight for increasing the third resolution of the third image. , a neural processing unit for image fusion.
According to claim 1,

The image fusion artificial neural network model,

A neural processing unit for image fusion, which is learned based on a learning data set having a form substantially similar to the first image and the second image.
According to claim 1,

The processing element array,

A neural processing unit for image fusion, configured to process convolutions and activation function operations.
According to claim 1,

The processing element array,

A neural processing unit for image fusion configured to process at least one of matrix multiplication, dilated convolution, transposed convolution, and bilinear interpolation operations for increasing the third resolution of the third image.
According to claim 1,

An output unit configured to output at least one inference operation result of the image fusion artificial neural network model learned to process at least one inference operation among classification, semantic segmentation, object detection, pose estimation, and prediction by the processing element array. Further comprising, a neural processing unit for image fusion.
According to claim 1,

The special function unit circuit further comprises at least one function of Skip-connection and Concatenation for artificial neural network fusion, a neural processing unit for image fusion.
According to claim 1,

The control unit further includes a scheduler,

The scheduler is configured to control the on-chip memory to preserve specific data stored in the on-chip memory until a specific operation step of the image fusion artificial neural network model based on data locality information of the image fusion artificial neural network model. , a neural processing unit for image fusion.
According to claim 1,

the array of processing elements further comprises a plurality of threads;

The control unit is configured to control the plurality of threads to process parallel sections of the image fusion artificial neural network model based on data locality of the image fusion artificial neural network model.
a first sensor that obtains a first image having a first resolution and a first image characteristic;

a second sensor configured to acquire a second image having a second resolution smaller than the first resolution and a second image characteristic different from the first image characteristic; and

a neural processing unit configured to process an image convergence artificial neural network model trained to output a new third image by inputting first and second images having different resolutions and image characteristics; including,

The third resolution of the third image has a value between the first resolution of the first image and the second resolution of the second image,

The artificial neural network system for image fusion, wherein the third image characteristic of the third image is at least partially the same as the first image characteristic of the first image or the second image characteristic of the second image.