KR20240030922A - Npu for distributing artificial neural networks based on mpeg-vcm and method thereof - Google Patents

Abstract

According to one disclosure of the present specification, a Neural Processing Unit (NPU) capable of performing encoding is provided. The NPU includes one or more Processing Elements (PEs) that perform operations on multiple layers within an artificial neural network to generate multiple output feature maps; And, it may include an encoder that encodes one or more random output feature maps among the plurality of output feature maps into a bitstream and then transmits it.

Description

NPU and method for distributed processing of artificial neural network based on MPEG-VCM {NPU FOR DISTRIBUTING ARTIFICIAL NEURAL NETWORKS BASED ON MPEG-VCM AND METHOD THEREOF}

This disclosure relates to an NPU and method for distributed processing of an artificial neural network.

Through the continued development of the information and communications industry, broadcasting services with HD (High Definition) resolution have spread globally.

Through this proliferation, many users have become accustomed to high-resolution and high-definition images and/or videos, and higher resolution, such as 4K or 8K or higher UHD (Ultra High Definition) images/videos, Demand for high-quality images/videos has increased in various fields.

The technology for coding such UHD video data was completed through HEVC (High Efficiency Video Coding), a standard technology in 2013.

HEVC is a next-generation video compression technology with a higher compression rate and lower complexity than the previous H.264/AVC technology, and is a core technology for effectively compressing massive data of HD and UHD video.

HEVC performs block-level encoding like previous compression standards. However, unlike H.264/AVC, there is only one profile. The core coding technologies included in HEVC's unique profile include a total of eight fields: hierarchical coding structure technology, transformation technology, quantization technology, intra-screen predictive coding technology, inter-screen motion prediction technology, entropy coding technology, loop filter technology, and other technologies. There is.

Since the establishment of the HEVC video codec in 2013, as realistic video and virtual reality services using 4K and 8K video images have expanded, the next-generation video codec, Versatile Video Coding (VVC), aims to improve performance by more than 2 times compared to HEVC. Video Coding) standards have been developed. VVC is called H.266.

H.266 (VVC) was developed with the goal of being more than twice as efficient as the previous generation codec, H.265 (HEVC). VVC was initially developed considering resolutions of 4K or higher, but was also developed for ultra-high resolution video processing at a whopping 16K level in order to respond to 360-degree videos due to the expansion of the VR market. In addition, in response to the gradual expansion of the HDR market due to the development of display technology, it supports not only 10-bit color depth but also 16-bit color depth, and supports brightness expressions of 1000 nits, 4000 nits, and 10000 nits. Additionally, since it is being developed with the VR market and 360-degree video market in mind, it supports partial frame rates ranging from 0 to 120 FPS.

<Development of artificial intelligence>

Artificial intelligence (AI) is also gradually developing. AI refers to artificially imitating human intelligence, that is, intelligence capable of recognition, classification, inference, prediction, control/decision making, etc. do.

Due to the development of artificial intelligence technology and the increase in Internet of Things (IOT) devices, traffic between machines is predicted to explode, and machine-dependent video analysis is predicted to be widely used.

However, as the number of images to be analyzed by machines is expected to increase exponentially, server load and power consumption problems are expected to arise.

Accordingly, the purpose of the present disclosure is to provide a Neural Processing Unit (NPU) to effectively perform image analysis by a machine.

In order to achieve the above-described object, according to one disclosure of the present specification, a Neural Processing Unit (NPU) capable of performing encoding is provided. The NPU includes one or more Processing Elements (PEs) that perform operations on multiple layers within an artificial neural network to generate multiple output feature maps; And, it may include an encoder that encodes one or more random output feature maps among the plurality of output feature maps into a bitstream and then transmits it.

The encoder may: select one or more random output feature maps among the plurality of output feature maps based on arbitrary information.

The arbitrary information may be received from a server or another NPU that receives the bitstream.

The random information may include identification information about the one or more random output feature maps or identification information about a layer corresponding to the random output feature map.

The arbitrary information is based on one or more of: size information of the output feature map, information on machine tasks performed in another NPU receiving the bitstream, and information on an artificial neural network model calculated in another NPU receiving the bitstream. It can be created.

The bitstream may be transmitted to enable another NPU to perform calculations for subsequent layers in the artificial neural network.

In order to achieve the above-described object, according to one disclosure of the present specification, a Neural Processing Unit (NPU) capable of performing decoding is provided. The NPU includes a decoder that decodes the received bitstream and restores it into one or more random feature maps; It may also include one or more PE (Processing Elements) that perform calculations for the artificial neural network model. The one or more reconstructed random feature maps may be output feature maps of a random layer among multiple layers in the artificial neural network. The one or more PEs may utilize the one or more reconstructed random feature maps as an input feature map for a subsequent layer of the random layer.

The NPU may transmit random information that allows the random layer or the one or more feature maps to be selected by another NPU transmitting the bitstream.

The random information may include identification information about the one or more random feature maps or identification information about a layer corresponding to the one or more random feature maps.

The random information may be generated based on one or more of: size information of the feature map, information on machine tasks performed in the NPU, and information on the artificial neural network model.

Also, to achieve the above-described object, according to one disclosure of the present specification, a system is presented. The system includes a first neural processing unit configured to process a first layer section of an artificial neural network model including a plurality of layers; and a second neural processing unit configured to process a second layer section of the artificial neural network model. The first neural processing unit may be configured to input the output feature map of the last layer of the first layer section to the second neural processing unit.

The size of the sum of the weights of the layers in the first layer section may be relatively smaller than the size of the sum of the weights of the layers in the second layer section.

There may be a plurality of first neural processing units, and the second neural processing unit may be configured to sequentially process each of the output feature maps output from the first neural processing unit.

The second neural processing unit may be configured to reuse weights corresponding to at least the number of the first neural processing units.

The size of the first memory of the first neural processing unit may be configured to be relatively smaller than the second memory of the second neural processing unit.

The first layer section and the second layer section of the artificial neural network model may be determined based on the size of the output feature map of a specific layer.

The first layer section and the second layer section of the artificial neural network model may be determined based on the size of the sum of the weights of each layer section.

The size of the first weight of the first layer section of the artificial neural network model may be relatively smaller than the size of the second weight of the second layer section.

According to the NPU presented by this disclosure, through distributed processing of an artificial neural network, the edge-type NPU can reduce manufacturing costs and power consumption, and the server-type NPU can reduce computational load.

1 schematically shows an example of a video/picture coding system.
Figure 2 is a diagram schematically explaining the configuration of a video/image encoding device.
Figure 3 is a diagram schematically explaining the configuration of a video/image decoding device.
4 is a schematic conceptual diagram explaining a neural processing unit according to the present disclosure.
FIG. 5A is a schematic conceptual diagram illustrating one processing element among a plurality of processing elements applicable to the present disclosure.
Figure 5b is a schematic conceptual diagram explaining an SFU applicable to the present disclosure.
FIG. 6 is an exemplary diagram showing a modified example of the neural processing unit 100 shown in FIG. 4.
7 is a schematic conceptual diagram illustrating an exemplary artificial neural network model.
Figure 8a is a diagram for explaining the basic structure of a convolutional neural network.
Figure 8b is a comprehensive diagram showing the operation of a convolutional neural network in an easy-to-understand manner.
Figures 9a to 9d are illustrations showing an NPU including a VCM encoder and an NPU including a VCM decoder.
Figures 10a and 10b are exemplary diagrams showing the positions of bitstreams in the artificial neural network model.
FIG. 11A shows an example of a VCM encoder compressing a feature map and then transmitting it as a bitstream, and a VCM decoder for restoring the received bitstream, according to an example, and FIG. 11B shows feature maps P2 to feature maps shown in FIG. 11A. An example of creating P5 is shown.
FIG. 11C is an example diagram illustrating in detail the MSFF execution unit shown in FIG. 11A.
FIG. 12 is an exemplary diagram showing a modification of FIG. 11A.
FIG. 13A is a conceptual diagram illustrating an example of distributed processing between an edge device and a cloud server according to an disclosure of the present specification, based on the size of data for each layer in an exemplary artificial neural network model.
FIG. 13B is an example table showing the data size for each layer in the example artificial neural network model shown in FIG. 13A.
FIG. 14A is an exemplary diagram illustrating feature maps from layer 9 to layer 13 of the artificial neural network model shown in FIG. 13A.
FIG. 14B is an exemplary diagram illustrating a VCM encoder that compresses the feature maps P9 to P12 shown in FIG. 14A and then transmits them as a bitstream, and a VCM decoder that restores the received bitstream, according to an example.
Figure 15a is an exemplary diagram showing the NPU according to the disclosure of the present specification from an operation perspective.
Figure 15b is a flowchart showing the signal flow between the transmitting NPU and the receiving NPU.
Figure 15c is a modified example of Figure 15b.
16 is an exemplary diagram illustrating a first exemplary scenario to which a disclosure of the present specification is applied.
17 is an exemplary diagram illustrating a second exemplary scenario to which a disclosure of the present specification is applied.

Specific structural or step-by-step descriptions of embodiments according to the concept of the present disclosure disclosed in the present specification or application are merely illustrative for the purpose of explaining embodiments according to the concept of the present disclosure, and the implementation according to the concept of the present disclosure Examples may be implemented in various forms, and embodiments according to the concept of the present disclosure may be implemented in various forms, and should not be construed as being limited to the embodiments described in this specification or application.

Since the embodiments according to the concept of the present disclosure may be subject to various changes and may have various forms, specific embodiments will be illustrated in the drawings and described in detail in the present specification or application. However, this is not intended to limit the embodiments according to the concept of the present disclosure to a specific disclosure form, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present disclosure.

Terms such as first and/or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to the concept of the present disclosure, a first component may be named a second component, and similarly The second component may also be referred to as the first component.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "immediately between" or "neighboring" and "directly adjacent to" should be interpreted similarly.

The terms used in this specification are merely used to describe specific embodiments and are not intended to limit the present disclosure. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the existence of a described feature, number, step, operation, component, part, or combination thereof, but are not intended to indicate the presence of one or more other features or numbers. It should be understood that this does not preclude the existence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which this disclosure pertains. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

In describing the embodiments, description of technical content that is well known in the technical field to which this disclosure belongs and that is not directly related to this disclosure will be omitted. This is to convey the gist of the present disclosure more clearly without obscuring it by omitting unnecessary explanation.

This document is about video/image coding. For example, the method/embodiment disclosed in this document may be applied to the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next-generation video/image coding standard after VVC, or other video coding-related standards ( For example, it may be related to the High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), essential video coding (EVC) standard, AVS2 standard, etc.).

This document presents various embodiments of video/image coding, and unless otherwise specified, the embodiments may be performed in combination with each other.

In this document, video may refer to a series of images over time. A picture generally refers to a unit representing one image in a specific time period, and a slice/tile is a unit that forms part of a picture in coding. A slice/tile may contain one or more coding tree units (CTUs). One picture may consist of one or more slices/tiles. One picture may consist of one or more tile groups. One tile group may include one or more tiles.

A pixel or pel may refer to the minimum unit that constitutes one picture (or video). Additionally, 'sample' may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel/pixel value of a luma component, or only a pixel/pixel value of a chroma component. Alternatively, a sample may mean a pixel value in the spatial domain, or when these pixel values are converted to the frequency domain, it may mean a conversion coefficient in the frequency domain.

A unit may represent the basic unit of image processing. A unit may include at least one of a specific area of a picture and information related to the area. One unit can include one luma block and two chroma (ex. cb, cr) blocks. In some cases, unit may be used interchangeably with terms such as block or area. In a general case, an MxN block may include a set (or array) of samples (or a sample array) or transform coefficients consisting of M columns and N rows.

<Definition of terms>

Hereinafter, in order to help understand the disclosures presented in this specification, the terms used in this specification will be briefly summarized.

NPU: Abbreviation for Neural Processing Unit, which may refer to a processor specialized for computing artificial neural network models, separate from the CPU (Central processing unit).

AI accelerator: As an AI calculation accelerator, it can refer to NPU.

ANN: An abbreviation for artificial neural network. In order to imitate human intelligence, neurons in the human brain are connected through synapses, forming a layer of nodes. It can mean a network connected in a structure.

Information about the structure of the artificial neural network: Information including information about the number of layers, the number of nodes within the layer, the value of each node, information about the calculation processing method, and information about the weight matrix applied to each node.

Information about data locality of artificial neural network: This is information that predicts the operation order of the artificial neural network model processed by the neural processing unit based on the order of data access requests made by the neural processing unit to a separate memory.

DNN: An abbreviation for Deep Neural Network, which may mean increasing the number of hidden layers of an artificial neural network to implement higher artificial intelligence.

CNN: An abbreviation for Convolutional Neural Network, it is a neural network that functions similar to image processing in the visual cortex of the human brain. Convolutional neural networks are known to be suitable for image processing and are known to be easy to extract features of input data and identify patterns of features.

Kernel: May refer to the weight matrix applied to CNN. The value of the kernel can be determined through machine learning.

Hereinafter, the present disclosure will be described in detail by explaining preferred embodiments of the present disclosure with reference to the accompanying drawings. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

1 schematically shows an example of a video/picture coding system.

Referring to FIG. 1, a video/image coding system may include a source device and a receiving device. The source device can transmit encoded video/image information or data in file or streaming form to a receiving device through a digital storage medium or network.

The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may be called a video/image encoding device, and the decoding device may be called a video/image decoding device. A transmitter may be included in the encoding device. A receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be composed of a separate device or external component.

A video source can acquire video/image through the process of capturing, compositing, or creating video/image. A video source may include a video/image capture device and/or a video/image generation device. A video/image capture device may include, for example, one or more cameras, a video/image archive containing previously captured video/images, etc. Video/image generating devices may include, for example, computers, tablets, and smartphones, and are capable of generating video/images (electronically). For example, a virtual video/image may be created through a computer, etc., and in this case, the video/image capture process may be replaced by the process of generating related data.

The encoding device can encode input video/image. The encoding device can perform a series of procedures such as prediction, transformation, and quantization for compression and coding efficiency. Encoded data (encoded video/image information) may be output in the form of a bitstream.

The transmitting unit may transmit the encoded video/image information or data output in the form of a bitstream to the receiving unit of the receiving device through a digital storage medium or network in the form of a file or streaming. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit may include elements for creating a media file through a predetermined file format and may include elements for transmission through a broadcasting/communication network. The receiving unit may receive/extract the bitstream and transmit it to the decoding device.

The decoding device can decode the video/image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction that correspond to the operation of the encoding device.

The renderer can render the decoded video/image. The rendered video/image may be displayed through the display unit.

Figure 2 is a diagram schematically explaining the configuration of a video/image encoding device.

Hereinafter, the video encoding device may include a video encoding device.

Referring to FIG. 2, the encoding device 10a includes an image partitioner (10a-10), a predictor (10a-20), a residual processor (10a-30), and an entropy encoding unit ( It may be configured to include an entropy encoder (10a-40), an adder (10a-50), a filter (10a-60), and a memory (10a-70). The prediction unit 10a-20 may include an inter prediction unit 10a-21 and an intra prediction unit 10a-22. The residual processing unit 10a-30 includes a transformer 10a-32, a quantizer 10a-33, a dequantizer 10a-34, and an inverse transformer 10a-35. can do. The residual processing unit 10a-30 may further include a subtractor 10a-31. The adder 10a-50 may be called a reconstructor or a reconstructed block generator. The above-described image segmentation unit 10a-10, prediction unit 10a-20, residual processing unit 10a-30, entropy encoding unit 10a-40, addition unit 10a-50, and filtering unit 10a- 60) may be configured by one or more hardware components (eg, an encoder chipset or processor) depending on the embodiment. Additionally, the memory 10a-70 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include a memory 10a-70 as an internal/external component.

The image segmentation unit 10a-10 may divide an input image (or picture, frame) input to the encoding device 10a into one or more processing units. As an example, the processing unit may be called a coding unit (CU). In this case, the coding unit will be split recursively according to the QTBTTT (Quad-tree binary-tree ternary-tree) structure from the coding tree unit (CTU) or the largest coding unit (LCU). You can. For example, one coding unit may be divided into a plurality of coding units of deeper depth based on a quad tree structure, binary tree structure, and/or ternary structure. In this case, for example, the quad tree structure may be applied first and the binary tree structure and/or ternary structure may be applied later. Alternatively, the binary tree structure may be applied first. The coding procedure according to this document can be performed based on the final coding unit that is no longer divided. In this case, based on coding efficiency according to video characteristics, the largest coding unit can be used directly as the final coding unit, or, if necessary, the coding unit is recursively divided into coding units of lower depth to determine the optimal coding unit. A coding unit of size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be divided or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving a transform coefficient and/or a unit for deriving a residual signal from the transform coefficient.

In some cases, unit may be used interchangeably with terms such as block or area. In a general case, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally represent a pixel or a pixel value, and may represent only a pixel/pixel value of a luminance (luma) component, or only a pixel/pixel value of a chroma component. A sample may be used as a term that corresponds to a pixel or pel of one picture (or video).

The subtractor 10a-31 subtracts the prediction signal (predicted block, prediction samples, or prediction sample array) output from the prediction unit 10a-20 from the input image signal (original block, original samples, or original sample array). A residual signal (residual block, residual samples, or residual sample array) can be generated by subtraction, and the generated residual signal is transmitted to the conversion unit 10a-32. The prediction unit 10a-20 may perform prediction on a block to be processed (hereinafter referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit 10a-20 may determine whether intra prediction or inter prediction is applied on a current block or CU basis. As will be described later in the description of each prediction mode, the prediction unit may generate various information related to prediction, such as prediction mode information, and transmit it to the entropy encoding unit 10a-40. Information about prediction may be encoded in the entropy encoding unit 10a-40 and output in the form of a bitstream.

The intra prediction unit 10a-22 may predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from the current block depending on the prediction mode. In intra prediction, prediction modes may include a plurality of non-directional modes and a plurality of directional modes. Non-directional modes may include, for example, DC mode and planar mode. The directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the level of detail of the prediction direction. However, this is an example and more or less directional prediction modes may be used depending on the setting. The intra prediction unit 10a-22 may determine the prediction mode applied to the current block using the prediction mode applied to the neighboring block.

The inter prediction unit 10a-21 may derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector in the reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted on a block, subblock, or sample basis based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, neighboring blocks may include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. A reference picture including the reference block and a reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, a collocated reference block, a collocated CU (colCU), etc., and a reference picture including the temporal neighboring block may be called a collocated picture (colPic). It may be possible. For example, the inter prediction unit 10a-21 constructs a motion information candidate list based on neighboring blocks and indicates which candidate is used to derive the motion vector and/or reference picture index of the current block. Information can be generated. Inter prediction may be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter prediction unit 10a-21 may use motion information of neighboring blocks as motion information of the current block. . In the case of skip mode, unlike merge mode, residual signals may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the surrounding block is used as a motion vector predictor, and the motion vector of the current block is predicted by signaling the motion vector difference. You can instruct.

The prediction unit 10a-20 may generate a prediction signal based on various prediction methods described later. For example, the prediction unit can not only apply intra prediction or inter prediction for prediction of one block, but also can apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). Additionally, the prediction unit may perform intra block copy (IBC) to predict a block. The intra block copy can be used, for example, for video/video coding of content such as games, such as SCC (screen content coding). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can use at least one of the inter prediction techniques described in this document.

The prediction signal generated through the inter prediction unit 10a-21 and/or the intra prediction unit 10a-22 may be used to generate a restored signal or a residual signal. The transform unit 10a-32 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transformation technique may include Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Graph-Based Transform (GBT), or Conditionally Non-linear Transform (CNT). Here, GBT refers to the transformation obtained from this graph when the relationship information between pixels is expressed as a graph. CNT refers to the transformation obtained by generating a prediction signal using all previously reconstructed pixels and obtaining it based on it. Additionally, the conversion process may be applied to square pixel blocks of the same size, or to non-square blocks of variable size.

The quantization unit 10a-33 quantizes the transform coefficients and transmits them to the entropy encoding unit 10a-40, and the entropy encoding unit 10a-40 encodes the quantized signal (information about the quantized transform coefficients) It can be output as a bitstream. Information about the quantized transform coefficients may be called residual information. The quantization unit 10a-33 may rearrange the quantized transform coefficients in block form into a one-dimensional vector form based on the coefficient scan order, and based on the quantized transform coefficients in the one-dimensional vector form, the Information about quantized transform coefficients may also be generated. The entropy encoding unit 10a-40 may perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC).

The entropy encoding unit 10a-40 may encode information necessary for video/image restoration (e.g., values of syntax elements, etc.) in addition to the quantized transformation coefficients together or separately. Encoded information (ex. encoded video/picture information) may be transmitted or stored in bitstream form in units of NAL (network abstraction layer) units. The video/image information may further include information about various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video/image information may further include general constraint information. Signaling/transmission information and/or syntax elements described later in this document may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream can be transmitted over a network or stored in a digital storage medium. Here, the network may include a broadcasting network and/or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The signal output from the entropy encoding unit 10a-40 may be configured as an internal/external element of the encoding device 10a through a transmission unit (not shown) to transmit and/or a storage unit (not shown) to store it, or The transmission unit may be included in the entropy encoding unit 10a-40.

The quantized transform coefficients output from the quantization unit 10a-33 can be used to generate a prediction signal. For example, the residual signal (residual block or residual samples) is restored by applying inverse quantization and inverse transformation to the quantized transformation coefficients through the inverse quantization unit 10a-34 and the inverse transformation unit 10a-35. can do. The adder 10a-50 adds the reconstructed residual signal to the prediction signal output from the prediction unit 10a-20, thereby creating a reconstructed signal (reconstructed picture, restored block, restored samples, or restored sample array). can be created. If there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as a restoration block. The generated reconstructed signal can be used for intra prediction of the next processing target block in the current picture, and can also be used for inter prediction of the next picture after filtering, as will be described later.

Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and/or restoration process.

The filtering unit 10a-60 may improve subjective/objective image quality by applying filtering to the restored signal. For example, the filtering unit 10a-60 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 10a-70, specifically the memory ( It can be stored in the DPB of 10a-70). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset (SAO), adaptive loop filter, bilateral filter, etc. The filtering unit 10a-60 may generate various information about filtering and transmit it to the entropy encoding unit 10a-90, as will be described later in the description of each filtering method. Information about filtering may be encoded in the entropy encoding unit 10a-90 and output in the form of a bitstream.

The modified reconstructed picture transmitted to the memory 10a-70 may be used as a reference picture in the inter prediction unit 10a-80. Through this, the encoding device can avoid prediction mismatch in the encoding device 10a and the decoding device when inter prediction is applied, and can also improve encoding efficiency.

The DPB of the memory 10a-70 can store the modified reconstructed picture for use as a reference picture in the inter prediction unit 10a-21. The memory 10a-70 may store motion information of a block from which motion information in the current picture is derived (or encoded) and/or motion information of blocks in an already reconstructed picture. The stored motion information can be transmitted to the inter prediction unit 10a-21 to be used as motion information of spatial neighboring blocks or motion information of temporal neighboring blocks. The memory 10a-70 can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 10a-22.

Figure 3 is a diagram schematically explaining the configuration of a video/image decoding device.

Referring to FIG. 3, the decoding device 10b includes an entropy decoder (10b-10), a residual processor (10b-20), a predictor (10b-30), and an adder (adder). , 10b-40), a filter (10b-50), and a memory (10b-60). The prediction unit 10b-30 may include an inter prediction unit 10b-31 and an intra prediction unit 10b-32. The residual processing unit 10b-20 may include a dequantizer 10b-21 and an inverse transformer 10b-21. The above-described entropy decoding unit 10b-10, residual processing unit 10b-20, prediction unit 10b-30, addition unit 10b-40, and filtering unit 10b-50 are one unit depending on the embodiment. It may be configured by hardware components (for example, a decoder chipset or processor). Additionally, the memory 10b-60 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include a memory 10b-60 as an internal/external component.

When a bitstream including video/image information is input, the decoding device 10b can restore the image in response to the process in which the video/image information is processed in the encoding device of FIG. 2. For example, the decoding device 10b may derive units/blocks based on block division-related information obtained from the bitstream. The decoding device 10b may perform decoding using a processing unit applied in the encoding device. Accordingly, the processing unit of decoding may for example be a coding unit, and the coding unit may be split along a quad tree structure, a binary tree structure and/or a ternary tree structure from a coding tree unit or a maximum coding unit. One or more transformation units can be derived from a coding unit. And, the restored video signal decoded and output through the decoding device 10b can be played through a playback device.

The decoding device 10b may receive the signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal may be decoded through the entropy decoding unit 10b-10. For example, the entropy decoder 10b-10 may parse the bitstream to derive information (e.g. video/picture information) necessary for image restoration (or picture restoration). The video/image information may further include information about various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video/image information may further include general constraint information.

The decoding device may decode a picture further based on information about the parameter set and/or the general restriction information. Signaled/received information and/or syntax elements described later in this document may be decoded through the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 10b-10 decodes information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and quantizes the values of syntax elements necessary for image restoration and transform coefficients related to residuals. The values can be output. In more detail, the CABAC entropy decoding method receives bins corresponding to each syntax element from the bitstream, and provides syntax element information to be decoded, decoding information of surrounding and target blocks to be decoded, or information of symbols/bins decoded in the previous step. You can use this to determine a context model, predict the probability of occurrence of a bin according to the determined context model, perform arithmetic decoding of the bin, and generate symbols corresponding to the value of each syntax element. there is. At this time, the CABAC entropy decoding method can update the context model using information on the decoded symbol/bin for the context model of the next symbol/bin after determining the context model. Information about prediction among the information decoded in the entropy decoding unit 10b-10 is provided to the prediction unit 10b-30, and information about the residual for which entropy decoding was performed in the entropy decoding unit 10b-10, that is, Quantized transform coefficients and related parameter information may be input to the inverse quantization unit 10b-21.

Additionally, information about filtering among the information decoded by the entropy decoding unit 10b-10 may be provided to the filtering unit 10b-50. Meanwhile, a receiving unit (not shown) that receives the signal output from the encoding device may be further configured as an internal/external element of the decoding device 10b, or the receiving unit may be a component of the entropy decoding unit 10b-10. there is. Meanwhile, the decoding device according to this document may be called a video/image/picture decoding device, and the decoding device can be divided into an information decoder (video/image/picture information decoder) and a sample decoder (video/image/picture sample decoder). It may be possible. The information decoder may include the entropy decoding unit 10b-10, and the sample decoder may include the inverse quantization unit 10b-21, the inverse transform unit 10b-22, the prediction unit 10b-30, and the addition unit. It may include at least one of a unit 10b-40, a filtering unit 10b-50, and a memory 10b-60.

The inverse quantization unit 10b-21 may inversely quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit 10b-21 may rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the reordering may be performed based on the coefficient scan order performed in the encoding device. The inverse quantization unit 10b-21 may perform inverse quantization on quantized transform coefficients using quantization parameters (eg, quantization step size information) and obtain transform coefficients.

The inverse transform unit 10b-22 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

The prediction unit may perform prediction on the current block and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on the information about the prediction output from the entropy decoding unit 10b-10, and may determine a specific intra/inter prediction mode. .

The prediction unit may generate a prediction signal based on various prediction methods described later. For example, the prediction unit can not only apply intra prediction or inter prediction for prediction of one block, but also can apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). Additionally, the prediction unit may perform intra block copy (IBC) to predict a block. The intra block copy can be used, for example, for video/video coding of content such as games, such as SCC (screen content coding). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can use at least one of the inter prediction techniques described in this document.

The intra prediction unit 10b-32 may predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from the current block depending on the prediction mode. In intra prediction, prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 10b-32 may determine the prediction mode applied to the current block using the prediction mode applied to the neighboring block.

The inter prediction unit 10b-31 may derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector in the reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted on a block, subblock, or sample basis based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information.

In the case of inter prediction, neighboring blocks may include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. For example, the inter prediction unit 10b-31 may construct a motion information candidate list based on neighboring blocks and derive a motion vector and/or reference picture index of the current block based on the received candidate selection information. there is. Inter prediction may be performed based on various prediction modes, and the information about the prediction may include information indicating the mode of inter prediction for the current block.

The adder 10b-40 adds the obtained residual signal to the prediction signal (predicted block, predicted sample array) output from the prediction unit 10b-30 to generate a restored signal (restored picture, restored block, restored sample array). ) can be created. If there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as a restoration block.

The addition unit 10b-40 may be called a restoration unit or a restoration block generation unit. The generated reconstructed signal may be used for intra prediction of the next processing target block in the current picture, may be output after filtering as described later, or may be used for inter prediction of the next picture.

Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

The filtering unit 10b-50 may improve subjective/objective image quality by applying filtering to the restored signal. For example, the filtering unit 10b-50 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 60, specifically the memory 10b- 60) can be transmitted to the DPB. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

The (corrected) reconstructed picture stored in the DPB of the memory 10b-60 can be used as a reference picture in the inter prediction unit 10b-31. The memory 10b-60 may store motion information of a block from which motion information in the current picture is derived (or decoded) and/or motion information of blocks in an already reconstructed picture. The stored motion information can be transmitted to the inter prediction unit 10b-31 to be used as motion information of spatial neighboring blocks or motion information of temporal neighboring blocks. The memory 10b-60 can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 10b-32.

In this specification, the embodiments described in the prediction unit 10b-30, the inverse quantization unit 10b-21, the inverse transform unit 10b-22, and the filtering unit 10b-50 of the decoding device 10b are encoded, respectively. The same or corresponding application may be applied to the prediction unit 10a-20, the inverse quantization unit 10a-34, the inverse transform unit 10a-35, and the filtering unit 10a-60 of the device 10a.

As described above, when performing video coding, prediction is performed to increase compression efficiency. Through this, a predicted block containing prediction samples for the current block, which is the coding target block, can be generated. Here, the predicted block includes prediction samples in the spatial domain (or pixel domain). The predicted block is derived equally from the encoding device and the decoding device, and the encoding device decodes information about the residual between the original block and the predicted block (residual information) rather than the original sample value of the original block itself. Video coding efficiency can be increased by signaling to the device. The decoding device may derive a residual block including residual samples based on the residual information, generate a restored block including restored samples by combining the residual block and the predicted block, and generate the restored blocks. A restored picture containing the image can be generated.

The residual information can be generated through transformation and quantization procedures. For example, the encoding device derives a residual block between the original block and the predicted block, and performs a transformation procedure on residual samples (residual sample array) included in the residual block to derive transformation coefficients. Then, a quantization procedure can be performed on the transform coefficients to derive quantized transform coefficients, and the related residual information can be signaled to the decoding device (via a bitstream). Here, the residual information may include information such as value information of the quantized transform coefficients, position information, transform technique, transform kernel, and quantization parameters. The decoding device may perform an inverse quantization/inverse transformation procedure based on the residual information and derive residual samples (or residual blocks). The decoding device can generate a restored picture based on the predicted block and the residual block. The encoding device may also dequantize/inverse transform the quantized transform coefficients to derive a residual block for reference for inter prediction of a subsequent picture, and generate a restored picture based on this.

<NPU(Neural Processing Unit)>

4 is a schematic conceptual diagram explaining a neural processing unit according to the present disclosure.

The neural processing unit (NPU) 100 shown in FIG. 4 is a processor specialized to perform operations for artificial neural networks.

An artificial neural network refers to a network of artificial neurons that, when multiple inputs or stimuli come in, multiply each by its weight and add it, and additionally transform and transmit the added value through an activation function. The artificial neural network learned in this way can be used to output inference results from input data.

The NPU 100 may be a semiconductor implemented as an electrical/electronic circuit. An electrical/electronic circuit may mean including numerous electronic elements (eg, transistors, capacitors).

In the case of a transformer and/or CNN-based artificial neural network model, the NPU 100 can select and process matrix multiplication operations, convolution operations, etc. according to the architecture of the artificial neural network.

For example, in each layer of a convolutional neural network (CNN), the input feature map corresponding to the input data and the kernel corresponding to the weight may be a matrix composed of a plurality of channels. . A convolution operation of the input feature map and the kernel is performed, and a convolution operation and a pooled output feature map are generated in each channel. An activation map for the corresponding channel is created by applying an activation function to the output feature map. Afterwards, pooling on the activation map may be applied. Here, the activation map may be generically referred to as the output feature map.

However, examples of the present disclosure are not limited to this, and the output feature map means that a matrix multiplication operation or convolution operation, etc. has been applied.

To elaborate, the output feature maps according to examples of the present disclosure should be interpreted in a comprehensive sense. For example, the output feature map may be the result of a matrix multiplication operation or convolution operation. Accordingly, the plurality of processing elements 110 may be modified to further include a processing circuit for an additional algorithm.

The NPU 100 may be configured to include a plurality of processing elements 110 for processing convolution and matrix multiplication required for the above-described artificial neural network operation.

The NPU 100 performs the matrix multiplication operation, convolution operation, activation function operation, pooling operation, stride operation, batch normalization operation, skip connection operation, concatenation operation, quantization operation, clipping operation, and padding operation required for the artificial neural network operation described above. It can be configured to include optimized respective processing circuits.

For example, the NPU 100 is for processing at least one of the activation function operation, pooling operation, stride operation, batch normalization operation, skip connection operation, concatenation operation, quantization operation, clipping operation, and padding operation among the above-described algorithms. It may be configured to include an SFU (150).

Specifically, the NPU 100 may include a plurality of processing elements (PE) 110, SFU 150, NPU internal memory 120, NPU controller 130, and NPU interface 140. there is. Each of the plurality of processing elements 110, SFU 150, NPU internal memory 120, NPU controller 130, and NPU interface 140 may be a semiconductor circuit to which numerous transistors are connected. Therefore, some of them may be difficult to identify and distinguish with the naked eye, and can only be identified through movement.

For example, an arbitrary circuit may operate as a plurality of processing elements 110 or may operate as an NPU controller 130. The NPU controller 130 may be configured to perform the function of a control unit configured to control the artificial neural network inference operation of the NPU 100.

The NPU 100 includes an NPU internal memory 120 configured to store parameters of an artificial neural network model that can be inferred from a plurality of processing elements 110 and the SFU 150, and a plurality of processing elements 110 and the SFU 150. ), and an NPU controller 130 including a scheduler configured to control the operation schedule of the NPU internal memory 120.

The NPU 100 may be configured to process a feature map corresponding to an encoding and decoding method using SVC or SFC.

The plurality of processing elements 110 may perform some operations for an artificial neural network.

SFU 150 may perform other parts of the operations for the artificial neural network.

The NPU 100 may be configured to accelerate the calculation of an artificial neural network model in hardware using a plurality of processing elements 110 and the SFU 150.

The NPU interface 140 may communicate with various components connected to the NPU 100, such as memory, through a system bus.

The NPU controller 130 is a scheduler configured to control the operation of the plurality of processing elements 110 for the inference operation of the neural processing unit 100, the operation of the SFU 150, and the read and write order of the NPU internal memory 120. may include.

The scheduler in the NPU controller 130 may be configured to control a plurality of processing elements 110, the SFU 150, and the NPU internal memory 120 based on data locality information or information on the structure of the artificial neural network model. .

The scheduler within the NPU controller 130 may analyze the structure of an artificial neural network model to operate in the plurality of processing elements 110 and the SFU 150 or may receive information that has already been analyzed. For example, the artificial neural network data that the artificial neural network model can include is node data (i.e. feature map) of each layer, arrangement data of the layers, locality information or structure information, and connecting the nodes of each layer. It may include at least some of the weight data (i.e., weight kernel) of each network. Data of the artificial neural network may be stored in a memory provided inside the NPU controller 130 or in the NPU internal memory 120.

The scheduler in the NPU controller 130 may schedule the operation order of the artificial neural network model to be performed by the NPU 100 based on information about the data locality or structure of the artificial neural network model.

The scheduler in the NPU controller 130 may obtain a memory address value where the feature map and weight data of the layer of the artificial neural network model are stored based on information about the data locality or structure of the artificial neural network model. For example, the scheduler in the NPU controller 130 may obtain the memory address value where the feature map and weight data of the layer of the artificial neural network model stored in the memory are stored. Therefore, the scheduler in the NPU controller 130 can retrieve the feature map and weight data of the layer of the artificial neural network model to be run from the main memory and store them in the NPU internal memory 120.

The feature map of each layer may have a corresponding memory address value.

Each weight data may have a corresponding memory address value.

The scheduler in the NPU controller 130 performs a plurality of processing based on data locality information or information on the structure of the artificial neural network model, for example, placement data, locality information, or information on the structure of the artificial neural network layers of the artificial neural network model. The operation order of the element 110 can be scheduled.

Since the scheduler in the NPU controller 130 performs scheduling based on data locality information or information on the structure of the artificial neural network model, it may operate differently from the scheduling concept of a general CPU. Typical CPU scheduling takes into account fairness, efficiency, stability, response time, etc. and operates to achieve the best efficiency. In other words, scheduling is performed to perform the most processing within the same amount of time, taking into account priority, computation time, etc.

Conventional CPUs used an algorithm to schedule tasks by considering data such as priority order of each processing and operation processing time.

Differently, the scheduler in the NPU controller 130 may control the NPU 100 in the processing order of the NPU 100 determined based on data locality information or information on the structure of the artificial neural network model.

Furthermore, the scheduler in the NPU controller 130 determines the processing order based on information about the data locality information or structure of the artificial neural network model and/or information about the data locality information or structure of the neural processing unit 100 to be used. The NPU (100) can be driven as is.

However, the present disclosure is not limited to information on data locality or structure of the NPU 100.

The scheduler in the NPU controller 130 may be configured to store information about data locality or structure of the artificial neural network.

In other words, the scheduler in the NPU controller 130 can determine the processing order even if only using at least information about the data locality information or structure of the artificial neural network model of the artificial neural network model.

Furthermore, the scheduler in the NPU controller 130 can determine the processing order of the NPU 100 by considering the information about the data locality information or structure of the artificial neural network model and the information about the data locality information or structure of the NPU 100. there is. Additionally, it is possible to optimize processing of the NPU 100 in the determined processing order.

The plurality of processing elements 110 refers to a configuration in which a plurality of processing elements (PE1 to PE12) configured to calculate feature maps and weight data of an artificial neural network are arranged. Each processing element may include a multiply and accumulate (MAC) operator and/or an Arithmetic Logic Unit (ALU) operator. However, examples according to the present disclosure are not limited thereto.

Each processing element may be configured to optionally further include additional special function units to process additional special functions.

For example, the processing element (PE) may be modified to further include a batch-normalization unit, an activation function unit, an interpolation unit, etc.

The SFU (150) performs activation function operation, pooling operation, stride operation, batch-normalization operation, skip-connection operation, concatenation operation, and quantization operation. , may include a circuit unit configured to select and process clipping operations, padding operations, etc. according to the architecture of the artificial neural network. That is, the SFU 150 may include a plurality of special function operation processing circuit units.

Although a plurality of processing elements are exemplarily shown in FIG. 4, instead of a MAC, inside one processing element, operators implemented as a plurality of multipliers and adder trees are arranged in parallel. possible. In this case, the plurality of processing elements 110 may also be referred to as at least one processing element including a plurality of operators.

The plurality of processing elements 110 are configured to include a plurality of processing elements (PE1 to PE12). The plurality of processing elements (PE1 to PE12) shown in FIG. 4 are merely examples for convenience of explanation, and the number of processing elements (PE1 to PE12) is not limited. The size or number of the plurality of processing elements 110 may be determined depending on the number of the plurality of processing elements (PE1 to PE12). The size of the plurality of processing elements 110 may be implemented in the form of an N x M matrix. Here N and M are integers greater than 0. The plurality of processing elements 110 may include N x M processing elements. That is, there may be one or more processing elements.

The size of the plurality of processing elements 110 can be designed considering the characteristics of the artificial neural network model on which the NPU 100 operates.

The plurality of processing elements 110 are configured to perform functions such as addition, multiplication, and accumulation required for artificial neural network operations. In other words, the plurality of processing elements 110 may be configured to perform a multiplication and accumulation (MAC) operation.

Hereinafter, the first processing element PE1 among the plurality of processing elements 110 will be described as an example.

FIG. 5A is a schematic conceptual diagram illustrating one processing element among a plurality of processing elements applicable to the present disclosure.

The NPU 100 according to an example of the present disclosure includes a plurality of processing elements 110, an NPU internal memory 120 configured to store an artificial neural network model that can be inferred from the plurality of processing elements 110, and a plurality of processing elements. (110) and an NPU controller 130 configured to control the NPU internal memory 120, a plurality of processing elements 110 are configured to perform a MAC operation, and the plurality of processing elements 110 are configured to perform the MAC operation result. It can be configured to quantize and output. However, the examples of the present disclosure are not limited thereto.

The NPU internal memory 120 can store all or part of the artificial neural network model depending on the memory size and the data size of the artificial neural network model.

The first processing element PE1 may include a multiplier 111, an adder 112, an accumulator 113, and a bit quantization unit 114. However, examples according to the present disclosure are not limited thereto, and the plurality of processing elements 110 may be modified and implemented in consideration of the computational characteristics of the artificial neural network.

The multiplier 111 multiplies the input (N) bit data and (M) bit data. The operation value of the multiplier 111 is output as (N+M) bit data.

The multiplier 111 may be configured to receive one variable and one constant as input.

The accumulator 113 accumulates the operation value of the multiplier 111 and the operation value of the accumulator 113 using the adder 112 the number of (L) loops. Therefore, the bit width of the data of the output and input portions of the accumulator 113 can be output as (N+M+log2(L)) bits. Here, L is an integer greater than 0.

When accumulation is completed, the accumulator 113 can receive an initialization reset signal and initialize the data stored inside the accumulator 113 to 0. However, examples according to the present disclosure are not limited thereto.

The bit quantization unit 114 can reduce the bit width of data output from the accumulator 113. The bit quantization unit 114 may be controlled by the NPU controller 130. The bit width of quantized data can be output as (X)bit. Here, X is an integer greater than 0. According to the above-described configuration, the plurality of processing elements 110 are configured to perform MAC operation, and the plurality of processing elements 110 have the effect of quantizing the MAC operation result and outputting it. In particular, this quantization has the effect of further reducing power consumption as (L)loops increase. Additionally, reducing power consumption has the effect of reducing heat generation. In particular, reducing heat generation has the effect of reducing the possibility of malfunction due to high temperature of the NPU (100).

The output data (X) bit of the bit quantization unit 114 may be node data of the next layer or input data of convolution. If the artificial neural network model is quantized, the bit quantization unit 114 may be configured to receive quantized information from the artificial neural network model. However, it is not limited to this, and the NPU controller 130 may be configured to extract quantized information by analyzing an artificial neural network model. Therefore, the output data (X) bit can be converted to the quantized bit width and output to correspond to the quantized data size. The output data (X) bit of the bit quantization unit 114 may be stored in the NPU internal memory 120 as a quantized bit width.

The plurality of processing elements 110 of the NPU 100 according to an example of the present disclosure include a multiplier 111, an adder 112, and an accumulator 113. The bit quantization unit 114 can be selected depending on whether quantization is applied.

Figure 5b is a schematic conceptual diagram explaining an SFU applicable to the present disclosure.

Referring to Figure 5b, SFU 150 includes several functional units. Each functional unit can be operated selectively. Each functional unit can be selectively turned on or off. That is, each functional unit can be set.

In other words, the SFU 150 may include various circuit units required for artificial neural network inference calculations.

For example, the circuit units of the SFU 150 include a functional unit for a skip-connection operation, a functional unit for an activation function operation, a functional unit for a pooling operation, and quantization. Functional unit for (quantization) operation, functional unit for NMS (non-maximum suppression) operation, functional unit for integer and floating point conversion (INT to FP32) operation, functional unit for batch-normalization operation. , it may include a functional unit for an interpolation operation, a functional unit for a concatenation operation, and a functional unit for a bias operation.

Functional units of the SFU 150 can be selectively turned on or turned off based on data locality information of the artificial neural network model. The data locality information of the artificial neural network model may include turn-off or control information related to the turn-off of the corresponding functional unit when an operation for a specific layer is performed.

Among the functional units of the SFU 150, an activated unit may be turned on. In this way, when some functional units of the SFU 150 are selectively turned off, the power consumption of the NPU 100 can be reduced. Meanwhile, power gating can be used to turn off some functional units. Alternatively, clock gating may be performed to turn off some functional units.

FIG. 6 is an exemplary diagram showing a modified example of the NPU 100 shown in FIG. 4.

Since the NPU 100 shown in FIG. 6 is substantially the same as the processing unit 100 shown by way of example in FIG. 4 except for a plurality of processing elements 110, the following is only for convenience of description. Redundant descriptions can be omitted.

The plurality of processing elements 110 illustratively shown in FIG. 6 include, in addition to a plurality of processing elements (PE1 to PE12), respective register files (RF1 to RF12) corresponding to each of the processing elements (PE1 to PE12). ) may further be included.

The plurality of processing elements (PE1 to PE12) and the plurality of register files (RF1 to RF12) shown in FIG. 6 are examples only for convenience of explanation, and the plurality of processing elements (PE1 to PE12) and the plurality of registers The number of files (RF1 to RF12) is not limited.

The size or number of the plurality of processing elements 110 may be determined depending on the number of processing elements (PE1 to PE12) and the number of register files (RF1 to RF12). The sizes of the plurality of processing elements 110 and the plurality of register files (RF1 to RF12) may be implemented in the form of an N x M matrix. Here N and M are integers greater than 0.

The array size of the plurality of processing elements 110 can be designed considering the characteristics of the artificial neural network model on which the NPU 100 operates. To elaborate, the memory size of the register file can be determined by considering the data size of the artificial neural network model to be operated, required operation speed, required power consumption, etc.

The register files (RF1 to RF12) of the NPU (100) are static memory units directly connected to the processing elements (PE1 to PE12). Register files (RF1 to RF12) may be composed of, for example, flip-flops and/or latches. Register files (RF1 to RF12) may be configured to store MAC operation values of corresponding processing elements (PE1 to PE12). Register files (RF1 to RF12) may be configured to provide or receive NPU internal memory 120, weight data, and/or node data.

The register files (RF1 to RF12) can also be configured to perform the function of the accumulator's temporary memory during MAC operation.

7 is a schematic conceptual diagram illustrating an exemplary artificial neural network model.

Hereinafter, the operation of the exemplary artificial neural network model 110-10 that can be operated on the NPU 100 will be described.

The exemplary artificial neural network model 110-10 in FIG. 7 may be an artificial neural network learned in the NPU 100 shown in FIG. 4 or 6 or in a separate machine learning device. An artificial neural network model may be an artificial neural network trained to perform various inference functions such as object recognition and voice recognition.

The artificial neural network model 110-10 may be a deep neural network (DNN).

However, the artificial neural network model 110-10 according to examples of the present disclosure is not limited to deep neural networks.

For example, artificial neural network models are trained to perform inferences such as Object Detection, Object Segmentation, Image/Video Reconstruction, Image/Video Enhancement, Object Tracking, Event Recognition, Event Prediction, Anomaly Detection, Density Estimation, Event Search, and Measurement. It could be a model to become.

For example, artificial neural network models can be Bisenet, Shelfnet, Alexnet, Densenet, Efficientnet, EfficientDet, Googlenet, Mnasnet, Mobilenet, Resnet, Shufflenet, Squeezenet, VGG, Yolo, RNN, CNN, DBN, RBM, LSTM, etc. there is. For example, artificial neural network models may be generative adversarial networks (GAN), transformer, etc. However, the present disclosure is not limited to this, and new artificial neural network models to operate on NPU are continuously being announced.

However, the present disclosure is not limited to the above-described models. Additionally, the artificial neural network model 110-10 may be an ensemble model based on at least two different models.

The artificial neural network model 110-10 may be stored in the NPU internal memory 120 of the NPU 100.

Hereinafter, with reference to FIG. 7, the inference process performed by the NPU 100 on the exemplary artificial neural network model 110-10 will be described.

The artificial neural network model (110-10) includes an input layer (110-11), a first connection network (110-12), a first hidden layer (110-13), a second connection network (110-14), and a second hidden layer ( 110-15), a third connection network 110-16, and an output layer 110-17. However, the present disclosure is not limited to the artificial neural network model shown in FIG. 7. The first hidden layer 110-13 and the second hidden layer 110-15 may also be referred to as a plurality of hidden layers.

The input layer 110-11 may illustratively include x1 and x2 input nodes. That is, the input layer 110-11 may include information about two input values. The scheduler in the NPU controller 130 shown in FIG. 4 or 6 assigns a memory address where information about the input value from the input layer 110-11 is stored to the NPU internal memory 120 shown in FIG. 4 or 6. It can be set to .

The first connection network 110-12 illustratively includes information on six weight values for connecting each node of the input layer 110-11 to each node of the first hidden layer 110-13. It can be included. The scheduler in the NPU controller 130 shown in FIG. 4 or 6 may set a memory address in the NPU internal memory 120 where information about the weight value of the first connection network 110-12 is stored. Each weight value is multiplied by the input node value, and the accumulated value of the multiplied values is stored in the first hidden layer 110-13. Here, nodes may be referred to as feature maps.

The first hidden layer 110-13 may exemplarily include nodes a1, a2, and a3. That is, the first hidden layer 110-13 may include information about three node values. The scheduler in the NPU controller 130 shown in FIG. 4 or 6 may set a memory address for storing information about the node value of the first hidden layer 110-13 in the NPU internal memory 120.

The scheduler in the NPU controller 130 may be configured to schedule an operation order so that the first processing element (PE1) performs the MAC operation of the a1 node of the first hidden layer (110-13). The scheduler in the NPU controller 130 may be configured to schedule an operation order so that the second processing element (PE2) performs the MAC operation of the a2 node of the first hidden layer (110-13). The scheduler in the NPU controller 130 may be configured to schedule an operation order so that the third processing element (PE3) performs the MAC operation of the a3 node of the first hidden layer (110-13). Here, the scheduler in the NPU controller 130 may pre-schedule the operation order so that the three processing elements each perform the MAC operation simultaneously in parallel.

The second connection network 110-14 illustratively includes nine weight values for connecting each node of the first hidden layer 110-13 to each node of the second hidden layer 110-15. May contain information. The scheduler in the NPU controller 130 shown in FIG. 4 or 6 may set a memory address for storing information about the weight value of the second connection network 110-14 in the NPU internal memory 120. The weight value of the second network 110-14 is multiplied by the node value input from the first hidden layer 110-13, and the accumulated value of the multiplied values is sent to the second hidden layer 110-15. It is saved.

The second hidden layer 110-15 may exemplarily include nodes b1, b2, and b3. That is, the second hidden layer 110-15 may include information about three node values. The scheduler within the NPU controller 130 may set a memory address for storing information about the node value of the second hidden layer 110-15 in the NPU internal memory 120.

The scheduler in the NPU controller 130 may be configured to schedule an operation order so that the fourth processing element (PE4) performs the MAC operation of the b1 node of the second hidden layer 110-15. The scheduler in the NPU controller 130 may be configured to schedule an operation order so that the fifth processing element (PE5) performs the MAC operation of the b2 node of the second hidden layer 110-15. The scheduler in the NPU controller 130 may be configured to schedule an operation order so that the sixth processing element (PE6) performs the MAC operation of the b3 node of the second hidden layer 110-15.

Here, the scheduler in the NPU controller 130 may pre-schedule the operation order so that the three processing elements each perform the MAC operation simultaneously in parallel.

Here, the scheduler in the NPU controller 130 may determine scheduling so that the operation of the second hidden layer (110-15) is performed after the MAC operation of the first hidden layer (110-13) of the artificial neural network model.

That is, the scheduler in the NPU controller 130 may be configured to control the plurality of processing elements 100 and the NPU internal memory 120 based on information about the data locality or structure of the artificial neural network model.

The third network 110-16 illustratively includes information on six weight values connecting each node of the second hidden layer 110-15 and each node of the output layer 110-17. can do. The scheduler within the NPU controller 130 may set a memory address for storing information about the weight value of the third connection network 110-16 in the NPU internal memory 120. The weight value of the third network 110-16 is multiplied by the node value input from the second hidden layer 110-15, and the accumulated value of the multiplied values is stored in the output layer 110-17.

The output layer 110-17 may illustratively include y1 and y2 nodes. That is, the output layer 110-17 may include information about two node values. The scheduler within the NPU controller 130 may set a memory address in the NPU internal memory 120 to store information about the node value of the output layer 110-17.

The scheduler in the NPU controller 130 may be configured to schedule an operation order so that the seventh processing element (PE7) performs the MAC operation of the y1 node of the output layer 110-17. The scheduler in the NPU controller 130 may be configured to schedule an operation order so that the eighth processing element (PE8) performs the MAC operation of the y2 node of the output layer 110-15.

Here, the scheduler in the NPU controller 130 may pre-schedule the operation order so that the two processing elements each perform the MAC operation simultaneously in parallel.

Here, the scheduler in the NPU controller 130 may determine scheduling so that the operation of the output layer (110-17) is performed after the MAC operation of the second hidden layer (110-15) of the artificial neural network model.

That is, the scheduler in the NPU controller 130 may be configured to control the plurality of processing elements 100 and the NPU internal memory 120 based on information about the data locality or structure of the artificial neural network model.

That is, the scheduler in the NPU controller 130 can analyze the structure of the artificial neural network model to operate in the plurality of processing elements 100 or receive the analyzed information. The artificial neural network information that the artificial neural network model can include is information about the node value of each layer, information about the locality or structure of the arrangement data of the layers, and information about the weight value of each network connecting the nodes of each layer. May contain information.

Since the scheduler in the NPU controller 130 is provided with information about the data locality information or structure of the exemplary artificial neural network model 110-10, the scheduler in the NPU controller 130 inputs the artificial neural network model 110-10. You can understand the operation sequence from start to output.

Accordingly, the scheduler within the NPU controller 130 may set the memory address where the MAC operation values of each layer are stored in the NPU internal memory 120 in consideration of the scheduling order.

The NPU internal memory 120 may be configured to preserve weight data of connection networks stored in the NPU internal memory 120 while the inference operation of the NPU 100 continues. Therefore, there is an effect of reducing memory read and write operations.

That is, the NPU internal memory 120 may be configured to reuse the MAC operation value stored in the NPU internal memory 120 while the inference operation continues.

Figure 8a is a diagram for explaining the basic structure of a convolutional neural network.

Referring to FIG. 8A, a convolutional neural network may be a combination of one or more convolutional layers, a pooling layer, and a fully connected layer.

In the example of the present disclosure, the convolutional neural network has a kernel for each channel that extracts features of the input image of the channel. The kernel may be composed of a two-dimensional matrix and performs convolution operations while traversing the input data. The size of the kernel can be determined arbitrarily, and the interval (stride) at which the kernel traverses the input data can also be arbitrarily determined. The convolution result for all input data per kernel may be referred to as a feature map or activation map. Hereinafter, the kernel may include one set of weight values or multiple sets of weight values. The number of kernels for each layer may be referred to as the number of channels. Kernels may be referred to as weights in matrix form, or kernels may be referred to as weights.

In this way, since the convolution operation is an operation consisting of a combination of input data and a kernel, an activation function to add nonlinearity can be applied later. When an activation function is applied to a feature map that is the result of a convolution operation, it may be referred to as an activation map.

Referring specifically to FIG. 8A, the convolutional neural network includes at least one convolutional layer, at least one pooling layer, and at least one fully connected layer.

For example, convolution is based on two main parameters: the size of the input data (usually 1Х1, 3Х3 or 5Х5 matrices) and the depth of the output feature map (number of kernels). can be defined. These key parameters can be computed by convolution. These convolutions may start at depth 32, continue to depth 64, and end at depth 128 or 256. The convolution operation may refer to an operation that slides a kernel of size 3Х3 or 5Х5 over the input image matrix, which is input data, multiplies each weight of the kernel by each element of the overlapping input image matrix, and then adds them all together.

An activation function is applied to the output feature map generated in this way, and the activation map can be finally output. Additionally, the weights used in the current layer can be passed to the next layer through convolution. The pooling layer can perform a pooling operation to reduce the size of the feature map by downsampling the output data (i.e., activation map). For example, the pooling operation may include, but is not limited to, max pooling and/or average pooling.

The maximum pooling operation uses a kernel, and the feature map and kernel are slid to output the maximum value in the area of the feature map that overlaps the kernel. The average pooling operation slides the feature map and the kernel to output the average value within the area of the feature map that overlaps the kernel. As the size of the feature map is reduced by the pooling operation, the number of weights in the feature map is also reduced.

The fully connected layer can classify data output through the pooling layer into multiple classes (i.e., estimated values) and output the classified classes and their scores. The data output through the pooling layer takes the form of a 3D feature map, and this 3D feature map can be converted into a 1D vector and input into a fully connected layer.

Figure 8b is a comprehensive diagram showing the operation of a convolutional neural network in an easy-to-understand manner.

Referring to FIG. 8B, the input image is exemplarily shown as a two-dimensional matrix with a size of 6 x 6. In addition, Figure 8b shows that three channels, that is, channel 1, channel 2, and channel 3, are used as an example.

First, let's explain the convolution operation.

The input image (illustratively shown to have a size of 6 x 6 in Figure 8B) is convolved with kernel 1 for channel 1 (illustrated to have a size of 3 x 3 in Figure 8B) at the first node, As a result, feature map 1 (illustratively shown as having a size of 4 x 4 in FIG. 8B) is output. Additionally, the input image (illustratively shown to have a size of 6 x 6 in FIG. 8B) is synthesized with kernel 2 for channel 2 (illustrated to have a size of 3 x 3 in FIG. 8B) at the second node. It is multiplied, and feature map 2 (illustratively shown as having a size of 4 x 4 in FIG. 8B) is output as a result. In addition, the input image is convolutioned with kernel 3 for channel 3 (illustratively shown as having a size of 3 x 3 in FIG. 8B) at the third node, and as a result, feature map 3 (illustratively shown in FIG. 8B (indicated to be 4 x 4 in size) is output.

In order to process each convolution, the processing elements (PE1 to PE12) of the NPU 100 are configured to perform MAC operation.

Next, we will explain the operation of the activation function.

An activation function can be applied to feature map 1, feature map 2, and feature map 3 (each size is exemplarily shown as 4 x 4 in FIG. 8B) output from the convolution operation. The output after the activation function is applied may, for example, have a size of 4 x 4.

Next, the polling operation will be explained.

Feature map 1, feature map 2, and feature map 3 output from the activation function (each size is exemplarily shown as 4 x 4 in FIG. 8B) are input as three nodes. Polling can be performed by receiving feature maps output from the activation function as input. The polling can reduce the size or emphasize specific values in the matrix. Polling methods include maximum polling, average polling, and minimum polling. Maximum polling can be used to collect the maximum value within a specific region of a matrix, and average polling can be used to find the average within a specific region.

In the example of Figure 8b, a feature map of size 4 x 4 is shown to be reduced to size of 2 x 2 by polling.

Specifically, the first node receives feature map 1 for channel 1 as input, performs polling, and outputs it as, for example, a 2 x 2 matrix. The second node receives feature map 2 for channel 2 as input, performs polling, and outputs it as, for example, a 2 x 2 matrix. The third node receives feature map 3 for channel 3 as input, performs polling, and outputs it as, for example, a 2 x 2 matrix.

The above-described convolution, activation function, and polling are repeated, and finally, fully connected can be output as shown in FIG. 8a. The output can be input back into an artificial neural network for image recognition. However, the present disclosure is not limited to the size of the feature map or kernel.

<VCM(Video coding for Machines)>

Recently, as various industrial fields such as Surveillance, Intelligent Transportation, Smart City, Intelligent Industry, and Intelligent Content develop, the amount of image or feature map data consumed by machines is increasing. In contrast, the traditional video compression method currently in use is a technology developed considering the characteristics of human vision as perceived by viewers, and therefore contains unnecessary information, making it inefficient in performing machine tasks. Therefore, research on video codec technology to efficiently compress feature maps for machine mission performance is required.

Video Coding for Machine (VCM) technology is being discussed at MPEG (Moving Picture Experts Group), an international standardization group for multimedia encoding. VCM is an image or feature map encoding technology that is based on the machine's data consumption vision (Machine Vision) rather than the human viewer's vision.

<Description of the disclosure of this specification>

Figures 9a to 9d are illustrations showing an NPU including a VCM encoder and an NPU including a VCM decoder.

Referring to FIG. 9A, the first NPU (100a) may include a VCM encoder (150a), and the second NPU (100b) may include a VCM decoder (150b).

When the VCM encoder (150a) in the first NPU (100a) encodes the video and/or feature map and transmits it as a bitstream, the VCM decoder (150b) in the second NPU (100b) can decode and output the bitstream. there is. At this time, the VCM decoder 150b within the second NPU 100b may output one or more videos and/or feature maps. For example, the VCM decoder 150b in the second NPU 100b can output a first feature map for analysis using a machine and output a first image for viewing by a user. The first image may have higher resolution than the first feature map.

Referring to FIG. 9B, the first NPU 100a may include a feature extractor and a VCM encoder 150a that extract a feature map.

The VCM encoder 150a within the first NPU 100a may include a feature encoder. The second NPU (100b) may include a VCM decoder (150b). The VCM decoder 150b in the second NPU 100b may include a feature decoder and a video reconstructor. The feature decoder may decode a feature map from a bitstream and output a first feature map for analysis using a machine. The video regenerator may reproduce and output a first video for viewing by a user from a bitstream.

Referring to FIG. 9C, the first NPU 100a may include a feature extractor and a VCM encoder 150a that extract a feature map.

The VCM encoder 150a within the first NPU 100a may include a feature encoder. The second NPU (100b) may include a VCM decoder (150b). The VCM decoder 150b within the second NPU 100b may include a feature decoder. The feature decoder can decode a feature map from a bitstream and output a first feature map for analysis using a machine. In other words, the bitstream can be encoded only as a feature map, not as an image. To elaborate, a feature map may be data containing information about features for processing a specific task of a machine based on an image.

Referring to FIG. 9D, the first NPU 100a may include a feature extractor and a VCM encoder 150a that extract a feature map.

The VCM encoder 150a within the first NPU 100a may include a feature converter and a video encoder. The second NPU (100b) may include a VCM decoder (150b). The VCM decoder 150b within the second NPU 100b may include a video decoder and an inverse converter.

The feature extractor shown in FIGS. 9B to 9D may correspond to a plurality of PEs 110 in the NPU 100 shown in FIG. 4 or 6. Alternatively, the feature extractor shown in FIGS. 9B to 9D may correspond to a combination of a plurality of PEs 110 in the NPU 100 and the SFU described above.

As described above, in FIGS. 9A to 9D, the first NPU (100a) includes at least a VCM encoder (150a), and the second NPU (100b) includes at least a VCM decoder (150b). However, the present invention is not limited to this, and the VCM encoder 150a may be modified to include the first NPU 100a, or the VCM decoder 150b may be modified to include the second NPU 100b.

The first NPU 100a may generate a feature map by processing an artificial intelligence operation (eg, convolution). The first NPU (100a) can process artificial intelligence operations, encode the feature map, and then transmit it.

The second NPU 100b may receive the encoded feature map. The second NPU 100b may decode the encoded feature map by processing an artificial intelligence operation (eg, deconvolution).

To process artificial intelligence calculations, an artificial neural network model with a specific structure can be used. For example, to extract feature maps, the NPU can process convolution operations. For example, to encode feature maps, the NPU can process convolution operations. For example, to decode an encoded feature map, the NPU can process a deconvolution operation.

The artificial neural network model may have a multi-layer structure, and the artificial neural network model may include a backbone network. The feature map generated through the artificial intelligence operation of the first NPU 100a may be a feature map generated in a specific layer of a multi-layer artificial neural network model. That is, the feature map may be at least one feature map generated from at least one layer of an artificial neural network model with a multi-layer structure. A feature map generated from a specific layer of a multi-layer artificial neural network model may be a feature map suitable for analysis using a specific machine.

Figures 10a and 10b are exemplary diagrams showing the positions of bitstreams in the artificial neural network model.

As can be seen with reference to FIG. 10A, when the first NPU (100a) or the VCM encoder receives video, it uses an artificial neural network model (e.g., a convolutional network model) to create each feature map for each layer. can be created. Figure 10a shows an example of transmitting a feature map in a fully connected layer corresponding to the last layer of the convolutional network model as a bitstream.

Then, the second NPU (100b) or the VCM decoder can decode the bitstream including the feature map using a deconvolution network model.

Meanwhile, referring to FIG. 10b, the feature maps from the fully connected layer are not transmitted as a bitstream, but the feature maps generated from the middle layer of the artificial neural network model (e.g., convolutional network model) are An example of transmission as a bitstream is shown.

FIG. 11A shows a VCM encoder that compresses a feature map and transmits it as a bitstream, and a VCM decoder that restores the received bitstream, according to an example, and FIG. 11B generates feature maps P2 to P5 shown in FIG. 11A. Shows an example. And, FIG. 11C is an example diagram showing the MSFF execution unit shown in FIG. 11A in detail.

As can be seen with reference to FIG. 11A, the VCM encoder 150a (or transmitter) includes a multi-scale feature fusion (MSFF) performing unit 150a-1 and a single-stream feature codec (SSFC) encoder 150a- 2) may be included. The VCM decoder 150b (or receiving end) may include an SSFC decoder 150b-1 and a Multi-Scale Feature Reconstruction (MSFR) performing unit 150b-2.

The MSFF performing unit 150a-1 aligns the feature maps P2 to P5 of the P layer and then concatenates them. Feature maps P2 to P5 of the P layer can be generated as shown in FIG. 11B. Referring to FIG. 11B , feature maps P2 to P5 may be generated using a combination of a first artificial neural network model, such as ResNet, and a second artificial neural network model, such as FPN (Feature Pyramid Network). The first artificial neural network model may be called a backbone, and the second artificial neural network model may be called a neck or head. As shown in FIG. 11B, when an image is input, the first artificial neural network model (eg, ResNet) can perform several stages (eg, Stage 1 to Stage 5) in a bottom-up manner. Each step may include, for example, convolution, batch normalization, ReLu (Rectified Linear Unit), etc. Convolution may be performed on the output from each stage of the first artificial neural network model (eg, ResNet) to generate feature maps C2 to C5 of the C layer. The second artificial neural network model (eg, FPN) may receive feature maps C2 to C5 of the C layer and accumulate them in a top-down manner to generate feature maps M2 to M5 of the M layer. By performing convolution on the feature maps M2 to M5 of the M layer, feature maps P2 to P5 of the P layer can be generated.

Referring again to FIG. 11A, the MSFF performing unit 150a-1 aligns the feature maps P2 to P5 of the P layer and then concatenates them. Specifically, the MSFF performing unit 150a-1 downsamples the size of the feature maps P2 to P4 to match the size of the feature map P5, and then concatenates them.

In detail with reference to FIG. 11C, the MSFF performing unit 150a-1 performs pooling on each of the feature maps P2 to P5. By the pooling, each of the feature maps P2 to P5 can be reduced to a specific size, for example, 64 x 64 or 32 x 32. Alternatively, the remaining feature maps (i.e., feature maps P2, feature maps P3, and P4) may be made to have the same size based on feature map P5, which has the smallest size among feature maps P2 to P5. That is, pooling can be performed on each of the remaining feature maps (i.e., feature maps P2, feature maps P3, and P4) to make the size the same as feature map P5.

Then, the MSFF performing unit 150a-1 can concatenate feature maps P2 to P5. For example, if feature maps P2 to P5 each have a cube shape of 64

The concatenated feature maps can pass through the SEblock.

Referring again to FIG. 11A, the SEBlock is a processing operation based on SE (Squeeze-and-Excitation), which calculates a weight for each channel of the feature map and multiplies this weight by the output feature map of the residual unit. Additionally, the MSFF performing unit may perform channel-wise reduction on the output of the SEBlock using convolution. The MSFF performing unit may finally transmit the output F to the SSFC encoder (150a-2). When the SSFC encoder (150a-2) receives the output F from the MSFF performing unit (150a-1), the channel After reducing the number, the bitstream can be transmitted. To this end, the SSFC encoder 150a-2 may include convolution, batch normalization, and Tanh functions.

The VCM decoder 150b (i.e., receiving end) may include an MSFR performing unit 150b-1 and an SSFC decoder 150b-2. When the SSFC decoder 150b-2 receives the bitstream, it can generate output F'. To this end, the SSFC decoder 150b-2 may include convolution, batch normalization, PReLu, etc.

The MSFR performing unit 150b-1 of the VCM decoder 150b can restore feature maps P2' to feature maps P5' of the P' layer from the output F' from the SSFC decoder (150b-2). To this end, the MSFR performing unit 150b-1 may perform up-sampling/down-sampling and accumulation.

FIG. 12 is an exemplary diagram showing a modification of FIG. 11A.

Hereinafter, only the parts that are different from those of FIG. 11A will be described, and the same parts will not be repeatedly explained, and the description of FIG. 11A will be used.

As can be seen with reference to FIG. 12, the VCM encoder 150a may further include a compressor 150a-3. And the VCM decoder 150b may further include a decompressor 150b-3.

The MSFF performing unit 150a-1 of the VCM encoder 150a may selectively receive one or more of the plurality of feature maps extracted from the feature map extractor shown in FIGS. 9B to 9D. For a specific example, let's assume that the feature map extractor shown in FIGS. 9B to 9D extracts feature maps P2 to P5 as shown in FIG. 11B. Then, the MSFF performing unit 150a-1 of the VCM encoder 150a can selectively receive one or more feature maps from among the feature maps P2 to P5 as needed. In the example of FIG. 12, among the feature maps P2 to P5, the MSFF performance unit 150a-1 of the VCM encoder 150a is shown to selectively receive feature maps P2, P3, and P5. It's messy.

The compressor 150a-3 in the VCM encoder 150a may compress the output from the SSFC encoder 150a-2. For the compression, the compressor 150a-3 in the VCM encoder 150a uses Versatile Video Coding (VVC) (or H.266), context-based adaptive binary arithmetic coding (CABAC), principal component analysis (PCA), DCT (discrete cosine transform), etc. can be performed. Alternatively, for the compression, the compressor 150a-3 in the VCM encoder 150a may use an entropy coding technique or the like.

When the compression is performed in this way, the size of the transmitted bitstream can be greatly reduced, thereby saving network bandwidth.

The decompressor 150b-3 of the VCM decoder 150b can decompress the received bitstream. To this end, the decompressor 150b-3 of the VCM decoder 150b can perform VVC (i.e., H.266), Inverse DCT (IDCT), Inverse PCA (IPCA), etc.

The MSFR performing unit 150b-1 of the VCM decoder 150b can restore feature maps P2' to feature maps P5' from the output F' from the SSFC decoder (150b-2). Additionally, the MSFR performing unit 150b-1 of the VCM decoder 150b may transmit one or more of the restored feature maps P2' to feature maps P5' to each task. The task may include, for example, object classification, object detection, object tracking, object segmentation, etc. That is, the MSFR performing unit 150b-1 of the VCM decoder 150b can transfer feature map P5' from among the restored feature maps P2' to feature maps P5' to task #1, feature map P2', and feature map P5'. Map P3' and feature map P4' can be passed to task #2.

FIG. 13A is a conceptual diagram illustrating an example of distributed processing between an edge device and a cloud server according to an disclosure of the present specification, based on the size of data for each layer in an exemplary artificial neural network model. FIG. 13B is an example table showing the data size for each layer in the example artificial neural network model shown in FIG. 13A.

The example artificial neural network model shown in FIG. 13A is shown to be Mobilenet V1. Referring to FIG. 13A, the horizontal axis sequentially represents layers in an exemplary artificial neural network model, and the vertical axis represents the size of data.

Referring to layer 1 shown in FIGS. 13A and 13B, it can be seen that the size of the output feature map (OFMAP_SIZE) is larger than the size of the input feature map (IFMAP_SIZE).

However, referring to layer 12 shown in Figures 13a and 13b, the size of the output feature map (OFMAP_SIZE) is 50,176 bytes (about 50Kb), which is not only smaller than the size of the input feature map (IFMAP_SIZE) of 200,704 bytes, but also the size of layer 12 The size of the output feature map (OFMAP_SIZE) is smaller than the sizes of the output feature maps of the front and rear adjacent layers.

Additionally, the total size of the weights from layer 1 to layer 12 is very small, at 133Kb. Additionally, the size of the output feature map of layer 12 is about 50Kb, which is also very small.

Therefore, the NPU of each surveillance camera can transmit the output feature map of layer 12 in bitstream form. If this method of transmitting small-sized feature maps is selected, even if the number of surveillance cameras greatly increases, it may not cause a large load or congestion on the communication network.

Therefore, as shown in FIG. 13A, it is possible to consider processing layers 1 to 12 in the NPU of the edge terminal and then transmitting the output feature map of layer 12 to the cloud server by wire or wirelessly.

The NPU of the cloud server can process layers 13 to 28 by using the received output feature map of layer 12 as the input feature map of layer 13.

Meanwhile, the output feature map of layer 12 may be transmitted in the form of a bitstream through the MSFF performing unit 150a-1 and the SSFC encoder 150a-2, as shown in FIG. 11A. That is, the size of the output feature map of layer 12 can be compressed through the MSFF performing unit and the SSFC encoder 150a-2 to become smaller.

In this way, by selecting and transmitting the output feature map with a smaller size among the layers in the artificial neural network model, the computing load can be distributed while reducing network traffic.

FIG. 14A is an exemplary diagram illustrating feature maps from layer 9 to layer 13 of the artificial neural network model shown in FIG. 13A.

Referring to FIG. 14A, when an arbitrary operation (eg, convolution operation) is performed on layer 9 of the artificial neural network model shown in FIG. 13A, feature map P9 is output, and the feature map P9 is input to layer 10. When a random operation (eg, convolution operation) is performed in layer 10, feature map P10 is output, and the feature map P10 is input to layer 11. When a random operation (eg, convolution operation) is performed in layer 11, feature map P11 is output and input to layer 12. Likewise, when an arbitrary operation (eg, convolution operation) is performed in layer 12, feature map P12 is output, and the feature map P12 is input to layer 13.

FIG. 14B is an exemplary diagram showing a VCM encoder that compresses the feature maps P9 to P12 shown in FIG. 14A and then transmits them as a bitstream, and a VCM decoder that restores the received bitstream, according to an example.

As can be seen with reference to FIG. 14b, the VCM encoder 150a (or transmitter) may include an MSFF performance unit 150a-1, an SSFC encoder 150a-2, and a compressor 150a-3. . And the VCM decoder 150b (or receiving end) may include an SSFC decoder 150b-1, an MSFR performance unit 150b-2, and a decompressor 150b-3. Hereinafter, only the parts that are different from those of FIGS. 11A and 12 will be described, and the same parts will not be repeatedly explained, and the descriptions of FIGS. 11A and 12 will be used.

The MSFF performing unit 150a-1 of the VCM encoder 150a may selectively receive one or more of the plurality of feature maps extracted from the feature map extractor shown in FIGS. 9B to 9D. For a specific example, let's assume that the feature map extractor shown in FIGS. 9B to 9D extracts feature maps P9 to P12 as shown in FIG. 14A. Then, the MSFF performing unit 150a-1 of the VCM encoder 150a can selectively receive one or more feature maps from among the feature maps P9 to P12 as needed. In the example of FIG. 14b, among the feature maps P9 to P12, the feature map P9, feature map P10, and feature map P12 are shown as being selectively received by the MSFF performing unit 150a-1 of the VCM encoder 150a. It's messy.

The MSFR performing unit 150b-1 of the VCM decoder 150b can restore feature maps P9' to feature maps P12' from the output F' from the SSFC decoder (150b-2). Additionally, the MSFR performing unit 150b-1 of the VCM decoder 150b may transmit one or more of the restored feature maps P9' to feature maps P12' to each task.

Figure 15a is an exemplary diagram showing the NPU according to the disclosure of the present specification from an operation perspective.

As can be seen with reference to FIG. 15A, the first NPU 100a on the transmission side includes a plurality of first PEs 110a, a first NPU internal memory 120a, a first NPU controller 130a, and a first NPU. It may include an interface 140a and a VCM encoder 150a.

The VCM encoder 150a may include an MSFF performing unit 150a-1, an SSFC encoder 150a-2, and a compressor 150a-3, as shown in FIG. 11, 12, or 14B.

And, the second NPU (100b) on the receiving side includes a plurality of second PEs (110b), a second NPU internal memory (120b), a second NPU controller (130b), a second NPU interface (140b), and a VCM decoder (150b). ) may include.

The VCM decoder 150b may include an SSFC decoder 150b-1, an MSFR performing unit 150b-2, and a decompressor 150b-3, as shown in FIG. 11, 12, or 14B. .

The depicted thick line represents the flow of data (e.g., video/image, feature map, or bitstream), and the dotted line represents the flow of control signals.

First, when the first NPU (100a) on the transmission side receives a video/image, the first NPU controller (130a) in the first NPU (100a) transmits the first NPU internal memory through the first NPU interface (140a). Save it at (120a).

Subsequently, the first NPU controller 130a in the first NPU 100a causes the plurality of first PEs 110a to read the video/image from the first NPU internal memory 120a, and then ANN Perform calculations for each layer of the model.

The plurality of first PEs 110a store feature maps output as a result of performing calculations for each layer of the ANN model according to a control signal from the first NPU controller 130a in the first NPU internal memory 120a. ) to save it. As feature maps are stored for each layer of the ANN model, a plurality of feature maps may be stored in the first NPU internal memory 120a.

The VCM encoder 150a may select one or more feature maps from among a plurality of feature maps stored in the first NPU internal memory 120a as a transmission target according to a control signal from the first NPU controller 130a. .

Then, the selected one or more feature maps pass through the SSFC decoder (150b-1), the MSFR performance unit (150b-2), and the decompressor (150b-3) within the VCM encoder (150a), and are transmitted through the VCM encoder (150a). It is converted into a bitstream as shown in FIG. 14B.

Thereafter, the VCM encoder 150a transmits the bitstream through the first NPU interface 140a according to a control signal from the first NPU controller 130a.

Meanwhile, when the second NPU (100b) on the receiving side receives the bitstream, the second NPU controller (130b) in the second NPU (100b) transmits the second NPU internal memory through the second NPU interface (140b). Save it at (120b).

The VCM decoder 150b in the second NPU 100b on the receiving side reads the bitstream from the second NPU internal memory 120b under the control of the second NPU controller 130b.

The read bitstream passes through the SSFC decoder (150b-1), the MSFR performance unit (150b-2), and the decompressor (150b-3) in the VCM decoder (150b) and is then displayed in Figures 11, 12, or 14B. As shown, it is restored into one or more feature maps.

Then, the VCM decoder (150b) stores the one or more restored feature maps in the second NPU internal memory (120b).

The second NPU controller 130b in the second NPU 100b causes the plurality of second PEs 110b to read the one or more restored feature maps from the second NPU internal memory 120b, Calculations are performed for each layer of the ANN model.

As shown in FIG. 14b, when there are a plurality of machine tasks, there may also be a plurality of ANN models.

Figure 15b is a flowchart showing the signal flow between the transmitting NPU and the receiving NPU.

Referring to Figure 15b, a transmitting NPU (100a) and a receiving NPU (100b) are shown. The receiving NPU (100b) may transmit feedback information to the transmitting NPU (100a). The feedback information may be information that allows the transmitting NPU (100a) to select one or more feature maps as a transmission target. For example, the feedback information may include information about the machine task used in the receiving NPU (100b) and the corresponding ANN model. Alternatively, the feedback information may include identification information for one or more layers among a plurality of layers in the corresponding ANN model. Alternatively, the feedback information may include identification information about one or more feature maps to be transmitted among a plurality of feature maps generated by the transmitting NPU (100a).

Then, the transmitting NPU (100a) may select one or more feature maps from a plurality of feature maps as a transmission target based on the feedback information.

In addition, the transmitting NPU (100a) may encode and/or compress the selected feature map and then transmit it in the form of a bitstream.

After receiving the bitstream, the receiving NPU (100b) may perform decoding and/or decompression to restore the feature map.

And the receiving NPU (100b) can use the restored feature map to perform calculations on subsequent layers of the ANN model and perform machine tasks.

Figure 15c is a modified example of Figure 15b.

Referring to FIG. 15C, a transmitting NPU (100a), a receiving NPU (100b), and a server 200 are shown.

First, the server 200 may transmit information about the ANN model to be used by the receiving NPU 100b to both or one of the transmitting NPU 100a and the receiving NPU 100b.

The receiving NPU (100b) may transmit feedback information. The feedback information may be information that allows the transmitting NPU (100a) to select one or more feature maps as a transmission target. For example, the feedback information may include identification information for one or more layers among a plurality of layers in the ANN model. Alternatively, the feedback information may include identification information about one or more feature maps to be transmitted among a plurality of feature maps generated by the transmitting NPU (100a).

Then, the transmitting NPU (100a) may select one or more feature maps from a plurality of feature maps as a transmission target based on the feedback information.

In addition, the transmitting NPU (100a) may encode and/or compress the selected feature map and then transmit it in the form of a bitstream.

After receiving the bitstream, the receiving NPU (100b) may perform decoding and/or decompression to restore the feature map.

And the receiving NPU (100b) can use the restored feature map to perform calculations on subsequent layers of the ANN model and perform machine tasks.

16 is an exemplary diagram illustrating a first exemplary scenario to which a disclosure of the present specification is applied.

As can be seen with reference to FIG. 16, a surveillance camera may be equipped with an edge NPU. The edge NPU is an NPU specialized for low power consumption.

The NPU can extract a feature map from a captured image. In general, the size of the feature map is very large, so transmitting it wired or wirelessly requires a significant amount of network bandwidth, causing network congestion. Additionally, power waste is also caused.

Therefore, according to one disclosure of the present specification, the NPU mounted on the surveillance camera can select and transmit the output feature map with a smaller size among the layers in the artificial neural network model. Before transmitting the output feature map, the NPU mounted on the surveillance camera determines the size through a multi-scale feature fusion (MSFF) performer and a single-stream feature codec (SSFC) encoder, as shown in Figure 11a. It can be reduced further.

In this way, if each surveillance camera selects a specific layer with a small data size, for example, layer 12, and transmits the feature map of layer 12, the use of about 4M bytes of weight per inference can be omitted.

Mobilenet V1 Layer sum of weights MAX
IFMAP+OFMAP MAC operation EDGE NPU 1~12 133,568 (Kbytes) 1,204,224 (Kbytes) 202,861,568 SERVER NPU 13~28 4,075,520(Kbytes) 200,704 (Kbytes) 365,878,784

Therefore, the NPU mounted on the surveillance camera does not need to process all layers of the artificial neural network, and thus complexity can be simplified. For example, the NPU mounted on the surveillance camera only needs to process a random layer among all layers of the artificial neural network, so it may include only a smaller number of PEs. Additionally, the NPU mounted on the surveillance camera may include a smaller SRAM. Furthermore, the NPU mounted on the camera may not include dynamic random-access memory (DRAM). Through this, the NPU to be mounted on the surveillance camera can significantly reduce manufacturing costs. Additionally, the NPU to be mounted on the surveillance camera can significantly reduce power consumption.

Each surveillance camera can select a specific layer with a small data size and then transmit the output feature map of that specific layer.

In this way, the output feature maps transmitted from multiple surveillance cameras can be multiplexed in a time division manner by a multiplexer (MUX) and transmitted to the cloud server.

The cloud server may include a first memory and an NPU.

The first memory of the cloud server temporarily stores a plurality of output feature maps that are multiplexed and received in a time division manner. Additionally, the first memory of the cloud server may store weights to be used in subsequent layers that have not been processed by the NPU of the surveillance camera. As can be seen with reference to FIGS. 12 and 13, the total size of the weights from layer 13 to layer 28 may be 4075Kb. As such, the total size of the weights from layer 13 to layer 28 is not significantly small, but as the number of surveillance cameras increases, the reuse rate of the weights also increases, so efficiency does not decrease.

A memory controller controlling the first memory may manage memory addresses so that the plurality of output feature maps can be sequentially read from the first memory.

Under the control of the memory controller, the first memory sequentially delivers the plurality of output feature maps to the NPU.

The NPU of the cloud server uses the output feature map of layer 12 from each camera read from the first memory as the input feature map of layer 13 and performs calculations using the weights read from the first memory.

Meanwhile, as the NPU of the cloud server processes layers 13 to 28, the sizes of the output feature map and input feature map written or read from the first memory are as can be seen with reference to FIGS. 12 and 13. Because it is not large (only 200 kb per surveillance camera), the cloud server can process artificial neural network calculations for a large number of cameras.

17 is an exemplary diagram illustrating a second exemplary scenario to which a disclosure of the present specification is applied.

The vehicle camera shown in FIG. 17 may include an edge NPU. As described above, the NPU mounted on the vehicle camera can select and transmit the output feature map with a smaller size among the layers in the artificial neural network model. Before transmitting the output feature map, the NPU mounted on the surveillance camera determines the size through a multi-scale feature fusion (MSFF) performer and a single-stream feature codec (SSFC) encoder, as shown in Figure 11a. It can be reduced further.

The cloud server may include a server NPU. The NPU of the cloud server uses the output feature map from each vehicle camera as an input feature map to perform calculations. And the NPU of the cloud server can generate a vehicle control signal as a result of the above calculation and transmit it to each vehicle.

Other details are the same as those described above, so they will not be explained repeatedly.

The examples of the present disclosure shown in the specification and drawings are merely provided as specific examples to easily explain the technical content of the present disclosure and aid understanding of the present disclosure, and are not intended to limit the scope of the present disclosure. It is obvious to those skilled in the art that in addition to the examples described so far, other modifications can be implemented.

The claims set forth herein may be combined in various ways. For example, the technical features of the method claims of this specification may be combined to implement a device, and the technical features of the device claims of this specification may be combined to implement a method. Additionally, the technical features of the method claims of this specification and the technical features of the device claims may be combined to implement a device, and the technical features of the method claims of this specification and technical features of the device claims may be combined to implement a method.

Claims (18)

As a Neural Processing Unit (NPU) capable of performing encoding,
One or more PEs (Processing Elements) that perform operations for multiple layers in an artificial neural network to generate multiple output feature maps; and
An NPU comprising an encoder that encodes one or more random output feature maps among the plurality of output feature maps into a bitstream and then transmits it. The method of claim 1, wherein the encoder
An NPU that selects one or more random output feature maps from among the plurality of output feature maps based on random information. The NPU of claim 2, wherein the arbitrary information is received from a server or another NPU that receives the bitstream. The NPU of claim 2, wherein the random information includes identification information about the one or more random output feature maps or identification information about a layer corresponding to the random output feature map. The method of claim 2, wherein the optional information:
An NPU generated based on one or more of size information of an output feature map, information on a machine task performed in another NPU receiving the bitstream, and information on an artificial neural network model calculated in another NPU receiving the bitstream. According to paragraph 1,
The bitstream is transmitted to enable another NPU to perform operations for subsequent layers in the artificial neural network. As a Neural Processing Unit (NPU) capable of performing decoding,
a decoder that decodes a received bitstream and restores it into one or more random feature maps; and
Contains one or more PE (Processing Elements) that perform calculations for an artificial neural network model,
The one or more reconstructed random feature maps are output feature maps of a random layer among a plurality of layers in the artificial neural network,
The one or more PEs utilize the one or more reconstructed random feature maps as an input feature map of a subsequent layer of the random layer. The NPU of claim 7, wherein the NPU transmits random information that allows the random layer or the one or more feature maps to be selected by another NPU transmitting the bitstream. The NPU of claim 8, wherein the random information includes identification information about the one or more random feature maps or identification information about a layer corresponding to the one or more random feature maps. The method of claim 8, wherein the optional information:
An NPU generated based on one or more of the size information of the feature map, information on machine tasks performed in the NPU, and information on the artificial neural network model. a first neural processing unit configured to process a first layer section of an artificial neural network model including a plurality of layers; and
A second neural processing unit configured to process a second layer section of the artificial neural network model,
The first neural processing unit is configured to input the output feature map of the last layer of the first layer section to the second neural processing unit. According to clause 11,
The system wherein the size of the sum of the weights of the layers in the first layer section is relatively smaller than the size of the sum of the weights of the layers in the second layer section. According to clause 11,
The system includes a plurality of first neural processing units, and the second neural processing unit is configured to sequentially process each of the output feature maps output from the first neural processing unit. According to clause 11,
The system wherein the second neural processing unit is configured to reuse weights at least corresponding to the number of the first neural processing unit. According to clause 11,
The system is configured such that the size of the first memory of the first neural processing unit is relatively smaller than the second memory of the second neural processing unit. According to clause 11,
The system wherein the first layer section and the second layer section of the artificial neural network model are determined based on the size of the output feature map of a specific layer. According to clause 11,
The system wherein the first layer section and the second layer section of the artificial neural network model are determined based on the size of the sum of the weights of each layer section. According to clause 11,
The system wherein the size of the first weight of the first layer section of the artificial neural network model is relatively smaller than the size of the second weight of the second layer section.

Images