KR20240015023A - Anchor enhancer based on neural acceleration with selective super-resolution video encoding acceleration function and anchor enhancement method using the same - Google Patents

Abstract

A neural accelerator-based anchor enhancer with a selective super-resolution video encoding acceleration function that provides efficient and scalable neural enhancement for live streams and an anchor enhancement method using the same are disclosed. A neural accelerator-based anchor enhancer with optional super-resolution video encoding acceleration selects the most advantageous anchor frames from the video stream, and for each video stream, a deep neural network (DNN) and anchors output the selected anchor frames. A preprocessor that preprocesses the DNN provided by the scheduler; When a decoded image is received in a GPU (Graphics Processing Unit) or NPU (Neural Processing Unit), an inference engine that converts it into a high-resolution image through neural network inference; and an encoder that reuses the input video stream and uses an image codec to compress only super-resolution anchor frames while offloading non-anchor frame reconstruction to the client, packaging the encoded video and super-resolution anchor frames into a single file.

Description

Neural accelerator-based anchor enhancer with selective super-resolution video encoding acceleration function and anchor enhancement method using the same

The present invention relates to a neural accelerator-based anchor enhancer with selective super-resolution video encoding acceleration and an anchor enhancement method using the same, and more specifically, to selective super-resolution video encoding that provides efficient and scalable neural enhancement for live streams. This relates to a neural accelerator-based anchor enhancer with video encoding acceleration function and an anchor enhancement method using the same.

The demand for live streaming has grown rapidly. Live video traffic is expected to account for 17% of Internet traffic by 2022. Current live streaming infrastructure relies on two key elements: Video is transmitted to the media server using a low-latency streaming protocol. On the distribution side, the client runs the Adaptive Bitrate (ABR) algorithm to select the highest quality video that can be streamed in real time.

Traditional live streaming is inadequate to consistently deliver high-quality video (e.g. 4K/8K), as ingestion video quality is unfortunately highly dependent on the streamer's uplink bandwidth. Congestion of the collection path directly reduces overall downstream video quality. However, video quality is the most important factor affecting user engagement in live streaming. For example, more than 50% of live viewers will abandon a video stream if video quality degrades for more than 90 seconds. This loss of viewers can have a huge impact on the revenue of live streaming providers.

Recent advances in neural enhanced streaming show great promise in leveraging computation in media servers to improve ingestion video quality. When the ingestion video quality deteriorates, the media server executes end-to-end neural enhancement, which consists of decoding the low-quality stream, applying a super-resolution Deep Neural Network (DNN), and super-resolved output encoding to recover high-quality video. . This provides dramatic quality improvement in downstream video.

However, neural enhancement is too expensive to support commercial-scale live streaming. For example, Twitch supports over 100,000 simultaneous live streams. Applying end-to-end neural enhancement in this setup requires tens of thousands of graphics processing units (GPUs) and costs more than $169,000 per hour in the public cloud. The cost breakdown shows that both video super-resolution and encoding are expensive. Neural super-resolution requires 100 to 1000 times more computation than DNNs used for discrimination tasks, and video encoding is up to 3.3 times slower than super-resolution.

Korean Patent No. 10-2313136 (2021. 10. 15.)

Accordingly, the technical problem of the present invention is focused on this point, and the purpose of the present invention is to provide efficient and scalable neural enhancement for live streams by using a lightweight codec that can compress high-resolution streams as fast as selective inference, The aim is to provide a neural accelerator-based anchor enhancer with optional super-resolution video encoding acceleration that can process end-to-end neural enhancement at selective super-resolution speed.

Another object of the present invention is to provide an anchor enhancing method using the anchor enhancer described above.

In order to realize the object of the present invention described above, according to one embodiment, a neural accelerator-based anchor enhancer with a selective super-resolution video encoding acceleration function selects the most advantageous anchor frame from the video stream, and selects the most advantageous anchor frame from the video stream, and uses a deep neural network for each video stream. (Deep Neural Network, DNN) and a preprocessor that preprocesses the DNN provided by the anchor scheduler that outputs the selected anchor frame; When a decoded image is received in a GPU (Graphics Processing Unit) or NPU (Neural Processing Unit), an inference engine that converts it into a high-resolution image through neural network inference; and an encoder that reuses the input video stream and uses an image codec to compress only super-resolution anchor frames while offloading non-anchor frame reconstruction to the client, packaging the encoded video and super-resolution anchor frames into a single file.

In one embodiment, the preprocessor may optimize GPU context switching by executing a super-resolution DNN in a framework optimized for inference.

In one embodiment, the preprocessor takes a randomly initialized mock DNN before live streaming begins, pre-optimizes it in the target accelerator, and then optimizes the DNN using the optimized mock DNN when optimization of the DNN is required during live streaming. You can create versions.

In one embodiment, the encoder may compress only super-resolution anchor frames while offloading non-anchor frame reconstruction to the client, packaging the encoded video and super-resolution anchor frames into a single file.

In one embodiment, the size of each of the anchor frames may be set identically to meet the bit rate constraints of live streaming.

In one embodiment, a new header attached to each frame may include the frame type and the super-resolution frame of the anchor frame.

In order to realize the other object of the present invention described above, the anchor enhancing method according to an embodiment is to: (i) select the most informative anchor frame from a video stream, and use a deep neural network (DNN) for each video stream; preprocessing; (ii) When the decoded image is received in the GPU (Graphics Processing Unit) or NPU (Neural Processing Unit), it is converted into a high-resolution image through neural network inference; and (iii) reusing the input video stream and using an image codec to compress only the super-resolution anchor frames while offloading non-anchor frame reconstruction to the client, packaging the encoded video and super-resolution anchor frames into a single file. Includes.

In one embodiment, step (i) includes (i-1) taking a randomly initialized mock DNN and pre-optimizing it in a target accelerator before live streaming begins; And (i-2) if optimization of the DNN is required during live streaming, it may include generating an optimized version using an offline optimized mock DNN.

In one embodiment, step (iii) includes (iii-1) reusing the input video stream and compressing only the super-resolution anchor frames using an image codec while offloading non-anchor frame reconstruction to the client; and (iii-2) packaging the encoded video and super-resolution anchor frame into a single file.

According to this neural accelerator-based anchor enhancer with selective super-resolution video encoding acceleration function and anchor enhancement method using the same, end-to-end neural enhancement requires re-encoding, but existing video encoders have high computational cost and are bottlenecks. Therefore, end-to-end neural enhancement can be processed at selective super-resolution rates by using a lightweight codec that can compress high-resolution streams as fast as selective inference to avoid bottlenecks while providing similar quality to existing codecs. You can.

Figure 1 is a conceptual diagram to schematically explain existing streaming and neural network-enhanced streaming (adaptive streaming case).
Figure 2 is a conceptual diagram schematically explaining selective super-resolution.
Figure 3 is a conceptual diagram schematically explaining a neuroscaler according to an embodiment of the present invention.
Figure 4 is a conceptual diagram schematically illustrating the deployment model of a neural scaler for adaptive streaming.
FIG. 5 is a conceptual diagram schematically explaining the anchor scheduler shown in FIG. 3.
FIG. 6 is a conceptual diagram schematically explaining the anchor selector shown in FIG. 3.
FIG. 7 is a conceptual diagram for explaining the overall operation of the hybrid video codec unit including the encoder shown in FIG. 3.

Hereinafter, with reference to the attached drawings, the neural scaler and the neural scaling method using the same according to the present invention will be described in more detail. Since the present invention can be subject to various changes and have various forms, specific embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to a specific disclosed form, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention.

While describing each drawing, similar reference numerals are used for similar components. In the attached drawings, the dimensions of the structures are enlarged from the actual size for clarity of the present invention.

Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may also be referred to as a first component. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Additionally, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

Hereinafter, the present invention will be described in detail with reference to the attached drawings.

Live streaming consists of collection and distribution aspects, as shown in Figure 1.

Figure 1 is a conceptual diagram to schematically explain existing streaming and neural network-enhanced streaming (adaptive streaming case).

Referring to Figure 1, on the collection side, the streamer captures live video and uploads it to a media server. In adaptive streaming, a streamer uploads a single video stream, and a media server transcodes it into multiple quality versions.

In multi-party video conferencing, broadcasters upload video streams of multiple qualities using simulcast or scalable video codecs (SVC), and media servers deliver the video streams to viewers. Next, on the distribution side, the client runs the Adaptive Bitrate (ABR) algorithm for video streaming quality control to select/download the highest quality video in the given network bandwidth. The ABR algorithm can be divided into heuristics-based methods and machine learning-based methods. Heuristic-based methods are classified into network bandwidth, client buffer occupancy, and hybrid based methods that combine the two.

A common limitation of traditional live streaming is that video quality is limited by the streamer's uplink bandwidth. Even if viewers have sufficient network bandwidth, limitations may deprive them of the opportunity to enjoy high-quality video.

Super-resolution is a type of technology that creates high-resolution images from low-resolution images. Recent research uses deep neural networks (DNNs) to learn mappings from low-resolution to high-resolution and shows dramatic quality improvements.

Neural-enhanced live streaming uses ultra-high-resolution DNNs to enhance the collected video stream, as shown in Figure 1. Unlike traditional live streaming, the media server implements end-to-end neural enhancement. When the video arrives, it is first decoded into raw frames. Then, the DNN is applied to all or some frames depending on the super-resolution method. The output is re-encoded into video before being delivered to the client.

To provide reliable quality improvement, content-aware DNNs are commonly used for each video stream and can also be updated during live streaming. Context switching between content-aware DNNs incurs two types of overhead.

First, while modern DNN compilers provide optimized inference for the target accelerator, this includes an upfront cost due to model optimization, which can take up to several minutes. Apply both graph-level and operator-level optimization to create a DNN tailored to your specific target.

Second, before executing inference, the DNN and frames must be loaded into the device memory of the target accelerator. Because neural accelerators are typically limited in memory size (for example, 16 GB for the NVIDIA T4), multiple content-aware DNNs, each requiring several GB of memory, must be replaced frequently.

Selective super-resolution (SR) reduces computing overhead by leveraging temporal redundancy across video frames. Neural inference applies only to selective frames, called anchor frames.

Figure 2 is a conceptual diagram schematically explaining selective super-resolution.

As can be seen in Figure 2, non-anchor frames are reconstructed by reusing previous super-resolution frames guided by codec level information (e.g., reference index, motion vector, residual). It includes lightweight bilinear interpolation and decoding, allowing real-time processing even on mobile devices.

Upscaling non-anchor frames by reusing previous super-resolution frames inevitably leads to quality loss (compared to frame-by-frame inference) due to time differences. This loss accumulates in successive non-anchor frames but is reset at the anchor frame. Therefore, it is important to select an informative set of anchor frames to maximize quality. To achieve this, the NEMO algorithm relies on expensive per-frame inference. Therefore, it cannot be used for live streaming where anchor frame selection must be performed online in real time.

Figure 3 is a conceptual diagram schematically explaining a neuroscaler according to an embodiment of the present invention.

Referring to FIG. 3, the NeuroScaler according to an embodiment of the present invention includes an anchor scheduler 100 and an anchor enhancer 200, and a DNN pair with the collected video stream. and output a high-resolution video stream. The DNN can also be dynamically updated through online learning, such as in LiveNAS.

Anchor scheduler 100 decodes the collected video stream and selects the most advantageous anchor frame from the video stream. Here, the anchor frame refers to a frame in which the optional super-resolution is extended to accelerate video super-resolution, but super-resolution is applied in real time using codec-level information. Then, the anchor scheduler 100, for each video stream, transfers the DNN and the selected anchor frame to the anchor enhancer 200 equipped with a neural accelerator.

The anchor enhancer 200 has an optional super-resolution video encoding acceleration function based on a neural accelerator, preprocesses the DNN, applies the preprocessed DNN to the anchor frame, and re-encodes the super-resolution output.

In this embodiment, all processes of anchor scheduler 100 and anchor enhancer 200 are pipelined and parallelized to maximize overall throughput.

According to the neuroscaler according to the present invention, by motivating the limitations of neural network-enhanced live streaming, the end-to-end neural network enhancement cost can be reduced and the overall quality improvement in the computing cluster can be maximized.

Figure 4 is a conceptual diagram schematically explaining the deployment model of a neuroscaler for adaptive streaming.

Referring to Figure 4, when a low-quality video stream (e.g., 360p or 720p) is uploaded to a media server, the neuroscaler generates a high-resolution video stream using real-time super-resolution.

Lower resolution versions (e.g., 240p to 720p) are also created from the collected video stream by using an existing transcoding pipeline. Using Neuroscaler, clients can view high-resolution video even when the collection path is congested. Deploying Neuroscaler for video conferencing is similar to the process above, but does not require multi-resolution transcoding.

FIG. 5 is a conceptual diagram schematically explaining the anchor scheduler 100 shown in FIG. 3.

3 and 5, the anchor scheduler 100 includes a decoder 110 and a resource management module 120.

The decoder 110 decodes an externally provided video stream and provides it to the resource management module 120. The decoded video stream may be provided directly to the resource management module 120 or may be provided to the resource management module 120 via memory.

Resource management module 120 includes an anchor selector 122 and an anchor-level load balancer 124 to distribute load while optimally allocating computing resources to each video stream. In this embodiment, the resource management module 120 distributes the load while optimally allocating computing resources to each video stream to maximize quality improvement when processing a large number of video streams. In particular, the resource management module 120 performs global anchor frame selection in all video streams to select an optimal set of anchor frames, and distributes the load to instances in units of anchor frames where the overhead of anchor frames can be accurately predicted. In this embodiment, the anchor scheduler 100 has been described as including an anchor selector 122 and an anchor-level load balancer 124, but this is only logically divided for convenience of explanation and not hardwarely divided. .

Anchor selector 122 runs on a centralized server to select the most advantageous anchor frames from all video streams. In particular, the anchor selector 122 selects the maximum number of anchor frames that can be processed in real time in the computing cluster as shown in Equation (1) below and periodically executes the zero-inference algorithm.

[Formula 1]

where is the anchor frame aligned with the anchor gain, is the DNN latency, is the anchor frame selection interval which is a configurable parameter, and is the number of compute instances in the cluster.

In this specification, unless otherwise stated, = 666ms (40 frames for 60fps video) is used. This global anchor frame selection can alleviate the imbalance of anchor frames per video stream.

In general, using longer scheduling intervals can improve quality by selecting more influential anchor frames. Therefore, the spacing must be adjusted according to application requirements.

Anchor-level load balancer 124 selects an anchor frame and then dynamically balances the load across compute instances at anchor frame granularity. To this end, the anchor scheduler 100 divides the selected anchor frames into groups for each instance and the frames of each group are delivered to the corresponding computing instance for processing.

To meet real-time constraints, the anchor scheduler 100 allocates a maximum number of anchor frames for each group whose total waiting time is less than the anchor frame selection interval (). Through this anchor-level load balancing, the load imbalance for each instance can be alleviated.

FIG. 6 is a conceptual diagram schematically explaining the anchor selector 122 shown in FIG. 5.

5 and 6, the anchor selector 122 includes a frame splitting unit 122a, an anchor gain estimating unit 122b, a group merging and sorting unit 122c, and an anchor frame selection unit 122d, Codec-level information (e.g., frame type, residual) is utilized to select informative anchor frames. In this embodiment, the anchor selector 122 has been described as including a frame dividing unit 122a, an anchor gain estimating unit 122b, a group merging and sorting unit 122c, and an anchor frame selection unit 122d. For convenience of explanation, they are divided logically and not hardware-wise.

The frame division unit 122a divides frames into groups for each video stream according to the type of each video stream. That is, the frame division unit 122a divides frames into key frame groups (key frame groups) according to the type of each video stream. ), alternate reference frame group ( ) and other frame groups ( ) divided by Here, the group is , and There is a priority order for selecting anchor frames.

The anchor gain estimation unit 122b is an alternative reference frame group ( ) and other frame groups ( ), we estimate the benefit of using an anchor frame, called the anchor gain. The anchor gain is calculated based on the accumulated residuals using Algorithm 1, described later. Key frame group ( ), since there are only a few key frames and they are generally all selected as anchor frames, it is assumed to have an anchor gain equal to the overall performance.

The anchor gain estimation unit 122b includes a residual calculation module (not shown), a residual calculation module (not shown), an anchor gain setting module (not shown), and an update module (not shown). In this embodiment, the anchor gain estimation unit 122b has been described as including a residual calculation module, a residual amount calculation module, an anchor gain setting module, and an update module, but these are only logically divided for convenience of explanation and hardwarely divided. I didn't do it. The residual calculation module calculates the residual accumulated over the entire frame. The residual amount calculation module calculates the residual amount by which each frame is reduced for each iteration. The anchor gain setting module selects the frame that reduces the residual the most and sets the anchor gain to the amount of the reduced residual. The update module updates the cumulative residual of each frame.

The group merging and sorting unit 122c merges groups for each video stream into a global group. That is, the keyframe global group ( ), alternative reference frame global group ( ), and other frame global groups ( ) includes key frames, alternate reference frames, and regular frames of all video streams, respectively. Frames of the same group are sorted according to anchor gain.

The anchor frame selection unit 122d is a key frame global group ( ) in the alternate reference frame global group ( ) and other frame global groups ( ) Iteratively selects anchor frames from the global group sorted.

In this embodiment, the neuroscaler measures the latency of the DNN once and selects the maximum number of anchor frames whose total latency is less than the available computing time to meet real-time constraints.

Hereinafter, we will describe how the neuroscaler uses residuals to estimate anchor gains through Algorithm 1.

Algorithm 1 shows how the neuroscaler estimates the gain of the anchor frame.

Algorithm 1 first calculates the residuals accumulated across frames (Algorithm 1, line #2).

Then, the most advantageous frame is iteratively selected and the anchor gain is estimated. That is, for each repetition, the remaining amount by which each frame is reduced is calculated as shown in the following equation (2) (lines #6-8).

[Formula 2]

Here, Res(F[j]) is the cumulative residual of the jth frame, and k is the index of the nearest frame where the residual is reset. The residuals are cleared from key frames or frames where the anchor gain was estimated from the previous iteration. If such a frame does not exist within a given frame, the algorithm predicts that the residual is reset at the key frame of the next video chunk.

Next, the frame that reduces the residual the most is selected and the anchor gain is set to the amount of the reduced residual (lines #12, 13).

Finally, the cumulative residual for each frame is updated to reflect the impact of the selected frame (line #14). To quickly estimate the anchor gain, the total residual pixel value is approximated by the size of the encoded residual frame. Since both values are highly correlated, they have minimal impact on quality (=in PSNR 0.05dB).

Referring again to FIG. 3, anchor enhancer 200 includes a preprocessor 210, an inference engine 220, and an encoder 230.

The preprocessor 210 preprocesses the DNN provided by the anchor scheduler 100.

The preprocessor 210 optimizes GPU (graphics processing unit) context switching to efficiently run ultra-high-resolution DNN in a framework optimized for inference (e.g., TensorRT, TVM). Naive use of the framework in neurally enhanced live streaming will result in severe speedup due to two types of GPU context switching overhead:

The first is the DNN update problem. Model optimization can make inference faster, but typically takes seconds to minutes, depending on the DNN size. Therefore, it is difficult to translate the benefits of model optimization to a live streaming context. Since DNNs can be updated online, optimization must occur in real time. In this embodiment, in order to reduce optimization waiting time, we note that the model optimization results have a low correlation with the actual weight values. Based on this, we propose a model pre-optimization method in which optimization is performed offline and occurs only once.

The preprocessor 210 obtains a randomly initialized mock DNN and pre-optimizes it in the target accelerator before live streaming begins.

Next, if optimization of the DNN is required during live streaming, the preprocessor 210 generates an optimized version using an offline optimized mock DNN. This involves replacing the parameters of the mock DNN with those of the target DNN. This scheme reduces latency from tens of seconds to milliseconds.

The second is the DNN loading problem. Since multiple DNNs/frames arrive at each anchor enhancer 200 per scheduling interval, device/host memory must be allocated and deallocated frequently. This overhead is similar to that of neural inference for high resolution (4K/8K), as each DNN requires several MB/GB of host and device memory. To avoid frequent memory allocation overhead, the preprocessor 210 pre-allocates host/device memory and configures memory pools when the program starts.

When a decoded image is input to the GPU or NPU, the inference engine 220 converts it into a high-resolution image through neural network inference.

Encoder 230 reuses the input video stream and uses an image codec to compress only super-resolution anchor frames while offloading non-anchor frame reconstruction to the client, packaging the encoded video and super-resolution anchor frames into a single file.

End-to-end neural enhancement requires re-encoding, but existing video encoders 230 (e.g., VPx, H. 26x) are computationally expensive and represent a major computational bottleneck. Therefore, we propose a lightweight codec unit that can compress high-resolution video streams as quickly as selective inference to avoid bottlenecks while providing similar quality to existing codec units. The lightweight codec unit allows end-to-end neural enhancement processing at selective super-resolution speeds. This is because non-anchor frames can be easily reconfigured on the client side, even on commercial mobile devices, without significant overhead.

This bypass allows the target video to be encoded very quickly. Additionally, since non-anchor frames are transmitted directly at low resolution without upscaling, there is more room in terms of bandwidth to encode anchor frames at high quality.

With this in mind, we designed a hybrid video codec unit specialized for encoding selective ultra-high resolution output.

FIG. 7 is a conceptual diagram for explaining the overall operation of the hybrid video codec unit 300 including the encoder 230 shown in FIG. 3.

Referring to FIGS. 3 and 7 , the hybrid video codec unit 300 includes a hybrid encoder 310 disposed on the server side and a hybrid decoder 320 disposed on the client side. In this embodiment, the hybrid encoder 310 corresponds to the encoder 230 disposed in the anchor enhancer 200 shown in FIG. 3, and the hybrid decoder 320 can be disposed in a client-side terminal that plays live streaming. there is.

Unlike existing encoders, the hybrid encoder 310 utilizes both video and image codec units in a synergistic manner. The hybrid encoder 310 reuses the input video stream and uses an image codec unit to compress only ultra-high resolution anchor frames while offloading non-anchor frame reconstruction to the client. Each anchor frame size is set identically to meet the bitrate constraints of live streaming. Next, the hybrid encoder 310 packages the encoded video and super-resolution anchor frames into a single file and later delivers it to the client. A new header attached to each frame includes the frame type (e.g., anchor or non-anchor frame) and the super-resolution frame of the anchor frame.

Hybrid encoder 310 has two main advantages.

First, computing overhead can be reduced by 78.6 to 235.8 times compared to video encoders. This is because the hybrid encoder 310 applies a lightweight image codec, which is ~6.25 times cheaper than a video encoder, to only a few anchor frames (5-10% of the total frame).

Second, because anchor frames are sparsely distributed within video chunks, applying an image codec to a few frames has minimal impact on compression efficiency. The Bjontegaard rate difference (BD-rate) is improved by 6.69% when using the hybrid encoder 310 compared to re-encoding the video using a conventional encoder.

The hybrid decoder 320 disposed on the client side decodes the video stream encoded by the hybrid encoder 310. When a compressed frame arrives, the hybrid decoder 320 first checks whether the current frame is an anchor frame. If the current frame is confirmed to be an anchor frame, the hybrid decoder 320 decodes the hyper-resolved frame and caches it in memory. Meanwhile, if it is determined that the current frame is not an anchor frame, the hybrid decoder 320 reuses the previous super-resolution frame to enlarge the current frame using codec-level information. Decoding hybrid encoded video streams incurs minimal overhead.

As described above, according to the present invention, end-to-end neural enhancement requires re-encoding, but existing video encoders are computationally expensive and create a bottleneck, so while providing similar quality to existing codecs, To prevent this from happening, by using a lightweight codec that can compress high-resolution streams as fast as selective inference, we can process end-to-end neural enhancement at selective super-resolution speeds.

Components according to embodiments of the present invention may be implemented in the form of software or hardware such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and may perform certain roles. However, 'components' are not limited to software or hardware, and each component may be configured to reside on an addressable storage medium or may be configured to run on one or more processors.

Thus, as an example, a component may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, and sub-processes. Includes routines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

Components and the functionality provided within them may be combined into a smaller number of components or further separated into additional components. At this time, it will be understood that each block in the drawings can be executed by computer program instructions. These computer program instructions may be mounted on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing device, so that the instructions performed through the processor of the computer or other programmable data processing device may perform the functions described in the block(s). It creates the means to carry out these tasks. These computer program instructions may be stored in a computer-readable memory or may be stored in a computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular manner. Instructions stored in memory may also produce manufactured items containing instruction means to perform the functions described in the block(s). Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer and can be processed by the computer or other programmable data processing equipment. Instructions that perform processing equipment may also provide steps for executing the functions described in the block(s).

Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). Additionally, it should be noted that in some alternative execution examples it is possible for the functions mentioned in the blocks to occur out of order. For example, it is possible for two blocks shown in succession to be performed substantially at the same time, or it is possible for the blocks to be performed in reverse order depending on the corresponding function.

At this time, the term '~unit' used in this embodiment refers to software or hardware components such as FPGA or ASIC, and the '~unit' performs certain roles. However, '~part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card.

Although the above has been described with reference to examples, those skilled in the art can make various modifications and changes to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand.

120: Resource Management Module 122: Anchor Selector
124: anchor-level load balancer 122a: frame divider
122b: Anchor gain estimation unit 122c: Group merging and sorting unit
122d: anchor frame selection unit 210: pre-processor
220: Inference Engine 230: Encoder
300: Hybrid video codec unit 310: Hybrid encoder
320: Hybrid decoder

Claims (9)

A preprocessor that selects the most informative anchor frames from a video stream and preprocesses a deep neural network (DNN) for each video stream and a DNN provided by an anchor scheduler that outputs the selected anchor frames;
When a decoded image is received in a GPU (Graphics Processing Unit) or NPU (Neural Processing Unit), an inference engine that converts it into a high-resolution image through neural network inference; and
It features an encoder that reuses the input video stream and uses an image codec to compress only super-resolution anchor frames while offloading non-anchor frame reconstruction to the client, packaging the encoded video and super-resolution anchor frames into a single file. A neural accelerator-based anchor enhancer with selective super-resolution video encoding acceleration. The anchor enhancer based on a neural accelerator according to claim 1, wherein the preprocessor optimizes GPU context switching by executing a super-resolution DNN in a framework optimized for inference. The method of claim 1, wherein the preprocessor imports a randomly initialized mock DNN before live streaming begins, pre-optimizes it in a target accelerator, and then optimizes it using the optimized mock DNN when optimization of the DNN is necessary during live streaming. A neural accelerator-based anchor enhancer with selective super-resolution video encoding acceleration, characterized in that it generates an enhanced version. 2. The selective super-resolution method of claim 1, wherein the encoder compresses only super-resolution anchor frames while offloading non-anchor frame reconstruction to the client and packaging the encoded video and super-resolution anchor frames into a single file. A neural accelerator-based anchor enhancer with video encoding acceleration capabilities. The anchor enhancer based on a neural accelerator with a selective super-resolution video encoding acceleration function according to claim 1, wherein the size of each anchor frame is set to be the same to meet the bit rate constraints of live streaming. The anchor enhancer based on a neural accelerator with selective super-resolution video encoding acceleration according to claim 1, wherein a new header attached to each frame includes a frame type and a super-resolution frame of the anchor frame. (i) selecting the most informative anchor frames from the video streams and preprocessing a deep neural network (DNN) for each video stream;
(ii) When the decoded image is received in the GPU (Graphics Processing Unit) or NPU (Neural Processing Unit), it is converted into a high-resolution image through neural network inference; and
(iii) reusing the input video stream and using an image codec to compress only the super-resolution anchor frames while offloading non-anchor frame reconstruction to the client, including packaging the encoded video and super-resolution anchor frames into a single file; An anchor enhancing method characterized by: The method of claim 7, wherein step (i) is:
(i-1) Before live streaming starts, take a randomly initialized mock DNN and pre-optimize it on the target accelerator; and
(i-2) An anchor enhancement method comprising the step of generating an optimized version using an offline optimized mock DNN when optimization of the DNN is required during live streaming. The method of claim 7, wherein step (iii) is:
(iii-1) reusing the input video stream and using an image codec to compress only super-resolution anchor frames while offloading non-anchor frame reconstruction to the client; and
(iii-2) An anchor enhancing method comprising packaging the encoded video and super-resolution anchor frame into a single file.