KR20230052378A - Decoding method for multi-view image based on super-resolution and decoder - Google Patents

Abstract

A decoding method for a multi-view video using super resolution includes the steps of: allowing a decoder to receive a bit stream of a target video that is down-sampled and encoded from an encoder; allowing the decoder to decode the target video of low resolution; and allowing the decoder to input the target image and a reference image for the target image into a previously trained neural network model and perform super resolution on the target image.

Description

Multi-view video decoding method and decoder using super-resolution

A technique described below relates to a technique of coding multi-view video using super-resolution. In particular, the technology described below relates to a method and a decoder for decoding multi-view video using super-resolution.

A multiview video is an image that provides users with various viewpoints in various directions by matching images taken through a plurality of cameras. Furthermore, along with the emergence of virtual reality (VR) technology, demand for 360-degree multi-view video content is also increasing.

Korean Patent Registration No. 10-2141319

Multi-view video basically uses images captured by multiple cameras. In order to provide a natural 3D video service, high-resolution multi-view images providing tens to hundreds of viewpoints are required. Therefore, an efficient compression technique for multi-view video is required.

The technology to be described below aims to provide a technique of encoding multi-view images into low-resolution images in an encoder and transmitting them, and providing high-resolution images by super-resolutioning the low-resolution images in a decoder.

A method of decoding a multi-view video using super-resolution includes receiving a bit stream of a target image that has been downsampled and encoded by a decoder in an encoder, decoding the target image having a low resolution by the decoder, and the target image being decoded by the decoder. and performing super-resolution on the target image by inputting a reference image for the target image to a previously trained neural network model.

A decoder that decodes a multi-view video using super-resolution is an input device that receives a bit stream of a target image that has been downsampled and encoded by an encoder, and a neural network model that super-resolutions the low-resolution image by receiving a low-resolution image and a reference image. and an arithmetic device that decodes the low-resolution target image, inputs the target image and a reference image for the target image to the neural network model, and performs super-resolution on the target image.

The technology to be described below reduces the amount of transmitted data by downsampling and encoding a high-resolution video in an encoder. Meanwhile, in the technique described below, a high-quality, high-resolution image can be provided by a decoder using a neural network model that performs super-resolution using various reference images existing in a multi-view image.

1 is a schematic example of a coding process of a multi-view image using super-resolution.
2 is an example of a process of encoding and decoding a multi-view image using super-resolution.
3 is an example of a neural network model performing super-resolution.
4 is an example of a decoder for restoring an image using super-resolution.

Since the technology to be described below can have various changes and various embodiments, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and it should be understood to include all modifications, equivalents, or substitutes included in the spirit and scope of the technology described below.

Terms such as first, second, A, B, etc. may be used to describe various elements, but the elements are not limited by the above terms, and are merely used to distinguish one element from another. used only as For example, without departing from the scope of the technology described below, a first element may be referred to as a second element, and similarly, the second element may be referred to as a first element. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

In terms used in this specification, singular expressions should be understood to include plural expressions unless clearly interpreted differently in context, and terms such as “comprising” refer to the described features, numbers, steps, operations, and components. , parts or combinations thereof, but it should be understood that it does not exclude the possibility of the presence or addition of one or more other features or numbers, step-action components, parts or combinations thereof.

Prior to a detailed description of the drawings, it is to be clarified that the classification of components in the present specification is merely a classification for each main function in charge of each component. That is, two or more components to be described below may be combined into one component, or one component may be divided into two or more for each more subdivided function. In addition, each component to be described below may additionally perform some or all of the functions of other components in addition to its main function, and some of the main functions of each component may be performed by other components. Of course, it may be dedicated and performed by .

In addition, in performing a method or method of operation, each process constituting the method may occur in a different order from the specified order unless a specific order is clearly described in context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

A multi-view video or a multi-view image is an image of a plurality of different viewpoints and refers to video content that provides various viewpoints. Accordingly, the multi-view video includes a 360-degree video, a multi-view 360-degree video, and the like. Furthermore, a multi-view video is meant to include a multi-view 3D video having a texture and a depth image for each view.

The low-resolution image refers to an image whose resolution is less than a reference value by constantly downscaling an image acquired by a camera at a specific point in time. Hereinafter, a low-resolution image is referred to as a low resolution (LR) image. The high-resolution image means an original image acquired by a camera at a corresponding specific point in time or an image having a resolution higher than a predetermined reference value. Hereinafter, a high resolution image is expressed as a high resolution (HR) image.

Super-resolution refers to a technique of converting a low resolution image into a higher resolution image. The technology described below compresses multi-view video into a low-resolution image and provides it, and restores the image using super-resolution in a decoding process.

The technology described below performs super-resolution using an artificial neural network model. There are various models such as recurrent neural networks (RNNs), feedforward neural networks (FFNNs), convolutional neural networks (CNNs), and the like. In the following description, the CNN-based model will be mainly described.

First, general descriptions of coding for multi-view video will be given. A multi-view video includes images from different viewpoints. Accordingly, multi-view video can be encoded using various video sources in terms of time and space.

Each view constituting a multi-view video can be classified as an independent view or a dependent view. In the independent view, encoding is performed only with information of the current view without using information of an adjacent view. That is, inter-time prediction encoding is performed and compressed. On the other hand, the dependent view can be coded using information of an independent view.

Meanwhile, an image of each view may have image information (texture) and depth information. Image information and depth information use the same viewpoint index. In the case of an independent view, image information is always encoded before related depth information. However, in the case of a dependent view, image information may be coded before or after related depth information having the same view index. Inter-layer prediction encoding between an image and depth information may be performed according to an encoding order. For example, if image information is encoded first, depth information may be encoded by referring to information used for encoding related image information.

Independent views and dependent views are coded through inter-view and inter-temporal prediction using a hierarchical B structure. The first frame of an independent view is compressed as an I-frame, and the inter-temporal compression technique is applied to subsequent frames. The first frame of a dependent view refers to an I frame using a P or B frame. The first frames of dependent views are coded using inter-view prediction or intra-picture compression techniques without using inter-time prediction. Frames of time from a dependent time point thereafter may use both inter-time and inter-view prediction techniques.

Hereinafter, a process of encoding and decoding a multi-view video using super-resolution will be described. An image processing device for encoding is an encoder, and an image processing device for decoding is a decoder. Hereinafter, an image processing device is used to mean including an encoder and/or a decoder. An image processing device is a computer device capable of processing image data. For example, the image processing device may be a PC, a smart device, a server on a network, a TV set-up box, a VR device, and the like.

1 is a schematic example of a coding process of a multi-view image using super-resolution. Figure 1 shows the video coding concept in a second plane with a time axis and a viewpoint axis. 1 shows time t and time t-1 as examples, and time points 1 and 2 as examples. 1 is a video frame at time t-1 and time 1 (F _1,t and F _{2,t - 1} ), it is assumed that encoding and decoding are completed. In FIG. 1, an encoding and decoding target is a video frame F _2,t .

In the step of encoding the video frame F _2,t at time t and time 2, the encoder downsamples the original video and encodes it into a low-resolution video (F' _2,t ).

The decoder performs a super-resolution process of restoring the downsampled video frame F' _2,t to its original resolution. Meanwhile, the decoder may use a reference picture-based super-resolution method that uses both video frames F _1,t and F _2,t-1 received in the super-resolution process.

2 is an example of a process 100 of encoding and decoding a multi-view image using super-resolution. As shown in FIG. 1 , encoding and decoding of the video frame F _2,t will be mainly described. 2 is an example for simultaneously explaining the operation of an encoder and a decoder. An encoder may also include a component for decoding a previous frame for motion compensation or the like in an encoding process.

It is assumed that the encoder encodes the first video frame F _1,t at the original resolution, and the decoder also decodes at the original resolution. Also, it is assumed that encoding at time t-1 is also completed.

In this process, the encoder receives the video frame F _1,t (210) and generates F' _1,t by downsampling F _1,t (120). The encoder may store the downsampled video frame F' _1,t in a decoded picture buffer (DPB). Meanwhile, the decoder may downsample the decoded video and store it in the DPB.

The encoder receives a video frame F _2,t (130).

The encoder generates a frame F' _2,t by downsampling F _2,t (140). The encoder performs down-sampling of the video frame F _2,t at the additional time point (time point 2) and performs encoding at a low resolution. At this time, the DPB is in a state where F' _{1, t} at time point 1 and F' 2,t-1 at time t _{-1 are stored, which have the same resolution as F' 2,t} .

For video frame encoding, various schemes utilizing previously encoded results are used. MCP (Motion Compensated Prediction) is a method of encoding a current time frame using a previous time frame. That is, MCP is an encoding method that transmits only residuals based on motion information that appears according to time differences. In addition, DCP (Disparity Compensated Prediction) means a method of transmitting only the residual by using a block at an adjacent view different from the current frame as a prediction block.

Accordingly, the encoder may encode F' _2,t using DCP and/or MCP using F' _1, t and F' _{2,t -1} , respectively. To encode F' _2,t, the encoder compares F' _2,t and F' _1,t to predict (DCP) the residual according to the time difference, or F' _2,t and F' _{2,t - 1} The residual according to the time difference is predicted (MCP) by comparing (150). The encoder may perform DCP or MCP, or perform both DCP and MCP simultaneously in some cases to determine only residuals that do not overlap in the two criteria.

The subsequent coding process is the same as a general video coding process. The encoder performs DCT (Discrete Cosine Transform) and quantization on the residual of F' _2,t (160). The encoder may perform entropy coding on the quantized residual in the frequency domain (170). An encoder produces the final bit stream.

The decoder receives the bit stream generated by the encoder (180). That is, the decoder receives the low-resolution image data (F' _2,t ) and decodes the image. 2 does not show details of the decoder operation. Briefly, the decoder receives the residual data of F' _2,t , and decodes F' _2,t by referring to previous data using a DCP or/and MCP method corresponding to the encoded method.

Thereafter, the decoder reconstructs F _2,t by upsampling F' _2,t to super-resolution (190). Even in the decoding process, the decoder retains F' _1,t and F' _{2,t - 1} in the DPB in the same way as in the coding process. This is because the image must be reconstructed using the corresponding information according to the residual prediction method. The decoder reconstructs F _2,t using an image in the DPB as a reference image.

Meanwhile, the multi-view image may have a format having a texture image and a depth image for one frame. In this case, the prediction used in the coding process uses texture-to-depth prediction instead of DCP. That is, the encoder may perform predictive encoding between the texture image and the depth image according to the encoding order. For example, if a texture image is encoded first, a depth image may be encoded by referring to information used for encoding a corresponding texture image. Furthermore, if the depth image is encoded first, the texture image may be encoded by referring to information used for encoding the corresponding depth image. Therefore, unlike FIG. 2 , the image processing device (encoder and decoder) may code the current depth image or texture image using the previously coded texture image or depth image.

In FIG. 2, the decoder upsamples an image of a specific frame using super-resolution using a reference image. The super-resolution process will be described below. The decoder may perform super-resolution using a neural network model.

3 is an example of a neural network model 200 performing super-resolution. The neural network model 200 of FIG. 3 receives a low-resolution image at a specific point in time and outputs a high-resolution image. A low-resolution image, which is a super-resolution target, is called a target low-resolution image. Meanwhile, the neural network model 200 super-resolutions a target low-resolution image using a reference image. Accordingly, the input images input to the neural network are the target low-resolution image E ^LR and the reference image E ^Ref . The output image is the high-resolution image E ^SR obtained by upsampling the target low-resolution image E ^LR .

The first convolution layer 211 receives E ^LR and outputs a feature t ^LR . The second convolution layer layer 212 receives E ^Ref and outputs a feature t ^Ref .

Disparity estimator 220 is E ^LR and E ^Ref extract the correlation between them. Correlation includes information in which related features are aligned among features extracted from each image. The disparity estimator 220 corresponds to a layer for estimating a correlation between two images.

The disparity estimator 220 calculates the correlation D _r→l and transfers the features of the reference image together with the low-resolution image without flow supervision.

The disparity estimator 220 extracts a first feature from t ^LR of the target low-resolution image, extracts a second feature from t ^Ref of the reference image, and estimates a flow from the correlation between the first feature and the second feature includes a configuration that

Briefly, the first encoder 221 receives t ^LR and extracts f ^LR as a basic feature, and the first latitude-aware convolution layer 222 calculates the disparity between images in f ^LR serves to reduce the difference. The first location-aware convolution layer 222 outputs s ^LR . The second encoder 223 receives t ^Ref and extracts f ^Ref as a basic feature, and the second location-aware convolution layer 224 serves to reduce a disparity difference between images in f ^Ref . The second location-aware convolution layer 224 outputs s ^Ref . Double-headed arrows between the first encoder 221 and the second encoder 223, and between the first location-aware convolution layer 222 and the second location-aware convolution layer 224 indicate a relationship in which kernel parameters are shared.

로 와핑(warping)한다(240).

는 참조 영상을 타깃 영상에 정렬한 결과에 해당한다.The image processing device performs a correlation calculation 225 of s ^LR and s ^Ref , and the flow estimator 226 receives the information obtained through the correlation calculation and finally calculates E ^LR and E ^Ref Outputs the correlation D _r→l between The image processing device uses D _r→l to set E ^Ref

Warping with (240).

corresponds to the result of aligning the reference image to the target image.

가 타깃과 관련하여 부정적인 특징(성능 저하 특징)을 갖는 경우, 초해상화 성능이 저하된다. 신경망 모델(200)은 이와 같은 성능 저하 특징을 제거하기 위한 구성을 더 포함할 수 있다. 마스크 생성기(230)는 t^LR 및 D_r→l을 입력받아

에 포함되는 성능 저하 특성을 제거하기 위한 마스크 M을 생성한다.

If has negative characteristics (performance degradation characteristics) with respect to the target, the super-resolution performance is degraded. The neural network model 200 may further include a configuration for removing such performance degradation characteristics. Mask generator 230 is t ^LR and D _r→l as input

A mask M for removing the performance degradation characteristic included in is generated.

을 요소별 곱셈(elementwise multiplication)하여 초해상화를 위한 최종적인 특징 데이터를 마련할 수 있다. The image processing device is M and

Final feature data for super-resolution may be prepared by elementwise multiplication.

또는 M과

을 곱한 결과를 입력받아 영상을 재구성하여 초해상화를 수행한다. 재구성 계층(250)은 잔차 블록(residual block)으로 구성될 수 있다. 잔차 블록은 입력 특징이 들어와 일정 값이 출력되는 과정에서 학습 효율에 따라 컨볼루션을 수행하거나 수행하지 않는 과정을 선택적으로 제공한다. 재구성 계층(250)은 E^LR을 초해상화한 E^SR을 출력한다.Reconstruction layer 250 includes (i) a low-resolution target image E ^LR (or t ^LR ) and (ii)

or with M

Reconstruct the image by receiving the result of multiplying by and performing super-resolution. The reconstruction layer 250 may be composed of residual blocks. The residual block selectively provides a process of performing or not performing convolution according to learning efficiency in the process of inputting input features and outputting a certain value. The reconstruction layer 250 outputs E ^SR obtained by super-resolving E ^LR .

In FIG. 2, the decoder receives the residual of F' _2,t according to the MCP method and restores F 2, _t using F' _{2,t - 1} . In this case, the neural network model 200 in the decoder calculates F _{2,t by receiving the target low-resolution image F' 2,t} and the reference image F' _{2,t - 1 .} Alternatively, the decoder receives the residual of F' _2,t according to the DCP method and restores F _{2,t using F' 1, t} . In this case, the neural network model 220 in the decoder calculates F _{2,t by receiving the target low-resolution image F' 2,t} and the reference image F' _{1, t} . It may be a method using an encoder or MCP and DCP selectively or in combination. In this case, the decoder may additionally receive information about the residual generation method used by the encoder and select a reference image according to the residual generation method.

Meanwhile, the DPB of the decoder holds data (reference image set) for a previously decoded frame at the time of decoding a specific target low-resolution image. Accordingly, the decoder may select a reference image for decoding a target low-resolution image among reference image sets based on a predetermined criterion. The decoder may select a reference image according to one of the following criteria. It is assumed that a reference image for F _2,t is selected and explained.

① The decoder may select the closest frame in order of F _2,t and viewpoint among the reference image sets stored in the DPB. For example, although F' _1,t is used in the previous description, the decoder may select F' _3,t for F' _2,t super-resolution.

② The decoder may select a frame closest to F _2,t in chronological order from among the reference image sets stored in the DPB.

③ The decoder may select an I frame from the reference picture set stored in the DPB. For example, an I frame may be selected for super-resolution of an independent view image.

④ The decoder may select a frame having the lowest temporal ID among reference image sets stored in the DPB. Temporal layer ID is information introduced to support temporal scalability in HEVC.

⑤ The decoder may select a frame using the lowest quantization parameter among reference image sets stored in the DPB.

4 is an example of a decoder 400 that reconstructs an image using super-resolution. The decoder 400 may be in the form of a VR device, PC, smart device, network server, and the like.

The decoder 400 may include a storage device 410, a memory 420, an arithmetic device 430, an interface device 440, and a communication device 450.

The storage device 410 may store programs or codes for operation of the decoder 400 .

The storage device 410 may store the aforementioned neural network model 200 .

The storage device 410 may store programs or codes for learning the neural network model 200 or 300.

The storage device 410 may store the decoded low-resolution image.

The storage device 410 may store previously decoded low-resolution images. The storage device 410 may store low-resolution images corresponding to super-resolution reference images.

The storage device 410 may store a high-resolution image generated by a neural network model.

The memory 420 may temporarily store data and information generated during the operation of the decoder 400 .

The interface device 440 is a device that receives certain commands and data from the outside. The interface device 440 may receive certain information from a physically connected input device or a physical interface (keypad, touch panel, etc.).

The interface device 440 may receive a neural network model, information for learning the neural network model, and learning data. The interface device 440 may also receive parameter values for updating the neural network model.

The interface device 440 may receive a bit stream generated by an encoder.

The interface device 440 may receive a reference image required to super-resolution a target low-resolution image.

The communication device 450 transmits and receives certain information through a wireless network. The communication device 450 may receive a neural network model, information for learning the neural network model, and learning data. The communication device 450 may receive parameter values for updating the neural network model.

The communication device 450 may receive the bit stream generated by the encoder.

The communication device 450 may receive a reference image required to super-resolution a target low-resolution image.

The communication device 450 may transmit a high-resolution image generated by a neural network model to an external object.

The interface device 440 and the communication device 450 may receive certain information and data from a user or an external object. In terms of receiving information, the interface device 440 and the communication device 450 may be collectively referred to as an input device.

The arithmetic device 430 controls the operation of the decoder 400 using programs or codes stored in the storage device 410 .

The arithmetic unit 430 may extract a residual of a target low-resolution image from a bit stream. The processing unit 430 may decode a target low-resolution image using an image of an adjacent view or an image of a previous view in a manner corresponding to a prediction method of an encoder. Furthermore, when the target low-resolution image is a texture image, the arithmetic device 430 may decode the image using the depth image. When the target low-resolution image is a depth image, the arithmetic unit 430 may decode the image using the texture image.

The arithmetic unit 430 upsamples the target low-resolution image by inputting the decoded target low-resolution image and the reference image to the neural network model. The arithmetic unit 430 may select a reference image according to a prediction method of an encoder.

Furthermore, the arithmetic device 430 may select any one reference image from among various reference image sets by using information related to a target low-resolution image. For example, the arithmetic device 430 may select a reference image closest to a target low-resolution image based on a viewpoint from a set of reference images stored in a storage device. Alternatively, the arithmetic device 430 may select a reference image closest to the target low-resolution image in chronological order from among the reference image sets stored in the storage device. Alternatively, the arithmetic device 430 may select an I frame from a set of reference images stored in a storage device. Alternatively, the arithmetic device 430 may select a reference image having the lowest temporal ID among the reference image sets stored in the storage device. Alternatively, the arithmetic device 430 may select a reference image using the lowest quantization parameter from among reference image sets stored in the storage device.

The arithmetic device 430 may be a device such as a processor, an AP, or a chip in which a program is embedded that processes data and performs certain arithmetic operations.

In addition, the multi-view video coding method and the multi-view video decoding method using super-resolution as described above may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be stored and provided in a temporary or non-transitory computer readable medium.

A non-transitory readable medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and can be read by a device. Specifically, the various applications or programs described above are CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), EPROM (Erasable PROM, EPROM) Alternatively, it may be stored and provided in a non-transitory readable medium such as EEPROM (Electrically EPROM) or flash memory.

Temporary readable media include static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (Enhanced SDRAM). SDRAM, ESDRAM), Synchronous DRAM (Synclink DRAM, SLDRAM) and Direct Rambus RAM (DRRAM).

This embodiment and the drawings accompanying this specification clearly represent only a part of the technical idea included in the foregoing technology, and those skilled in the art can easily understand it within the scope of the technical idea included in the specification and drawings of the above technology. It will be obvious that all variations and specific examples that can be inferred are included in the scope of the above-described technology.

Claims (8)

receiving, by a decoder, a bit stream of a target image that has been down-sampled and encoded by an encoder;
decoding the target image having a low resolution by the decoder; and
The decoder inputs the target image and a reference image for the target image to a pre-learned neural network model to perform super-resolution on the target image,
The method of decoding a multi-view video using super-resolution, wherein the target image is an image of a specific viewpoint among multi-view videos at time t. According to claim 1,
The method of decoding multi-view video using super-resolution, wherein the reference image is determined according to a prediction technique used when the target image is encoded in the encoder. According to claim 1,
Among the reference image sets previously decoded by the decoder, the reference image includes (i) an image closest to the target image in terms of viewpoint, (ii) an image closest to the target image in terms of time, (iii) an I-frame image, (iv) A method of decoding a multi-view video using super-resolution, which is any one of an image having the lowest temporal ID and (v) an image using the lowest quantization parameter. According to claim 1,
The neural network model is
a first convolution layer receiving the target image and extracting a first feature;
a second convolution layer receiving the reference image and extracting a second feature;
a dispatch estimation layer receiving the first feature and the second feature and outputting a correlation between the target image and the reference image; and
A method of decoding a multi-view video using super-resolution including an image obtained by warping the reference image so as to be aligned with the target image based on the correlation and a reconstruction layer receiving the target image and reconstructing the target image. an input device that receives a bit stream of a target image that has been downsampled and encoded by an encoder;
a storage device that receives a low resolution image and a reference image and stores a neural network model that super-resolutions the low resolution image; and
An arithmetic device that decodes the target image of low resolution and inputs the target image and a reference image for the target image to the neural network model to perform super-resolution on the target image,
The target image decodes a multi-view video using super-resolution, which is an image of a specific viewpoint among multi-view videos at time t. According to claim 5,
The reference image decodes a multi-view video using super-resolution that is determined according to a prediction technique used when the target image is encoded in the encoder. According to claim 5,
Among the reference image sets previously decoded by the decoder, the reference image includes (i) an image closest to the target image in terms of viewpoint, (ii) an image closest to the target image in terms of time, (iii) an I-frame image, A decoder for decoding a multi-view video using super-resolution, which is any one of (iv) an image having the lowest temporal ID and (v) an image using the lowest quantization parameter. According to claim 5,
The neural network model is
a first convolution layer receiving the target image and extracting a first feature;
a second convolution layer receiving the reference image and extracting a second feature;
a dispatch estimation layer receiving the first feature and the second feature and outputting a correlation between the target image and the reference image; and
A decoder for decoding a multi-view video using super-resolution including an image obtained by warping the reference image so as to be aligned with the target image based on the correlation and a reconstruction layer receiving the target image and reconstructing the target image .

Images