KR102500199B1 - Method and device of processing multi-view video - Google Patents

Abstract

A multi-view image encoding method according to an embodiment includes dividing an image of a plurality of views into a base view image and an additional view image; calculating a ratio of an additional view image to a background view image; obtaining transform coefficients by performing a prediction operation and a transform operation on the base view image and the additional view image, and obtaining a quantization transform coefficient by quantizing the transform coefficients of the base view image and the additional view image in a plurality of quantization steps, respectively; selecting quantization transform coefficients quantized in a first quantization step and a second quantization step for a base view image and an additional view image, respectively, from among quantization transform coefficients quantized in a plurality of quantization steps, based on the ratio; and entropy-coding the selected quantization transform coefficients.

Description

Multi-view image processing method and apparatus {METHOD AND DEVICE OF PROCESSING MULTI-VIEW VIDEO}

This specification relates to a method and apparatus for processing a multi-view image.

Immersive video is a technology that allows users to enjoy with a higher degree of freedom by using images from multiple viewpoints and depth map information. In order to smoothly provide such an immersive video service, since a large number of images are required, a higher bandwidth is required.

Currently, the moving picture experts group (MPEG), a standardization organization, standardizes immersive media technology that can provide users with motion parallax according to screen switching by acquiring and processing images that exist in various locations at various times. is in progress

As part of this standardization, at the MPEG 126th meeting in March 2019, the MPEG-I (Immersive) subgroup developed a Test Model for Immersive Video (TMIV) that supports immersive media that provides 6 degrees of freedom (DoF). defined.

6DoF immersive media technology standardization continues to explore the performance of MIV (Model for Immersive Video) technology for the purpose of processing multiple images for immersive media. Since 6DoF technology supports viewpoints according to the user's free gaze and movement, high bandwidth and computation are required in the process of acquiring and synthesizing images from the user's viewpoint by simultaneously compressing and transmitting images acquired from various locations.

In the current MPEG-I, in order to reduce the number of video encoders/decoders required for processing multiple images, multiple images are combined into a basic view (or basic view) and an additional view (or additional view) (Additional view). view) and encoding method is adopted.

However, until now, it has been difficult to efficiently adjust the bandwidth in an adaptive streaming scenario by processing the base view and the additional view as a single image.

This specification takes this situation into account, and an object of this specification is to provide a method for efficiently transmitting a background view image and an additional view image while reducing quality loss of a final output image in a limited bandwidth situation.

Another object of this specification is to provide a method for selecting an optimal quantization step for a base view image and an additional view image in order to save transmission bandwidth.

A multi-view image encoding method according to an embodiment of the present specification includes dividing an image of a plurality of views into a base view image and an additional view image; calculating a ratio of an additional view image to a background view image; obtaining transform coefficients by performing a prediction operation and a transform operation on the base view image and the additional view image, and obtaining a quantization transform coefficient by quantizing the transform coefficients of the base view image and the additional view image in a plurality of quantization steps, respectively; Selecting quantization transform coefficients quantized in the first quantization step and the second quantization step for the base view image and the additional view image, respectively, from among the quantization transform coefficients quantized in a plurality of quantization steps based on the ratio. do.

A multi-view image encoding apparatus according to another embodiment of the present specification overlaps a background view image in a background view atlas including a background view image selected as a background view among images of a plurality of views and additional view images of remaining unselected views. an atlas generation unit for generating a background view atlas including a patch image generated by removing a patch image and calculating a ratio between the background view image and the patch image; a texture encoder for encoding the base view image included in the base view atlas and the patch image included in the additional view atlas using a 2D codec; and a metadata construction unit for encoding configuration information of the base view atlas and the additional view atlas and/or control information for controlling a synthesis process of the base view atlas and the additional view atlas, and outputting the encoded metadata as metadata, wherein the texture encoder is configured. obtains transformation coefficients by performing a prediction operation and transformation operation on the background view image and the patch image, obtains a quantization transformation coefficient by quantizing the transformation coefficients of the background view image and the patch image in a plurality of quantization steps, respectively, and calculates the ratio Based on this, quantization transform coefficients quantized in the first quantization step and the second quantization step are selected for the background view image and the patch image among the quantization transform coefficients quantized in a plurality of quantization steps, respectively, and the selected quantization transform coefficients are entropy-coded. to be characterized.

A multi-view image decoding apparatus according to another embodiment of the present specification includes a texture decoder for generating a base view atlas and an additional view atlas by decoding encoded texture data of a base view and an additional view using a 2D codec; a metadata parser for interpreting metadata including configuration information of the base view atlas and the additional view atlas and/or control information for controlling a synthesis process of the base view atlas and the additional view atlas; an atlas patch occupancy map generation unit for generating a patch occupancy map by determining a location and direction in which a patch image included in an additional viewpoint atlas is to be placed using metadata; and a reproduction unit for generating an image corresponding to the motion of the viewer from the base view atlas and the additional view atlas using the metadata and the patch occupancy map, wherein the metadata is a quantization conversion coefficient of the base view image and the patch image. It further includes information related to the quantization step applied to the generation, and the texture decoder uses the information related to the quantization step to inverse quantize the quantization transform coefficients of the base view image and the patch image, and inverse transform the inverse quantized transform coefficients to obtain residual samples. and restores the base view image and the patch image by generating reconstructed samples using the residual samples and prediction samples generated by the prediction operation, and the quantization step applied to the transform coefficient of the base view image and the transform coefficient of the patch image The quantization step applied to is characterized in that different values are determined based on the ratio of the background view image and the patch image.

Accordingly, it is possible to reduce transmission bandwidth while minimizing quality loss. In addition, it is possible to transmit a high-quality video using a more efficient bandwidth by adaptively selecting a quantization step according to the characteristics or importance of the video.

1 shows the configuration of a multi-view video encoder according to an embodiment of this specification in functional blocks;
2 shows the configuration of a multi-view image decoder according to an embodiment of this specification in functional blocks;
3 illustrates an example in which MPEG Immersive Video (MIV) technology divides a multi-viewpoint image into a base viewpoint image and an additional viewpoint image and processes it.
FIG. 4 shows an example in which the amount of information of the additional viewpoint image is small due to the large overlap between the background viewpoint image and the additional viewpoint image.
5 shows an example in which the amount of information of the additional view image is large because the overlap between the background view image and the additional view image is small,
6 shows the configuration of a texture encoder according to an embodiment of this specification,
7 is an operational flowchart of a method of selecting data to which different quantization steps are applied according to viewpoints according to an embodiment of the present specification;
8 shows the difference between synthesized images when both the background view image and the additional view image are compressed singly with QP1 and QP2, respectively.
9 illustrates a difference between synthesized images when a background view image is compressed by QP1 and an additional view image is compressed by lowering the quality to QP2.

Hereinafter, preferred embodiments of a multi-view image processing method and apparatus according to this specification will be described in detail with reference to the accompanying drawings.

The technology disclosed in this specification can be applied to a virtual reality image transmission technology based on a user's point of view. However, the technology disclosed in this specification is not limited thereto, and may be applied to all electronic devices and methods to which the technical spirit of the technology may be applied.

It should be noted that technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the spirit of the technology disclosed in this specification. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the field to which the technology disclosed in this specification belongs, unless specifically defined otherwise in this specification. It should not be interpreted in an overly comprehensive sense or in an excessively reduced sense. In addition, when the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the technology disclosed in this specification, it is a technical term that can be correctly understood by those of ordinary skill in the field to which the technology disclosed in this specification belongs. should be replaced with In addition, general terms used in this specification should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Terms including ordinal numbers such as first and second used herein may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted.

In addition, in describing the technology disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the technology disclosed in this specification, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only intended to facilitate understanding of the spirit of the technology disclosed in this specification, and should not be construed as limiting the spirit of the technology by the accompanying drawings.

Recently, MPEG-I, a subgroup of MPEG, an international standardization organization, has developed 'immersion' for 6DoF (degrees of freedom) technology, which processes images from multiple viewpoints and responds to the user's location and viewpoint according to screen transitions beyond 360-degree images. Standardization is underway under the name of Immersive Video.

In order to implement 6DoF video streaming beyond simple 360 VR video, it is necessary to respond to all positions and viewing angles of the viewer using images acquired from various viewpoints and depth images (or depth maps). In order to correspond to the viewer's point of view, a virtual point of view synthesizing process of synthesizing and processing images for various points of view is performed.

In the current MIV, after acquiring multiple images with a depth map based on the depth map-based image rendering (DIBR) technique, based on the depth map, the viewpoint and distance of each image are calculated to synthesize virtual image viewpoints. It goes on. Therefore, it is possible to synthesize a viewpoint to match the user's line of sight and position, and as a result, the user can enjoy more interactive and immersive media.

To this end, MPEG-I adopts a method of dividing a plurality of images into a base view and an additional view.

FIG. 1 illustrates the configuration of a multi-view image encoder according to an embodiment of this specification in functional blocks, and FIG. 2 illustrates the configuration of a multi-view image decoder according to an embodiment of this specification in functional blocks.

Referring to FIG. 1 , a multi-view image encoder 100 includes a view optimizer 110, an atlas constructor 120, an encoder 130, and a metadata configuration unit ( Metadata Composer) 140 may be included.

The viewpoint optimization unit 110 receives source images and depth images (Source Views (T+D)) from various viewpoints, and determines the number of background viewpoints necessary by considering direction deviation, field of view, distance, and overlap between viewpoints In addition, the background viewpoint can be selected by considering the position between viewpoints and overlapping with each other.

The atlas generation unit 120 configures the background viewpoints selected by the viewpoint optimization unit 110 as one complete single atlas, and generates the remaining patches (or residual patches) after removing overlapping parts based on the background viewpoints for the additional viewpoints. can be configured.

The atlas generator 120 preserves background viewpoints using a mask while generating the atlas, removes overlapping parts of additional viewpoints, updates the mask used in the video frame according to the temporal order, and packs each patch atlas. Finally, an atlas of additional viewpoints can be created.

As shown in FIG. 3, the original view image (or atlas) (BV) includes all pixels in the texture (including depth information) as it is, but the image (or atlas) (AV of the additional view) ) can remove overlapping pixels based on the background view and the additional view images of the prior order, and configure only the remaining non-overlapping textures (including depth information) in the form of a block patch. The decoder may reconstruct a block patch to reconstruct a video frame of an additional view.

In FIG. 3, there are source images and depth images of 16 viewpoints, and among them, images of 4 viewpoints are selected as a prototype and combined into one atlas to generate a single background viewpoint image (including texture and depth images, respectively). The images of the remaining 12 viewpoints are determined as additional images, and only the part different from the background viewpoint is generated as one additional viewpoint image (including texture and depth images, respectively) in the form of a patch.

The encoder 130 uses a 2D codec, for example, High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC), to generate a texture image T and a depth image of a base view (BV) and an additional view (AV). A bitstream can be generated by encoding (D).

The metadata constructing unit 140 is configured to generate a base view and an additional view atlas so that the multi-view image decoder 200 can dynamically reproduce a viewport image without interruption according to the viewer's motion through post-processing and view synthesis. Configuration information and/or control information for controlling the synthesis process of the atlas of the basic viewpoint and the additional viewpoint may be encoded and generated as metadata.

Referring to FIG. 2 , the multiview image decoder 200 includes a decoder 210, a metadata parser 220, an atlas patch occupancy map generator 230, and a playback unit. (Renderer) 240 may be included.

The decoder 210 may generate a base view atlas and an additional view atlas by decoding encoded texture data and depth data of the base view and the additional view using HEVC or VVC.

The metadata parser 220 may interpret metadata and provide the atlas patch occupancy map generation unit 230 and playback unit 240 .

The atlas patch occupancy map generation unit 240 determines the position and direction of the patch of the atlas of the additional viewpoint using the metadata provided by the metadata parser 220 to generate a patch occupancy map, and reproduces the patch occupancy map 240 ) can be provided.

The playback unit 240 uses the metadata provided by the metadata parser 220 and the patch occupancy map provided by the atlas patch occupancy map generator 240 to determine the movement of the viewer from the base view atlas and the additional view atlas. A corresponding image may be generated and output.

Meanwhile, the additional view images are segmented images, and have a high bit rate because they contain a lot of high frequencies, but their ratios in the entire image are different for each content. In addition, the effect of the base view image and the additional view image on the image quality of the synthesized image synthesized by the decoder may vary according to the ratio of the additional view image in the image reconstructed or synthesized by the decoder.

FIG. 4 illustrates an example in which the amount of information of the additional view image is small due to the large amount of overlap between the background view image and the additional view image, and FIG. it is depicted In FIGS. 4 and 5, the image on the left is a background view and the image on the right is an image of an additional view.

In FIG. 4 , information of the additional view image is small because there are many pixels overlapping between the background view image and the additional view image, whereas in FIG. 5 there are not many overlapping pixels between the background view image and the additional view image, so the information of the additional view image is large.

In the case of FIG. 4 , since the amount of information included in the image of the additional view is small, the effect of the image of the additional view on the image quality of the synthesized video synthesized by the decoder is inevitably small. On the other hand, in the case of FIG. 5 , since the amount of information included in the image of the additional viewpoint is relatively large, the effect of the image of the additional viewpoint on the image quality of the synthesized image may be great.

In immersive video, unlike general 360-degree video, there is an image synthesis process, so even if the quality of low-important images is lowered, the effect on the final synthesized image is small, and the loss in the final synthesized image can be relatively reduced.

When synthesizing images on the decoder side, since the effect of the images of the background view and the additional view on the entire synthesized image is inevitably different depending on the content and/or the amount of information of the background and additional view, the image of the background view and the additional view It is not efficient to encode the texture by considering it as one image.

Considering this point, texture encoding is performed by considering the image as a separate image according to the viewpoint, that is, depending on whether the image is a base viewpoint or an additional viewpoint. The overall bit rate may be adjusted by adaptively varying the image quality according to the importance of the viewpoint.

The importance of each view can be calculated as the ratio of the number of pixels in the background view image to the number of pixels in the additional view image or the ratio of pixels in the background view image and pixels in the additional view image to the reconstructed image. Encoding quality of viewpoint images may be adjusted differently.

In an embodiment of this specification, the encoding quality of the base view image and the additional view image can be adjusted differently by asymmetrically quantizing the base view image and the additional view image. In this case, the number of pixels of the base view image versus the pixels of the additional view image The quantization step is adjusted in consideration of the number ratio, but the quantization step can be adjusted within a range that satisfies a target rate-distortion cost (RD-cost) condition.

6 illustrates the configuration of a texture encoder according to an embodiment of this specification.

The texture encoder 130 generates a bitstream by applying HEVC or VVC to the background view image (or atlas) and the additional view image (or atlas) as well as the background/additional view depth image (or map). The additional viewpoint image may be divided into separate images and processed.

The texture encoder 130 may include a prediction unit 131, a transform unit 132, a quantization unit 133, and an entropy encoding unit 134.

The prediction unit 131 is a prediction sample for a coding unit (hereinafter simply expressed as a block) divided into a plurality of coding units based on a quad tree structure and/or a binary tree structure by an image segmentation unit (not shown). (or prediction block), and a residual sample (residual or residual block) corresponding to the difference between the original sample and the predicted sample may be generated.

The prediction unit 131 includes techniques used in HEVC or VVC, for example, an intra-prediction technique referring to neighboring samples of a current block within a current picture, an inter-prediction technique referring to samples of a picture other than the current picture, and a current picture. An intra block copy (IBC) technique that refers to samples of other blocks in a picture may be applied.

The transform unit 132 may generate transform coefficients by applying a transform technique to residual samples. The transform technique may include discrete cosine transform (DCT), discrete sine transform (DST), Karhunen-Loeve transform (KLT), GBT ( graph-based transform), or conditionally non-linear transform (CNT).

The quantization unit 133 quantizes transform coefficients. Based on a coefficient scan order, the quantized transform coefficients in block form may be rearranged into a 1D vector form, and the quantized transform coefficients in 1D vector form may be rearranged. Information on quantized transform coefficients may be generated based on .

The entropy encoding unit 134 encodes information about quantized transform coefficients and outputs it as a bitstream. For example, exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary) arithmetic coding), etc., and information necessary for video/image reconstruction (eg values of syntax elements, etc.) may be encoded together or separately, except for quantized transform coefficients. there is.

The bitstream may be transmitted through a network or may be stored in a digital storage medium, where the network may include a broadcasting network and/or a communication network, and the digital storage medium may include USB, SD, CD, DVD, and Blu-ray. , HDD, SSD, etc. may include various storage media.

The quantization unit 133 according to an embodiment of this specification includes a quality controller, and converts samples (residual samples) of a base view image and an additional view image into a plurality of quantization steps (or quantization parameters). It is quantized and stored (QP1, QP2, QP3, etc. in FIG. 6), and based on the ratio of the number of pixels between the base view image and the additional view image (or the ratio of the number of pixels of the synthesized image synthesized in the decoder and the number of pixels of the additional view image) A base view image and an additional view image to which different quantization steps are applied can be selected.

In FIG. 6 , the quality control unit of the quantization unit 133 selects data quantized by QP1 for the background view image BV, and selects data quantized by QP2 for the additional view image AV.

In addition, the quality control unit of the quantization unit 133 may change the quantization step of the background view image (BV) and the additional view image (AV) on a frame-by-frame basis. For example, in the first frame, the background view image (BV) For the second frame, select the data quantized by QP1 for the additional viewpoint image (AV), select the data quantized by QP2 for the additional viewpoint image (AV), and select the data quantized by QP1 for the background viewpoint image (BV) in the second frame, and select the additional viewpoint image ( AV), data quantized with QP3 can be selected.

In addition, the quality control unit of the quantization unit 133 divides the ratio of the number of pixels between the base view image and the additional view image into a plurality of ranges, determines a quantization step suitable for each of the base view image and the additional view image for each range, and determines the quantization step. can be applied

For example, if the pixel number ratio is 1:0.5, the quantization steps of the base and additional view images are set to QP1 and QP2, and if the pixel ratio is 1:0.25, the quantization steps of the base and additional view images are set to QP1. and QP3 (a quantization step with lower quality than QP2), and when the pixel count ratio is 1:0.125, the quantization steps of the base view image and the additional view image are set to QP1 and QP4 (quantization step with a lower quality than QP3). can

Alternatively, the quality control unit of the quantization unit 133 divides the ratio of the number of pixels between the background view image and the additional view image into a plurality of ranges, and for each range, a difference value of a quantization step suitable for the base view image and the additional view image is determined. It can be prepared and applied. When the pixel number ratio is 1:0.5, the quantization steps of the base view image and the additional view image are QP1 and QP2, but QP2 corresponds to a quality lower than QP1 by one level, and the pixel number ratio is 1: In the case of 0.25, QP3 has a quality lower than that of QP1 and QP2, and when the pixel number ratio is 1:0.125, QP4 has a quality lower than QP1 and QP3.

In the above example, the quantization step of the background view image is fixed to QP1 and the quantization step of the additional view image is changed. Depending on the quantization step of the base view image that is changed, the quantization step of the additional view image may be additionally changed.

In the above example, the pixel number ratio value of the background view image and the additional view image, the value of the quantization step suitable for the background view image and the additional view image, or the difference value thereof are merely examples, and the embodiments of this specification value is not limited.

The pixel number ratio of the base view image and the additional view image may be transmitted from the atlas generator 120 constituting the multi-view image encoder 100 of FIG. 1 . The atlas generator 120 may calculate a pixel number ratio of the base view image and the additional view image in units of GOP (Group Of Picture) and transmit the calculated ratio to the texture encoder 130 .

The quantization unit 133 transfers information related to the quantization step applied to the base view image and the additional view image to the metadata constructing unit 140, so that the metadata constructing unit 140 converts the information related to the asymmetric quantization step into the metadata. can include

Alternatively, information related to the quantization step may be recorded as, for example, a syntax element instead of being transmitted to the metadata constructing unit 140 and encoded together with the quantized transform coefficient of the video of the corresponding viewpoint by the entropy encoding unit 134. there is.

The multi-view image decoder 200 extracts information related to a quantization step from syntax elements entropy-encoded into metadata and/or texture data, and adds quantized coefficients of a base view image based on the extracted quantization step related information. The quantized coefficients of the viewpoint image may be inversely quantized.

In addition, the multi-view image decoder 200 reconstructs residual samples of the current block by inverse transforming the inverse quantized coefficients for the base view and the additional view, and restores the residual samples and the predicted samples predicted for the current block. Based on the reconstructed sample of the current block, a base view image and an additional view image (patch image of the additional view) may be generated.

Thereafter, the multi-view image decoder 200 generates an image corresponding to a viewpoint corresponding to the motion of the viewer by combining the base viewpoint image and the patch image of the additional viewpoint based on the patch occupancy map generated based on the metadata. can

7 is a flowchart illustrating an operation of a method of selecting data to which different quantization steps are applied depending on viewpoints according to an embodiment of the present specification.

The atlas generating unit 120 generates an atlas for a basic viewpoint selected by the viewpoint optimizing unit 110 and an atlas for additional viewpoints with respect to input images of multiple viewpoints (S710). The atlas generating unit 120 determines the number of pixels of the background view image in consideration of the number of target image pixels (the number of pixels of the synthesized image displayed on the display considering the environment of the decoder-side terminal), and determines the number of pixels of the background view image and An additional viewpoint image from which redundancy is removed is generated, and the number of pixels of the additional viewpoint image may be determined in consideration of the number of target image pixels.

For example, in an image of 16 viewpoints with a resolution of 2048x2048, the background viewpoint image is composed of 4 viewpoints and the remaining additional viewpoint images are generated at medium resolution, or the background viewpoint image is composed of 8 viewpoints and the remaining additional viewpoint images are generated. can be created at a low degree of resolution. In the case of configuring the background view image with 4 views, the number of pixels is not greater than in the case of configuring the image with 8 views, so the terminal can process it with low complexity, but the restoration accuracy (image quality) is degraded.

Therefore, the atlas generator 120 determines the resolution of the background view image according to the hardware configuration of the target decoder terminal or the request of the decoder terminal, and converts the input images of a predetermined number of views into the base view and the additional view. It is possible to create an atlas image by dividing the image.

The atlas generating unit 120 calculates the ratio between the background view image and the additional view image, that is, the ratio between the number of pixels of the background view image and the number of pixels of the additional view image per frame (S720). For example, the background view per frame. When the number of pixels of the image is 2,073,600 and the number of pixels of the additional view image is 298,741, the ratio of the number of pixels of the synthesized (or reconstructed) image to the number of pixels of the base view image in the decoder can be calculated as 298,741/(2,073,600 + 298,741).

The atlas generator 120 may determine the ratio between the background view image and the additional view image in units of GOPs, and accordingly calculate the ratio between the number of pixels of the background view image and the number of pixels of the additional view image per frame.

The texture encoder 130 encodes the textures of the base-view image and the additional-view image according to HEVC, VVC, or other video coding techniques to generate a bitstream, and the quantization unit 133 converts the base-view image and the additional-view image Quantization coefficient data may be generated by quantizing the quantized coefficients through a plurality of quantization steps (QP) (S730).

The quality control unit of the quantization unit 133 selects quantization coefficient data for a base view image and quantization coefficient data for an additional view image from among data quantized in a plurality of quantization steps, and selects one timing (frame at the corresponding timing). Constructing) quantization coefficient data, but quantization coefficient data of the additional viewpoint image to which a quantization step different from that applied to the quantization coefficient data selected for the base viewpoint image is applied based on the pixel number ratio of the base and additional viewpoint images can be selected (S740). At this time, the quantization coefficient data of the background and additional view images must satisfy the target RD-cost condition.

The quality control unit of the quantization unit 133 asymmetrically selects a quantization step for the base view image and the additional view image according to the ratio of the number of pixels, and determines whether the quantized transformation data quantized by the selected quantization step satisfies the RD-cost condition. After checking, and when the RD-cost condition is not satisfied, quantization steps for the base view image and the additional view image may be selected until the RD-cost condition is satisfied. Satisfaction of the RD-cost condition may be performed in units of frames.

If the set RD-cost condition is not satisfied even though the quantization step for the background and additional view images is selected with a certain value, the target value of the number of pixels of the background view image is adjusted to obtain the base view image and the additional view image from step S710. It can be performed again from the process of creating .

The quantization unit 133 transfers information related to the quantization step applied to the base view image and the additional view image to the metadata constructing unit 140, and the metadata constructing unit 140 reflects the information related to the quantization step to the metadata. to change the metadata, and the multi-view image encoder 100 may generate and output the data generated by the encoder 130 and the metadata configuration unit 140 as a bit stream in a multi-view image format (S750).

The bitstream generated by the multi-view image encoder 100 may be transmitted to a client device including a multi-view image decoder through a network or stored in a digital storage medium.

8 shows the difference between synthetic images when both the background view image and the additional view image are individually compressed with QP1 and QP2, respectively, and FIG. It shows the difference of the synthesized image when it is done.

In order to verify the efficiency of the method proposed by the embodiments of this specification, three images of Common Test Conditions defined in MPEG-I were selected and simulated. HEVC test model (HM) version 16.20 was used for image encoding, and an image including one or more patches was defined as an additional viewpoint.

The quantization parameters applied to the transformation data of the background and additional view images consisted of QP1 to QP5, and in the case of the additional view, the value of the quantization parameter was increased by one step (with a quantization parameter of lower quality) than the background view, and encoding was performed.

Unlike FIG. 8, in FIG. 9, it can be confirmed that loss occurs only in the part that has an effect on synthesis, and this means that the quality loss in compression may occur in a different way from existing images according to the amount of information of the image of the additional view point. suggests

Based on the common experimental conditions, the base view and the additional view were encoded, decoded again, and the quality evaluation was calculated on the average when the original images of each view were reconstructed with the base and additional view. It was confirmed that a bitrate gain of -10 to 30% compared to the previous method could be obtained when compression was applied asymmetrically to the video and the additional view video.

That is, in immersive video, unlike general 360 video, there is an image synthesis process, so even if the quality of low-important images is lowered, it is confirmed that the loss can be relatively small in the final synthesized image, and additional viewpoint images of low importance are created. It can be seen that the bandwidth transmission efficiency compared to the quality actually increases when the quality is lowered.

The asymmetric quantization allocation technique and transmission technique disclosed in this specification can be used for 6DoF immersive video image streaming. As many images are needed to synthesize a virtual viewpoint corresponding to the user's viewpoint, the required bandwidth is also very large. Through the technique according to the embodiment of this specification, it is possible to effectively reduce the transmitted bit rate while minimizing the quality loss of the video adaptively enjoyed according to the importance of each video.

The technology of the embodiment disclosed in this specification can also be applied to MPEG DASH, which is a streaming service that divides and transmits a bitstream, Microsoft's Smooth Streaming, Apple's HLS (HTTP Live Streaming), and the like.

Various embodiments of the multi-view image processing method and apparatus of this specification will be briefly and clearly described as follows.

A multi-view image encoding method according to an embodiment includes dividing an image of a plurality of views into a base view image and an additional view image; calculating a ratio of an additional view image to a background view image; obtaining transform coefficients by performing a prediction operation and a transform operation on the base view image and the additional view image, and obtaining a quantization transform coefficient by quantizing the transform coefficients of the base view image and the additional view image in a plurality of quantization steps, respectively; selecting quantization transform coefficients quantized in a first quantization step and a second quantization step for a base view image and an additional view image, respectively, from among quantization transform coefficients quantized in a plurality of quantization steps, based on the ratio; and entropy-coding the selected quantization transform coefficients.

In an embodiment, the ratio may be calculated as a ratio occupied by an additional view image in a synthesized image synthesized by a decoder using a base view image and an additional view image.

In an embodiment, the ratio may be a ratio of the number of pixels of the synthesized image to the number of pixels of the additional viewpoint image.

In one embodiment, the selecting step may select quantization transform coefficients quantized by the first quantization step and the second quantization step to satisfy a target rate-distortion cost.

In one embodiment, whether or not the rate-distortion cost is satisfied may be made in units of frames.

In one embodiment, when there is no quantization transform coefficient quantized in a quantization step satisfying a rate-distortion cost among quantization transform coefficients quantized in a plurality of quantization steps in the selection step, changing the number of pixels of a base view image; By applying the changed number of pixels, the steps of dividing images of a plurality of views into a base view image and an additional view image, calculating a ratio, obtaining quantization transform coefficients, and selecting quantization transform coefficients may be performed again.

In one embodiment, the calculating step may calculate the ratio in units of GOP (Group Of Picture).

In an embodiment, in the selecting step, a quantization transform coefficient may be determined by dividing the ratio into a plurality of ranges and applying a predetermined quantization step to each of the base view image and the additional view image for each range.

In an embodiment, the ratio is a first value and a second value smaller than the first value, respectively, and a second value applied when generating a quantization transform coefficient selected based on the first value and the second value for the additional view image. When the quantization steps are the 2-1st step and the 2-2nd step, respectively, the 2-2nd step may be a quantization step with lower quality than the 2-1st step.

In an embodiment, the dividing step may include selecting one or more background view images from among a plurality of view images to generate a background view atlas, and images of remaining views that are not selected as one or more background view images from among the plurality of view images. In this method, an additional viewpoint atlas can be created with the remaining patch image after removing overlap with the background viewpoint image.

In an embodiment, the multi-view image encoding method includes quantization transformation selected for metadata including configuration information of a base view image and an additional view image and/or control information for generating a synthesized image from the base view image and the additional view image. A step of reflecting information related to a quantization step applied to generation of coefficients may be further included.

An apparatus for encoding a multi-view image according to another embodiment includes a background view atlas including a background view image selected as a background view among images of a plurality of views and a portion overlapping with a background view image in additional view images of the remaining unselected views. an atlas generation unit for generating a background view atlas including the patch image generated by removing the patch, and calculating a ratio between the background view image and the patch image; a texture encoder for encoding the base view image included in the base view atlas and the patch image included in the additional view atlas using a 2D codec; and a metadata construction unit for encoding configuration information of the base view atlas and the additional view atlas and/or control information for controlling a synthesis process of the base view atlas and the additional view atlas, and outputting the encoded metadata as metadata, wherein the texture encoder is configured. obtains transformation coefficients by performing a prediction operation and transformation operation on the background view image and the patch image, obtains a quantization transformation coefficient by quantizing the transformation coefficients of the background view image and the patch image in a plurality of quantization steps, respectively, and calculates the ratio Based on this, quantization transform coefficients quantized in the first quantization step and the second quantization step are selected for the background view image and the patch image, respectively, among the quantization transform coefficients quantized in a plurality of quantization steps, and the selected quantization transform coefficients can be entropy-coded. there is.

In an embodiment, the ratio may be calculated as a ratio occupied by the additional view atlas in a synthesized image synthesized by the decoder using the base view atlas and the additional view atlas.

In an embodiment, the ratio may be a ratio of the number of pixels of the synthesized image to the number of pixels of the additional viewpoint atlas.

In one embodiment, the texture encoder may select quantization transform coefficients quantized in the first quantization step and the second quantization step to satisfy a target rate-distortion cost.

In one embodiment, the texture encoder may perform an operation of determining whether a rate-distortion cost is satisfied in units of frames.

In one embodiment, the atlas generating unit may calculate the ratio in units of Group Of Pictures (GOP).

In an embodiment, the texture encoder may divide the ratio into a plurality of ranges and determine quantization transform coefficients by applying predetermined quantization steps to each of the base view image and the patch image for each range.

In an embodiment, the ratio is a first value and a second value smaller than the first value, respectively, and a second value applied when generating a quantization transform coefficient selected based on the first value and the second value for the additional view image. When the quantization steps are the 2-1st step and the 2-2nd step, respectively, the 2-2nd step may be a quantization step with lower quality than the 2-1st step.

In one embodiment, the metadata configuration unit may reflect information related to a quantization step applied to generation of the selected quantization transform coefficient in metadata.

An apparatus for decoding a multi-view image according to another embodiment includes a texture decoder configured to generate a base view atlas and an additional view atlas by decoding encoded texture data of a base view and an additional view using a 2D codec; a metadata parser for interpreting metadata including configuration information of the base view atlas and the additional view atlas and/or control information for controlling a synthesis process of the base view atlas and the additional view atlas; an atlas patch occupancy map generation unit for generating a patch occupancy map by determining a location and direction in which a patch image included in an additional viewpoint atlas is to be placed using metadata; and a reproduction unit for generating an image corresponding to the motion of the viewer from the base view atlas and the additional view atlas using the metadata and the patch occupancy map, wherein the metadata is a quantization conversion coefficient of the base view image and the patch image. It further includes information related to the quantization step applied to the generation, and the texture decoder uses the information related to the quantization step to inverse quantize the quantization transform coefficients of the base view image and the patch image, and inverse transform the inverse quantized transform coefficients to obtain residual samples. and restores the base view image and the patch image by generating reconstructed samples using the residual samples and prediction samples generated by the prediction operation, and the quantization step applied to the transform coefficient of the base view image and the transform coefficient of the patch image The quantization step applied to can be determined as different values based on the ratio of the background view image and the patch image.

The embodiments of this specification may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the medium may be those specially designed and configured for the embodiments of this specification, or those known and usable to those skilled in computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs, DVDs, and BDs, and magneto-optical media such as floptical disks. -optical), ROM (ROM), RAM (RAM), flash memory, etc. may be included. Examples of the program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler. A hardware device may be configured to act as one or more software modules to perform the operations of the embodiments herein, and vice versa.

Through the above description, those skilled in the art will know that various changes and modifications are possible without departing from the spirit of the present invention. Therefore, the technical scope of the present invention is not limited to the contents described in the detailed description of the specification, but should be defined by the claims.

100: Multi-View Video Encoder 110: View Optimization Unit
120: atlas generator 130: encoder
131: prediction unit 132: conversion unit
133: quantization unit 134: entropy encoding unit
140: metadata construction unit 200: multi-view image decoder
210: decoder 220: metadata parser
230: Atlas patch occupancy map generation unit
240: regeneration unit

Claims (21)

dividing an image of a plurality of viewpoints into a base viewpoint image and an additional viewpoint image;
calculating a ratio of the background view image to the additional view image;
obtaining transform coefficients by performing a prediction operation and a transform operation on the base view image and the additional view image, and obtaining a quantization transform coefficient by quantizing the transform coefficients of the base and additional view images in a plurality of quantization steps, respectively;
Selecting quantized transform coefficients quantized in a first quantization step and a second quantization step for the base view image and the additional view image, respectively, from among the quantized transform coefficients quantized in the plurality of quantization steps, based on the ratio. ; and
Entropy coding the selected quantization transform coefficients,
The ratio is a ratio of the number of pixels of the synthesized image to the number of pixels of the additional view image in a synthesized image synthesized by a decoder using the base view image and the additional view image. delete delete According to claim 1,
The selecting step selects quantization transform coefficients quantized in the first quantization step and the second quantization step to satisfy a target rate-distortion cost. According to claim 4,
The multi-view video encoding method, characterized in that whether the rate-distortion cost is satisfied is performed in units of frames. According to claim 4,
In the selecting step, when there is no quantization transform coefficient quantized in the quantization step satisfying the rate-distortion cost among the quantization transform coefficients quantized in the plurality of quantization steps, the number of pixels of the base view image is changed; The steps of dividing the images of the plurality of views into the base view image and the additional view image by applying the changed number of pixels, calculating the ratio, obtaining the quantization transform coefficients, and selecting the quantization transform coefficients again. A multi-view video encoding method characterized in that it is performed. According to claim 1,
The calculating step is a multi-view video encoding method, characterized in that for calculating the ratio in units of GOP (Group Of Picture). According to claim 1,
The selecting step divides the ratio into a plurality of ranges, and determines the quantization transform coefficient by applying a predetermined quantization step to each of the base view image and the additional view image for each range. Point image encoding method. According to claim 1,
A second value applied when the ratio is a first value and a second value smaller than the first value, and the selected quantization transform coefficient is generated based on the first value and the second value for the additional view image. When the quantization steps are steps 2-1 and 2-2, respectively, the step 2-2 is a quantization step having a lower quality than the step 2-1. According to claim 1,
The dividing step may include selecting one or more background viewpoint images from among the plurality of viewpoint images to generate a background viewpoint atlas, and generating a background viewpoint atlas from images of remaining viewpoints not selected as the one or more background viewpoint images among the plurality of viewpoint images. A method of encoding a multi-view image, characterized in that an additional view atlas is generated from the patch image remaining after removing the overlap with the background view image. According to claim 1,
A quantization step applied to generation of the selected quantization transform coefficient to metadata including configuration information of the base view image and the additional view image and/or control information for generating a synthesized image from the base view image and the additional view image. A multi-view video encoding method characterized in that it further comprises the step of reflecting information related to. A background view atlas including a background view image selected as a background view from among images of a plurality of view points and a background view including a patch image generated by removing portions overlapping with the background view image from additional view images of the remaining unselected views an atlas generator for generating a viewpoint atlas and calculating a ratio between the background viewpoint image and the patch image;
a texture encoder for encoding the background view image included in the background view atlas and the patch image included in the additional view atlas using a 2D codec; and
It is configured to include a metadata configuration unit for encoding configuration information of the base view atlas and the additional view atlas and/or control information for controlling a process of synthesizing the base view atlas and the additional view atlas and outputting them as metadata; ,
The texture encoder obtains transform coefficients by performing a prediction operation and a transform operation on the base view image and the patch image, and quantizes the transform coefficients of the base view image and the patch image in a plurality of quantization steps, respectively. A transform coefficient is obtained, and based on the ratio, quantized transform coefficients quantized in the first quantization step and the second quantization step are selected for the background view image and the patch image, respectively, from among the quantized transform coefficients quantized in the plurality of quantization steps. and entropy-coding the selected quantization transform coefficients.
The ratio is a ratio of the number of pixels of the synthesized image to the number of pixels of the additional view image in a synthesized image synthesized by a decoder using the base view image and the additional view image. delete delete According to claim 12,
The texture encoder selects quantized transform coefficients quantized in the first quantization step and the second quantization step to satisfy a target rate-distortion cost. According to claim 15,
The multi-view video encoding apparatus of claim 1 , wherein the texture encoder performs an operation of determining whether the rate-distortion cost is satisfied in units of frames. According to claim 12,
The atlas generating unit calculates the ratio in a group of picture (GOP) unit. According to claim 12,
The texture encoder divides the ratio into a plurality of ranges, and determines the quantization transform coefficient by applying a predetermined quantization step to each of the base view image and the patch image for each range. encoding device. According to claim 12,
A second value applied when the ratio is a first value and a second value smaller than the first value, and the selected quantization transform coefficient is generated based on the first value and the second value for the additional view image. When the quantization steps are the 2-1 step and the 2-2 step, respectively, the multi-view video encoding apparatus, characterized in that the 2-2 step is a quantization step with lower quality than the 2-1 step. According to claim 12,
The multi-view video encoding apparatus of claim 1 , wherein the metadata construction unit reflects information related to a quantization step applied to the generation of the selected quantization transformation coefficient in the metadata. a texture decoder for generating a basic view atlas and an additional view atlas by decoding encoded texture data of a basic view and an additional view using a 2D codec;
a metadata parser for interpreting metadata including configuration information of the basic view atlas and the additional view atlas and/or control information for controlling a synthesis process of the basic view atlas and the additional view atlas;
an atlas patch occupancy map generating unit configured to generate a patch occupancy map by determining a position and direction in which a patch image included in the atlas of the additional viewpoint is to be placed using the metadata; and
A playback unit configured to generate an image corresponding to a motion of a viewer from the base view atlas and the additional view atlas using the metadata and the patch occupancy map;
The metadata further includes information related to a quantization step applied to generation of quantization transformation coefficients of the base view image and the patch image;
The texture decoder inversely quantizes quantization transform coefficients of the base view image and the patch image using information related to the quantization step, inversely transforms the inversely quantized transform coefficient, and restores residual samples, Restoring the base view image and the patch image by generating reconstructed samples using prediction samples generated by the prediction operation and the prediction operation;
A quantization step applied to a transform coefficient of the background view image and a quantization step applied to a transform coefficient of the patch image are determined to be different values based on a ratio between the background view image and the patch image;
The ratio is a ratio of the number of pixels of the synthesized image to the number of pixels of the additional view image in a synthesized image synthesized by a decoder using the base view image and the additional view image.

Images