KR102312285B1 - A method for seletively decoding a syncronized multi view video by using spatial layout information - Google Patents

Abstract

An image decoding method according to an embodiment of the present invention includes: receiving a bitstream including an encoded image; obtaining spatial structure information corresponding to the synchronized multi-view image; and selectively decoding at least a portion of the bitstream based on the spatial structure information.

Description

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a method and apparatus for encoding/decoding an image. More specifically, the present invention relates to a method for selectively decoding a synchronized multi-view image using spatial structure information, a method for encoding, and an apparatus therefor.

With the recent development of digital image processing and computer graphics technology, research on virtual reality (VR) technology, which reproduces the real world and allows you to experience it realistically, is being actively conducted.

In particular, a recent VR system such as a head mounted display (HMD) can provide a 3D stereoscopic image to both eyes of a user and track the viewpoint in all directions, so 360-degree rotational viewing is possible. It is receiving a lot of attention in that it can provide reality (VR) video content.

However, since 360 VR contents are composed of simultaneous omnidirectional multi-view image information in which temporal and binocular images are spatially compositely synchronized, in the production and transmission of images, two large images synchronized for binocular space at all viewpoints are used. It is encoded, compressed, and transmitted. This increases complexity and bandwidth burden, and in particular, in the decoding apparatus, there is a problem in that unnecessary processes are wasted because decoding is performed even for an area that is not actually viewed beyond the user's point of view.

Accordingly, there is a need for an efficient encoding method that reduces the amount and complexity of transmission data of an image, and also consumes bandwidth and a battery of a decoding apparatus.

The present invention provides a method and apparatus for efficiently encoding/decoding a synchronized multi-view image such as a 360-degree camera or VR image by using spatial structure information of the synchronized multi-view image. Its purpose is to provide

As a technical means for achieving the above technical problem, an image decoding method according to an embodiment of the present invention includes: receiving a bitstream including an encoded image; obtaining spatial structure information corresponding to the synchronized multi-view image; and selectively decoding at least a portion of the bitstream based on the spatial structure information.

In addition, as a technical means for achieving the above technical problem, an image decoding apparatus according to an embodiment of the present invention obtains spatial structure information corresponding to a synchronized multi-view image from a bitstream including an encoded image, and a decoding processing unit that selectively decodes at least a portion of the bitstream based on the spatial structure information.

And, as a technical means for achieving the above technical problem, an image encoding method according to an embodiment of the present invention includes: obtaining a synchronized multi-view image; generating spatial structure information of the synchronized multi-view image; encoding the synchronized multi-view image; and transmitting a bitstream including the encoded multiview image and the spatial structure information to a decoding system, wherein the decoding system selectively decodes at least a portion of the bitstream based on the spatial structure information characterized in that

Meanwhile, the video processing method may be implemented as a computer-readable recording medium in which a program for execution by a computer is recorded.

According to an embodiment of the present invention, by extracting and signaling spatial structure information optimized for encoding and transmission from a synchronized multi-view image, it is possible to efficiently reduce the transmission data amount, bandwidth, and complexity of an image.

In addition, when a synchronized multi-view image is received, the decoding end can perform some optimized selective decoding for each view according to the signaling information, thereby reducing system waste and efficient encoding/decoding in terms of complexity and battery consumption. A decryption method and apparatus may be provided.

In addition, according to an embodiment of the present invention, it is possible to support spatial structure information for various types of synchronized images, thereby enabling appropriate image reproduction according to the specification of a decoding device, thereby improving device compatibility.

1 shows the overall system structure according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a time-synchronized multi-view video encoding apparatus according to an embodiment of the present invention.
3 to 6 are diagrams illustrating an example of a spatial structure of a synchronized multi-view image according to an embodiment of the present invention.
7 to 9 are diagrams for explaining a method of signaling spatial structure information according to various embodiments of the present disclosure.
10 is a diagram for explaining the configuration of spatial structure information according to an embodiment of the present invention.
11 to 12 are diagrams for explaining a type index table of spatial structure information according to an embodiment of the present invention.
13 is a diagram for explaining a viewpoint information table of spatial structure information according to an embodiment of the present invention.
14 is a flowchart illustrating a decoding method according to an embodiment of the present invention.
15 and 18 are diagrams illustrating that a scanning order of a decoder is determined according to signaling of spatial structure information according to an embodiment of the present invention.
17 is a diagram for describing an independent sub-image and a dependent sub-image that are distinguished according to signaling of spatial structure information.
18 to 19 show that a boundary region between sub-images is decoded with reference to an independent sub-image according to spatial structure information.
20 to 24 are diagrams illustrating a decoding system and its operation according to an embodiment of the present invention.
25 to 26 are diagrams for explaining encoding and decoding processing according to an embodiment of the present invention.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily implement them. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be "connected" with another part, it includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. do.

Throughout this specification, when a member is said to be positioned on another member, this includes not only a case in which a member is in contact with another member but also a case in which another member exists between the two members.

Throughout this specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. As used throughout this specification, the terms "about," "substantially," and the like are used in a sense at or close to the numerical value when presented with manufacturing and material tolerances inherent in the stated meaning, and are intended to enhance the understanding of this application. To help, precise or absolute figures are used to prevent unfair use by unscrupulous infringers of the stated disclosure. The term “step of” or “step of” to the extent used throughout this specification does not mean “step for”.

Throughout this specification, the term of a combination thereof included in the expression of the Markush form means one or more mixtures or combinations selected from the group consisting of the components described in the expression of the Markush form, and the components are It means to include one or more selected from the group consisting of.

In an embodiment of the present invention, as an example of a method for encoding a synchronized image, the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG), which have the highest encoding efficiency among the video encoding standards developed so far, jointly standardize Encoding may be performed using one HEVC (High Efficiency Video Coding), but is not limited thereto.

In general, the encoding apparatus includes an encoding process and a decoding process, and the decoding apparatus includes a decoding process. The decoding process of the decoding apparatus is the same as the decoding process of the encoding apparatus. Therefore, the encoding apparatus will be mainly described below.

1 shows the overall system structure according to an embodiment of the present invention.

Referring to FIG. 1 , the entire system according to an embodiment of the present invention includes a pre-processing unit 10 , an encoding unit 100 , a decoding unit 200 , and a post-processing unit 20 .

A system according to an embodiment of the present invention includes a preprocessing unit 10 for obtaining a synchronized video frame by preprocessing a plurality of views by merging or stitching images, and the synchronized video frame. An encoding apparatus 100 for encoding and outputting a bitstream, a decoding apparatus 200 for receiving the bitstream and decoding the synchronized video frame, and post-processing of the video frame, each synchronized image for each view It may be configured to include a post-processing device 20 to be output to the display of.

Here, the input image may include individual images for each multi-viewpoint, for example, sub-image information of various viewpoints captured by one or more cameras in time and space synchronized state. Accordingly, the pre-processing apparatus 10 may obtain synchronized image information by spatially merging or stitching the acquired multi-viewpoint sub-image information according to time.

In addition, the encoding apparatus 100 may generate a bitstream by scanning and predictive encoding the synchronized image information, and the generated bitstream may be transmitted to the decoding apparatus 200 . In particular, the encoding apparatus 100 according to an embodiment of the present invention may extract spatial structure information from the synchronized image information, and may signal to the decoding apparatus 200 .

Here, the spatial layout information may include basic information about the properties and arrangement of each sub-image as one or more sub-images are merged from the pre-processing unit 10 to form one video frame. . In addition, additional information about each sub-image and a relationship between the sub-images may be further included, which will be described later.

Accordingly, spatial structure information according to an embodiment of the present invention may be transmitted to the decoding apparatus 200 . In addition, the decoding apparatus 200 may determine a decoding target and a decoding order of the bitstream with reference to spatial structure information and user viewpoint information, which may induce efficient decoding.

Then, the decoded video frame is again separated into sub-images for each display through the post-processing device 20 and provided to a plurality of synchronized display systems such as HMD. A multi-view video can be provided.

2 is a block diagram illustrating a configuration of a time-synchronized multi-view video encoding apparatus according to an embodiment of the present invention.

Referring to FIG. 2 , the encoding apparatus 100 according to an embodiment of the present invention includes a synchronized multi-view image acquisition unit 110 , a spatial structure information generation unit 120 , a spatial structure information signaling unit 130 , and an image encoding unit. and a transmission processing unit 150 .

The synchronized multi-viewpoint image acquisition unit 110 acquires a synchronized multi-viewpoint image using a synchronized multi-viewpoint image acquisition means such as a 360-degree camera. The synchronized multi-view image may include a plurality of sub-images synchronized in time and space, and may be received from the preprocessing unit 10 or may be received from a separate external input device.

Then, the spatial structure information generating unit 120 divides the synchronized multi-view image into video frames in units of time, and extracts spatial structure information for the video frame. The spatial structure information may be determined according to the property and arrangement state of each sub-image, and may be determined according to information obtained from the preprocessing unit 10 .

In addition, the spatial structure information signaling unit 130 performs information processing for signaling the spatial structure information to the decoding apparatus 200 . For example, the spatial structure information signaling unit 130 performs one or more processes to be included in the image data encoded by the image encoder, to configure a separate data format, or to be included in the metadata of the encoded image. can

Then, the image encoder encodes the synchronized multi-view image according to the passage of time. Also, the image encoder may determine an image scanning order and a reference image by using the spatial structure information generated by the spatial structure information generator 120 as reference information.

Accordingly, the image encoder may perform encoding using HEVC (High Efficiency Video Coding) as described above, but may be improved in a more efficient manner for a synchronized multi-view image according to spatial structure information.

In addition, the transmission processing unit 150 combines the encoded image data and the spatial structure information inserted from the spatial structure information signaling unit 130 to perform one or more transformation and transmission processes for transmission to the decoding apparatus 200 . can

3 to 6 are diagrams illustrating an example of a spatial structure and image configuration of a synchronized multi-view image according to an embodiment of the present invention.

Referring to FIG. 3 , a multi-view image according to an embodiment of the present invention may include a plurality of temporally synchronized and spatially synchronized image frames.

Each frame may be synchronized according to a unique spatial layout, and a layout of sub-images corresponding to one or more scenes, perspectives, or views to be displayed at the same time may be configured.

Accordingly, spatial layout information may be obtained by combining sub-images constituting a synchronized multi-view image into one input image through merging, stitching, etc., or a simultaneous multi-view image (for example, , as a plurality of images synchronized at the same time (corresponding to various views corresponding to the same POC) as input images, arrangement information of the multi-view image or sub-images, location information and angle information of the capture camera, The sub-images and their relationship information, such as merging information, information on the number of sub-images, scanning order information, acquisition time information, camera parameter information, and reference dependency information between sub-images, may be included.

For example, as shown in FIG. 4 , image information can be captured through a divergent type camera arrangement, and a 360-degree observable spatial image is constructed through stitching on the arrangement image. can do.

As shown in FIG. 4, images A', B', C', ... taken corresponding to each camera arrangement A, B, C ... may be arranged according to a one-dimensional or two-dimensional spatial structure. In addition, left, right, upper and lower region relation information for stitch processing between the arranged images may be exemplified as spatial structure information.

Accordingly, the spatial structure information generating unit 120 can extract spatial structure information including the various properties as described above from the input image, and the spatial structure information signaling unit 130 provides an optimized spatial structure information to be described later. method can be signaled.

* The spatial structure information generated and signaled in this way can be utilized as useful reference information as described above.

For example, if the content captured by each camera is a pre-stitched image, the pre-stitched images are overlapped before encoding to constitute one scene. On the other hand, the scene may be divided according to each view, and mutual compensation may be made between the separated images according to the type.

Accordingly, in the case of a pre-stitched image that merges and stitches one or more images taken from multiple viewpoints into one image in the pre-processing process and delivers it as an input to the encoder, scene information of the merged and stitched input image, spatial layout configuration information etc. may be delivered to the encoding step and the decoding step through separate spatial structure information signaling.

In addition, even in the case of a non-stitched image image type in which images acquired from multi-views are transmitted as one or more input images at a temporally synchronized viewpoint and encoded and decoded, referencing and compensation according to the spatial structure information in the encoding and decoding steps For this purpose, various spatial layout information and data fields corresponding thereto may be required. In addition, the data field may be encoded together with the compression information of the input image or may be transmitted in separate metadata.

In addition, the data field including spatial layout information may be utilized in the image post-processing apparatus 20 and the rendering process of the display.

To this end, the data field including spatial layout information may include position coordinate information and color difference information obtained at the time of image acquisition from each camera.

For example, information such as 3D coordinate information and color difference information (X, Y, Z), (R, G, B) of an image acquired at the time of acquiring image information from each camera is added to each sub-image It can be acquired and transmitted as information, and this information can be used in post-processing and rendering of images after decoding is performed.

Also, the data field including spatial layout information may include camera information of each camera.

As shown in FIGS. 5 to 6 , one or more cameras may be disposed to provide a spatial image by photographing a three-dimensional space.

For example, as shown in FIG. 5 , the positions of one or more cameras may be fixed to a central position and respective directions may be set in a form of acquiring surrounding objects at one point in a three-dimensional space when acquiring an image. .

In addition, as shown in FIG. 6 , one or more cameras may be arranged to photograph one object from various angles. At this time, in a VR display device that plays a 3D image based on coordinate information (X, Y, Z) and distance information at the time of image acquisition, user's movement information (Up/Down, Left/Right, Zoom in/Zoom Out) ), etc., and decoding or post-processing a portion of the corresponding image, it is possible to restore the desired viewpoint or partial image. On the other hand, as described above, the In a system such as compression, transmission, and reproduction, a separate image conversion tool module, etc. may be added to a necessary part according to a type or characteristic of an image, a characteristic of a decoding apparatus, and the like.

For example, when the image obtained from the camera is of an equirectangular type, the image encoding unit 140 converts it into an image type of a method such as Icosahedron/Cube Map through a conversion tool module according to the compression performance and encoding efficiency of the image, etc., Through this, encoding can be performed. At this time, the conversion tool module may be utilized in the pre-processing device 10 and the post-processing device 20 as well, and conversion information according to the conversion is included in the spatial structure information, etc. in the form of metadata in the decoding device 200 or the post-processing device. (20) or may be transmitted to a VR display device.

Meanwhile, in order to deliver a synchronized multi-view image according to an embodiment of the present invention, a separate VR image compression method for supporting scalability between the encoding apparatus 100 and the decoding apparatus 200 may be required.

Accordingly, in order to scalably compress the VR image, the encoding apparatus 100 may compression-encode the image in a manner that separates the base layer and the enhancement layer.

In this way, when one input image is compressed with high-resolution VR images acquired through various cameras, the base layer performs compression on the original image, and the enhancement layer divides a single picture into regions such as Slice / Tile, etc. Thus, encoding can be performed for each sub-image.

In this case, the encoding apparatus 100 may process compression encoding through an inter-layer prediction technique that increases encoding efficiency by using the reconstructed image of the base layer as a reference image.

On the other hand, the decoding apparatus 200 decodes the base layer and, when it is necessary to quickly decode a specific image according to the user's movement, decodes a part of the enhancement layer, so that the partial image decoding according to the user's movement can be performed quickly. .

In this scalable compression method, the encoding apparatus 100 encodes the base layer, but in the base layer, scale down or down sampling of the original image is performed at an arbitrary ratio. can be compressed. In this case, the enhancement layer adjusts the size of the image to the same resolution through scale-up or up-sampling of the reconstructed image of the base layer, and uses the reconstructed image of the base layer corresponding thereto as a reference picture. By doing so, encoding/decoding can be performed.

According to a processing structure supporting such scalability, the decoding apparatus 200 decodes the entire bitstream of the base layer compressed to a low bit or low resolution, and improves only some images of the entire bitstream according to the user's movement. It can be decrypted in layers. In addition, since the entire image is not completely decoded, the VR image can be reconstructed with low complexity.

In addition, according to an image compression method supporting separate scalability with different resolutions, the encoding apparatus 100 may perform compression on an original image or an image according to the intention of the image producer in the base layer, and improve Encoding may be performed based on an inter-layer prediction method in which encoding is performed by referring to a reconstructed image of a base layer in a layer.

In this case, the input image of the enhancement layer may be an image obtained by segmenting one input image through an image segmentation method and encoding the image into a plurality of regions. One divided region may include at most one sub-image, and a plurality of divided regions may be composed of one sub-image. The compressed bitstream encoded through this division method can process two or more outputs in the service and application stages. For example, in the service, the entire image is restored and output through decoding of the base layer, and in the enhancement layer, only a partial region and some sub-images are reflected by reflecting the user's movement, viewpoint change, and manipulation through the service or application. can be decrypted.

7 to 9 are diagrams for explaining a method of signaling spatial structure information according to various embodiments of the present disclosure.

7 to 9, spatial structure information is NAL (NETWORK ABSTRACTION LAYER) on HLS such as SPS (SEQUENCE PARAMETER SET) or VPS (VIDEO PARAMETER SET) defined as encoding parameters in general video encoding. It can be signaled as one class type of UNIT format.

First, FIG. 7 shows a NAL UNIT type into which a synchronized video encoding flag is inserted according to an embodiment of the present invention. A flag may be inserted.

Accordingly, FIG. 8 shows an embodiment of inserting the spatial structure information flag into a VIDEO PARAMETER SET (VPS) according to an embodiment of the present invention.

As shown in FIG. 8 , the spatial structure information signaling unit 130 according to an embodiment of the present invention may insert a flag for checking the type of a separate input image on the VPS. The encoding apparatus 100 may insert a flag indicating that synchronized multi-view image encoding such as VR content is performed using vps_other_type_coding_flag through the spatial structure information signaling unit 130 and that the spatial structure information is signaled.

In addition, as shown in FIG. 9 , the spatial structure information signaling unit 130 according to an embodiment of the present invention may signal a multi-view synchronized image encoded image on a Sequence PARAMETER SET (SPS).

For example, as shown in FIG. 9 , the spatial structure information signaling unit 130 inserts the input image type (INPUT_IMAGE_TYPE), so that index information of the synchronized multi-view image may be included in the SPS and transmitted.

Here, when INPUT_IMAGE_TYPE_INDEX on SPS is not -1, or when the INDEX value is -1, or when the value is designated as 0 and semantically corresponds to -1, INPUT_IMAGE_TYPE is synchronized again according to an embodiment of the present invention. It can be indicated that it is a point image.

In addition, when the type of the input image is a synchronized multi-view image, the spatial structure information signaling unit 130 signals by including the viewpoint information (PERSPECTIVE INFORMATION) in the SPS to signal a part of the spatial structure information of the synchronized multi-view image. can be transmitted by inserting into the SPS. The viewpoint information is information that an image layout for each time period is signaled according to a 3D rendering processing process of a 2D image, and may include order information such as top, bottom, and side.

Accordingly, the decoding apparatus 200 may decode the flag of the VPS or SPS to identify whether the corresponding image has been encoded using spatial structure information according to an embodiment of the present invention. For example, in the case of the VPS of FIG. 5 , by extracting VPS_OTHER_TYPE_CODING_FLAG, it can be checked whether the corresponding image is a synchronized multi-view image encoded using spatial structure information.

In addition, in the case of the SPS of FIG. 9, by decoding the PERSPECTIVE_INFORMATION_INDEX information, actual spatial structure information such as a layout can be identified.

At this time, the spatial structure information may be configured in the form of parameters, for example, the spatial structure parameter information is included differently on HLS such as SPS and VPS, or syntax is configured in the form of a separate function, It may be defined as an SEI message.

Also, according to an embodiment, the spatial structure information may be included in a PPS (PICTURE PARAMETER SET) and transmitted. In this case, attribute information for each sub-image may be included. For example, the independence of the sub-image may be signaled. Independence may indicate that a corresponding image can be encoded and decoded without referring to another image, and sub-images of the synchronized multi-view image may include an independent (INDEPENDENT) sub-image and a dependent (DEPENDENT) sub-image. The dependent sub-image may be decoded with reference to the independent sub-image. The spatial structure information signaling unit 130 may signal an independent sub image on the PPS in the form of an independent sub image list.

In addition, the spatial structure information may be defined and signaled as an SEI message. 10 illustrates an SEI message as spatial structure information, parameterized spatial structure information may be inserted using a spatial layout information descriptor.

As shown in FIG. 10 , the spatial structure information includes type index information (INPUT IMAGE TYPE INDEX) that can indicate the spatial layout of the input image, viewpoint information (PERSPECTIVE INFORMATION), camera parameter information (CAMERA PARAMETER), It may include at least one of scene angle information (SCEN ANGLE), scene dynamic range information (SCENE DYNAMIC RANGE), independent sub image information (INDEPENDENT SUB IMAGE), and scene time information (SCENE TIME INFORMATION). Various information necessary for efficiently encoding the image may be further added. Such parameters may be defined in the form of an SEI message in the form of one descriptor, and the decoding apparatus 200 may parse it and efficiently utilize the spatial structure information in decoding, post-processing, and rendering steps.

And, as described above, the spatial structure information may be transmitted to the decoding apparatus 200 in the form of SEI or metadata.

Also, for example, spatial structure information may be signaled by a selection option such as configuration in the encoding step.

As a first option, spatial structure information may be included in VPS / SPS / PPS or Coding unit syntax on HLS according to coding efficiency on syntax.

As a second option, the spatial structure information may be signaled at once as metadata in the form of SEI on syntax.

Hereinafter, an efficient video encoding and decoding method according to a synchronized multi-view image format according to an embodiment of the present invention will be described in more detail with reference to FIGS. 11 to 19 .

As described above, a plurality of images for each viewpoint generated in the preprocessing step may be synthesized and encoded into one input image. In this case, one input image may include a plurality of sub images. Each of the sub-images may be synchronized to the same time point, and may correspond to a different view, a visual point of view (PERSPECTIVE), or a scene, respectively. This has the effect of supporting various views in the same POC (PICTURE ORDER COUNT) without using separate depth information as in the past, and the overlapping area between each sub-image is limited to the boundary area.

In particular, the spatial structure information of the input image may be signaled in the form described above, and the encoding apparatus 100 and the decoding apparatus 200 may parse the spatial structure information and use it to perform efficient encoding and decoding. That is, the encoding apparatus 100 may process multi-view image encoding using the spatial structure information in the encoding step, and the decoding apparatus 200 may process decoding using the spatial structure information in decoding, pre-processing, and rendering steps. can

11 to 12 are diagrams for explaining a type index table of spatial structure information according to an embodiment of the present invention.

As described above, the sub-images of the input image may be arranged in various ways. Accordingly, the spatial structure information may separately include a table index for signaling the arrangement information. For example, as shown in FIG. 8 , for the synchronized multi-view image, layouts such as CUBIC LAYOUT 4 X 3, CUBIC LAYOUT 3 X 2, ICOSAHEDRON, EQUIRECTANGULAR, etc. may be exemplified, and spatial structure information corresponds to each layout. The table index shown in FIG. 9 may be inserted.

However, the table shown in FIG. 12 is arbitrarily arranged according to the input image, and may be changed according to encoding efficiency and content distribution in the market.

Accordingly, the decoding apparatus 200 may parse the separately signaled table index and use it for the decoding process.

In particular, in an embodiment of the present invention, each of the layout information may be usefully used for partial decoding of an image. That is, sub-image arrangement information such as CUBIC LAYOUT can be used to distinguish an independent sub-image from a dependent sub-image, and accordingly, it can be used to determine an efficient encoding and decoding scanning order or to perform partial decoding for a specific viewpoint.

13 is a flowchart illustrating a decoding method according to an embodiment of the present invention.

Referring to FIG. 13 , first, the decoding apparatus 200 receives an image bitstream ( S101 ).

Then, the decoding apparatus 200 checks whether the image is a synchronized multi-view image (S103).

Here, the decoding apparatus 200 may identify whether it is a synchronized multi-view image from a flag signaled from the spatial structure information signaling unit 130 from the image bitstream. For example, the decoding apparatus 200 may identify in advance whether an image is a synchronized multi-view image from the above-described VPS, SPS, or the like.

If it is not a synchronized multi-view image, general image decoding is performed (S113).

Then, in the case of the synchronized multi-view image, the decoding apparatus 200 decodes the table index from the spatial structure information (S105).

Here, the decoding apparatus 200 may identify whether it is an EQUIRECTANGULAR image from the table index (S107).

This is because the EQUIRECTANGULAR image among the synchronized multi-view images may not be divided into separate sub images, and the decoding apparatus 200 decodes the entire image for the EQUIRECTANGULAR image (S113).

If it is not an EQUIRECTANGULAR image, the decoding apparatus 200 decodes the remaining total spatial structure information (SPATIAL LAYOUT INFORMATION) (S109), and performs image decoding processing based on the spatial structure information (S111).

14 is a diagram for explaining a viewpoint information table of spatial structure information according to an embodiment of the present invention.

The spatial structure information according to an embodiment of the present invention may include a table for viewpoint information (PERSPECTIVE INFORMATION).

The encoding apparatus 100 and the decoding apparatus 200 may use the viewpoint information table as information for discriminating reference between images, encoding and decoding orders, and independent sub-images.

In addition, when rendering in the multi-viewpoint display device, the viewpoint information table may be transmitted to the system layer of the device together with the decoded image, and using the information, the user can match the phase according to the intention of the contents provider. You will be able to watch the video.

More specifically, the spatial structure information signaling unit 130 may signal a viewpoint (PERSPECTIVE) of each sub-image. In particular, the spatial structure information signaling unit 130 signals only information on the top, bottom, and front images according to the type of image, and the image information on the remaining sides is transmitted to the decoding device 200 by the top scene and front perspective scene. , it can be guided by using the Bottom Scene information. Accordingly, only minimal information can be signaled.

15 and 16 are diagrams illustrating that a scanning order of a decoder is determined according to signaling of spatial structure information according to an embodiment of the present invention.

15 and 16 , the scanning order of sub-images may be transmitted together according to the type index of the spatial structure information, and effective decoding and rendering may be performed through the transmitted scanning order information.

15 and 16 show that the original image is scanned in the order of A->B->C->D->E->F.

For this signaling, the spatial structure information signaling unit 130 may signal only order information for some sub-images, such as a top view, a bottom view, and a front view, among scanning sequences. The decoding apparatus 200 may derive an overall order by using order information of the partial sub-images.

In addition, the scanning order may be changed as shown in FIG. 15 or 16 according to the type, parallelism, and reference structure of the transmitted image.

Assuming that Top is A, Bottom is F, and Front is B, the scanning order of FIG. 15 may be A -> F -> B -> C -> D -> E, and the scanning order of FIG. 16 is A->B It can be ->C->D->E->F. This can be usefully used when different scanning orders are determined in consideration of encoding efficiency.

17 is a diagram for describing an independent sub-image and a dependent sub-image that are distinguished according to signaling of spatial structure information.

As shown in FIG. 17 , each sub-image disposed in the spatial structure may be divided into a dependent sub image and an independent sub image in consideration of referentiality and parallelism. The independent sub-image has a characteristic of being decoded without referring to other sub-images, and the dependent sub-image can be reconstructed with reference to the adjacent independent sub-image or the adjacent dependent sub-image.

Accordingly, the independent sub-image may have a characteristic that must be encoded or decoded before the dependent sub-image.

In an embodiment, the independent sub-image may be encoded or decoded with reference to a previously encoded or decoded independent sub-image that is not identical in the time axis, and the dependent sub-image is encoded with reference to the same or non-identical independent sub-image in the time axis Or it can be decrypted. In addition, whether it is independent may be signaled by a separate index according to spatial structure information.

18 to 19 show that a boundary region between sub-images is decoded with reference to an independent sub-image according to spatial structure information.

As described above, the encoding apparatus 100 or the decoding apparatus 200 may process encoding and decoding by simultaneously considering a scanning order and dependency information of a sub-image.

If A and F are independent sub-images as shown in Fig. 18, the scanning order can be transmitted or derived as follows: A -> F -> B -> C -> D -> E, in this case For A and F, decoding should be performed first compared to other dependent sub-images. And when the remaining B, C, D, and E are decoded, the adjacent boundary region of each sub-image can be decoded by referring to the independent sub-image. Accordingly, the boundary region of the previously decoded independent sub-image or the previously decoded dependent sub-image may be referred to for decoding of the remaining sub-images.

In addition, the above-described independent sub-image may be used not only for one image frame but also for performing intra/inter encoding and decoding in a boundary region of an adjacent picture.

*However, as shown in FIG. 19, 1:1 mapping may not be possible due to different resolutions. (In the case of a general image, the ratio of Width is wider than Height)

In this case, in order to refer to the adjacent surface, a scale according to the resolution is adjusted through an image processing technique such as rotation and up-sampling for the target sub-image to be referenced for encoding or decoding of the boundary region. can

For example, in encoding/decoding of the upper region boundary of C in FIG. 19 , the side value of A may be referred to. To this end, the encoding apparatus 100 or the decoding apparatus 200 up-samples the side value (Height) of A at a ratio corresponding to the width of C to generate a reference block value according to the corresponding position, and encodes and decodes it can be performed.

20 to 24 are diagrams illustrating a decoding system and its operation according to an embodiment of the present invention.

Referring to FIG. 20 , the decoding system 300 according to an embodiment of the present invention receives the entire synchronized multi-view image bitstream and spatial structure information received from the encoding apparatus 100 or an external server as described above. , a client system that provides one or more decoded pictures to the user's virtual reality display device 400 may be configured.

To this end, the decryption system 300 includes a decryption processing unit 310 , a user motion analysis unit 320 , and an interface unit 330 . However, although the decryption system 300 is described as a separate system in this specification, it is all or all of the above-described decryption apparatus 200 and post-processing apparatus 20 for performing the necessary decryption processing and post-processing. It may be configured by a combination of some modules, and may be configured by expanding the decryption apparatus 200 . Therefore, it is not limited to the name.

Accordingly, the decoding system 300 according to an embodiment of the present invention selectively decodes a part of the entire bitstream based on spatial structure information received from the encoding apparatus 100 and user viewpoint information according to user motion analysis. can be done In particular, according to the selective decoding described in FIG. 20 , the decoding system 300 uses spatial structure information to view input images having a plurality of viewpoints at the same time (POC, Picture of Count) based on a predetermined direction of the user. It can correspond to the viewpoint (PERSPECTIVE). Also, based on this, partial decoding may be performed on Region Of Interest (ROI) pictures determined by the user's viewpoint.

To this end, the decoding system 300 may include an interface layer for receiving and analyzing user information, and selectively selects the viewpoint mapping, post-processing, rendering, etc. between the viewpoint supported by the currently decoded image and the VR display apparatus 400 . can be done More specifically, the interface layer may include one or more processing modules for the post-processing and rendering, an interface unit 330 , and a user motion analysis unit 320 .

The interface unit 330 may receive motion information from the VR display device 400 worn by the user.

The interface unit 330 may, for example, connect at least one of an environment sensor, a proximity sensor, a motion detection sensor, a position sensor, a gyroscope sensor, an acceleration sensor, and a geomagnetic sensor of the user's VR display device 400 by wire or wirelessly. one or more data communication modules for receiving.

Then, the user motion analysis unit 320 analyzes the user motion information received from the interface unit 330 to determine the user's viewpoint (PERSPECTIVE), and selection information for adaptively selecting a decoded picture group corresponding thereto may be transmitted to the decryption processing unit 310 .

Accordingly, the decoding processing unit 310 may set an ROI mask for selecting an ROI (Region Of Intrest) picture based on the selection information transmitted from the user motion analysis unit 320 , and may set an ROI mask corresponding to the set ROI mask. Only the picture region can be decoded. For example, a picture group may correspond to at least one of a plurality of sub-images or reference images in the above-described image frame.

For example, as shown in FIG. 20 , when sub-images of a specific POC decoded by the decoding processing unit 310 exist from 1 to 8, the decoding processing unit 310 corresponds to the user's viewpoint (PERSPECTIVE). By decoding only the sub-image areas 6 and 7, the processing speed and efficiency can be improved in real time.

21 exemplifies the entire encoded bitstream and GROUP OF PICTURES (GOP) according to an embodiment of the present invention, and FIG. 22 shows that the decoded picture group among the entire bitstream is selectively changed according to the user's viewpoint. .

As described above, the decoding system 300 according to an embodiment of the present invention receives the entire encoded bitstream of the synchronized multi-view image, and converts it into a sub-bitstream corresponding to the user's perspective view. decryption can be performed.

In this case, the spatial structure information of the synchronized multi-view image may be signaled in the form of SEI or HLS as described above. In particular, according to an embodiment, the decoding system 300 obtains a reference structure using an independent sub image, a dependent sub image, and a spatial random access picture (SRAP) identified from the signaling information. can be created and built. And, using the reference structure, it is possible to select and decode some bitstreams according to a change in a visual viewpoint (PERSPECTIVE).

To this end, the encoding apparatus 100 may select a picture as a spatial random access picture (SRAP, SPATIAL RANDAOM ACCESS PICTURE) by NAL type or PPS, and the SRAP picture is used for intra prediction encoding of sub-images of other pictures that are not selected. may be used as a reference picture.

Accordingly, the encoding apparatus 100 and the decoding apparatus 200 select one or more of the SRAP pictures for a picture that is not selected as SPAP by a specific group of picture (GOP) or NAL type or PPS to perform encoding and decoding processing. can

More specifically, in the SRAP picture, for decoding of some bitstreams among the entire bitstream, a single input image composed of multiple views for the same time is sub-in the SRAP picture, regardless of the decoding of images for different times. It may mean a picture that can be independently decoded by using data redundancy between images.

In addition, in the picture selected by SRAP, each sub-image can be encoded/decoded through intra coding, etc., and at least one SRAP picture in a certain GOP is included in order to decode some bitstreams of the entire bitstream. can

Also, pictures not selected as SRAP pictures may be used to perform decoding according to a user's viewpoint through an inter prediction method by using pictures selected as SRAP pictures as reference pictures.

Accordingly, the decoding system 300 can decode one or more sub-images selected in response to the user's visual viewpoint (PERSPECTIVE) through the decoding processing unit 310 using the reference picture.

Meanwhile, the picture selected by the SRAP may be fully intra-prediction-encoded of a single image according to a general encoding and decoding method without reference.

Therefore, the decoding system 300 or the decoding apparatus 200 needs to store at least one SRAP picture in a DECODED PICTURE BUFFER (DPB). In addition, a picture previously decoded in one GOP or a sub-image between SRAP pictures may be decoded using a picture stored in the SRAP as a reference picture.

Accordingly, FIG. 22 shows a picture designated as SRAP and a change in a selective decoding area according to a change in a user's viewpoint. As shown in FIG. 19 , the period during which the SRAP picture is decoded may be a predetermined period of time.

In addition, with reference to the SRAP picture that is periodically decoded, the decoded sub-image group may be changed according to the occurrence time of a USER PERSPECTIVE OF CHANGE. For example, as shown in FIG. 19 , the decoded ROI region may be changed from the first sub-image group to the second image sub-image group when the user viewpoint change occurs.

Meanwhile, FIG. 23 is a diagram for explaining a sub-image decoding method according to an embodiment of the present invention.

As described above, in the method of decoding some pictures according to the user's perspective, the dependency (DEPENDENCY) of each sub-image may be considered as shown in FIG. 20 .

As described above, the encoding apparatus 100 and the decoding apparatus 200 may grant independence to some pictures (sub-images) according to encoding efficiency or a scan order of the sub-images. In addition, by encoding and decoding the independent sub-image to which the independence is given as an in-screen picture, dependence on other sub-images can be eliminated.

Meanwhile, the encoding apparatus 100 and the decoding apparatus 200 may designate the remaining sub-images as dependent sub-images. Independent sub-images having the same POC as the dependent sub-image may be added to a reference picture list of the dependent sub-image, and the dependent sub-images are intra prediction for adjacent independent sub-images. Encoding or decoding through (intra coding), inter prediction using an independent sub-image having the same POC as a reference picture, or inter-prediction method for independent sub-images having different POCs can be encoded or decoded.

More specifically, in response to the dependent sub-images, independent sub-images having different POCs may also be added to the reference picture list. The dependent sub-image may be used as a reference picture in intra prediction encoding or inter prediction encoding referring to independent sub-images added to a list.

Accordingly, decoding through the intra prediction method may be performed on the dependent sub-images by using the independent sub-image as a reference picture. Also, decoding may be performed through an inter prediction method by designating the same POC as the sub-image currently being decoded as a reference picture.

For example, since the dependent sub-image has a high similarity to the independent sub-images having the same POC in the boundary region, the encoding apparatus 100 determines the boundary region of the dependent sub-image within the screen with the independent sub-image. Encoding using a prediction method may be performed.

In this case, independent sub-images to be referenced may be changed according to a unit such as PPS or SPS on HLS, and information on this may be separately signaled. And, the decoding apparatus 200 may derive dependent sub-images based on the independent sub-images. Also, the decoding apparatus 200 may receive the independence of all sub-images in the form of a separate list. .

In addition, according to the SRAP and independence-based encoding and decoding processing as described above, the decoding system 300 easily selects some bitstreams according to the user's perspective from among the entire bitstreams, and performs adaptive and selective decoding. can be performed efficiently.

In addition, independent sub-images or dependent sub-images may be separately signaled or indexed, and a signal capable of identifying them may be transmitted to the decoding apparatus 200 . In this case, the decoding apparatus 200 directly encodes a region of some or all of the sub-images to be decoded according to the user's viewpoint with reference to an adjacent independent sub-image or an adjacent region between the sub-images with reference to previously decoded sub-images. The mode can be guessed and restoration can be performed on the corresponding area.

Meanwhile, as shown in FIG. 24 , in such intra prediction and inter prediction encoding, a coding method in consideration of encoding efficiency may be exemplified.

Referring to FIG. 24 , the encoding apparatus 100 according to an embodiment of the present invention measures a distortion such as a PSNR of an image or an error value of an RDO process (encoding process), an area according to the characteristics of an image. A metric for applying different weighting factors may be performed.

As shown in FIG. 24 , a user viewing a synchronized multi-view image using a VR display may generally have a wide left and right viewing range based on a front view, but may have a narrow vertical viewing range. Accordingly, the encoding apparatus 100 converts a multi-viewpoint image into a two-dimensional image, and sets the upper and lower regions of the field of view as LOW WEIGHT FACTOR REGIONs, thereby increasing the encoding efficiency.

24 shows a user view region of various synchronized multi-view images, and shows a region to which a low weighting factor is applied in the case where the dual multi-view image is an equirectangular type.

In particular, for weight application, first, the encoding apparatus 100 may perform transcoding on a synchronized multi-view image into a 2D image, and a weighting region may be determined differently depending on the type of the multi-view image. can In addition, the weight region information may be included in the above-described spatial structure information or may be separately signaled through HLS. A weighting factor may also be signaled on a separate HLS or transmitted in the form of a list.

The weight area according to the embodiment of the present invention may be set on the assumption that an area on which a human can focus is limited due to characteristics of a natural image and a refracted 3D image.

Accordingly, the encoding apparatus 100 may perform encoding by applying a relatively low QP as a base QP to a relatively low weight region. Also, when measuring the PSNR, the encoding apparatus 100 may measure the PSNR for a relatively low weight region to be low. Conversely, the encoding apparatus 100 may perform an operation of applying a relatively high QP as a base QP or measuring a relatively high PSNR to a region having a relatively high weight.

In addition, according to an embodiment of the present invention, as described above, for intra block copy prediction, decoding with reference to images having different viewpoints within the same POC may be performed, and the referenced images are independent decoding possible. A sub image may be exemplified.

In particular, intra prediction for the boundary region may be efficiently applied, and the weight factor may be used to maximize encoding efficiency for the boundary region. For example, the encoding apparatus 100 designates the weight factor that allows the area referenced from the independent sub-image area to be patched according to the distance, and applies a scaling value to the weighting factor to determine the prediction value of the neighboring block. can be efficiently determined.

Also, the encoding apparatus 100 may apply a weight to the boundary area when configuring the MPM (MOST PROBABLE MODE) for intra-picture encoding. Accordingly, the encoding apparatus 100 may derive the prediction direction of the MPM centering on the boundary between the sub-images in the image, even if they are not structurally adjacent.

Meanwhile, the encoding apparatus 100 may also consider an independent sub-image in INTER CODING. For example, the encoding apparatus 100 may refer to a sub-image co-lacated in a related region in correspondence to the independent sub-image. Accordingly, when configuring a reference picture for inter-screen encoding, only some sub-images of an image can be added to the reference picture list.

25 to 26 are diagrams for explaining encoding and decoding processing according to an embodiment of the present invention.

25 is a block diagram showing the configuration of a moving picture encoding apparatus according to an embodiment of the present invention. Each sub-image or entire frame of a synchronized multi-view image according to an embodiment of the present invention is received as an input video signal. can be processed

Referring to FIG. 25 , the video encoding apparatus 100 according to the present invention includes a picture division unit 160 , a transform unit, a quantization unit, a scanning unit, an entropy encoding unit, an intra prediction unit 169 , and an inter prediction unit 170 . , an inverse quantization unit, an inverse transform unit, a post-processing unit 171 , a picture storage unit 172 , a subtraction unit and an addition unit 168 .

The picture divider 160 analyzes the input video signal, divides the picture into coding units of a predetermined size for each largest coding unit (LCU), determines a prediction mode, and the size of the prediction unit for each coding unit. to decide

Then, the picture splitter 160 transmits the prediction unit to be encoded to the intra prediction unit 169 or the inter prediction unit 170 according to the prediction mode (or prediction method). Also, the picture dividing unit 160 sends a prediction unit to be encoded to the subtracting unit.

A picture may be composed of a plurality of slices, and a slice may be composed of a plurality of largest coding units (LCUs).

The LCU may be divided into a plurality of coding units (CUs), and the encoder may add information (flag) indicating whether to divide to a bitstream. The decoder may recognize the location of the LCU using an address (LcuAddr).

When splitting is not allowed, the coding unit (CU) is regarded as a prediction unit (PU), and the decoder may recognize the position of the PU using the PU index.

A prediction unit (PU) may be divided into a plurality of partitions. Also, the prediction unit (PU) may be composed of a plurality of transform units (TUs).

In this case, the picture divider 160 may transmit the image data to the subtractor in block units (eg, PU units or TU units) of a predetermined size according to the determined encoding mode.

A Coding Tree Block (CTB) is used as a video coding unit, and CTB is defined in various square shapes. CTB is called a coding unit (CU).

The coding unit (CU) may have the form of a quad tree according to division. In addition, in the case of QTBT (Quadtree plus binary tree) partitioning, the coding unit may have the form of the quadtree or binary tree partitioned binary in the terminal node, and the maximum size according to the encoder standard is 256X256 to 64XVIII. It may consist of 64.

For example, when the maximum size is 64X64, the picture divider 160 sets the depth to 0 when the maximum coding unit is the largest coding unit (LCU) until the depth becomes 3, that is, the Encoding is performed by finding the optimal prediction unit recursively up to the coding unit (CU). Also, for example, for a coding unit of a terminal node divided into QTBT, a prediction unit (PU) and a transform unit (TU) may have the same form as the segmented coding unit or may have a further segmented form.

A prediction unit that performs prediction is defined as a prediction unit (PU), and each coding unit (CU) performs prediction of a unit divided into a plurality of blocks, and performs prediction by dividing it into square and rectangular shapes.

The transform unit transforms a residual block that is a residual signal of the original block of the input prediction unit and the prediction block generated by the intra prediction unit 169 or the inter prediction unit 170 . The residual block is composed of a coding unit or a prediction unit. A residual block composed of a coding unit or a prediction unit is divided and transformed into an optimal transform unit. Different transform matrices may be determined according to prediction modes (intra or inter). In addition, since the residual signal of the intra prediction has directionality according to the intra prediction mode, the transform matrix may be adaptively determined according to the intra prediction mode.

A transformation unit may be transformed by two (horizontal and vertical) one-dimensional transformation matrices. For example, in the case of inter prediction, one predetermined transform matrix is determined.

On the other hand, in the case of intra prediction, when the intra prediction mode is horizontal, the probability that the residual block has a vertical direction increases, so a DCT-based integer matrix is applied in the vertical direction and DST-based in the horizontal direction. Alternatively, a KLT-based integer matrix is applied. When the intra prediction mode is vertical, a DST-based or KLT-based integer matrix is applied in a vertical direction and a DCT-based integer matrix is applied in a horizontal direction.

In the case of DC mode, DCT-based integer matrix is applied in both directions. Also, in the case of intra prediction, the transform matrix may be adaptively determined depending on the size of the transform unit.

The quantization unit determines a quantization step size for quantizing coefficients of the residual block transformed by the transform matrix. The quantization step size is determined for each coding unit (hereinafter, referred to as a quantization unit) having a size greater than or equal to a predetermined size.

The predetermined size may be 8x8 or 16x16. Then, the coefficients of the transform block are quantized using a quantization matrix determined according to the determined quantization step size and prediction mode.

The quantization unit uses the quantization step size of a quantization unit adjacent to the current quantization unit as a quantization step size predictor of the current quantization unit.

The quantization unit searches in the order of the left quantization unit, the upper quantization unit, and the upper left quantization unit of the current quantization unit, and uses one or two valid quantization step sizes to generate a quantization step size predictor of the current quantization unit.

For example, a valid first quantization step size searched in the above order may be determined as a quantization step size predictor. In addition, the average value of two valid quantization step sizes searched in the above order may be determined as the quantization step size predictor, or if only one is valid, it may be determined as the quantization step size predictor.

When the quantization step size predictor is determined, a difference value between the quantization step size of the current coding unit and the quantization step size predictor is transmitted to the entropy encoder.

On the other hand, there is a possibility that all of the left coding unit, the upper coding unit, and the upper left coding unit of the current coding unit do not exist. On the other hand, a previously existing coding unit in the coding order within the largest coding unit may exist.

Accordingly, in the quantization units adjacent to the current coding unit and the largest coding unit, the quantization step size of the quantization unit immediately preceding in the coding order may be a candidate.

In this case, 1) the left quantization unit of the current coding unit, 2) the upper quantization unit of the current coding unit, 3) the upper left quantization unit of the current coding unit, 4) the order of the quantization unit immediately preceding in the coding order. can The order may be changed, and the upper-left quantization unit may be omitted.

The quantized transform block is provided to an inverse quantizer and a scanning unit.

The scanning unit scans the coefficients of the quantized transform block and converts them into one-dimensional quantized coefficients. Since the coefficient distribution of the transform block after quantization may depend on the intra prediction mode, the scanning method is determined according to the intra prediction mode.

Also, the coefficient scanning method may be determined differently according to the size of the transform unit. The scan pattern may vary according to a directional intra prediction mode. The scan order of the quantization coefficients is reversed.

When the quantized coefficients are divided into a plurality of subsets, the same scan pattern is applied to the quantized coefficients in each subset. A zigzag scan or a diagonal scan is applied for the scan pattern between subsets. The scan pattern is preferably scanned from the main subset including DC to the remaining subsets in a forward direction, but the reverse direction is also possible.

Also, the scan pattern between the subsets may be set to be the same as the scan pattern of the quantized coefficients in the subset. In this case, the scan pattern between the subsets is determined according to the intra prediction mode. Meanwhile, the encoder transmits information indicating the position of the last non-zero quantization coefficient in the transform unit to the decoder.

Information that may indicate the position of the last non-zero quantization coefficient in each subset may also be transmitted to the decoder.

The inverse quantization 135 inversely quantizes the quantized quantization coefficient. The inverse transform unit restores the inverse quantized transform coefficient to a residual block in the spatial domain. The adder generates a reconstructed block by combining the residual block reconstructed by the inverse transform unit and the prediction block received from the intra prediction unit 169 or the inter prediction unit 170 .

The post-processing unit 171 includes a deblocking filtering process for removing a blocking effect occurring in the reconstructed picture, an adaptive offset application process for compensating for a difference value from the original image in units of pixels, and a difference from the original image as a coding unit. An adaptive loop filtering process is performed to supplement the values.

The deblocking filtering process is preferably applied to a boundary between a prediction unit and a transform unit having a size greater than or equal to a predetermined size. The size may be 8x8. The deblocking filtering process includes determining a boundary to be filtered, determining a boundary filtering strength to be applied to the boundary, determining whether to apply a deblocking filter, and the deblocking filter and selecting a filter to be applied to the boundary when it is determined to apply .

Whether the deblocking filter is applied is determined by i) whether the boundary filtering intensity is greater than 0, and ii) a value indicating the degree of change in pixel values at the boundary of two blocks (P block, Q block) adjacent to the boundary to be filtered. It is determined by whether it is smaller than a first reference value determined by the quantization parameter.

The filter is preferably at least two or more. When the absolute value of the difference value between two pixels located at the block boundary is greater than or equal to the second reference value, a filter that performs relatively weak filtering is selected.

The second reference value is determined by the quantization parameter and the boundary filtering strength.

The process of applying the adaptive offset is to reduce the distortion between the pixel in the image to which the deblocking filter is applied and the original pixel. It may be determined whether to perform the adaptive offset application process in units of pictures or slices.

A picture or slice may be divided into a plurality of offset regions, and an offset type may be determined for each offset region. The offset type may include a predetermined number (eg, four) of edge offset types and two band offset types.

When the offset type is an edge offset type, an edge type to which each pixel belongs is determined, and an offset corresponding thereto is applied. The edge type is determined based on the distribution of values of two pixels adjacent to the current pixel.

The adaptive loop filtering process may perform filtering based on a value obtained by comparing the original image with the reconstructed image that has undergone the deblocking filtering process or the adaptive offset application process. In the adaptive loop filtering, the determined ALF may be applied to all pixels included in a 4x4 or 8x8 block.

Whether to apply the adaptive loop filter may be determined for each coding unit. The size and coefficient of the loop filter to be applied according to each coding unit may vary. Information indicating whether the adaptive loop filter is applied for each coding unit may be included in each slice header.

In the case of a color difference signal, it may be determined whether the adaptive loop filter is applied in units of pictures. The shape of the loop filter may also have a rectangular shape unlike the luminance.

Whether to apply the adaptive loop filtering for each slice may be determined. Accordingly, information indicating whether adaptive loop filtering is applied to the current slice is included in the slice header or the picture header.

If it indicates that adaptive loop filtering is applied to the current slice, the slice header or the picture header additionally includes information indicating the filter length in the horizontal and/or vertical direction of the luminance component used in the adaptive loop filtering process.

The slice header or the picture header may include information indicating the number of filter sets. In this case, if the number of filter sets is two or more, filter coefficients may be encoded using a prediction method. Accordingly, the slice header or the picture header may include information indicating whether filter coefficients are encoded by the prediction method, and include the predicted filter coefficients when the prediction method is used.

Meanwhile, not only luminance but also chrominance components may be adaptively filtered. Accordingly, the slice header or the picture header may include information indicating whether each chrominance component is filtered. In this case, information indicating whether to filter Cr and Cb may be jointly coded (ie, multiplexed coding) in order to reduce the number of bits.

In this case, in the case of color difference components, it is most likely that neither Cr nor Cb is filtered to reduce complexity. Therefore, when neither Cr nor Cb is filtered, the smallest index is assigned to perform entropy encoding. .

Then, when both Cr and Cb are filtered, entropy encoding is performed by allocating the largest index.

The picture storage unit 172 receives the post-processed image data from the post-processing unit 171 to restore and store the image in units of pictures. A picture may be an image in units of frames or images in units of fields. The picture storage unit 172 includes a buffer (not shown) capable of storing a plurality of pictures.

The inter prediction unit 170 performs motion estimation using at least one or more reference pictures stored in the picture storage unit 172 , and determines a reference picture index and a motion vector indicating the reference picture.

Then, according to the determined reference picture index and motion vector, from a reference picture used for motion estimation among a plurality of reference pictures stored in the picture storage unit 172, a prediction block corresponding to a prediction unit to be encoded is extracted and output. .

The intra prediction unit 169 performs intra prediction encoding by using the reconstructed pixel values inside the picture including the current prediction unit.

The intra prediction unit 169 receives a current prediction unit to be prediction-encoded, selects one of a preset number of intra prediction modes according to the size of the current block, and performs intra prediction.

The intra prediction unit 169 adaptively filters reference pixels to generate an intra prediction block. When the reference pixel is not available, the reference pixels may be generated using the available reference pixels.

The entropy encoding unit entropy-encodes the quantized coefficients quantized by the quantization unit, intra prediction information received from the intra prediction unit 169 , and motion information received from the inter prediction unit 170 .

Although not shown, the inter prediction encoding apparatus may include a motion information determiner, a motion information encoding mode determiner, a motion information encoder, a prediction block generator, a residual block generator, a residual block encoder, and a multiplexer. .

The motion information determining unit determines motion information of the current block. The motion information includes a reference picture index and a motion vector. The reference picture index indicates any one of previously encoded and reconstructed pictures.

When the current block is unidirectional inter prediction coded, it indicates any one of reference pictures belonging to list 0 (L0). On the other hand, when the current block is bi-predictively coded, it may include a reference picture index indicating one of the reference pictures of list 0 (L0) and a reference picture index indicating one of the reference pictures of list 1 (L1). .

In addition, when the current block is bi-predictively encoded, an index indicating one or two of the reference pictures of the composite list LC generated by combining the list 0 and the list 1 may be included.

The motion vector indicates the position of the prediction block in the picture indicated by each reference picture index. The motion vector may be a pixel unit (integer unit) or a sub-pixel unit.

For example, it may have a resolution of 1/2, 1/4, 1/8, or 1/16 pixels. When the motion vector is not an integer unit, a prediction block is generated from pixels of an integer unit.

The motion information encoding mode determiner determines whether to encode the motion information of the current block in the skip mode, the merge mode, or the AMVP mode.

The skip mode is applied when a skip candidate having the same motion information as the motion information of the current block exists and the residual signal is 0. In addition, the skip mode is applied when the current block has the same size as the coding unit. The current block can be viewed as a prediction unit.

The merge mode is applied when there is a merge candidate having the same motion information as the motion information of the current block. The merge mode is applied when a residual signal exists when the size of the current block is different from that of the coding unit or when the size is the same. The merge candidate and the skip candidate may be the same.

AMVP mode is applied when skip mode and merge mode are not applied. An AMVP candidate having a motion vector most similar to the motion vector of the current block is selected as the AMVP predictor.

The motion information encoder encodes the motion information according to a method determined by the motion information encoding mode determiner. When the motion information encoding mode is the skip mode or the merge mode, a merge motion vector encoding process is performed. When the motion information encoding mode is AMVP, an AMVP encoding process is performed.

The prediction block generator generates a prediction block by using motion information of the current block. When the motion vector is an integer unit, a prediction block of the current block is generated by copying the block corresponding to the position indicated by the motion vector in the picture indicated by the reference picture index.

However, when the motion vector is not an integer unit, pixels of the prediction block are generated from integer unit pixels in the picture indicated by the reference picture index.

In this case, in the case of a luminance pixel, a prediction pixel may be generated using an 8-tap interpolation filter. In the case of a chrominance pixel, a prediction pixel may be generated using a 4-tap interpolation filter.

The residual block generator generates a residual block by using the current block and the prediction block of the current block. When the size of the current block is 2Nx2N, a residual block is generated using the current block and a prediction block having a size of 2Nx2N corresponding to the current block.

However, when the size of the current block used for prediction is 2NxN or Nx2N, after obtaining a prediction block for each of the two 2NxN blocks constituting 2Nx2N, the final prediction block of the 2Nx2N size is obtained using the two 2NxN prediction blocks. can create

In addition, a 2Nx2N residual block may be generated using the 2Nx2N prediction block. In order to solve the discontinuity of the boundary between two prediction blocks having a size of 2NxN, the pixels of the boundary portion may be overlapped and smoothed.

The residual block encoder divides the generated residual block into one or more transform units. Then, each transform unit is transcoded, quantized, and entropy coded. In this case, the size of the transform unit may be determined in a quadtree method according to the size of the residual block.

The residual block encoder transforms the residual block generated by the inter prediction method using an integer-based transform matrix. The transformation matrix is an integer-based DCT matrix.

The residual block encoder uses a quantization matrix to quantize coefficients of the residual block transformed by the transform matrix. The quantization matrix is determined by a quantization parameter.

The quantization parameter is determined for each coding unit having a predetermined size or larger. The predetermined size may be 8x8 or 16x16. Accordingly, when the current coding unit is smaller than the predetermined size, only the quantization parameter of the first coding unit in the coding order among the plurality of coding units within the predetermined size is encoded, and the quantization parameters of the remaining coding units are the same as the parameters, so the encoding is performed. no need to do

Then, the coefficients of the transform block are quantized using a quantization matrix determined according to the determined quantization parameter and the prediction mode.

A quantization parameter determined for each coding unit having a size greater than or equal to the predetermined size is predictively encoded using a quantization parameter of a coding unit adjacent to the current coding unit. A quantization parameter predictor of the current coding unit may be generated using one or two valid quantization parameters by searching in the order of the left coding unit and the upper coding unit of the current coding unit.

For example, a valid first quantization parameter retrieved in the above order may be determined as a quantization parameter predictor. In addition, a first valid quantization parameter may be determined as a quantization parameter predictor by searching in the order of the left coding unit and the coding unit immediately preceding in the coding order.

The coefficients of the quantized transform block are scanned and transformed into one-dimensional quantized coefficients. The scanning method may be set differently according to the entropy encoding mode. For example, when coded with CABAC, inter prediction coded quantization coefficients may be scanned in one predetermined method (zigzag or raster scan in a diagonal direction). On the other hand, in the case of CAVLC encoding, scanning may be performed in a method different from the above method.

For example, the scanning method may be determined according to the zigzag mode in the case of inter and the intra prediction mode in the case of intra. Also, the coefficient scanning method may be determined differently according to the size of the transform unit.

The scan pattern may vary according to a directional intra prediction mode. The scan order of the quantization coefficients is reversed.

The multiplexer multiplexes the motion information encoded by the motion information encoder and the residual signals encoded by the residual block encoder. The motion information may vary according to an encoding mode.

That is, in the case of skip or merge, only the index indicating the predictor is included. However, in case of AMVP, the reference picture index, differential motion vector, and AMVP index of the current block are included.

Hereinafter, an embodiment of the operation of the intra prediction unit 169 will be described in detail.

First, prediction mode information and a size of a prediction block are received by the picture divider 160 , and the prediction mode information indicates an intra mode. The size of the prediction block may be a square such as 64x64, 32x32, 16x16, 8x8, 4x4, but is not limited thereto. That is, the size of the prediction block may be non-square rather than square.

Next, the reference pixel is read from the picture storage unit 172 to determine the intra prediction mode of the prediction block.

It is determined whether the reference pixel is created by examining whether the unavailable reference pixel exists. The reference pixels are used to determine the intra prediction mode of the current block.

When the current block is located at the upper boundary of the current picture, pixels adjacent to the upper side of the current block are not defined. Also, when the current block is located at the left boundary of the current picture, pixels adjacent to the left of the current block are not defined.

It is determined that these pixels are not available pixels. Also, when the current block is located at the slice boundary and pixels adjacent to the upper or left side of the slice are not pixels that are first encoded and reconstructed, it is determined that they are not available pixels.

As described above, when pixels adjacent to the left or upper side of the current block do not exist or there are no pre-encoded and reconstructed pixels, the intra prediction mode of the current block may be determined using only available pixels.

However, reference pixels at positions that are not available may be generated using the available reference pixels of the current block. For example, when the pixels of the upper block are not available, the upper pixels may be generated using some or all of the pixels on the left, and vice versa.

That is, it is possible to generate a reference pixel by copying an available reference pixel at a position closest to it in a predetermined direction from a reference pixel at an unavailable position. When there is no reference pixel available in the predetermined direction, the reference pixel may be generated as the reference pixel by copying the reference pixel available in the nearest position in the opposite direction.

Meanwhile, even when pixels above or to the left of the current block exist, they may be determined as unavailable reference pixels according to the encoding mode of the block to which the pixels belong.

For example, when a block to which a reference pixel adjacent to the upper side of the current block belongs is an inter-encoded and reconstructed block, the pixels may be determined as unavailable pixels.

In this case, usable reference pixels may be generated using pixels belonging to a block in which a block adjacent to the current block is intra-encoded and reconstructed. In this case, information indicating that the encoder determines available reference pixels according to the encoding mode must be transmitted to the decoder.

Next, an intra prediction mode of the current block is determined using the reference pixels. The number of intra prediction modes permissible for the current block may vary according to the size of the block. For example, when the size of the current block is 8x8, 16x16, or 32x32, 34 intra prediction modes may exist, and when the size of the current block is 4x4, 17 intra prediction modes may exist.

The 34 or 17 intra prediction modes may include at least one non-directional mode and a plurality of directional modes.

The one or more non-directional modes may be DC mode and/or planar mode. When the DC mode and the planar mode are included as the non-directional mode, 35 intra prediction modes may exist regardless of the size of the current block.

In this case, two non-directional modes (DC mode and planar mode) and 33 directional modes may be included.

In the planner mode, a prediction block of the current block is generated using at least one pixel value (or a predicted value of the pixel value, hereinafter referred to as a first reference value) and reference pixels located at the bottom-right side of the current block. .

As described above, the configuration of the video decoding apparatus according to an embodiment of the present invention may be derived from the configuration of the video encoding apparatus described with reference to FIGS. 1, 2 and 25, for example, referring to FIGS. 2 and 25 . An image can be decoded by performing a reverse process of the encoding process as described with reference to.

26 is a block diagram showing the configuration of a video decoding apparatus according to an embodiment of the present invention.

Referring to FIG. 26 , the video decoding apparatus according to the present invention includes an entropy decoding unit 210 , an inverse quantization/inverse transformation unit 220 , an adder 270 , a deblocking filter 250 , a picture storage unit 260 , It includes an intra prediction unit 230 , a motion compensation prediction unit 240 , and an intra/inter switching switch 280 .

The entropy decoding unit 210 decodes the encoded bit stream transmitted from the video encoding apparatus, and separates the encoded bit stream into an intra prediction mode index, motion information, a quantization coefficient sequence, and the like. The entropy decoding unit 210 supplies the decoded motion information to the motion compensation prediction unit 240 .

* The entropy decoding unit 210 supplies the intra prediction mode index to the intra prediction unit 230 and the inverse quantization/inverse transform unit 220 . Also, the entropy decoding unit 210 supplies the inverse quantization coefficient sequence to the inverse quantization/inverse transformation unit 220 .

The inverse quantization/inverse transform unit 220 transforms the quantization coefficient sequence into inverse quantization coefficients of a two-dimensional array. One of a plurality of scanning patterns is selected for the transformation. One of a plurality of scanning patterns is selected based on at least one of a prediction mode (ie, any one of intra prediction and inter prediction) of the current block and an intra prediction mode.

The intra prediction mode is received from an intra prediction unit or an entropy decoder.

The inverse quantization/inverse transform unit 220 restores the quantization coefficients by using a quantization matrix selected from among a plurality of quantization matrices for the inverse quantization coefficients of the two-dimensional array. Different quantization matrices are applied according to the size of the current block to be reconstructed, and a quantization matrix is selected based on at least one of a prediction mode and an intra prediction mode of the current block even for a block of the same size.

Then, the residual block is reconstructed by inverse transforming the reconstructed quantization coefficient.

The adder 270 reconstructs an image block by adding the residual block reconstructed by the inverse quantization/inverse transform unit 220 and the prediction block generated by the intra prediction unit 230 or the motion compensation prediction unit 240 .

The deblocking filter 250 performs a deblocking filter process on the reconstructed image generated by the adder 270 . Accordingly, it is possible to reduce the deblocking artifact caused by the image loss caused by the quantization process.

The picture storage unit 260 is a frame memory that maintains a local decoded image on which the deblocking filter process has been performed by the deblocking filter 250 .

The intra prediction unit 230 reconstructs the intra prediction mode of the current block based on the intra prediction mode index received from the entropy decoding unit 210 . Then, a prediction block is generated according to the reconstructed intra prediction mode.

The motion compensation prediction unit 240 generates a prediction block for the current block from the picture stored in the picture storage unit 260 based on the motion vector information. When motion compensation of fractional precision is applied, a prediction block is generated by applying the selected interpolation filter.

The intra/inter switching switch 280 provides the prediction block generated by any one of the intra prediction unit 230 and the motion compensation prediction unit 240 to the adder 270 based on the encoding mode.

The current block is reconstructed using the prediction block of the current block reconstructed in this way and the residual block of the decoded current block.

A video bitstream according to an embodiment of the present invention is a unit used to store encoded data in one picture, and may include parameter sets (PS) and slice data.

PS (parameter sets) is divided into a picture parameter set (hereinafter simply referred to as PPS) and a sequence parameter set (hereinafter simply referred to as SPS), which are data corresponding to the head of each picture. The PPS and SPS may include initialization information required to initialize each encoding, and may include spatial structure information (SPATIAL LAYOUT INFORMATION) according to an embodiment of the present invention.

The SPS is common reference information for decoding all pictures encoded by the random access unit (RAU), and may include a profile, the maximum number of pictures usable for reference, and a picture size.

The PPS may include a type of a variable length coding method, an initial value of a quantization step, and a plurality of reference pictures as reference information for decoding a picture for each picture encoded by a random access unit (RAU).

Meanwhile, the slice header SH includes information on a corresponding slice when coding in a slice unit.

The above-described method according to the present invention may be produced as a program to be executed on a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape. , floppy disks, and optical data storage devices.

The computer-readable recording medium is distributed in a network-connected computer system, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the art to which the present invention pertains.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims Various modifications may be made by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims (4)

disposing a plurality of images constituting an omnidirectional image of a three-dimensional space on a two-dimensional image;
transforming the arrangement of at least some of the plurality of regions included in the two-dimensional image;
encoding the converted two-dimensional image; and
Including; transmitting the encoded two-dimensional image and the SEI message;
The SEI message includes placement information including an arrangement method of an omnidirectional image with respect to the two-dimensional image, information on the transformation, and viewpoint information according to the intention of the content provider,
The arrangement information is at least one of a Cube Map method of arranging the sub-images of the omnidirectional image in the two-dimensional image according to the Cube Map format, and an Equirectangular method of arranging the sub-images of the omnidirectional image in the two-dimensional image according to the Equirectangular format. contains an SEI message syntax indicating one,
In the SEI message, the 2D image region is mapped to a 3D space when the bitstream of the 2D image in which the omnidirectional image is pre-encoded is performed according to at least a part of the SEI message and user viewpoint information. It contains information that allows it to be reproduced and reproduced;
Time information according to the intention of the content provider,
A user viewing the omnidirectional image restored from the encoded two-dimensional image, including viewpoint information of the omnidirectional image that allows the user to view the omnidirectional image according to the phase according to the intention of a content provider or image producer characterized by
Image processing method. an image acquisition unit that arranges a plurality of images constituting an omnidirectional image of a three-dimensional space on a two-dimensional image, and transforms the arrangement of at least a portion of a plurality of regions included in the two-dimensional image;
an image encoder for encoding the converted two-dimensional image; and
Including; a transmission processing unit for transmitting the encoded two-dimensional image and the SEI message,
The SEI message includes placement information including an arrangement method of an omnidirectional image with respect to the two-dimensional image, information on the transformation, and viewpoint information according to the intention of the content provider,
The arrangement information includes a Cube Map method of disposing the sub-images of the omnidirectional image in the two-dimensional image according to the Cube Map format, and Equirectangular methods of disposing the sub-images of the omnidirectional image in the two-dimensional image according to the Equirectangular format. an SEI message syntax indicating at least one;
In the SEI message, the 2D image region is mapped to a 3D space when the bitstream of the 2D image in which the omnidirectional image is pre-encoded is performed according to at least a part of the SEI message and user viewpoint information. It contains information that allows it to be reproduced and reproduced;
Time information according to the intention of the content provider,
A user viewing the omnidirectional image restored from the encoded two-dimensional image, including viewpoint information of the omnidirectional image that allows the user to view the omnidirectional image according to the phase according to the intention of a content provider or image producer characterized by
image processing device. obtaining an encoded two-dimensional image and an SEI message;
decoding the encoded two-dimensional image; and
Reproducing a 3D spatial image according to a user's point of view from the decoded 2D image;
The SEI message includes placement information including an arrangement method of an omnidirectional image with respect to the two-dimensional image, conversion information, and viewpoint information according to the intention of the content provider,
The arrangement information is at least one of a Cube Map method of arranging the sub-images of the omnidirectional image in the two-dimensional image according to the Cube Map format, and an Equirectangular method of arranging the sub-images of the omnidirectional image in the two-dimensional image according to the Equirectangular format. contains an SEI message syntax indicating one,
The transformation information is information indicating that the arrangement of at least some of the plurality of regions included in the two-dimensional image is transformed,
The decryption step is
According to at least a part of the SEI message and user viewpoint information, when decoding of the bitstream of the pre-encoded 2D image of the omnidirectional image is performed, the 2D image region is mapped to a 3D space and reproduced includes
Time information according to the intention of the content provider,
The user viewing the omnidirectional image restored from the encoded two-dimensional image, including the viewpoint information of the omnidirectional image that allows the user to view the omnidirectional image according to the phase according to the intention of the content provider or image producer
Image processing method. A decoding unit that obtains an encoded two-dimensional image and an SEI message, decodes the encoded two-dimensional image, and processes coordinate mapping for reproducing a three-dimensional spatial image according to a user's point of view from the decoded two-dimensional image, ,
The SEI message includes placement information including an arrangement method of an omnidirectional image with respect to the two-dimensional image, conversion information, and viewpoint information according to the intention of the content provider,
The arrangement information is at least one of a Cube Map method of arranging the sub-images of the omnidirectional image in the two-dimensional image according to the Cube Map format, and an Equirectangular method of arranging the sub-images of the omnidirectional image in the two-dimensional image according to the Equirectangular format. contains an SEI message syntax indicating one,
The transformation information is information indicating that the arrangement of at least some of the plurality of regions included in the two-dimensional image is transformed,
The decryption unit,
According to at least a part of the SEI message and user viewpoint information, when decoding of the bitstream of the 2D image in which the omnidirectional image is pre-encoded is performed, the 2D image region is mapped to a 3D space and reproduced,
Time information according to the intention of the content provider,
A user viewing the omnidirectional image restored from the encoded two-dimensional image, including viewpoint information of the omnidirectional image that allows the user to view the omnidirectional image according to the phase according to the intention of a content provider or image producer characterized by
image processing device.

Images