KR20190101930A - A method for seletively decoding a syncronized multi view video by using spatial layout information - Google Patents

Abstract

According to an embodiment of the present invention, an image decoding method comprises the following steps: receiving a bit stream including an encoded image; obtaining spatial structure information corresponding to a synchronized multi-view video; and selectively decoding at least a part of the bit stream based on the spatial structure information.

Description

Selective decoding method, encoding method, and apparatus for synchronized multi-view video using spatial structure information TECHNICAL FIELD

The present invention relates to a method and apparatus for encoding / decoding an image. More specifically, the present invention relates to a selective decoding method, an encoding method, and an apparatus of a synchronized multiview image using spatial structure information.

Recently, with the development of digital image processing and computer graphics technology, researches on virtual reality (VR) technology for reproducing the real world and experiencing it realistically are being actively conducted.

In particular, modern VR systems, such as HMD (Head Mounted Display), can not only provide three-dimensional stereoscopic images in both eyes of the user, but can also track the view in all directions. It is attracting much attention in that it can provide reality (VR) video content.

However, since 360 VR content is composed of simultaneous omnidirectional multi-view image information in which spatial and binocular images are spatially complex-synchronized, two large images synchronized to the binocular space at all points in time are produced and transmitted. Encode, compress, and pass. This adds complexity and bandwidth burden, and in particular, the decoding apparatus wastes unnecessary processes because decoding is performed on an area not actually viewed outside the user's viewpoint.

Accordingly, there is a need for an efficient encoding method that reduces the amount and complexity of the transmission data of an image and is also effective in terms of bandwidth and battery consumption of the decoding apparatus.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problem, and a method and apparatus for efficiently encoding / decoding a synchronized multiview image such as a 360 degree camera or a VR image using spatial structure information of the synchronized multiview image. The purpose is to provide.

As a technical means for achieving the above technical problem, an image decoding method according to an embodiment of the present invention, receiving a bitstream including an encoded image; Obtaining spatial structure information corresponding to the synchronized multi-view image; And selectively decoding at least a portion of the bitstream based on the spatial structure information.

In addition, as a technical means for achieving the above technical problem, the image decoding apparatus according to an embodiment of the present invention, obtains the spatial structure information corresponding to the synchronized multi-view image from the bitstream including the encoded image, And a decoding processing unit for selectively decoding at least a part of the bitstream based on the spatial structure information.

In addition, as a technical means for achieving the above technical problem, an image encoding method according to an embodiment of the present invention, the step of obtaining a synchronized multi-view image; Generating spatial structure information of the synchronized multi-view image; Encoding the synchronized multi-view image; And transmitting a bitstream including the encoded multiview image and the spatial structure information to a decoding system, wherein the decoding system selectively decodes at least a portion of the bitstream based on the spatial structure information. Characterized in that.

On the other hand, the video processing method may be implemented as a computer-readable recording medium recording a program for execution in a computer.

According to an embodiment of the present invention, by extracting and signaling spatial structure information optimized for encoding and transmission from a synchronized multiview image, it is possible to efficiently reduce the amount of transmission data, bandwidth and complexity of the image.

In addition, when the synchronized multi-view image is received, the decoding end may perform some selective decoding for each view according to the signaling information, thereby reducing system waste, and thus, may be efficiently encoded / complicated in terms of complexity and battery consumption. A decoding method and apparatus can be provided.

In addition, according to an embodiment of the present invention, it is possible to support spatial structure information on the synchronized video in various manners, thereby enabling proper video reproduction according to the decoding device specification, thereby improving device compatibility.

Figure 1 shows the overall system structure according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a time-synchronized multiview image encoding apparatus according to an embodiment of the present invention.
3 to 6 are diagrams illustrating an example of a spatial structure of a synchronized multiview image according to an exemplary embodiment of the present invention.
7 to 9 are diagrams for describing a signaling method of spatial structure information according to various embodiments of the present disclosure.
10 is a view for explaining the configuration of spatial structure information according to an embodiment of the present invention.
11 to 12 are diagrams for describing a type index table of spatial structure information according to an embodiment of the present invention.
FIG. 13 is a diagram for describing a viewpoint information table of spatial structure information according to an embodiment of the present invention.
14 is a flowchart illustrating a decoding method according to an embodiment of the present invention.
15 and 18 are diagrams illustrating that a scanning order of a decoding end is determined according to signaling of spatial structure information according to an embodiment of the present invention.
FIG. 17 illustrates an independent sub image and a dependent sub image classified according to signaling of spatial structure information.
18 to 19 illustrate that boundary regions between sub-images are decoded with reference to independent sub-images according to spatial structure information.
20 to 24 are diagrams illustrating a decoding system and its operation according to an embodiment of the present invention.
25 to 26 are diagrams for describing an encoding and decoding process, according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present disclosure. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted for simplicity of explanation, and like reference numerals designate like parts throughout the specification.

Throughout this specification, when a portion is "connected" to another portion, this includes not only "directly connected" but also "electrically connected" with another element in between. do.

Throughout this specification, when a member is located on another member, this includes not only when a member is in contact with another member but also when another member exists between two members.

Throughout this specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding the other components unless specifically stated otherwise. As used throughout this specification, the terms "about", "substantially" and the like are used at, or in the sense of, numerical values when a manufacturing and material tolerance inherent in the stated meanings is indicated, Accurate or absolute figures are used to assist in the prevention of unfair use by unscrupulous infringers. As used throughout this specification, the term "step to" or "step of" does not mean "step for."

Throughout this specification, the term combinations thereof included in the representation of the Markush form refers to one or more mixtures or combinations selected from the group consisting of the components described in the representation of the Markush form, wherein It means to include one or more selected from the group consisting of.

In an embodiment of the present invention, as an example of a method of encoding a synchronized image, standardization is jointly performed by a Moving Picture Experts Group (MPEG) and a Video Coding Experts Group (VCEG) having the highest coding efficiency among video coding standards developed to date. Encoding may be performed using one High Efficiency Video Coding (HEVC), but is not limited thereto.

In general, the encoding apparatus includes an encoding process and a decoding process, and the decoding apparatus includes a decoding process. The decoding process of the decoding apparatus is the same as the decoding process of the encoding apparatus. Therefore, the following description focuses on the encoding apparatus.

Figure 1 shows the overall system structure according to an embodiment of the present invention.

Referring to FIG. 1, the entire system according to an embodiment of the present invention includes a preprocessing apparatus 10, an encoding apparatus 100, a decoding apparatus 200, and a post-processing apparatus 20.

The system according to an embodiment of the present invention, the pre-processing device 10 for obtaining a synchronized video frame by pre-processing a plurality of viewpoint-specific images through a merge or stitch operation, and the synchronized video frame The encoding apparatus 100 that encodes and outputs a bitstream, the decoding apparatus 200 that receives the bitstream, and decodes the synchronized video frame, and the synchronized image for each view through post-processing of the video frame, respectively. It may be configured to include a post-processing device 20 to be output to the display of.

Here, the input image may include individual images for each multiview, and may include, for example, sub-image information of various viewpoints in which one or more cameras are photographed in a time and space synchronized state. Accordingly, the preprocessing apparatus 10 may obtain synchronized image information by spatially merging or stitching the acquired multi-view sub image information according to time.

The encoding apparatus 100 may generate a bitstream by scanning and predictively encoding the synchronized image information, and the generated bitstream may be transmitted to the decoding apparatus 200. In particular, the encoding apparatus 100 according to an embodiment of the present invention may extract spatial structure information from the synchronized image information, and may signal the decoding apparatus 200.

Here, the spatial layout information may include basic information on the properties and arrangement of each sub-image, as one or more sub-images are merged from the preprocessing apparatus 10 to form a single video frame. . In addition, each of the sub-images and the additional information on the relationship between the sub-images may further include, which will be described later.

Accordingly, spatial structure information according to an embodiment of the present invention can be delivered to the decoding apparatus 200. The decoding apparatus 200 may determine the decoding target and the decoding order of the bitstream with reference to the spatial structure information and the user viewpoint information, which may induce efficient decoding.

The decoded video frame is further divided into sub-images for each display through the post-processing device 20 and provided to a plurality of synchronized display systems such as HMDs. Multiview images can be provided.

2 is a block diagram illustrating a configuration of a time-synchronized multiview image encoding apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the encoding apparatus 100 according to an exemplary embodiment of the present invention may include a synchronized multiview image acquisition unit 110, a spatial structure information generation unit 120, a spatial structure information signaling unit 130, and image encoding. And a transmission processor 150.

The synchronized multiview image acquisition unit 110 acquires a synchronized multiview image using a synchronized multiview image acquisition means such as a 360 degree camera. The synchronized multi-view image may include a plurality of time and space synchronized sub-images, and may be received from the preprocessing device 10 or from a separate external input device.

The spatial structure information generator 120 divides the synchronized multi-view image into video frames in units of time, and extracts spatial structure information on the video frames. The spatial structure information may be determined according to the properties and the arrangement of the respective sub images, or may be determined according to the information obtained from the preprocessing device 10.

In addition, the spatial structure information signaling unit 130 performs information processing for signaling the spatial structure information to the decoding apparatus 200. For example, the spatial structure information signaling unit 130 may perform one or more processes for including in the image data encoded by the image encoder, configuring a separate data format, or including the metadata in the encoded image. Can be.

The image encoder encodes the synchronized multi-view image over time. In addition, the image encoder may determine the image scanning order, the reference image, etc., using the spatial structure information generated by the spatial structure information generator 120 as reference information.

Accordingly, the image encoder may perform encoding using HEVC (High Efficiency Video Coding) as described above, but may be improved in a more efficient manner with respect to the synchronized multi-view image according to the spatial structure information.

The transmission processor 150 may perform one or more transform and transmission processes for combining the encoded image data and the spatial structure information inserted from the spatial structure information signaling unit 130 and transmitting the combined structure data to the decoding apparatus 200. Can be.

3 to 6 are diagrams illustrating an example of a spatial structure and an image configuration of a synchronized multiview image according to an exemplary embodiment of the present invention.

Referring to FIG. 3, a multi-view image according to an embodiment of the present invention may include a plurality of image frames that are temporally synchronized and spatially synchronized.

Each frame may be synchronized according to a unique spatial layout, and may configure a layout of sub-images corresponding to one or more scenes, perspectives, or views to be displayed at the same time.

Accordingly, the spatial layout information is composed of one input image through merging, stitching, etc. of each sub-image constituting the synchronized multi-view image, or a simultaneous multi-view image (for example, , When a plurality of images synchronized at the same time and corresponding to various views corresponding to the same POC are configured as input images, arrangement information of the multi-view image or sub-images, position information and angle information of a capture camera, Sub-images and their relationship information may be included, such as merge information, information on the number of sub-images, scanning order information, acquisition time information, camera parameter information, and reference dependency information between the sub-images.

For example, as shown in FIG. 4, image information may be captured through a divergent camera arrangement, and a stitched image may be configured to form a spatial image that may be observed 360 degrees. can do.

As shown in Fig. 4, images A ', B', C ', ... photographed corresponding to each camera array A, B, C ... can be arranged according to a one-dimensional or two-dimensional spatial structure. In addition, left and right and top and bottom region relationship information for stitch processing between arranged images may be illustrated as spatial structure information.

Accordingly, the spatial structure information generator 120 may extract spatial structure information including the various attributes as described above from the input image, and the spatial structure information signaling unit 130 may be optimized to describe the spatial structure information later. Can be signaled in a manner.

The spatial structure information generated and signaled as described above may be used as useful reference information as described above.

For example, if the content photographed by each camera is a pre-stitched image, the pre-stiched images overlap each other before encoding to form a scene. On the other hand, the scene may be separated according to each view, and mutual compensation may be performed between each separated image according to a type.

Accordingly, in the case of the pre-stitched image which merges and stitches one or more images taken from a multi-view point into one image in the preprocessing process and delivers them to the input of the encoder, scene information and spatial layout configuration information of the merged and stitched input images Etc. may be delivered to the encoding step and the decoding step through separate spatial structure information signaling.

In addition, even in the case of a non-stitched image image type in which images acquired in a multiview are transmitted to one or more input images at a time-synchronized time point and encoded and decoded, reference and compensation are performed according to the spatial structure information in the encoding and decoding step. To this end, various spatial layout information and corresponding data fields may be required. The data field may be encoded together with the compressed information of the input image, or may be transmitted in a separate metadata.

In addition, the data field including the spatial layout information may be utilized in the post-processing apparatus 20 of the image and the rendering process of the display.

To this end, the data field including the spatial layout information may include position coordinate information and color difference information acquired at the time of image acquisition from each camera.

For example, three-dimensional coordinate information and color difference information (X, Y, Z), (R, G, B), etc. of an image acquired at the time of acquisition of image information from each camera are added to each sub-image. The information may be acquired and transmitted, and the information may be used in post-processing and rendering of an image after decoding.

In addition, the data field including the spatial layout information may include camera information of each camera.

As illustrated in FIGS. 5 to 6, one or more cameras that photograph a 3D space and provide a spatial image may be disposed.

For example, as shown in FIG. 5, the position of one or more cameras may be fixed to a central position and each direction may be set in the form of acquiring surrounding objects at a point in a three-dimensional space when an image is acquired. .

In addition, as illustrated in FIG. 6, one or more cameras may be arranged to photograph one object at various angles. At this time, the user's motion information (Up / Down, Left / Right, Zoom in / Zoom Out) in the VR display device to play the 3D image based on the coordinate information (X, Y, Z) and distance information at the time of image acquisition. And the like, and decode or post-process a portion of the image corresponding thereto to restore the viewpoint or partial image desired by the user. Meanwhile, as described above, the synchronized multi-view image exemplified as the VR image can be restored. In a system such as compression, transmission, and playback, a separate image conversion tool module or the like may be added to a required portion depending on the type or characteristic of an image, characteristics of a decoding apparatus, and the like.

For example, when the image acquired from the camera is of the Equirectangular type, the image encoder 140 converts the image type into an image type such as Icosahedron / Cube Map through a conversion tool module according to the compression performance and the encoding efficiency of the image. Through this, encoding can be performed. In this case, the conversion tool module may be used in the preprocessing device 10 and the post-processing device 20, and the conversion information according to the conversion is included in the spatial structure information, etc., so that the decoding device 200 or the post-processing device is in a metadata format. 20 or the VR display device.

Meanwhile, in order to deliver a synchronized multi-view image according to an embodiment of the present invention, a separate VR image compression method may be required to support scalability between the encoding apparatus 100 and the decoding apparatus 200.

Accordingly, the encoding apparatus 100 may compress-code the image in a manner of dividing the base layer and the enhancement layer in order to scalablely compress the VR image.

In this way, a single input image compresses high-resolution VR images acquired through various cameras, the base layer compresses the original image, and the enhancement layer divides one picture into areas such as Slice / Tile. Encoding may be performed for each sub-image.

In this case, the encoding apparatus 100 may process compression encoding by using an inter layer prediction technique to increase encoding efficiency by using the reconstructed image of the base layer as a reference image.

Meanwhile, when the decoding apparatus 200 needs to quickly decode a specific image according to a user's movement while decoding the base layer, the decoding apparatus 200 may decode a partial region of the enhancement layer and quickly decode some images according to the user's movement. .

In this scalable compression scheme, the encoding apparatus 100 encodes a base layer, but performs scale down or down sampling on the original image at an arbitrary ratio in the base layer. Can be compressed. In this case, the enhancement layer adjusts the size of the image at the same resolution through scale up or up sampling of the reconstructed image of the base layer, and uses the reconstructed image of the corresponding base layer as a reference picture. By doing so, encoding / decoding can be performed.

According to a processing structure supporting such scalability, the decoding apparatus 200 decodes the entire bitstream of the base layer compressed with a low bit or low resolution, and improves only a part of the images of the entire bitstream according to the user's movement. Can be decoded into layers In addition, since not all decoding is performed on the entire image, the VR image can be reconstructed with only low complexity.

In addition, according to an image compression scheme that supports separate scalability with different resolutions, the encoding apparatus 100 may perform compression on an original image or an image according to an intention of an image producer in a base layer. The encoding may be performed based on an inter-layer prediction method that performs encoding by referring to the reconstructed image of the base layer in the layer.

In this case, the input image of the enhancement layer may be an image obtained by dividing one input image through an image segmentation method and encoding the same into a plurality of regions. One divided area may include at most one sub image, and a plurality of divided areas may be configured as one sub image. The compressed bitstream encoded through this division method may process two or more outputs at the service and application stages. For example, the service restores and outputs the entire image by decoding the base layer, and the enhancement layer reflects only the partial region and some sub-images by reflecting the user's movement, viewpoint change, and manipulation through the service or application. Can be decrypted

7 to 9 are diagrams for describing a signaling method of spatial structure information according to various embodiments of the present disclosure.

As shown in FIGS. 7 to 9, the spatial structure information is NAL (NETWORK ABSTRACTION LAYER) on HLS such as SPS (SEQUENCE PARAMETER SET) or VPS (VIDEO PARAMETER SET) defined as coding parameters in general video encoding. It can be signaled as one class type of UNIT format.

First, FIG. 7 illustrates a NAL UNIT type into which a synchronized video encoding flag is inserted according to an embodiment of the present invention. For example, a synchronized video encoding according to an embodiment of the present invention may be performed on a video parameter set (VPS). Flags may be inserted.

Accordingly, FIG. 8 illustrates an embodiment of inserting a spatial structure information flag into a video parameter set (VPS) according to an embodiment of the present invention.

As shown in FIG. 8, the spatial structure information signaling unit 130 according to an embodiment of the present invention may insert a flag for checking the type of a separate input image on the VPS. The encoding apparatus 100 may insert a flag indicating that synchronized multi-view image encoding such as VR content is performed using the spatial structure information signaling unit 130 and the spatial structure information is signaled using vps_other_type_coding_flag.

In addition, as shown in FIG. 9, the spatial structure information signaling unit 130 according to the embodiment of the present invention may signal that the image is a multi-view synchronized video image on a SPS (SEQUENCE PARAMETER SET).

For example, as illustrated in FIG. 9, the spatial structure information signaling unit 130 may insert the type (INPUT_IMAGE_TYPE) of the input image so that index information of the synchronized multiview image may be included in the SPS and transmitted.

Here, if the INPUT_IMAGE_TYPE_INDEX on the SPS is not -1, or the INDEX value is -1, or if the value is 0 and semantically corresponding to -1, the INPUT_IMAGE_TYPE is synchronized again according to an embodiment of the present invention. This may indicate a point image.

In addition, when the type of the input image is a synchronized multi-view image, the spatial structure information signaling unit 130 includes a signal of PERSPECTIVE INFORMATION in the SPS to signal a part of the spatial structure information of the synchronized multi-view image. Can also be inserted into the SPS for transmission. The viewpoint information is information in which image layout of each time zone is signaled according to a 3D rendering processing process of a 2D image, and may include order information such as top, bottom, and side surfaces.

Accordingly, the decoding apparatus 200 may decode the flag of the VPS or the SPS to identify whether the corresponding image has been encoded using the spatial structure information according to the embodiment of the present invention. For example, in the case of the VPS of FIG. 5, VPS_OTHER_TYPE_CODING_FLAG may be extracted to determine whether the corresponding image is a synchronized multiview image encoded using spatial structure information.

In the case of the SPS of FIG. 9, the actual spatial structure information such as the layout can be identified by decoding the PERSPECTIVE_INFORMATION_INDEX information.

In this case, the spatial structure information may be configured in the form of a parameter. For example, the spatial structure parameter information may be differently included on an HLS such as an SPS or a VPS, or a syntax may be configured in the form of a separate function. Can be defined as an SEI message.

In addition, according to an embodiment, the spatial structure information may be included in the PPS (PICTURE PARAMETER SET) and transmitted. In this case, attribute information for each sub image may be included. For example, independence of the sub-images may be signaled. Independence may indicate that a corresponding picture may be encoded and decoded without referring to another picture, and the sub-images of the synchronized multi-view image may include an independent sub-image and a dependent sub-image. The dependent subimage may be decoded with reference to the independent subimage. The spatial structure information signaling unit 130 may signal an independent sub image in the form of an independent sub image list on the PPS.

In addition, the spatial structure information may be defined and signaled as an SEI message. 10 illustrates an SEI message as spatial structure information, and parameterized spatial structure information may be inserted using a spatial layout information descriptor.

As shown in FIG. 10, the spatial structure information includes INPUT IMAGE TYPE INDEX, PERSPECTIVE INFORMATION, CAMERA PARAMETER, which may indicate a spatial layout of an input image. Scene angle information (SCEN ANGLE), scene dynamic range information (SCENE DYNAMIC RANGE), independent sub-image information (INDEPENDENT SUB IMAGE), and scene time information (SCENE TIME INFORMATION) may include at least one other Various pieces of information necessary to efficiently encode the image can be added. Such parameters may be defined in the form of an SEI message in the form of a descriptor, and the decoding apparatus 200 may parse the data and efficiently utilize the spatial structure information in the decoding, post-processing and rendering steps.

As described above, the spatial structure information may be transmitted to the decoding apparatus 200 in the form of SEI or metadata.

Also, for example, the spatial structure information may be signaled by a selection option such as configuration in the encoding step.

As a first option, the spatial structure information may be included in the VPS / SPS / PPS or Coding unit syntax on the HLS according to the coding efficiency on the syntax.

As a second option, the spatial structure information may be signaled at once in metadata in SEI form on the syntax.

Hereinafter, an efficient video encoding and decoding method according to a synchronized multiview image format according to an embodiment of the present invention will be described in detail with reference to FIGS. 11 to 19.

As described above, a plurality of viewpoint-specific images generated in the preprocessing step may be synthesized and encoded into one input image. In this case, one input image may include a plurality of sub images. Each of the sub-images may be synchronized at the same time point, and may correspond to different views, visual views, or scenes, respectively. This has the effect of supporting various VIEWs on the same POC (PICTURE ORDER COUNT) without using separate depth information, and overlapping area between each sub-image is limited to a boundary area.

In particular, the spatial structure information of the input image may be signaled in the form described above, and the encoding apparatus 100 and the decoding apparatus 200 may parse and use the spatial structure information to perform efficient encoding and decoding. That is, the encoding apparatus 100 may process multi-view image encoding using the spatial structure information in an encoding step, and the decoding apparatus 200 may process decoding using the spatial structure information in a decoding, preprocessing, and rendering step. Can be.

11 to 12 are diagrams for describing a type index table of spatial structure information according to an embodiment of the present invention.

As described above, the sub-images of the input image may be arranged in various ways. Accordingly, the spatial structure information may separately include a table index for signaling placement information. For example, as shown in FIG. 8, the synchronized multiview image may include layouts of CUBIC LAYOUT 4 X 3, CUBIC LAYOUT 3 X 2, ICOSAHEDRON, EQUIRECTANGULAR, and the like, and the spatial structure information corresponds to each layout. The table index shown in FIG. 9 may be inserted.

However, the table illustrated in FIG. 12 is arbitrarily disposed according to the input image, and may be changed according to coding efficiency and market content distribution.

Accordingly, the decoding apparatus 200 may parse the table index signaled separately and use it for the decoding process.

In particular, in the embodiment of the present invention, each of the layout information may be usefully used for decoding a part of an image. That is, the sub image arrangement information such as CUBIC LAYOUT may be used to distinguish the independent sub image from the dependent sub image, and thus may be used to determine an efficient encoding and decoding scanning order or to perform some decoding for a specific viewpoint.

13 is a flowchart illustrating a decoding method according to an embodiment of the present invention.

Referring to FIG. 13, first, the decoding apparatus 200 receives an image bitstream (S101).

In operation S103, the decoding apparatus 200 determines whether the image is a synchronized multi-view image.

Here, the decoding apparatus 200 may identify whether the multiview image is synchronized from a flag signaled from the spatial structure information signaling unit 130 from the image bitstream. For example, the decoding apparatus 200 may identify in advance whether the image is a synchronized multiview image from the VPS, the SPS, or the like as described above.

If the multiview video is not synchronized, general full image decoding is performed (S113).

In the case of the synchronized multiview image, the decoding apparatus 200 decodes the table index from the spatial structure information (S105).

Here, the decoding apparatus 200 may identify whether the image is an EQUIRECTANGULAR image from the table index (S107).

This is because the EQUIRECTANGULAR image among the synchronized multiview images may not be divided into separate sub-images, and the decoding apparatus 200 performs decoding of the entire image on the EQUIRECTANGULAR image (S113).

If it is not an EQUIRECTANGULAR image, the decoding apparatus 200 decodes the remaining total spatial structure information (S109), and performs image decoding processing based on the spatial structure information (S111).

14 is a view for explaining a viewpoint information table of spatial structure information according to an embodiment of the present invention.

The spatial structure information according to an embodiment of the present invention may include a table for PERSPECTIVE INFORMATION.

The encoding apparatus 100 and the decoding apparatus 200 may use the viewpoint information table as information for distinguishing the reference, encoding, decoding order, and independent sub-image between images.

In addition, when rendering in a multi-view display device, the viewpoint information table may be delivered to the system layer of the device together with the decoded image, and by using the corresponding information, the user may adjust the phase according to the intention of the contents provider. You can watch the video.

More specifically, the spatial structure information signaling unit 130 may signal a PERSPECTIVE of each sub image. In particular, the spatial structure information signaling unit 130 signals only information on the top, bottom, and front images according to the type of the image, and information of the image on the remaining sides is displayed in the top scene and the front perspective scene in the decoding apparatus 200. It can be derived by using Bottom Scene information. Thus, only minimal information can be signaled.

15 and 16 are diagrams illustrating that a scanning order of a decoding end is determined according to signaling of spatial structure information according to an embodiment of the present invention.

As shown in FIG. 15 and FIG. 16, the scanning order of sub-images may be transmitted together according to the type index of the spatial structure information, and the effective decoding and rendering may be performed through the transmitted scanning order information.

15 and 16 illustrate that the original image is scanned in the order of A-> B-> C-> D-> E-> F.

For signaling, the spatial structure information signaling unit 130 may signal only order information on some sub-images such as a top view, a bottom view, and a front view of the scanning order. The decoding apparatus 200 may derive the entire order using the order information of the some sub-images.

Also, the scanning order may be changed as shown in FIG. 15 or 16 according to the type, parallelism, and reference structure of the transmitted image.

Assuming Top is A, Bottom is F, and Front is B, the scanning sequence of FIG. 15 may be A-> F-> B-> C-> D-> E, and the scanning sequence of FIG. 16 is A-> B. -> C-> D-> E-> F This may be useful when the scanning order is determined differently in consideration of coding efficiency.

FIG. 17 illustrates an independent sub image and a dependent sub image classified according to signaling of spatial structure information.

As illustrated in FIG. 17, each of the sub-images arranged in the spatial structure may be divided into a dependent sub image and an independent sub image in consideration of reference and parallelism. The independent subimage has a property of being decoded without referring to another subimage, and the dependent subimage may be reconstructed by referring to an adjacent independent subimage or an adjacent dependent subimage.

Therefore, the independent sub image may have a characteristic that should be encoded or decoded before the dependent sub image.

In an embodiment, the independent sub-image may be encoded or decoded by referring to an independent sub-image that is previously encoded or decoded on the time axis, and the dependent sub-image is encoded by referring to the independent sub-image that is the same or not the same on the time axis. Or may be decrypted. In addition, whether or not independent may be signaled as a separate index according to spatial structure information.

18 to 19 illustrate that boundary regions between sub-images are decoded with reference to independent sub-images according to spatial structure information.

As described above, the encoding apparatus 100 or the decoding apparatus 200 may process encoding and decoding in consideration of the scanning order and dependency information of the sub-image at the same time.

As shown in FIG. 18, when A and F are independent sub-images, the scanning order may be transmitted or derived as A-> F-> B-> C-> D-> E, in which case A and F must first be decoded relative to other dependent sub-images. When the remaining B, C, D, and E are decoded, adjacent boundary regions of each sub image may be decoded with reference to the independent sub image. Accordingly, the boundary region of the pre-decoded independent sub-image or the pre-decoded dependent sub-image may be referred to the decoding of the remaining sub-images.

In addition, the independent sub-image described above may be used not only for one image frame but also for performing intra / inter encoding and decoding in the boundary region of an adjacent picture.

However, as shown in FIG. 19, there may be a case where 1: 1 mapping may not be performed due to different resolutions. (For a general image, a ratio of width is wider than height.)

In this case, in order to refer to the adjacent surface, the scale according to the resolution is adjusted through image processing techniques such as rotation and up sampling for the target sub-image to refer to encoding or decoding of the boundary region. Can be.

For example, in encoding / decoding of the upper region boundary of C in FIG. 19, the side value of A may be referred to. To this end, the encoding apparatus 100 or the decoding apparatus 200 upsamples the side value (Height) of A at a ratio corresponding to the width of C to generate a reference block value according to the corresponding position, and encodes and decodes the same. Can be performed.

20 to 24 are diagrams illustrating a decoding system and its operation according to an embodiment of the present invention.

Referring to FIG. 20, the decoding system 300 according to an embodiment of the present invention receives the entire synchronized multiview image bitstream and spatial structure information received from the encoding apparatus 100 or the external server as described above. The client system may provide one or more decoded pictures to the user's virtual reality display apparatus 400.

To this end, the decryption system 300 includes a decryption processor 310, a user motion analysis unit 320, and an interface unit 330. However, although the decoding system 300 is described as a separate system in the present specification, all or constituting the above-described decoding apparatus 200 and the post-processing apparatus 20 for performing necessary decoding processing and post-processing or It may be configured by a combination of some modules, or may be configured by extending the decoding apparatus 200. Therefore, it is not limited to the name.

Accordingly, the decoding system 300 according to an embodiment of the present invention performs selective decoding on a part of the entire bitstream based on spatial structure information received from the encoding apparatus 100 and user viewpoint information according to user motion analysis. Can be done. In particular, according to the selective decoding described with reference to FIG. 20, the decoding system 300 uses spatial structure information to input input images having a plurality of viewpoints of the same time (POC, Picture of Count) based on a predetermined direction. It can be associated with a PERSPECTIVE point of view. Also, based on this, some decoding may be performed on regions of interest (ROI) pictures determined by a user's viewpoint.

To this end, the decoding system 300 may include an interface layer for receiving and analyzing user information. The decoding system 300 may selectively include a viewpoint supported by an image currently decoded and a viewpoint mapping, post-processing, and rendering of the VR display device 400. Can be done. More specifically, the interface layer may include one or more processing modules for post-processing and rendering, an interface unit 330, and a user motion analyzer 320.

The interface unit 330 may receive motion information from the VR display device 400 worn by the user.

The interface unit 330 may, for example, wire or wirelessly connect at least one of an environmental sensor, a proximity sensor, a motion sensor, a position sensor, a gyroscope sensor, an acceleration sensor, and a geomagnetic sensor of the user's VR display device 400. It may include one or more data communication modules for receiving.

The user motion analysis unit 320 analyzes user motion information received from the interface unit 330 to determine a user's viewpoint, and selection information for adaptively selecting a decoded picture group corresponding thereto. May be transferred to the decryption processor 310.

Accordingly, the decoding processor 310 may set an ROI mask for selecting a region of intrest (ROI) picture based on the selection information transmitted from the user motion analyzer 320, and corresponds to the set ROI mask. Only the picture area can be decoded. For example, the picture group may correspond to at least one of the plurality of sub-images or reference images in the above-described image frame.

For example, as shown in FIG. 20, when there are 1 to 8 sub-images of a specific POC decoded by the decoding processing unit 310, the decoding processing unit 310 corresponds to a user's visual viewpoint (PERSPECTIVE). By decoding only the sub image regions 6 and 7, it is possible to improve the processing speed and efficiency in real time.

21 is a diagram illustrating an entire coded bitstream and a group of graphics (GOP) according to an embodiment of the present invention, and FIG. 22 illustrates that a picture group to be decoded among all bitstreams is selectively changed according to a user's viewpoint. .

As described above, the decoding system 300 according to an embodiment of the present invention receives the entire encoded bitstream of the received synchronized multi-view image, and transmits the encoded bitstream to the sub-bitstream corresponding to the user's perspective view. Decoding can be performed.

In this case, the spatial structure information of the synchronized multiview image may be signaled in the form of SEI or HLS as described above. In particular, according to an embodiment, the decoding system 300 uses a reference structure using an independent sub image, a dependent sub image, and a spatial random access picture (SRAP) identified from the signaling information. Can be created and built In addition, some bitstreams may be selected and decoded according to a change in visual viewpoint using the reference structure.

To this end, the encoding apparatus 100 may select a picture as a spatial random access picture (SRAP) by a NAL type or a PPS, and the SRAP picture is used for intra prediction encoding of sub-images of other unselected pictures. Can be used as a reference picture.

Accordingly, the encoding apparatus 100 and the decoding apparatus 200 may select and encode and decode one or more of the SRAP pictures for a picture not selected as a SPAP by a specific group of picture (GOP) or NAL type or PPS. Can be.

More specifically, an SRAP picture includes a single input image composed of multiviews for the same time for decoding of some bitstreams of the entire bitstream, regardless of the decoding of the images for different times. The data may be independently decoded using data redundancy between images.

In addition, a picture selected as an SRAP may be encoded / decoded by each sub-image through intra coding, and at least one SRAP picture in a certain GOP may be included to decode some bitstreams of the entire bitstream. Can be.

In addition, pictures not selected as SRAP pictures may be used to perform decoding according to a user's viewpoint through an inter prediction method using pictures selected as SRAP pictures as reference pictures.

Accordingly, the decoding system 300 may decode one or more sub-images selected corresponding to the visual perspective of the user through the decoding processor 310 using the reference picture.

On the other hand, the picture selected as the SRAP may be all intra prediction encoded of a single image according to a general encoding and decoding method without reference.

Accordingly, the decoding system 300 or the decoding apparatus 200 may need to store at least one SRAP picture in a DPB (DECODED PICTURE BUFFER). A sub-image between a pre-decoded picture or an SRAP picture in one GOP may be decoded using the picture stored in the SRAP as a reference picture.

Accordingly, FIG. 22 illustrates a change in the selective decoding region according to a picture designated by SRAP and a change in the user's viewpoint. As shown in FIG. 19, the period in which the SRAP picture is decoded may be a predetermined time period.

The sub-image group to be decoded may be changed according to a generation time of USER PERSPECTIVE OF CHANGE with reference to the SRAP picture that is periodically decoded. For example, as illustrated in FIG. 19, the decoding ROI region may be changed from the first sub image group to the second image sub image group at the time when the user view change occurs.

23 is a diagram for describing a sub image decoding method according to an exemplary embodiment.

As described above, in the method of decoding some pictures according to a user perspective, the dependency DEPENDENCY of each sub-image may be considered as shown in FIG. 20.

As described above, the encoding apparatus 100 and the decoding apparatus 200 may give independence to some pictures (sub images) according to coding efficiency or a scanning order of sub images. The independent sub-image to which independence is provided can be encoded and decoded into an intra picture to remove dependency on other sub-images.

Meanwhile, the encoding apparatus 100 and the decoding apparatus 200 may designate the remaining sub image as a dependent sub image. Independent subimages having the same POC as the dependent subimage may be added to a reference picture list of the dependent subimage, and the dependent subimages are predicted in picture with respect to adjacent independent subimages. Intra coding is coded or decoded using Intra Coding, or an inter sub prediction using independent sub images having the same POC as a reference picture, or an inter sub prediction with independent POCs. It can be encoded or decoded through.

More specifically, corresponding sub-images, independent sub-images having different POCs may also be added to the reference picture list. The dependent subimage may be used as a reference picture in intra prediction encoding or inter prediction encoding referring to independent sub images added to a list.

Accordingly, decoding using an intra prediction method may be performed using the independent sub image as a reference picture for the dependent sub images. In addition, by designating the same POC as the reference picture as the sub-image currently performing decoding, decoding may be performed through an inter prediction method.

For example, since the dependent sub-images have high similarity in independent sub-images having the same POC in the boundary region, the encoding apparatus 100 may change the in-screen with the independent sub-images for the boundary region of the dependent sub-image. Coding may be performed using a prediction method.

In this case, the independent sub-images to be referenced may be changed according to a unit such as PPS or SPS on the HLS, and information on this may be signaled separately. In addition, the decoding apparatus 200 may derive dependent sub-images based on the independent sub-images. In addition, the decoding apparatus 200 may receive whether or not the entire sub-images are independent in a separate list form. .

In addition, according to the above-described SRAP and independence-based encoding and decoding processing, the decoding system 300 easily selects some bitstreams according to a user's perspective among all bitstreams, and performs adaptive and selective decoding. It can be done efficiently.

 In addition, independent sub-images or dependent sub-images may be separately signaled or indexed and a signal capable of identifying them may be transmitted to the decoding apparatus 200. In this case, the decoding apparatus 200 directly encodes a portion or the entire region of the sub-images to be decoded according to the user's viewpoint with respect to adjacent regions between the sub-images by referring to adjacent independent sub-images or previously decoded sub-images. You can guess the mode and perform the restoration for that area.

Meanwhile, as illustrated in FIG. 24, in such intra prediction and inter prediction encoding, a coding method considering encoding efficiency may be exemplified.

Referring to FIG. 24, the encoding apparatus 100 according to an embodiment of the present invention may measure a distortion such as PSNR of an image or an error value of an RDO process (encoding process) according to a feature of an image. A metric that applies different weighting factors may be performed.

As shown in FIG. 24, a user viewing a synchronized multi-view image using a VR display may generally have a wide left and right viewing range based on a front view, but may have a narrow up and down viewing range. Accordingly, the encoding apparatus 100 may increase the coding efficiency by converting the multiview image into a 2D image and setting the low weight region to the low weight region for the upper and lower regions of the field of view.

FIG. 24 illustrates a user view area of various synchronized multiview images, and illustrates a low weighting factor application region when the multiview image is an Equirectangular type.

In particular, in order to apply a weight, the encoding apparatus 100 may first perform transform encoding of a synchronized multiview image into a 2D image, and according to the type of a multiview image, a weighting region may be differently determined. Can be. In addition, the weight region information may be included in the above-described spatial structure information or may be separately signaled through the HLS. The weighting factor may also be signaled on a separate HLS or transmitted in the form of a list.

Such a weighted area according to an embodiment of the present invention may be set on the assumption that the area where the human can concentrate is limited due to the characteristics of the natural image and the characteristics of the refracted 3D image.

Accordingly, the encoding apparatus 100 may perform encoding by applying a relatively low QP as a base QP to a relatively low weight region. In addition, when measuring the PSNR, the encoding apparatus 100 may measure a lower PSNR for a relatively low weight region. On the contrary, the encoding apparatus 100 may apply a relatively high QP as a base QP or measure a PSNR relatively high in a region having a relatively high weight.

According to an embodiment of the present invention, as described above, for intra block copy prediction, decoding may be performed by referring to images having different viewpoints within the same POC, and the referenced image may be independently decoded independently. Sub-images can be illustrated.

In particular, intra prediction for a boundary region may be applied efficiently, and the weighting factor may be used to maximize coding efficiency for the boundary region. For example, the encoding apparatus 100 specifies the weighting factor for patching a region referenced from an independent sub-image region according to a distance, and applies a scaling value to the predictive value of a neighboring block. You can decide efficiently.

In addition, the encoding apparatus 100 may apply a weight to a boundary region in configuring an MMT PROBABLE MODE (MPM) for intra picture coding. Accordingly, the encoding apparatus 100 may induce the prediction direction of the MPM based on the boundary between sub-images in the image even though the structures are not neighboring.

Meanwhile, the encoding apparatus 100 may consider an independent sub-image even in inter coding. For example, the encoding apparatus 100 may refer to a sub-image co-lacated corresponding to the independent sub-image. Accordingly, when configuring a reference picture for inter-screen encoding, only some sub-images of an image can be added to the reference picture list.

25 to 26 are diagrams for describing an encoding and decoding process, according to an embodiment of the present invention.

FIG. 25 is a block diagram illustrating a configuration of a video encoding apparatus according to an embodiment of the present invention, and receives each sub-image or an entire frame of a synchronized multi-view image according to an embodiment of the present invention as an input video signal. Can be processed.

Referring to FIG. 25, the video encoding apparatus 100 according to the present invention may include a picture splitter 160, a transform unit, a quantizer, a scanning unit, an entropy encoder, an intra predictor 169, and an inter predictor 170. , An inverse quantization unit, an inverse transform unit, a post-processing unit 171, a picture storage unit 172, a subtraction unit, and an addition unit 168.

The picture dividing unit 160 analyzes the input video signal and divides the picture into coding units having a predetermined size for each largest coding unit (LCU) to determine a prediction mode, and sizes of prediction units for each coding unit. Determine.

The picture splitter 160 sends the prediction unit to be encoded to the intra predictor 169 or the inter predictor 170 according to a prediction mode (or a prediction method). In addition, the picture dividing unit 160 sends a prediction unit to be encoded to the subtracting unit.

The picture may be composed of a plurality of slices, and the slice may be composed of a plurality of largest coding units (LCUs).

The LCU may be divided into a plurality of coding units (CUs), and the encoder may add information (flag) indicating whether to divide the bitstream. The decoder can recognize the location of the LCU using the address LcuAddr.

The coding unit (CU) in the case where splitting is not allowed is regarded as a prediction unit (PU), and the decoder can recognize the location of the PU using the PU index.

The prediction unit PU may be divided into a plurality of partitions. In addition, the prediction unit PU may include a plurality of transform units (TUs).

In this case, the picture dividing unit 160 may transmit the image data to the subtracting unit in a block unit (for example, PU unit or TU unit) of a predetermined size according to the determined encoding mode.

A coding tree block (CTB) is used as a video coding unit, and the CTB is defined by various square shapes. CTB is called a coding unit CU (Coding Unit).

The coding unit (CU) may have the form of a quad tree according to division. In addition, in the case of quadtree plus binary tree (QTBT) splitting, the coding unit may have the form of a binary tree binary divided in the quadtree or the terminal node, and the maximum size is 64x256 to 64 ㅧ according to the standard of the encoder. 64 can be configured.

For example, when the maximum size is 64X64, the picture division unit 160 has a depth of 0 when the maximum coding unit is LCU (Largest Coding Unit) until the depth becomes 3, that is, 8 × 8 size. The coding is performed by finding the optimal prediction unit recursively up to the coding unit (CU). In addition, for example, for a coding unit of a terminal node divided into QTBTs, a prediction unit (PU) and a transformation unit (TU) may have the same form as the divided coding unit or may have a more divided form.

A prediction unit for performing prediction is defined as a PU (Prediction Unit), and each coding unit (CU) performs prediction of a unit divided into a plurality of blocks, and performs prediction by dividing into a square and a rectangle.

The transformer converts the residual block, which is a residual signal of the original block of the input prediction unit, and the prediction block generated by the intra predictor 169 or the inter predictor 170. The residual block consists of a coding unit or a prediction unit. The residual block composed of the coding unit or the prediction unit is divided and transformed into an optimal transform unit. Different transformation matrices may be determined according to the prediction mode (intra or inter). In addition, since the residual signal of intra prediction has a direction according to the intra prediction mode, the transformation matrix may be adaptively determined according to the intra prediction mode.

The transformation unit may be transformed by two (horizontal and vertical) one-dimensional transformation matrices. For example, in the case of inter prediction, one predetermined transformation matrix is determined.

On the other hand, in the case of intra prediction, when the intra prediction mode is horizontal, the probability of the residual block having the directionality in the vertical direction increases, so a DCT-based integer matrix is applied in the vertical direction and DST-based in the horizontal direction. Or apply a KLT-based integer matrix. When the intra prediction mode is vertical, an integer matrix based on DST or KLT is applied in the vertical direction and a DCT based integer matrix in the horizontal direction.

In DC mode, a DCT-based integer matrix is applied in both directions. In addition, in the case of intra prediction, a transform matrix may be adaptively determined depending on the size of the transform unit.

The quantization unit determines a quantization step size for quantizing the coefficients of the residual block transformed by the transform matrix. The quantization step size is determined for each coding unit (hereinafter, referred to as a quantization unit) having a predetermined size or more.

The predetermined size may be 8x8 or 16x16. The coefficients of the transform block are quantized using a quantization matrix determined according to the determined quantization step size and prediction mode.

The quantization unit uses the quantization step size of the quantization unit adjacent to the current quantization unit as the quantization step size predictor of the current quantization unit.

The quantization unit may search in the order of the left quantization unit, the upper quantization unit, and the upper left quantization unit of the current quantization unit to generate a quantization step size predictor of the current quantization unit using one or two valid quantization step sizes.

For example, the first valid quantization step size retrieved in the above order may be determined as a quantization step size predictor. In addition, the average value of the two valid quantization step sizes searched in the above order may be determined as the quantization step size predictor. If only one is valid, the average value may be determined as the quantization step size predictor.

When the quantization step size predictor is determined, a difference value between the quantization step size of the current coding unit and the quantization step size predictor is transmitted to an entropy encoder.

On the other hand, there is a possibility that all of the left coding unit, the upper coding unit, and the upper left coding unit of the current coding unit do not exist. On the other hand, there may be a coding unit previously present in the coding order within the largest coding unit.

Accordingly, candidates may be quantization step sizes of the quantization units adjacent to the current coding unit and the quantization unit immediately before the coding order within the maximum coding unit.

In this case, 1) the left quantization unit of the current coding unit, 2) the top quantization unit of the current coding unit, 3) the top left quantization unit of the current coding unit, and 4) the order of the prior quantization unit immediately prior to the coding order. Can be. The order may be reversed and the upper left quantization unit may be omitted.

The quantized transform block is provided to an inverse quantization unit and a scanning unit.

The scanning unit scans the coefficients of the quantized transform block and converts them into one-dimensional quantization coefficients. Since the coefficient distribution of the transform block after quantization may be dependent on the intra prediction mode, the scanning scheme is determined according to the intra prediction mode.

In addition, the coefficient scanning scheme may be determined differently according to the size of the transform unit. The scan pattern may vary depending on the directional intra prediction mode. The scan order of the quantization coefficients scans in the reverse direction.

When the quantized coefficients are divided into a plurality of subsets, the same scan pattern is applied to the quantized coefficients in each subset. Scan patterns between subsets apply a zigzag scan or a diagonal scan. The scan pattern is preferably scanned in the forward subset from the main subset containing DC in the forward direction, but vice versa.

Also, scan patterns between subsets may be set in the same manner as scan patterns of quantized coefficients in the subset. In this case, the scan pattern between the subsets is determined according to the intra prediction mode. The encoder, on the other hand, sends information to the decoder that can indicate the location of the last non-zero quantization coefficient in the transform unit.

Information that can indicate the position of the last non-zero quantization coefficient in each subset can also be transmitted to the decoder.

Inverse quantization 135 inverse quantizes the quantized quantization coefficients. The inverse transform unit restores the dequantized transform coefficients to the residual block of the spatial domain. The adder combines the residual block reconstructed by the inverse transform unit and the received prediction block from the intra predictor 169 or the inter predictor 170 to generate a reconstructed block.

The post-processing unit 171 performs a deblocking filtering process for removing a blocking effect occurring in the reconstructed picture, an adaptive offset application process for compensating a difference value from the original image in pixel units, and a difference from the original image in a coding unit. Adaptive loop filtering is performed to complement the value.

The deblocking filtering process is preferably applied to the boundary of the prediction unit and the transform unit having a size of a predetermined size or more. The size may be 8x8. The deblocking filtering process includes determining a boundary to filter, determining a boundary filtering strength to be applied to the boundary, determining whether to apply a deblocking filter, and the deblocking filter. If it is determined to apply, the method includes selecting a filter to apply to the boundary.

Whether or not the deblocking filter is applied is i) whether the boundary filtering intensity is greater than 0, and ii) a value indicating the degree of change of pixel values in two block (P block, Q block) boundary portions adjacent to the boundary to be filtered. It is determined by whether or not it is smaller than the first reference value determined by this quantization parameter.

It is preferable that the said filter is at least 2 or more. When the absolute value of the difference between two pixels positioned at the block boundary is greater than or equal to the second reference value, a filter that performs relatively weak filtering is selected.

The second reference value is determined by the quantization parameter and the boundary filtering intensity.

The adaptive offset application process is to reduce the distortion between the original pixel and the pixel in the image to which the deblocking filter is applied. Whether to perform the adaptive offset application process on a picture or slice basis may be determined.

The picture or slice may be divided into a plurality of offset regions, and an offset type may be determined for each offset region. The offset type may include a predetermined number (eg, four) edge offset types and two band offset types.

If the offset type is an edge offset type, the edge type to which each pixel belongs is determined, and an offset corresponding thereto is applied. The edge type is determined based on a distribution of two pixel values adjacent to the current pixel.

The adaptive loop filtering process may perform filtering based on a value obtained by comparing the reconstructed image and the original image that have undergone the deblocking filtering process or the adaptive offset application process. In the adaptive loop filtering, the determined ALF may be applied to all pixels included in a 4x4 or 8x8 block.

Whether to apply the adaptive loop filter may be determined for each coding unit. The size and coefficients of the loop filter to be applied according to each coding unit may vary. Information indicating whether the adaptive loop filter is applied to each coding unit may be included in each slice header.

In the case of a chrominance signal, it may be determined whether to apply an adaptive loop filter on a picture basis. The shape of the loop filter may have a rectangular shape unlike the luminance.

Adaptive loop filtering may determine whether to apply per slice. Therefore, information indicating whether adaptive loop filtering is applied to the current slice is included in the slice header or the picture header.

Indicating that adaptive loop filtering is applied to the current slice, the slice header or picture header additionally includes information indicating the filter length in the horizontal and / or vertical direction of the luminance component used in the adaptive loop filtering process.

The slice header or picture header may include information indicating the number of filter sets. In this case, if the number of filter sets is 2 or more, the filter coefficients may be encoded using a prediction method. Accordingly, the slice header or the picture header may include information indicating whether filter coefficients are encoded by the prediction method, and include the predicted filter coefficients when the prediction method is used.

Meanwhile, not only the luminance but also the color difference components may be adaptively filtered. Therefore, the slice header or the picture header may include information indicating whether each of the color difference components is filtered. In this case, in order to reduce the number of bits, information indicating whether to filter on Cr and Cb may be joint coded (ie, multiplexed coding).

In this case, in the case of chrominance components, it is most likely not to filter both Cr and Cb in order to reduce complexity, so that entropy coding is performed by allocating the smallest index when both Cr and Cb are not filtered. .

In the case of filtering both Cr and Cb, entropy encoding is performed by allocating the largest index.

The picture storage unit 172 receives the post-processed image data from the post-processing unit 171 and restores and stores the image in picture units. The picture may be an image in a frame unit or an image in a field unit. The picture storage unit 172 includes a buffer (not shown) that can store a plurality of pictures.

The inter prediction unit 170 performs motion estimation using at least one reference picture stored in the picture storage unit 172, and determines a reference picture index and a motion vector indicating the reference picture.

Based on the determined reference picture index and the motion vector, a prediction block corresponding to a prediction unit to be encoded is extracted from a reference picture used for motion estimation among a plurality of reference pictures stored in the picture storage unit 172 and output. .

The intra prediction unit 169 performs intra prediction encoding by using the reconstructed pixel value inside the picture in which the current prediction unit is included.

The intra prediction unit 169 receives the current prediction unit to be predictively encoded, selects one of a preset number of intra prediction modes according to the size of the current block, and performs intra prediction.

The intra predictor 169 adaptively filters the reference pixel to generate an intra prediction block. If the reference pixel is not available, the reference pixels can be generated using the available reference pixels.

The entropy encoding unit entropy encodes the quantization coefficient quantized by the quantization unit, intra prediction information received from the intra prediction unit 169, motion information received from the inter prediction unit 170, and the like.

Although not shown, the inter prediction encoding apparatus may include a motion information determiner, a motion information encoding mode determiner, a motion information encoder, a prediction block generator, a residual block generator, a residual block encoder, and a multiplexer. .

The motion information determiner determines motion information of the current block. The motion information includes a reference picture index and a motion vector. The reference picture index represents any one of the previously coded and reconstructed pictures.

When the current block is unidirectional inter prediction coded, it indicates any one of the reference pictures belonging to list 0 (L0). On the other hand, when the current block is bidirectional predictively coded, the current block may include a reference picture index indicating one of the reference pictures of list 0 (L0) and a reference picture index indicating one of the reference pictures of list 1 (L1). .

In addition, when the current block is bidirectional predictively coded, the current block may include an index indicating one or two pictures of reference pictures of the composite list LC generated by combining the list 0 and the list 1.

The motion vector indicates the position of the prediction block in the picture indicated by each reference picture index. The motion vector may be pixel unit (integer unit) or sub pixel unit.

For example, it may have a resolution of 1/2, 1/4, 1/8 or 1/16 pixels. If the motion vector is not an integer unit, the prediction block is generated from pixels of an integer unit.

The motion information encoding mode determiner determines whether to encode the motion information of the current block in the skip mode, the merge mode, or the AMVP mode.

The skip mode is applied when a skip candidate having the same motion information as the motion information of the current block exists and the residual signal is zero. Also, the skip mode is applied when the current block is the same size as the coding unit. The current block can be viewed as a prediction unit.

The merge mode is applied when there is a merge candidate having the same motion information as that of the current block. The merge mode is applied when a residual signal exists when the current block has a different size or the same size as the coding unit. The merge candidate and the skip candidate may be the same.

AMVP mode is applied when skip mode and merge mode are not applied. An AMVP candidate having a motion vector most similar to the motion vector of the current block is selected as an AMVP predictor.

The motion information encoder encodes motion information according to a method determined by the motion information encoding mode determiner. If the motion information encoding mode is the skip mode or the merge mode, the merge motion vector encoding process is performed. When the motion information encoding mode is AMVP, the AMVP encoding process is performed.

The prediction block generator generates a prediction block by using motion information of the current block. If the motion vector is an integer unit, the block corresponding to the position indicated by the motion vector in the picture indicated by the reference picture index is copied to generate a prediction block of the current block.

However, when the motion vector is not an integer unit, pixels of the prediction block are generated from integer unit pixels in the picture indicated by the reference picture index.

In this case, in the case of the luminance pixel, the prediction pixel may be generated using an interpolation filter of 8 taps. In the case of chrominance pixels, a prediction pixel may be generated using a 4-tap interpolation filter.

The residual block generator generates a residual block by using the current block and the prediction block of the current block. When the size of the current block is 2Nx2N, the residual block is generated by using the current block and a prediction block having a size of 2Nx2N corresponding to the current block.

However, when the size of the current block used for prediction is 2NxN or Nx2N, after obtaining the prediction blocks for each of the two 2NxN blocks constituting the 2Nx2N, and using the two 2NxN prediction blocks, the final prediction block of the size 2Nx2N Can be generated.

In addition, a residual block of 2Nx2N may be generated by using the prediction block having a size of 2Nx2N. In order to solve the discontinuity of the boundary portion of two prediction blocks having a size of 2N × N, overlapping pixels of the boundary portion may be smoothed.

The residual block encoder divides the generated residual block into one or more transform units. Each transform unit is then transform coded, quantized, and entropy coded. In this case, the size of the transform unit may be determined by the quadtree method according to the size of the residual block.

The residual block encoder transforms the residual block generated by the inter prediction method using an integer-based transform matrix. The transformation matrix is an integer based DCT matrix.

The residual block encoder uses a quantization matrix to quantize coefficients of the residual block transformed by the transform matrix. The quantization matrix is determined by the quantization parameter.

The quantization parameter is determined for each coding unit of a predetermined size or more. The predetermined size may be 8x8 or 16x16. Therefore, when the current coding unit is smaller than the predetermined size, only the quantization parameter of the first coding unit in the coding order among the plurality of coding units within the predetermined size is encoded, and since the quantization parameter of the remaining coding units is the same as the parameter, the encoding is performed. There is no need to do it.

The coefficients of the transform block are quantized using the quantization matrix determined according to the determined quantization parameter and the prediction mode.

The quantization parameter determined for each coding unit of the predetermined size or more is predictively coded using the quantization parameter of the coding unit adjacent to the current coding unit. The left coding unit of the current coding unit and the upper coding unit may be searched to generate a quantization parameter predictor of the current coding unit using one or two valid quantization parameters.

For example, the first valid quantization parameter found in the above order may be determined as a quantization parameter predictor. Further, the first valid quantization parameter may be determined as a quantization parameter predictor by searching the left coding unit and the coding unit immediately before the coding order.

The coefficients of the quantized transform block are scanned and converted into one-dimensional quantization coefficients. The scanning method may be set differently according to the entropy encoding mode. For example, when coded with CABAC, the inter prediction coded quantization coefficients may be scanned in one predetermined manner (zigzag or diagonal raster scan). On the other hand, when encoded by CAVLC, scanning may be performed in a manner different from that described above.

For example, the scanning method may be determined according to a zigzag in case of inter and an intra prediction mode in case of intra. In addition, the coefficient scanning scheme may be determined differently according to the size of the transform unit.

The scan pattern may vary depending on the directional intra prediction mode. The scan order of the quantization coefficients scans in the reverse direction.

The multiplexer multiplexes the motion information encoded by the motion information encoder and the residual signals encoded by the residual block encoder. The motion information may vary according to an encoding mode.

That is, in case of skip or merge, only the index indicating the predictor is included. However, in the case of AMVP, it includes a reference picture index, a differential motion vector, and an AMVP index of the current block.

Hereinafter, an embodiment of the operation of the intra prediction unit 169 will be described in detail.

First, the picture division unit 160 receives the prediction mode information and the size of the prediction block, and the prediction mode information indicates the intra mode. The size of the prediction block may be square, such as 64x64, 32x32, 16x16, 8x8, 4x4, but is not limited thereto. That is, the size of the prediction block may be non-square rather than square.

Next, the reference pixel is read from the picture storage unit 172 to determine the intra prediction mode of the prediction block.

It is determined whether a reference pixel is generated by examining whether there is a reference pixel that is not available. The reference pixels are used to determine the intra prediction mode of the current block.

When the current block is located at the upper boundary of the current picture, pixels adjacent to the upper side of the current block are not defined. In addition, when the current block is located at the left boundary of the current picture, pixels adjacent to the left side of the current block are not defined.

It is determined that these pixels are not available pixels. In addition, even if the current block is located at the slice boundary and the pixels adjacent to the upper or left side of the slice are not pixels that are first encoded and reconstructed, it is determined that they are not available pixels.

As described above, when there are no pixels adjacent to the left side or the upper side of the current block or there are no pre-coded and reconstructed pixels, the intra prediction mode of the current block may be determined using only the available pixels.

However, it is also possible to generate reference pixels at locations that are not available using the available reference pixels of the current block. For example, if the pixels of the upper block are not available, some or all of the left pixels may be used to generate the upper pixels, and vice versa.

That is, the available reference pixels of the nearest position in the predetermined direction can be copied to the reference pixels from the reference pixels of the unavailable positions. If there are no reference pixels available in the predetermined direction, the available reference pixels in the closest positions in the opposite direction may be copied and generated as reference pixels.

Meanwhile, even when the upper or left pixels of the current block exist, it may be determined as a reference pixel that is not available according to the encoding mode of the block to which the pixels belong.

For example, when a block to which a reference pixel adjacent to an upper side of the current block belongs is a block that is inter-coded and reconstructed, the pixels may be determined as unavailable pixels.

In this case, a block adjacent to the current block may be intra-encoded to generate usable reference pixels using pixels belonging to the reconstructed block. In this case, information indicating that the encoder determines available reference pixels according to the encoding mode should be transmitted to the decoder.

Next, the intra prediction mode of the current block is determined using the reference pixels. The number of intra prediction modes allowable for the current block may vary depending on the size of the block. For example, if the size of the current block is 8x8, 16x16, 32x32, there may be 34 intra prediction modes. If the size of the current block is 4x4, there may be 17 intra prediction modes.

The 34 or 17 intra prediction modes may include at least one non-directional mode and a plurality of directional modes.

One or more non-directional modes may be DC mode and / or planar mode. When the DC mode and the planner mode are included in the non-directional mode, there may be 35 intra prediction modes regardless of the size of the current block.

This may include two non-directional modes (DC mode and planner mode) and 33 directional modes.

The planner mode generates a prediction block of the current block by using at least one pixel value (or a prediction value of the pixel value, hereinafter referred to as a first reference value) and reference pixels positioned at the bottom-right side of the current block. .

As described above, the configuration of the video decoding apparatus according to an embodiment of the present invention may be derived from the configuration of the video encoding apparatus described with reference to FIGS. 1, 2, and 25. For example, FIGS. An image may be decoded by performing an inverse process of the encoding process as described above with reference.

26 is a block diagram illustrating a configuration of a video decoding apparatus according to an embodiment of the present invention.

Referring to FIG. 26, the video decoding apparatus according to the present invention includes an entropy decoding unit 210, an inverse quantization / inverse transform unit 220, an adder 270, a deblocking filter 250, a picture storage unit 260, An intra predictor 230, a motion compensation predictor 240, and an intra / inter switch 280 are provided.

The entropy decoder 210 decodes an encoded bit stream transmitted from a video encoding apparatus and separates the encoded bit stream into an intra prediction mode index, motion information, a quantization coefficient sequence, and the like. The entropy decoder 210 supplies the decoded motion information to the motion compensation predictor 240.

The entropy decoder 210 supplies the intra prediction mode index to the intra predictor 230 and the inverse quantizer / inverse transformer 220. In addition, the entropy decoder 210 supplies the inverse quantization coefficient sequence to the inverse quantization / inverse transform unit 220.

The inverse quantization / inverse transform unit 220 converts the quantization coefficient sequence into inverse quantization coefficients of a two-dimensional array. One of the plurality of scanning patterns is selected for the conversion. One of a plurality of scanning patterns is selected based on at least one of a prediction mode (ie, one of intra prediction and inter prediction) and an intra prediction mode of the current block.

The intra prediction mode is received from an intra predictor or an entropy decoder.

The inverse quantization / inverse transform unit 220 restores the quantization coefficients by using a quantization matrix selected from a plurality of quantization matrices for the inverse quantization coefficients of the two-dimensional array. Different quantization matrices are applied according to the size of the current block to be reconstructed, and the quantization matrix is selected based on at least one of the prediction mode and the intra prediction mode of the current block for the same size block.

The residual block is reconstructed by inversely transforming the reconstructed quantization coefficients.

The adder 270 reconstructs the image block by adding the residual block reconstructed by the inverse quantization / inverse transform unit 220 and the prediction block generated by the intra predictor 230 or the motion compensation predictor 240.

The deblocking filter 250 performs deblocking filter processing on the reconstructed image generated by the adder 270. Accordingly, it is possible to reduce the deblocking artifacts caused by the image loss due to the quantization process.

The picture storage unit 260 is a frame memory that holds a local decoded image on which the deblocking filter process is performed by the deblocking filter 250.

The intra predictor 230 restores the intra prediction mode of the current block based on the intra prediction mode index received from the entropy decoder 210. The prediction block is generated according to the reconstructed intra prediction mode.

The motion compensation predictor 240 generates a prediction block for the current block from the picture stored in the picture storage unit 260 based on the motion vector information. When motion compensation with decimal precision is applied, the prediction block is generated by applying the selected interpolation filter.

The intra / inter switch 280 provides the adder 270 with the prediction block generated by either the intra predictor 230 or the motion compensation predictor 240 based on the encoding mode.

The current block is reconstructed using the prediction block of the current block reconstructed in this manner and the residual block of the decoded current block.

A video bitstream according to an embodiment of the present invention is a unit used to store encoded data in one picture and may include parameter sets (PS) and slice data.

The PS (parameter sets) is divided into a picture parameter set (hereinafter simply referred to as PPS) which is data corresponding to the head of each picture and a sequence parameter set (hereinafter simply referred to as SPS). The PPS and the SPS may include initialization information necessary to initialize each encoding, and may include spatial layout information according to an embodiment of the present invention.

The SPS is common reference information for decoding all pictures encoded by the random access unit (RAU) and may include a profile, a maximum number of pictures available for reference, a picture size, and the like.

The PPS may include, for each picture encoded by the random access unit (RAU), a type of a variable length coding method, an initial value of a quantization step, and a plurality of reference pictures as reference information for decoding a picture.

On the other hand, the slice header (SH) includes information about the slice when coding in the slice unit.

The method according to the present invention described above may be stored in a computer-readable recording medium that is produced as a program for execution on a computer, and examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape , Floppy disks, optical data storage, and the like.

The computer readable recording medium can be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the art to which the present invention belongs.

In addition, although the preferred embodiment of the present invention has been shown and described above, the present invention is not limited to the specific embodiments described above, but the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

Claims (4)

Disposing a plurality of images constituting the omnidirectional image of the three-dimensional space on the two-dimensional image;
Converting an arrangement of at least some of the plurality of regions included in the two-dimensional image;
Encoding the converted two-dimensional image; And
And transmitting the encoded two-dimensional image and an SEI message.
The SEI message includes layout information including a layout method of an omnidirectional image with respect to the 2D image, information about the transformation, and viewpoint information according to an intention of a content provider.
The arrangement information includes at least one of a Cube Map method for arranging sub-images of the omnidirectional image in the 2D image according to a Cube Map format, and an Equirectangular method for arranging sub-images of the omnidirectional image in the 2D image according to an Equirectangular format. Includes an SEI message syntax indicating one,
The SEI message is mapped to the 3D space in the 2D image region when decoding of the bitstream of the 2D image, which is pre-coded the omnidirectional image, is performed according to at least some of the SEI message and user viewpoint information. Contains information that can be played back,
The viewpoint information according to the intention of the content provider is
It includes the view information of the omnidirectional image to enable the user to view the omnidirectional image reconstructed from the encoded two-dimensional image, according to the phase according to the intention of the content provider or image producer Characterized by
Image processing method. An image acquisition unit for arranging a plurality of images constituting an omnidirectional image of a three-dimensional space on a two-dimensional image, and converting an arrangement of at least some of the plurality of regions included in the two-dimensional image;
An image encoder which encodes the converted two-dimensional image; And
And a transmission processor for transmitting the encoded two-dimensional image and the SEI message.
The SEI message includes layout information including a layout method of an omnidirectional image with respect to the 2D image, information about the transformation, and viewpoint information according to an intention of a content provider.
The arrangement information includes a Cube Map method for arranging sub-images of the omnidirectional image in the two-dimensional image according to a Cube Map format, and an Equirectangular method for disposing sub-images of the omnidirectional image in the two-dimensional image according to an Equirectangular format. Includes an SEI message syntax indicating at least one,
The SEI message is mapped to the 3D space in the 2D image region when decoding of the bitstream of the 2D image, which is pre-coded the omnidirectional image, is performed according to at least some of the SEI message and user viewpoint information. Contains information that can be played back,
The viewpoint information according to the intention of the content provider is
It includes the view information of the omnidirectional image to enable the user to view the omnidirectional image reconstructed from the encoded two-dimensional image, according to the phase according to the intention of the content provider or image producer Characterized by
Image processing device. Obtaining an encoded two-dimensional image and an SEI message;
Decoding the encoded two-dimensional image; And
And reproducing a 3D spatial image according to a user's viewpoint from the decoded 2D image.
The SEI message includes layout information including a layout method of an omnidirectional image with respect to the 2D image, information about the transformation, and viewpoint information according to an intention of a content provider.
The arrangement information includes at least one of a Cube Map method for arranging sub-images of the omnidirectional image in the 2D image according to a Cube Map format, and an Equirectangular method for arranging sub-images of the omnidirectional image in the 2D image according to an Equirectangular format. Includes an SEI message syntax indicating one,
The conversion information is information indicating that the arrangement of at least some of the plurality of areas included in the two-dimensional image is converted,
The decoding step,
According to at least some of the SEI message and user viewpoint information, decoding the bitstream of the two-dimensional image pre-coded the omnidirectional image so that the two-dimensional image region is mapped and reproduced in a three-dimensional space; Including
The viewpoint information according to the intention of the content provider is
It includes the view information of the omnidirectional image to enable the user to view the omnidirectional image reconstructed from the encoded two-dimensional image, according to the phase according to the intention of the content provider or image producer
Image processing method. A decoder configured to acquire an encoded 2D image and an SEI message, decode the encoded 2D image, and process coordinate mapping for reproducing a 3D spatial image according to a user's viewpoint from the decoded 2D image; ,
The SEI message includes layout information including a layout method of an omnidirectional image with respect to the 2D image, information about the transformation, and viewpoint information according to an intention of a content provider.
The arrangement information includes at least one of a Cube Map method for arranging sub-images of the omnidirectional image in the 2D image according to a Cube Map format, and an Equirectangular method for arranging sub-images of the omnidirectional image in the 2D image according to an Equirectangular format. Includes an SEI message syntax indicating one,
The conversion information is information indicating that the arrangement of at least some of the plurality of areas included in the two-dimensional image is converted,
The decoding unit,
According to at least a part of the SEI message and user viewpoint information, decoding of the bitstream of the two-dimensional image, which is pre-coded the omnidirectional image, is performed to map and reproduce the two-dimensional image region in a three-dimensional space.
The viewpoint information according to the intention of the content provider is
It includes the view information of the omnidirectional image to enable the user to view the omnidirectional image reconstructed from the encoded two-dimensional image, according to the phase according to the intention of the content provider or image producer Characterized by
Image processing device.

Images